From 8df073b8acf8e15ae1e028d19b472b4b6cc4a839 Mon Sep 17 00:00:00 2001 From: Jerome Petazzoni Date: Mon, 9 Apr 2018 04:17:00 -0500 Subject: [PATCH 1/3] Add a chapter about centralized logging Explain the purpose of centralized logging. Describe the EFK stack. Deploy a simplified EFK stack through a YAML file. Use it to view container logs. Profit. --- slides/kube/logs-centralized.md | 144 ++++++++++++++++++++++++++++++++ 1 file changed, 144 insertions(+) create mode 100644 slides/kube/logs-centralized.md diff --git a/slides/kube/logs-centralized.md b/slides/kube/logs-centralized.md new file mode 100644 index 00000000..e01af147 --- /dev/null +++ b/slides/kube/logs-centralized.md @@ -0,0 +1,144 @@ +# Centralized logging + +- Using `kubectl` or `stern` is simple; but it has drawbacks: + + - when a node goes down, its logs are not available anymore + + - we can only dump or stream logs; we want to search/index/count... + +- We want to send all our logs to a single place + +- We want to parse them (e.g. for HTTP logs) and index them + +- We want a nice web dashboard + +-- + +- We are going to deploy an EFK stack + +--- + +## What is EFK? + +- EFK is three components: + + - ElasticSearch (to store and index log entries) + + - Fluentd (to get container logs, process them, and put them in ElasticSearch) + + - Kibana (to view/search log entries with a nice UI) + +- The only component that we need to access from outside will be Kibana + +--- + +## Deploying EFK on our cluster + +- We are going to use a YAML file describing all the required resources + +.exercise[ + +- Load the YAML file into our cluster: + ```bash + kubectl apply -f https://goo.gl/MUZhE4 + ``` + +] + +If we [look at the YAML file](https://goo.gl/MUZhE4), we see that +it creates a daemon set, two deployments, two services, +and a few roles and role bindings (to give fluentd the required permissions). + +--- + +## The itinerary of a log line (before Fluentd) + +- A container writes a line on stdout or stderr + +- Both are typically piped to the container engine (Docker or otherwise) + +- The container engine reads the line, and sends it to a logging driver + +- The timestamp and stream (stdout or stderr) is added to the log line + +- With the default configuration for Kubernetes, the line is written to a JSON file + + (`/var/log/containers/pod-name_namespace_container-id.log`) + +- That file is read when we invoke `kubectl logs`; we can access it directly too + +--- + +## The itinerary of a log line (with Fluentd) + +- Fluentd runs on each node (thanks to a deamon set) + +- It binds-mounts `/var/log/containers` from the host (to access these files) + +- It continuously scans this directory for new files; reads them; parses them + +- Each log line becomes a JSON object, fully annotated with extra information: +
container id, pod name, Kubernetes labels ... + +- These JSON objects are stored into ElasticSearch + +- ElasticSearch indexes the JSON objects + +- We can access the logs through Kibana (and perform searches, counts, etc.) + +--- + +## Accessing Kibana + +- Kibana offers a web interface that is relatively straightforward + +- Let's check it out! + +.exercise[ + +- Check which `NodePort` was allocated to Kibana: + ```bash + kubectl get svc kibana + ``` + +- With our web browser, connect to Kibana + +] + +--- + +## Using Kibana + +*Note: this is not a Kibana workshop! So this section is deliberately very terse.* + +- The first time you connect to Kibana, you must "configure an index pattern" + +- Just use the one that is suggested, `@timestamp` + +- Then click "Discover" (in the top-left corner) + +- You should see container logs + +- Advice: in the left column, select a few fields to display, e.g.: + + `kubernetes.host`, `kubernetes.pod_name`, `stream`, `log` + +--- + +## Caveat emptors + +We are using EFK because it is relatively straightforward +to deploy on Kubernetes, without having to redeploy or reconfigure +our cluster. But it doesn't mean that it will always be the best +option for your use-case. If you are running Kubernetes in the +cloud, you might consider using check the cloud provider's logging +infrastructure (if it can be integrated with Kubernetes). + +The deployment method that we will use here has been simplified: +there is only one ElasticSearch node. In a real deployment, you +might use a cluster, both for performance and reliability reasons. +But this is outside of the scope of this chapter. + +The YAML file that we used creates all the resources in the +`default` namespace, for simplicity. In a real scenario, you will +create the resources in the `kube-system` or in a dedicated namespace. From 0256dc86401d992a28d8ea9dfcc219b088e6acb7 Mon Sep 17 00:00:00 2001 From: Bridget Kromhout Date: Mon, 9 Apr 2018 11:22:43 -0500 Subject: [PATCH 2/3] Update logs-centralized.md A few typo fixes --- slides/kube/logs-centralized.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/slides/kube/logs-centralized.md b/slides/kube/logs-centralized.md index e01af147..ae36c499 100644 --- a/slides/kube/logs-centralized.md +++ b/slides/kube/logs-centralized.md @@ -28,7 +28,7 @@ - Kibana (to view/search log entries with a nice UI) -- The only component that we need to access from outside will be Kibana +- The only component that we need to access from outside the cluster will be Kibana --- @@ -71,7 +71,7 @@ and a few roles and role bindings (to give fluentd the required permissions). ## The itinerary of a log line (with Fluentd) -- Fluentd runs on each node (thanks to a deamon set) +- Fluentd runs on each node (thanks to a daemon set) - It binds-mounts `/var/log/containers` from the host (to access these files) @@ -125,13 +125,13 @@ and a few roles and role bindings (to give fluentd the required permissions). --- -## Caveat emptors +## Caveat emptor We are using EFK because it is relatively straightforward to deploy on Kubernetes, without having to redeploy or reconfigure our cluster. But it doesn't mean that it will always be the best option for your use-case. If you are running Kubernetes in the -cloud, you might consider using check the cloud provider's logging +cloud, you might consider using the cloud provider's logging infrastructure (if it can be integrated with Kubernetes). The deployment method that we will use here has been simplified: From 2dfa5a96604e745d694a8073e0b0c5a7deeaf042 Mon Sep 17 00:00:00 2001 From: Bridget Kromhout Date: Mon, 9 Apr 2018 11:59:19 -0500 Subject: [PATCH 3/3] Update logs-centralized.md --- slides/kube/logs-centralized.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/slides/kube/logs-centralized.md b/slides/kube/logs-centralized.md index ae36c499..d316b2fa 100644 --- a/slides/kube/logs-centralized.md +++ b/slides/kube/logs-centralized.md @@ -80,7 +80,7 @@ and a few roles and role bindings (to give fluentd the required permissions). - Each log line becomes a JSON object, fully annotated with extra information:
container id, pod name, Kubernetes labels ... -- These JSON objects are stored into ElasticSearch +- These JSON objects are stored in ElasticSearch - ElasticSearch indexes the JSON objects @@ -141,4 +141,4 @@ But this is outside of the scope of this chapter. The YAML file that we used creates all the resources in the `default` namespace, for simplicity. In a real scenario, you will -create the resources in the `kube-system` or in a dedicated namespace. +create the resources in the `kube-system` namespace or in a dedicated namespace.