diff --git a/slides/k8s/prometheus.md b/slides/k8s/prometheus.md new file mode 100644 index 00000000..7a3e12e1 --- /dev/null +++ b/slides/k8s/prometheus.md @@ -0,0 +1,486 @@ +# Collecting metrics with Prometheus + +- Prometheus is an open-source monitoring system including: + + - multiple *service discovery* backends to figure out which metrics to collect + + - a *scraper* to collect these metrics + + - an efficient *time series database* to store these metrics + + - a specific query language (PromQL) to query these time series + + - an *alert manager* to notify us according to metrics values or trends + +- We are going to deploy it on our Kubernetes cluster and see how to query it + +--- + +## Why Prometheus? + +- We don't endorse Prometheus more or less than any other system + +- It's relatively well integrated within the Cloud Native ecosystem + +- It can be self-hosted (this is useful for tutorials like this) + +- It can be used for deployments of varying complexity: + + - one binary and 10 lines of configuration to get started + + - all the way to thousands of nodes and millions of metrics + +--- + +## Exposing metrics to Prometheus + +- Prometheus obtains metrics and their values by querying *exporters* + +- An exporter serves metrics over HTTP, in plain text + +- This is was the *node exporter* looks like: + + http://demo.robustperception.io:9100/metrics + +- Prometheus itself exposes its own internal metrics, too: + + http://demo.robustperception.io:9090/metrics + +- If you want to expose custom metrics to Prometheus: + + - serve a text page like these, and you're good to go + + - libraries are available in various languages to help with quantiles etc. + +--- + +## How Prometheus gets these metrics + +- The *Prometheus server* will *scrape* URLs like these at regular intervals + + (by default: every minute; can be more/less frequent) + +- If you're worried about parsing overhead: exporters can also use protobuf + +- The list of URLs to scrape (the *scrape targets*) is defined in configuration + +--- + +## Defining scrape targets + +This is maybe the simplest configuration file for Prometheus: +```yaml +scrape_configs: + - job_name: 'prometheus' + static_configs: + - targets: ['localhost:9090'] +``` + +- In this configuration, Prometheus collects its own internal metrics + +- A typical configuration file will have multiple `scrape_configs` + +- In this configuration, the list of targets is fixed + +- A typical configuration file will use dynamic service discovery + +--- + +## Service discovery + +This configuration file will leverage existing DNS `A` records: +```yaml +scrape_configs: + - ... + - job_name: 'node' + dns_sd_configs: + - names: ['api-backends.dc-paris-2.enix.io'] + type: 'A' + port: 9100 +``` + +- In this configuration, Prometheus resolves the provided name(s) + + (here, `api-backends.dc-paris-2.enix.io`) + +- Each resulting IP address is added as a target on port 9100 + +--- + +## Dynamic service discovery + +- In the DNS example, the names are re-resolved at regular intervals + +- As DNS records are created/updated/removed, scrape targets change as well + +- Existing data (previously collected metrics) is not deleted + +- Other service discovery backends work in a similar fashion + +--- + +## Other service discovery mechanisms + +- Prometheus can connect to e.g. a cloud API to list instances + +- Or to the Kubernetes API to list nodes, pods, services ... + +- Or a service like Consul, Zookeeper, etcd, to list applications + +- The resulting configurations files are *way more complex* + + (but don't worry, we won't need to write them ourselves) + +--- + +## Time series database + +- We could wonder, "why do we need a specialized database?" + +- One metrics data point = metrics ID + timestamp + value + +- With a classic SQL or noSQL data store, that's at least 160 bits of data + indexes + +- Prometheus is way more efficient, without sacrificing performance + + (it will even be gentler on the I/O subsystem since it needs to write less) + +FIXME link to Goutham's talk + +--- + +## Running Prometheus on our cluster + +We need to: + +- Run the Prometheus server in a pod + + (using e.g. a Deployment to ensure that it keeps running) + +- Expose the Prometheus server web UI (e.g. with a NodePort) + +- Run the *node exporter* on each node (with a Daemon Set) + +- Setup a Service Account so that Prometheus can query the Kubernetes API + +- Configure the Prometheus server + + (storing the configuration in a Config Map for easy updates) + +--- + +## Helm Charts to the rescue + +- To make our lives easier, we are going to use a Helm Chart + +- The Helm Chart will take care of all the steps explained above + + (including some extra features that we don't need, but won't hurt) + +--- + +## Step 1: install Helm + +- If we already installed Helm earlier, these commands won't break anything + +.exercice[ + +- Install Tiller (Helm's server-side component) on our cluster: + ```bash + helm init + ``` + +- Give Tiller permission to deploy things on our cluster: + ```bash + kubectl create clusterrolebinding add-on-cluster-admin \ + --clusterrole=cluster-admin --serviceaccount=kube-system:default + ``` + +] + +--- + +## Step 2: install Prometheus + +- Skip this if we already installed Prometheus earlier + + (in doubt, check with `helm list`) + +.exercice[ + +- Install Prometheus on our cluster: + ```bash + helm install stable/prometheus \ + --set server.service.type=NodePort \ + --set server.persistentVolume.enabled=false + ``` + +] + +The provided flags: + +- expose the server web UI (and API) on a NodePort + +- use an ephemeral volume for metrics storage +
+ (instead of requesting a Persistent Volume through a Persistent Volume Claim) + +--- + +## Connecting to the Prometheus web UI + +- Let's connect to the web UI and see what we can do + +.exercise[ + +- Figure out the NodePort that was allocated to the Prometheus server: + ```bash + kubectl get svc prometheus-server + ``` + +- With your browser, connect to that port + +] + +--- + +## Querying some metrics + +- This is easy ... if you are familiar with PromQL + +.exercise[ + +- Click on "Graph", and in "expression", paste the following: + ``` + sum by (instance) ( + irate( + container_cpu_usage_seconds_total{ + pod_name=~"worker.*" + }[5m] + ) + ) + ``` + +] + +- Click on the blue "Execute" button and on the "Graph" tab just below + +- We see the cumulated CPU usage of worker pods for each node +
+ (if we just deployed Prometheus, there won't be much data to see, though) + +--- + +## Getting started with PromQL + +- We can't learn PromQL in just 5 minutes + +- But we can cover the basics to get an idea of what is possible + + (and have some keywords and pointers) + +- We are going to break down the query above + + (building it one step at a time) + +--- + +## Graphing one metric across all tags + +This query will show us CPU usage across all containers: +``` +container_cpu_usage_seconds_total +``` + +- The suffix of the metrics name tells us: + + - the unit (seconds of CPU) + + - that it's the total used since the container creation + +- Since it's a "total", it is an increasing quantity + + (we need to compute the derivative if we want e.g. CPU % over time) + +- We see that the metrics retrieved have *tags* attached to them + +--- + +## Selecting metrics with tags + +This query will show us only metrics for worker containers: +``` +container_cpu_usage_seconds_total{pod_name=~"worker.*"} +``` + +- The `=~` operator allows regex matching + +- We select all the pods with a name starting with `worker` + + (it would be better to use labels to select pods; more on that later) + +- The result is a smaller set of containers + +--- + +## Transforming counters in rates + +This query will show us CPU usage % instead of total seconds used: +``` +100*irate(container_cpu_usage_seconds_total{pod_name=~"worker.*"}[5m]) +``` + +- The [`irate`](https://prometheus.io/docs/prometheus/latest/querying/functions/#irate) operator computes the "per-second instant rate of increase" + + - `rate` is similar but allows decreasing counters and negative values + + - with `irate`, if a counter goes back to zero, we don't get a negative spike + +- The `[5m]` tells how far to look back if there is a gap in the data + +- And we multiply with `100*` to get CPU % usage + +--- + +## Aggregation operators + +This query sums the CPU usage per node: +``` +sum by (instance) ( + irate(container_cpu_usage_seconds_total{pod_name=~"worker.*"}[5m]) +) +``` + +- `instance` corresponds to the node on which the container is running + +- `sum by (instance) (...)` computes the sum for each instance + +- Note: all the other tags are collapsed + + (in other words, the resulting graph only shows the `instance` tag) + +- PromQL supports many more [aggregation operators](https://prometheus.io/docs/prometheus/latest/querying/operators/#aggregation-operators) + +--- + +## What kind of metrics can we collect? + +- Node metrics (related to physical or virtual machines) + +- Container metrics (resource usage per container) + +- Databases, message queues, load balancers, ... + + (check out this [list of exporters](https://prometheus.io/docs/instrumenting/exporters/)!) + +- Instrumentation (=deluxe `printf` for our code) + +- Business metrics (customers served, revenue, ...) + +--- + +class: extra-details + +## Node metrics + +- CPU, RAM, disk usage on the whole node + +- Total number of processes running, and their states + +- Number of open files, sockets, and their states + +- I/O activity (disk, network), per operation or volume + +- Physical/hardware (when applicable): temperature, fan speed ... + +- ... and much more! + +--- + +class: extra-details + +## Container metrics + +- Similar to node metrics, but not totally identical + +- RAM breakdown will be different + + - active vs inactive memory + - some memory is *shared* between containers, and accounted specially + +- I/O activity is also harder to track + + - async writes can cause deferred "charges" + - some page-ins are also shared between containers + +For details about container metrics, see: +
+http://jpetazzo.github.io/2013/10/08/docker-containers-metrics/ + +--- + +class: extra-details + +## Application metrics + +- Arbitrary metrics related to your application and business + +- System performance: request latency, error rate ... + +- Volume information: number of rows in database, message queue size ... + +- Business data: inventory, items sold, revenue ... + +--- + +class: extra-details + +## Detecting scrape targets + +- Prometheus can leverage Kubernetes service discovery + + (with proper configuration) + +- Services or pods can be annotated with: + + - `prometheus.io/scrape: true` to enable scraping + - `prometheus.io/port: 9090` to indicate the port number + - `prometheus.io/path: /metrics` to indicate the URI (`/metrics` by default) + +- Prometheus will detect and scrape these (without needing a restart or reload) + +--- + +## Querying labels + +- What if we want to get metrics for containers belong to pod tagged `worker`? + +- The cAdvisor exporter does not give us Kubernetes labels + +- Kubernetes labels are exposed through another exporter + +- We can see Kubernetes labels through metrics `kube_pod_labels` + + (each container appears as a time series with constant value of `1`) + +- Prometheus *kind of* supports "joins" between time series + +- But only if the names of the tags match exactly + +--- + +## Unfortunately ... + +- The cAdvisor exporter uses tag `pod_name` for the name of a pod + +- The Kubernetes service endpoints exporter uses tag `pod` instead + +- And this is why we can't have nice things + +- See [Prometheus issue #2204](https://github.com/prometheus/prometheus/issues/2204) for the rationale + + ([this comment](https://github.com/prometheus/prometheus/issues/2204#issuecomment-261515520) in particular if you want a workaround involving relabeling) + +- Then see [this blog post](https://www.robustperception.io/exposing-the-software-version-to-prometheus) or [this other one](https://www.weave.works/blog/aggregating-pod-resource-cpu-memory-usage-arbitrary-labels-prometheus/) to see how to perform "joins" + +- There is a good chance that the situation will improve in the future diff --git a/slides/new-content.yml b/slides/new-content.yml index 907443f6..a8a66293 100644 --- a/slides/new-content.yml +++ b/slides/new-content.yml @@ -18,10 +18,6 @@ chapters: - - k8s/authn-authz.md - k8s/ingress.md - k8s/gitworkflows.md - - | - # metrics - - prometheus - + - k8s/prometheus.md - - k8s/healthchecks.md