From 128a5a2340285880b1dfbbd7b319aedbebd31b1a Mon Sep 17 00:00:00 2001 From: Julien Girardin Date: Tue, 28 Jan 2020 16:16:36 +0100 Subject: [PATCH] Opentelemetry and a few things on prometheus --- slides/3.yml | 1 + slides/k8s/opentelemetry.md | 72 +++++++++++++++++++++++++++ slides/k8s/prometheus-endpoint.md | 31 ++++++++++++ slides/k8s/software-dev-banalities.md | 2 +- 4 files changed, 105 insertions(+), 1 deletion(-) create mode 100644 slides/k8s/opentelemetry.md diff --git a/slides/3.yml b/slides/3.yml index f5a548e1..91ee2b66 100644 --- a/slides/3.yml +++ b/slides/3.yml @@ -49,6 +49,7 @@ chapters: - k8s/monitoring-intro.md #- k8s/prometheus-intro.md - k8s/prometheus-endpoint.md + - k8s/opentelemetry.md #- diff --git a/slides/k8s/opentelemetry.md b/slides/k8s/opentelemetry.md new file mode 100644 index 00000000..a22153a7 --- /dev/null +++ b/slides/k8s/opentelemetry.md @@ -0,0 +1,72 @@ +# OpenTelemetry + +*Opentelemetry* is a "tracing" framework. It's a fusion of two other frameworks: +*opentracing* and *opencensus*. + +The goal is to have deep integration with software languages and application framework to +enable deep dive tracing of different events accross different components + +--- +## Span ! span ! span ! + +- A unit a tracing is called a `span`. + +- A span has a start time and and stop time and an ID. + +- It represent an action that took some time to complete + + ex: call to function `B`, DB transation, REST call to a backend... + +- A span could have a parent and could be parent of multiple child spans. + + ex: during the call to function `B`, a sub call to `C` and `D` has been issued + +- Think of it as a "tree" of calls + +--- +## Distributed tracing + +- This could be applied to multiple components + + ex: If microservice `A` send REST call to microservice `B` + + - `A` will have a span for the call to `B` + - `B` will have a span for the call from `A` + (that normally starts shortly after, and finishes shortly before) + - the span of `A` will be the parent of the span of `B`, + so that they join the same "tree" of call + +details: `A` will send headers (depends of the protocol used) to tag the span ID, +so that `B` can generate child span and joining the same tree of call + +--- +## Centrally stored + +- We do have "spans", ok. But what do we do with that ? + +- We store them. + +- In the previous exemple: + + - `A` will send trace to it's local agent + - `B` will do the same + - Every span will ends up in the same DB so that we can reconstruct the "tree" of call + later on and analyze it. + +- there is multiple implementation of those agents + DB + WebUI. The most famous opensource ones: + + - Zipkin + - Jaeger + +--- +## Distributed sampled + +- Huh, we store all of them ? (that could be a lot of storage) + +- No, we could apply sampling, to reduce storage/network footprint. + +- Smart sampling is applied directly in the application to save CPU if span is not needed. + +- It also insures that if a span is mark as sampled, all child-span are sampled together + + (so that the tree of call is complete) diff --git a/slides/k8s/prometheus-endpoint.md b/slides/k8s/prometheus-endpoint.md index 1ffa6a06..4b398416 100644 --- a/slides/k8s/prometheus-endpoint.md +++ b/slides/k8s/prometheus-endpoint.md @@ -1,3 +1,10 @@ +# Prometheus + +Prometheus is monitoring system with small storage io footprint. It's quite ubiquitous + +in the kubernetes world. This section is not a description + + ## Prometheus endpoint The goal here is to expose an HTTP endoint for prometheus. Sample response: @@ -53,3 +60,27 @@ Links (do you see a pattern ?): - Try to not expose monitoring channel more than needed. Often localhost is enough (sidecars run in the same network namespace as other containers) + +--- +## Ok! and then change prometheus conf ? + +- Well, not really. It achievable this way, but... + +- Prometheus has good service discovery paired with kubernetes. + +- Depending on how we installed prometheus, we just need: + + - pods annotations: + + ``` + annotations: + prometheus.io/port: 9090 + prometheus.io/path: /metrics + ``` + + - *service monitor* custom resource object +.small[ + https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#servicemonitor +] + +*Note: More on prometheus next day* diff --git a/slides/k8s/software-dev-banalities.md b/slides/k8s/software-dev-banalities.md index 1d41a050..86d3a0bc 100644 --- a/slides/k8s/software-dev-banalities.md +++ b/slides/k8s/software-dev-banalities.md @@ -1,4 +1,4 @@ -# Software development +## Software development From years, decades, (centuries !), software development has followed the same principles: