The rendering code checks whether endpoint IPs are part of
cluster-local networks. Due to the prevalence of endpoints - medium
sized reports can contain many thousands of endpoints - this is
performance critical. Alas the existing code performs the check via a
linear scan of a list of networks. That is slow when there are more
than a few, which will be the case in the context of k8s, since there
the probes register service IPs as local /32 networks.
Here we change representation of the set of networks to a prefix
tree (aka trie), which is well-suited for IP network membership checks
since networks are in fact a bitstring prefixes.
The specific representation is a crit-bit tree, but that choice was
purely based on implementation convenience - the chosen library is the
only one I could find that directly supports IP networks.
The rendering code checks whether endpoint IPs are part of
cluster-local networks. Due to the prevalence of endpoints - medium
sized reports can contain many thousands of endpoints - this is
performance critical. Alas the existing code performs the check via a
linear scan of a list of networks. That is slow when there are more
than a few. Unfortunately in some common k8s network setups, e.g. on
AWS, a cluster can contain hundreds of networks, due to /32 networks
derived from interfaces with multiple IPs.
Here we change representation of the set of networks to a prefix
tree (aka trie), which is well-suited for IP network membership checks
since networks are in fact a bitstring prefixes.
The specific representation is a crit-bit tree, but that choice was
purely based on implementation convenience - the chosen library is the
only one I could find that directly supports IP networks.
the information is constant and already present in the id, so we can
extract it from there.
That reduces the report size and improves report encoding/decoding
performance. It should reduce memory usage too and improve report
merging performance too.
NB: Probes with this change are incompatible with old apps.
We have to introduce the kinda hacky concept of a 'No Stack' stack
to reconcile it with the idea of a 'default' k8s namespace. This is important
because swarm services without a stack don't have the same docker labels as ones that do.
Curiously, they still have what appears to be a stack name 'prefix' on their names,
but I can't isolate that name anywhere easily so they'll just have to make do.
I basically copy-pasted updateFilters to make this work, todo go back and refactor
to not duplicate 90% of the code.
We replace the existing data structure with a simpler one that
only specifies how to get the parent label, which is the only
part of the Parent struct that can't be generated from the node info alone.
Future work: Standardize this concept of a label and put it in the topology instead.
Though that already exists...so just use it?
The default sets the node label to the node ID.
This is likely to not look very good, but the intent is that it creates an obvious problem,
ie. that the node ID is being used as the label, rather than a silent omission or more subtle problem.
Possible future work:
* For single-component IDs, extract the component automatically and use that instead.
* Instead of functions, in simple cases just have a LUT by topology with common behaviours like
'stack = true or false', 'label = this key in node.Latest'
The latter opens up to eventually moving this info inside the report itself ala topology templates,
or at least centralizing it in the source.
Currently, if a topology does not have any specific info in nodeSummariesByID,
any children of the node that belong to that topology will be silently omitted.
This change adds a default behaviour for such topologies, with no special columns
but at least it is displayed at all.
Unlisted topologies are displayed after all listed ones, in arbitrary order.
Note that completely bogus or other special cases (eg. topology = Pseudo) still will not
be displayed as report.Topology() will fail.
This gives us a single source of truth in a variety of situations where we want to
know what view to direct a user to in order to 'open' a particular node.
I wanted to put this in app/api_topologies where the views are defined, but that creates
a circular import.
This field changes the option group behaviour depending on its value.
Currently only supports two values:
"one" (default): Old behaviour, one option can be selected
"union": Any number of options can be selected, and the filters are OR-ed togther
It is written in such a way as to easily enable a future "intersection" option,
as per union but AND-ing the filters. But this is not done here. YAGNI.
This reverts commit 76ddc75fb8, reversing
changes made to 3ade2933eb.
We are rolling this back for now because it's causing a bug where sub-topologies
would have ~3000 repeated cases of the k8s filters, causing performance issues clientside.
To facilitate this, we replace the existing functionality of updateFilters which
sets k8s topologies to have the filters [namespace, managed], to instead append the namespace filter
to any existing. This lets it apply to both k8s and container topologies without overwriting existing
container filters. We instead set the managed filter in the static definition.
This however has the side effect that the ordering of the namespace filter and the managed filter
in k8s topologies has been reversed, so it reads:
Show Unmanaged | Hide Unmanaged
foo | bar | default | baz | All Namespaces
instead of:
foo | bar | default | baz | All Namespaces
Show Unmanaged | Hide Unmanaged
Based on work from Lorenzo, updated by Iago, Alban, Alessandro and
Michael.
This PR adds connection tracking using eBPF. This feature is not enabled by default.
For now, you can enable it by launching scope with the following command:
```
sudo ./scope launch --probe.ebpf.connections=true
```
This patch allows scope to get notified of every connection event,
without relying on the parsing of /proc/$pid/net/tcp{,6} and
/proc/$pid/fd/*, and therefore improve performance.
We vendor https://github.com/iovisor/gobpf in Scope to load the
pre-compiled ebpf program and https://github.com/weaveworks/tcptracer-bpf
to guess the offsets of the structures we need in the kernel. In this
way we don't need a different pre-compiled ebpf object file per kernel.
The pre-compiled ebpf program is included in the vendoring of
tcptracer-bpf.
The ebpf program uses kprobes/kretprobes on the following kernel functions:
- tcp_v4_connect
- tcp_v6_connect
- tcp_set_state
- inet_csk_accept
- tcp_close
It generates "connect", "accept" and "close" events containing the
connection tuple but also pid and netns.
Note: the IPv6 events are not supported in Scope and thus not passed on.
probe/endpoint/ebpf.go maintains the list of connections. Similarly to
conntrack, it also keeps the dead connections for one iteration in order
to report short-lived connections.
The code for parsing /proc/$pid/net/tcp{,6} and /proc/$pid/fd/* is still
there and still used at start-up because eBPF only brings us the events
and not the initial state. However, the /proc parsing for the initial
state is now done in foreground instead of background, via
newForegroundReader().
NAT resolution on connections from eBPF works in the same way as it did
on connections from /proc: by using conntrack. One of the two conntrack
instances is only started to get the initial state and then it is
stopped since eBPF detects short-lived connections.
The Scope Docker image size comparison:
- weaveworks/scope in current master: 22 MB (compressed), 68 MB
(uncompressed)
- weaveworks/scope with this patchset: 23 MB (compressed), 69 MB
(uncompressed)
Fixes#1168 (walking /proc to obtain connections is very expensive)
Fixes#1260 (Short-lived connections not tracked for containers in
shared networking namespaces)
Fixes#1962 (Port ebpf tracker to Go)
Fixes#1961 (Remove runtime kernel header dependency from ebpf tracker)
This is important for two reasons:
* It prevents nasty false-equality bugs when two different services from different ECS clusters
are present in the same report
* It allows us to retrieve the cluster and service name - all the info we need to look up the service -
using only the node ID. This matters, for example, when trying to handle a control request.
