It's a dynamic type check of sorts.
In the process we stop abusing ParseNodeID for extracting the host,
which in turn allows us to clarify its purpose.
Since we seed the joinResult with the nodes from the topology we are
mapping to, we know the 'create' function is only called when there is
no node with the specified id.
This neatly makes the 'create' function only do what it says,
i.e. return _new_ nodes.
We used to ignore source endpoints that are associated with multiple
destination endpoints, which is a partial workaround for our inability
to correctly represent two connections from the same source ip/port
but different processes, or the same destination ip/port but different
processes. See #2665.
However, that condition is too coarse. In particular, we end up
ignoring endpoints that are connected to NATed destinations, since the
latter are represented by two (or more) endpoints.
The change here corrects that.
This eliminates the awkward distinction between ProcessRenderer and
ColorConnectedProcessRenderer.
It also ensures that processes resulting from direct rendering of the
process topology (/api/topology/processes is invoking
ProcessWithContainerNameRenderer and /api/topology/processes-by-name
is invoking ProcessNameRenderer) are colored and hence summarising
them correctly sets the 'linkable' property. This was the behaviour
prior to the revamping of the rendering pipeline. However, it doesn't
actually make a practical difference since process detail panels only
show other processes as connection endpoints, and these are always
marked linkable anyway.
This is preparatory to future refactorings: all existing calls are to
Endpoints which have no children and where we don't want a Counter.
We make addChildAndChildren an obvious extension of addChild even
though it adds a dead code path (we never call addChildAndChildren
with an endpoint).
and add a comment indicating non-memoisation of other, not shared
top-level renderers.
This memoisation is effective when the browser requests multiple
topologies for the same report.
The filtering of endpoints causes some connections to get missed for
non-eBPF-tracked connections. Furthermore, the filtering of endpoints
is entirely pointless when the probes run eBPF since the filters just
pass through eBPF-tracked endpoints (for good reason too; because
otherwise some connections would be missed). So in that case it is
just costing CPU and removing it actually improves performance.
Note that removing the filtering does not result in over-counting
connections since that is done by source ip:port pairs.
Fixes#2551.
Fixes#2558.
ProcessRenderer was coloring connected nodes because we need that info
for rendering details panels. However, the main process topology view
renderers depending on ProcessRenderer were also doing coloring
themselves. For the 'processes' topology that was literally
duplicating work. For the 'processes-by-name' topology that was
throwing away the process coloring, and then coloring at the name
level.
Solution: remove the coloring from the ProcessRenderer, thus
eliminating the duplicate/thrown-away work, and introduce a
ColorConnectedProcessRenderer which is only used in places that
populate details panels.
the information is constant and already present in the id, so we can
extract it from there.
That reduces the report size and improves report encoding/decoding
performance. It should reduce memory usage too and improve report
merging performance too.
NB: Probes with this change are incompatible with old apps.
Based on work from Lorenzo, updated by Iago, Alban, Alessandro and
Michael.
This PR adds connection tracking using eBPF. This feature is not enabled by default.
For now, you can enable it by launching scope with the following command:
```
sudo ./scope launch --probe.ebpf.connections=true
```
This patch allows scope to get notified of every connection event,
without relying on the parsing of /proc/$pid/net/tcp{,6} and
/proc/$pid/fd/*, and therefore improve performance.
We vendor https://github.com/iovisor/gobpf in Scope to load the
pre-compiled ebpf program and https://github.com/weaveworks/tcptracer-bpf
to guess the offsets of the structures we need in the kernel. In this
way we don't need a different pre-compiled ebpf object file per kernel.
The pre-compiled ebpf program is included in the vendoring of
tcptracer-bpf.
The ebpf program uses kprobes/kretprobes on the following kernel functions:
- tcp_v4_connect
- tcp_v6_connect
- tcp_set_state
- inet_csk_accept
- tcp_close
It generates "connect", "accept" and "close" events containing the
connection tuple but also pid and netns.
Note: the IPv6 events are not supported in Scope and thus not passed on.
probe/endpoint/ebpf.go maintains the list of connections. Similarly to
conntrack, it also keeps the dead connections for one iteration in order
to report short-lived connections.
The code for parsing /proc/$pid/net/tcp{,6} and /proc/$pid/fd/* is still
there and still used at start-up because eBPF only brings us the events
and not the initial state. However, the /proc parsing for the initial
state is now done in foreground instead of background, via
newForegroundReader().
NAT resolution on connections from eBPF works in the same way as it did
on connections from /proc: by using conntrack. One of the two conntrack
instances is only started to get the initial state and then it is
stopped since eBPF detects short-lived connections.
The Scope Docker image size comparison:
- weaveworks/scope in current master: 22 MB (compressed), 68 MB
(uncompressed)
- weaveworks/scope with this patchset: 23 MB (compressed), 69 MB
(uncompressed)
Fixes#1168 (walking /proc to obtain connections is very expensive)
Fixes#1260 (Short-lived connections not tracked for containers in
shared networking namespaces)
Fixes#1962 (Port ebpf tracker to Go)
Fixes#1961 (Remove runtime kernel header dependency from ebpf tracker)
* Add filters for pseudo nodes.
- Don't filter the internet node as a pseudo node.
- Rename pseudo filter to unmanaged/uncontained.
- Review feedback
- Move the FilterFoo funcs into the tests
- Drop the 'nodes' from filter labels.
* Fix experimental
Squash of:
* We have to keep all the container hostnames until the end so we can
count how many we've filtered
* Adding tests for ContainerHostnameRenderer and PodServiceRenderer with
filters
* Because we filter on image name we need the image name before
filtering
* Alternative approach to passing decorators.
* Refactor out some of the decorator capture
* Don't memoise decorated calls to Render
* Fixing filtered counts on containers topology
Tricky, because we need the filters to be silent sometimes (when they're
in the middle), but not when they're at the top, so we take the "top"
filter's stats. However, this means we have to compose all
user-specified filters into a single Filter layer, so we can get all
stats.
There are no more Silent filters, as all filters are silent (unless they
are at the top).
Additionally, I clarified some of the filters as their usage/terminology
was inconsistent and confused. Now Filter(IsFoo, ...) *keeps* only nodes
where IsFoo is true.
