The change is necessitated by the removal of procspied/ebpf endpoint
filtering in the renderers, as a result of which the odd
conntracked-only, unconnected pseudo node can sneak through.
This new way of doing things also makes renderers more composable and
robust, and more directly reflects the objective:
- in the process topologies, filter out all unconnected nodes
- in all other topologies, filter out unconnected pseudo nodes
The filtering of endpoints causes some connections to get missed for
non-eBPF-tracked connections. Furthermore, the filtering of endpoints
is entirely pointless when the probes run eBPF since the filters just
pass through eBPF-tracked endpoints (for good reason too; because
otherwise some connections would be missed). So in that case it is
just costing CPU and removing it actually improves performance.
Note that removing the filtering does not result in over-counting
connections since that is done by source ip:port pairs.
Fixes#2551.
Fixes#2558.
* Maps metrics if there is a single pod in the controller, as per all other views.
* Also added heavy commenting on the increasingly-complex render chain
Since there are multiple types in the same topology, displaying the type is important.
We do this in multiple places:
* Add node type to minor label
* Add node type as metadata and include in metadata template.
Even though this will always be the same for every node of that topology, this was
the easiest way to add it so it displays in the table view.
Note we can't control ordering of columns in table view, it's always alphabetical.
The existing technique of "reducing" the two rendered graphs for daemonsets and deployments
had a glaring issue that no connections would ever be made between nodes of different types,
since that information would've been discarded earlier in the process.
It also makes it hard to identify "parentless" pods.
This commit extends the Map2Parent function, teaching it:
* To check multiple topologies for parents
* To pass through nodes with no parents found without modification
Since we already had two 'modes' for what to do with nodes without parents,
and it would've been clunky to try to encode the third option into the existing PseudoNodeID
arg in some way, we instead split it into two args, with the first being an enum specifying
either the old pseudo node behaviour, the old drop behaviour, or the new keep behaviour.
We then use the new Map2Parent to map pods to:
* A replica set, if it has one
* A daemonset, if it has one
* Itself, if neither of the above
and then map again from the results to any deployment, leaving as-is any nodes that
don't map to a deployment. Hence we are left with:
* Deployments
* Daemonsets
* Replica sets, but only if they map to no deployment
* Pods, but only if they map to none of the above
and connections between all these will be calculated correctly.
ProcessRenderer was coloring connected nodes because we need that info
for rendering details panels. However, the main process topology view
renderers depending on ProcessRenderer were also doing coloring
themselves. For the 'processes' topology that was literally
duplicating work. For the 'processes-by-name' topology that was
throwing away the process coloring, and then coloring at the name
level.
Solution: remove the coloring from the ProcessRenderer, thus
eliminating the duplicate/thrown-away work, and introduce a
ColorConnectedProcessRenderer which is only used in places that
populate details panels.
The rendering code checks whether endpoint IPs are part of
cluster-local networks. Due to the prevalence of endpoints - medium
sized reports can contain many thousands of endpoints - this is
performance critical. Alas the existing code performs the check via a
linear scan of a list of networks. That is slow when there are more
than a few, which will be the case in the context of k8s, since there
the probes register service IPs as local /32 networks.
Here we change representation of the set of networks to a prefix
tree (aka trie), which is well-suited for IP network membership checks
since networks are in fact a bitstring prefixes.
The specific representation is a crit-bit tree, but that choice was
purely based on implementation convenience - the chosen library is the
only one I could find that directly supports IP networks.
The rendering code checks whether endpoint IPs are part of
cluster-local networks. Due to the prevalence of endpoints - medium
sized reports can contain many thousands of endpoints - this is
performance critical. Alas the existing code performs the check via a
linear scan of a list of networks. That is slow when there are more
than a few. Unfortunately in some common k8s network setups, e.g. on
AWS, a cluster can contain hundreds of networks, due to /32 networks
derived from interfaces with multiple IPs.
Here we change representation of the set of networks to a prefix
tree (aka trie), which is well-suited for IP network membership checks
since networks are in fact a bitstring prefixes.
The specific representation is a crit-bit tree, but that choice was
purely based on implementation convenience - the chosen library is the
only one I could find that directly supports IP networks.
the information is constant and already present in the id, so we can
extract it from there.
That reduces the report size and improves report encoding/decoding
performance. It should reduce memory usage too and improve report
merging performance too.
NB: Probes with this change are incompatible with old apps.
We have to introduce the kinda hacky concept of a 'No Stack' stack
to reconcile it with the idea of a 'default' k8s namespace. This is important
because swarm services without a stack don't have the same docker labels as ones that do.
Curiously, they still have what appears to be a stack name 'prefix' on their names,
but I can't isolate that name anywhere easily so they'll just have to make do.
I basically copy-pasted updateFilters to make this work, todo go back and refactor
to not duplicate 90% of the code.
We replace the existing data structure with a simpler one that
only specifies how to get the parent label, which is the only
part of the Parent struct that can't be generated from the node info alone.
Future work: Standardize this concept of a label and put it in the topology instead.
Though that already exists...so just use it?
The default sets the node label to the node ID.
This is likely to not look very good, but the intent is that it creates an obvious problem,
ie. that the node ID is being used as the label, rather than a silent omission or more subtle problem.
Possible future work:
* For single-component IDs, extract the component automatically and use that instead.
* Instead of functions, in simple cases just have a LUT by topology with common behaviours like
'stack = true or false', 'label = this key in node.Latest'
The latter opens up to eventually moving this info inside the report itself ala topology templates,
or at least centralizing it in the source.
Currently, if a topology does not have any specific info in nodeSummariesByID,
any children of the node that belong to that topology will be silently omitted.
This change adds a default behaviour for such topologies, with no special columns
but at least it is displayed at all.
Unlisted topologies are displayed after all listed ones, in arbitrary order.
Note that completely bogus or other special cases (eg. topology = Pseudo) still will not
be displayed as report.Topology() will fail.
This gives us a single source of truth in a variety of situations where we want to
know what view to direct a user to in order to 'open' a particular node.
I wanted to put this in app/api_topologies where the views are defined, but that creates
a circular import.
This field changes the option group behaviour depending on its value.
Currently only supports two values:
"one" (default): Old behaviour, one option can be selected
"union": Any number of options can be selected, and the filters are OR-ed togther
It is written in such a way as to easily enable a future "intersection" option,
as per union but AND-ing the filters. But this is not done here. YAGNI.
This reverts commit 76ddc75fb8, reversing
changes made to 3ade2933eb.
We are rolling this back for now because it's causing a bug where sub-topologies
would have ~3000 repeated cases of the k8s filters, causing performance issues clientside.
To facilitate this, we replace the existing functionality of updateFilters which
sets k8s topologies to have the filters [namespace, managed], to instead append the namespace filter
to any existing. This lets it apply to both k8s and container topologies without overwriting existing
container filters. We instead set the managed filter in the static definition.
This however has the side effect that the ordering of the namespace filter and the managed filter
in k8s topologies has been reversed, so it reads:
Show Unmanaged | Hide Unmanaged
foo | bar | default | baz | All Namespaces
instead of:
foo | bar | default | baz | All Namespaces
Show Unmanaged | Hide Unmanaged
