Limiting env vars, docker&k8s labels, and weave net connection entries
to 20 is problematic because
- the truncation is arbitrary - there is a good chance that if you
care about a specific entry it won't be there
- the truncation is not consistent - different entries get truncated
at different times
- some of the rendering logic depends on specific labels, for example
namespace filtering of containers depends on the
`io.kubernetes.pod.namespace` label.
In practice, there should never be a huge number of labels, or Weave
Net connection entries. So there is no need to truncate them.
That leaves env vars. These are of limited use, so we now omit them by
default. If they are included they are included in full, so they are
actually useful.
Fixes#3127
If we have the node-name then we can save work requesting the list of
local pods from kubelet. And the best way to get the node name is via
the "downwards api", i.e. put this in the Daemonset Yaml:
- name: KUBERNETES_NODENAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
scope-app:
* Adds `-app.metrics-graph` cli flag for configuring the base url to
use for graph links; supports :orgID and :query placeholders
* Assigns query URLs to existing metrics and appends empty metrics if missing
scope-ui:
* Extends <CloudFeature /> with option alwaysShow
* Adds <CloudLink /> to simplify routing when in cloud vs not in cloud
* Links metric graphs in the ui's node details view for all k8s
toplogies and containers so far
* Tracks metric graph click in mixpanel `scope.node.metric.click`
* Uses percentages and MB for CPU/Memory urls
* Passes timetravel timestamp to cortex in deeplink
The figure is based on our experience of running scope probes in
production systems. Below this we see a lot of "conntrack
stderr:WARNING: We have hit ENOBUFS! We are losing events."
Changed default for flag `-app.docker` to use the DOCKER_* env variables
instead of hardcoded /var/run/docker.sock; uses docker's default if
no DOCKER_HOST defined, for both probe and app.
Fixes#1975
Based on work from Lorenzo, updated by Iago, Alban, Alessandro and
Michael.
This PR adds connection tracking using eBPF. This feature is not enabled by default.
For now, you can enable it by launching scope with the following command:
```
sudo ./scope launch --probe.ebpf.connections=true
```
This patch allows scope to get notified of every connection event,
without relying on the parsing of /proc/$pid/net/tcp{,6} and
/proc/$pid/fd/*, and therefore improve performance.
We vendor https://github.com/iovisor/gobpf in Scope to load the
pre-compiled ebpf program and https://github.com/weaveworks/tcptracer-bpf
to guess the offsets of the structures we need in the kernel. In this
way we don't need a different pre-compiled ebpf object file per kernel.
The pre-compiled ebpf program is included in the vendoring of
tcptracer-bpf.
The ebpf program uses kprobes/kretprobes on the following kernel functions:
- tcp_v4_connect
- tcp_v6_connect
- tcp_set_state
- inet_csk_accept
- tcp_close
It generates "connect", "accept" and "close" events containing the
connection tuple but also pid and netns.
Note: the IPv6 events are not supported in Scope and thus not passed on.
probe/endpoint/ebpf.go maintains the list of connections. Similarly to
conntrack, it also keeps the dead connections for one iteration in order
to report short-lived connections.
The code for parsing /proc/$pid/net/tcp{,6} and /proc/$pid/fd/* is still
there and still used at start-up because eBPF only brings us the events
and not the initial state. However, the /proc parsing for the initial
state is now done in foreground instead of background, via
newForegroundReader().
NAT resolution on connections from eBPF works in the same way as it did
on connections from /proc: by using conntrack. One of the two conntrack
instances is only started to get the initial state and then it is
stopped since eBPF detects short-lived connections.
The Scope Docker image size comparison:
- weaveworks/scope in current master: 22 MB (compressed), 68 MB
(uncompressed)
- weaveworks/scope with this patchset: 23 MB (compressed), 69 MB
(uncompressed)
Fixes#1168 (walking /proc to obtain connections is very expensive)
Fixes#1260 (Short-lived connections not tracked for containers in
shared networking namespaces)
Fixes#1962 (Port ebpf tracker to Go)
Fixes#1961 (Remove runtime kernel header dependency from ebpf tracker)
* Add options to hide args and env vars
To allow for use of weave-scope in an unauthenticated environment,
add options to the probe to hide comand line arguments and
environment variables, which might contain secret data.
Fixes#2222
* Change docker.NewRegistry arguments to be a struct
* Remove redundant declarations of default values
* Move registry options outside to improve readability
Now you can launch the scope app with something like
./prog/scope --mode=app --weave=false --app.collector=file:///tmp/reports
and if the specified dir contains reports with filenames in the form
<timestamp>.{msgpack|json}[.gz],
e.g. "1488557088545489008.msgpack.gz", then these reports are replayed
in a loop at a sequence and speed determined by the timestamps.
User can control the webserver listen address with `-app.http.address`.
If they do, `scope --mode probe ...` should use the port part of address
in the default target and not ':4040'. Otherwise we encounter a
`dial tcp 127.0.0.1:4040: getsockopt: connection refused` error and
`scope-probe` cannot report.
Signed-off-by: Michael Schubert <michael@kinvolk.io>
gofmt load_container_filters.go
removed the environment variable for container label filters
Added the --app.container-label-filter command line argument, and load_container_filters.go now uses the results from that
Changed init() to InitializeTopologies()
Changed init() to InitializeTopologies() so that it can be called after the container filters are loaded from the command line argument. init() executes before main() in prog/main.go, so the flag parsing isn't finished before init() is called
Applied lint fixes
fixed lint issues
brought back the init function for api_topologies.go
Addressed many of the PR comments, except escaping colons
Renamed IsDesired to HasLabel in render/filters.go
Allows for the user to escape colons
added registry function for modifying the container filters
created a separate function that parses the container filter flags
simplified registry.addContainerFilters()
addressed review comments
switched API Topology Description IDs to constants
addressed review comments
joined constants
added test functions
addressed most of the review comments
Changed containerLabelFilters to an array of APItopologyOptions, placing the parsing in the Set() function. Removed parsing from HasLabel in render/filters.go
refactored code
added test that applies to the container filtering by labels
applied golint
made Registry items private and added a MakeRegistry() function
fixed usage of topologyRegistry.RendererForTopology
Added container label filters by exclusion
minor update to report_fixture
Modified container labels test to use existing report
I added labels to the existing containers in the fixed report for testing.
refactored code
refactored code
further code refactoring
addressed @ijsnellf's review comments
unexported Registry, and reduced duplicate code
addressed @ijsnellf's review comments
Addressed review comments
Addressed final review comments
Also:
- Parse targets on startup and catch badly formed ones before Scope can start.
- If no port is specified, use default port for scheme; if no scheme is specificed, use 4040.
- Use username as probe token
