Commit Graph

488 Commits

Author SHA1 Message Date
Bryan Boreham
b5376facf2 Cache merged groups of reports, to reduce the number we handle in parallel
Previously we would merge all reports in a 15-second window.
Now we use a 'quantum' of 3 seconds, similar to the single-user app.

E.g. a 30-node cluster will have 150 individual reports over 15
seconds, but the new code will merge 5 pre-merged reports plus 20-ish
very recent individual ones.

This limits the max heap size used for deserialising, since we only do
3 seconds at once per instance.

Individual reports are still put into the cache, but should get
displaced by the pre-merged ones under LRU.
2019-09-09 10:00:26 +00:00
Bryan Boreham
70550ca34a Refactor: pull userid fetch up out of getReportKeys() 2019-09-09 08:19:55 +00:00
Bryan Boreham
589c4c4d0b Refactor: pull time interval computation up out of getReportKeys() 2019-09-08 12:27:57 +00:00
Bryan Boreham
26c8760877 Merge pull request #3605 from weaveworks/defer-metrics-registration
Defer metrics registration until we need it
2019-07-16 15:45:30 +01:00
Bryan Boreham
89363f5dcf Defer metrics registration until we need it
This avoids app-specific metrics appearing in the probe.
2019-07-04 14:24:22 +00:00
Bryan Boreham
1e2206963a Merge pull request #3599 from weaveworks/per-tenant-metrics
Add metrics for report size and count per tenant
2019-05-15 13:44:42 +01:00
Bryan Boreham
870b52eec0 Review feedback: metric description 2019-05-15 12:43:22 +00:00
Bryan Boreham
711aa66bd5 Add OpenTracing span for report.ReadBinary()
So we can see the timing and size in Jaeger.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2019-05-10 15:34:53 +00:00
Bryan Boreham
4c74f8b1cf Add metrics for report size and count per tenant
In a multitenant system it is useful to be able to drill into which
tenants have the most or biggest reports.

Signed-off-by: Bryan Boreham <bryan@weave.works>
2019-05-10 14:49:57 +00:00
Filip Barl
050017085e Use Time Travel context when downloading raw reports. 2019-03-14 15:54:50 +01:00
Filip Barl
c5022bd2bb Code cleanup. 2019-02-22 14:56:44 +01:00
Filip Barl
0f1b7e5972 Prepare to filter node summaries post-render. 2019-02-21 16:18:24 +01:00
Filip Barl
3c5320ef09 Polished the code and applied censoring to other API endpoints. 2019-02-21 14:34:05 +01:00
Filip Barl
97fdcdc525 Option to censor raw reports by command line args and env vars. 2019-02-15 17:31:08 +01:00
Bryan Boreham
ee0ce7b087 Merge pull request #3384 from weaveworks/drop-big-topologies
In multitenant app, drop all nodes for big topologies
2018-11-01 17:21:55 +00:00
Bryan Boreham
3be8cf71dd Add more Opentracing detail to the app (#3383)
* Pass Go context down to Renderers

This is useful for cancellation or tracing.

* Add tracing spans to app

Also log things like number of nodes in Map, total number of reports.
2018-10-26 11:21:33 +05:30
Akash Srivastava
5b8a8c9f56 Merge branch 'master' into volume-snapshot 2018-10-17 23:36:09 +05:30
Akash Srivastava
81a682893f Add kubernetes volume snapshot support
- Add a new client in probe/kubernetes/client.go
  which will be used to fetch details of Snapshot CRD
- Visualize VolumeSnapshot and VolumeSnapshotData
- Add adjacency between PV and VolumeSnapshot
- Add adjacency between VolumeSnapshot and VolumeSnapshotData
- Add snapshot filter to hide and show snapshots

Signed-off-by: Akash Srivastava <akash.srivastava@openebs.io>
2018-10-16 01:04:44 +05:30
Bryan Boreham
05b350850f Drop topologies which are way too big 2018-10-11 17:20:16 +00:00
Bryan Boreham
27047c3297 Embed AWSCollectorConfig instead of duplicating the fields
This simplifies adding more fields later.
2018-10-11 15:57:45 +00:00
Bryan Boreham
724ccd9826 Use the Context in the request 2018-10-09 15:50:59 +00:00
Marc Carré
2ba50b8b3d Update golang.org/x/net/context to latest
```
$ gvt delete golang.org/x/net/context
$ gvt fetch golang.org/x/net/context
2018/07/23 18:03:49 Fetching: golang.org/x/net/context
$ git grep -l "golang.org/x/net/context" | grep -v vendor | xargs sed -i '' 's:golang.org/x/net/context:context:g'
$ git grep -l "context/ctxhttp" | grep -v vendor | xargs sed -i '' 's:context/ctxhttp:golang.org/x/net/context/ctxhttp:g'
$ gofmt -s -w app
$ gofmt -s -w common
$ gofmt -s -w probe
$ gofmt -s -w prog
$ gofmt -s -w tools
```
fixed a bunch of:
```
cannot use func literal (type func("github.com/weaveworks/scope/vendor/golang.org/x/net/context".Context) error) as type func("context".Context) error
```
2018-07-23 20:10:18 +02:00
Marc Carré
d46c2266ce Change Sirupsen/logrus to sirupsen/logrus
```
$ git grep -l Sirupsen | grep -v vendor | xargs sed -i '' 's:github.com/Sirupsen/logrus:github.com/sirupsen/logrus:g'
$ gofmt -s -w app
$ gofmt -s -w common
$ gofmt -s -w probe
$ gofmt -s -w prog
$ gofmt -s -w tools
```
2018-07-23 20:10:14 +02:00
Bryan Boreham
dad34df575 Merge pull request #3267 from weaveworks/pool-gzwriters
Re-use gzip writers in a pool
2018-07-13 16:09:24 +01:00
Alfonso Acosta
0a5fb7aad3 Remove ProcessWithContainerNameRenderer, it wasn't working (#3263) 2018-07-13 13:34:37 +02:00
Bryan Boreham
14dc98343e Use newer gzip handler for performance
"github.com/NYTimes/gziphandler" keeps a pool of gzip writers, which
saves on garbage-collection.

Also the old one, github.com/PuerkitoBio/ghost, is marked "Status : Unmaintained"
2018-07-12 19:13:47 +00:00
Bryan Boreham
a06d82ccf8 Shuffle reports before merging, in benchmark
So that we have equal chance of merging older into newer or vice-versa
2018-07-09 21:34:21 +00:00
Bryan Boreham
3c07c93847 Merge pull request #3236 from weaveworks/fast-merger
Faster report merging through mutating objects
2018-06-22 16:37:07 +01:00
Bryan Boreham
3309d09ad8 Remove slower mergers 2018-06-22 11:59:43 +00:00
Bryan Boreham
126a171f62 Make 'fast' merger the default 2018-06-22 11:59:43 +00:00
Bryan Boreham
1706746a32 Faster report merging through mutating objects
When we know we have the only reference to a Report or Node object we
can avoid copying the data to change it. Add "Unsafe" variants of
various Merge operations which mutate the receiver, and a new Merger
which takes advantage of them.
2018-06-22 11:59:43 +00:00
Bryan Boreham
06c895267c Reports: streamline report serialization
Move the creation of the buffer and the choice of compression level
(which never changes) into WriteBinary(), to simplify the code.
2018-06-19 22:59:01 +00:00
Bryan Boreham
8c18a785e7 Merge pull request #3132 from openebs/add-pv-pvc-support
Add Kubernetes Storage (pv and pvc) support in Weave Scope
2018-06-08 12:41:21 +01:00
Satyam Zode
d9bd17af8a Refactor edge frontend code and reporter
- Refactor dotted edge logic.
- Change Storage view to show storage components as well as all
  the pods.
- Sentence case storage related variables.

Signed-off-by: Satyam Zode <satyam.zode@openebs.io>
2018-06-08 16:37:48 +05:30
Satyam Zode
24741bec2a Simplify volume rendering code in render
- Remove the complex volume rendering code.
- Remove volume specific code from Transform.
- Rename filtering methods
- Remove extra params from PV, PVC nodes to make table view look good.

Signed-off-by: Satyam Zode <satyam.zode@openebs.io>
2018-06-08 16:37:42 +05:30
Satyam Zode
d26b2c3805 Add Kubernetes storage class resource to weave scope
This will:

- Add StorageClass resource. Storage classes are mentioned
in the PVC spec. We're using storage class name from PVC spec to
add adjacency to the PVC node.
- Add square sheet shape for StorageClass.
- Add storage filter in the PODS topology.
Storage Filter will allow user to see distinct view of
stateful applications.
- Add visually distinct edge to show storage adjacency.

Signed-off-by: Satyam Zode <satyam.zode@openebs.io>
2018-06-08 16:36:29 +05:30
Satyam Zode
23210a6a77 Add Kubernetes volumes support in Weave Scope
This will

- Add Kubernetes volume resources such as PV, PVC.
- Add shapes for Kubernetes PV and PVC
- Add `Cylinder` shape for PV and `Dotted Cylinder` shape for PVC.

Signed-off-by: Satyam Zode <satyam.zode@openebs.io>
2018-06-07 17:01:29 +05:30
Simon
4647ce1901 Merge pull request #3165 from weaveworks/default-filter
Use the default value for a TopologyOption if omitted
2018-05-28 14:36:25 +02:00
Roland Schilter
f012c23ca1 Sentence cased text everywhere (#3166)
* Sentence cased text everywhere

Follows Weave Cloud's direction of sentence case on most things.

* More space between sorter caret and label

* Use full topology name for table header
2018-05-17 17:30:38 -07:00
Bryan Boreham
a64db66010 Use the default value for an APITopologyOption if omitted 2018-05-09 19:21:03 +00:00
Bryan Boreham
7da7183a37 Refactor: simplify default type of option 2018-05-09 19:07:38 +00:00
Marcus Cobden
ba81924278 Add CLI flag for SQS RPC timeout 2018-05-04 10:11:25 +01:00
Bryan Boreham
593952cd6e Move report creation outside of timing loop 2018-04-12 10:02:50 +00:00
Matthias Radestock
5b30b668ae refactor: don't return receiver in Topology.AddNode()
This had little use and was obscuring the mutating nature of
AddNode().
2018-02-19 05:10:04 +00:00
Roberto Bruggemann
710d665c41 Upgrade k8s.io/client-go to kubernetes-1.9.1
Upgraded from 99c19923, branch release-3.0.

This required fetching or upgrading the following:
* k8s.io/api to kubernetes-1.9.1
* k8s.io/apimachinery to kubernetes-1.9.1
* github.com/juju/ratelimit to 1.0.1
* github.com/spf13/pflag to 4c012f6d

Also, update Scope's imports/function calls to be compatible with the new client.
2018-01-30 10:14:42 +00:00
Roberto Bruggemann
d3c53bacd6 'updateKubeFilters` returns early if there are no namespaces
This change makes sure an empty `APITopologyOptionGroup` struct is not created for namespaces, if there are no namespaces to report.
2018-01-08 17:37:49 +00:00
Roberto Bruggemann
1fb23dd8f9 Extract namespaces from report.Namespace
This avoids unnecessary loops.
2018-01-03 13:55:27 +00:00
Matthias Radestock
d1bb4aa45e refactor: remove APITopologyDesc.filter
It's always set to render.FilterUnconnectedPseudo, so we can simply
use that constant in the one function (RendererForTopology) that
looked at that value.
2018-01-02 10:33:57 +00:00
Matthias Radestock
ec7c468758 make it possible for users to see unconnected processes
We prevented this because rendering them was expensive. It still is,
but less so and actually works well, especially in table mode.
2018-01-02 09:40:18 +00:00
Matthias Radestock
286453b9a9 refactor: move RenderContext where it belongs
It shouldn't be in 'report' because that is shared between the probe
and the app, and RenderContext, as the name suggests, is only needed
in the app.
2017-12-27 16:22:52 +00:00