Commit Graph

75 Commits

Author SHA1 Message Date
Matthias Radestock
1f95e3efd4 obtain historic_reports capability from Reporter 2017-07-31 09:58:31 +01:00
Alfonso Acosta
7b4b410f4a Review Feedback 2017-07-20 11:01:44 +00:00
Alfonso Acosta
c6d5a6a646 Make linter happy 2017-07-20 10:05:48 +00:00
Alfonso Acosta
f66d79d427 Backoff when writing to Dynamo and S3 2017-07-18 15:28:08 +00:00
Filip Barl
b6dfe25499 Time travel control (#2524)
* Hacky working prototype.

* Operate with time.Duration offset instead of fixed timestamp.

* Polished the backend code.

* Made a nicer UI component.

* Small refactorings of the websockets code.

* Fixed the backend tests.

* Better websocketing and smoother transitions

* Small styling refactoring.

* Detecting empty topologies.

* Improved error messaging.

* Addressed some of David's comments.

* Moved nodesDeltaBuffer to a global state to fix the paused status rendering bug.

* Small styling changes

* Changed the websocket global state variables a bit.

* Polishing & refactoring.

* More polishing.

* Final refactoring.

* Addressed a couple of bugs.

* Hidden the timeline control behind Cloud context and a feature flag.

* Addressed most of @davkal's comments.

* Added mixpanel tracking.
2017-06-12 11:22:17 +02:00
Marcus Cobden
9fded9d34e Add weavenet metrics to billing 2017-05-09 15:33:12 +00:00
Matthias Radestock
46f288bb6d close s3 response body to prevent leaks
Fixes #2018
2017-04-14 00:27:41 +01:00
Alfonso Acosta
a404d82a91 Merge pull request #2419 from weaveworks/name-unnamed
Make various anonymous fields named
2017-04-06 14:16:29 +02:00
Paul Bellamy
fbf74b8d40 Add NodeSeconds to billing emitter 2017-04-04 10:33:19 +01:00
Bryan Boreham
515f4b1a47 Make various anonymous fields named
Anonymous fields make any methods on the inner object visible on the
outer, so they should only be used when the outer is-a inner.
2017-04-01 11:35:10 +00:00
Paul Bellamy
9228f7a7fa Container Seconds should not be Container Nanoseconds (#2372) 2017-03-22 17:34:46 +00:00
Paul Bellamy
e68ffd467e Add app/multitenant/billing_emitter to emit billing events 2017-03-22 10:41:34 +00:00
Adam Harrison
83c073fad0 Correct dynamoDB metric label
Discovered whilst attempting to resolve github.com/weaveworks/service/issues/1157
2017-03-17 12:09:40 +00:00
Bryan Boreham
b085c80ef3 Decode via byte slice for memcache and file read
This is more efficient, since the decoder can read field names in-place.
It also appears to be absolutely faster.
2017-03-14 14:54:17 +00:00
Jonathan Lange
e8085b01b6 Use 'common' library 2016-12-07 11:22:38 +00:00
Alban Crequy
3c40892add linter: fix punctuation and capitalization
Symptoms:
> app/control_router.go:44:38: error strings should not be capitalized or end with punctuation or a newline

This is blocking the build on:
https://circleci.com/gh/kinvolk/scope/363
2016-11-23 17:17:14 +01:00
Tom Wilkie
21ee72a384 Add OpenTracing support to TimeRequestHistogram (#2023)
* Vendor open tracing

* Extend TimeRequestHistorgram to also emit opentracing spans.

* Update some of the labels
2016-11-22 14:43:15 +00:00
Matthias Radestock
0610936296 plug goroutine leak in control
Fixes #2002.
2016-11-10 17:45:16 +00:00
Tom Wilkie
32edfc9112 Don't reencode reports in the collector (#1819)
* Don't reencode reports in the collector

* Review feedback

* Fix comment

* Update alpine URLs so it will build

* Fix tests
2016-08-22 17:37:41 +01:00
Matthias Radestock
6ba812e82f restore compatibility of old probes with new m/t app
This got broken in #1682
2016-08-16 23:53:07 +01:00
Matthias Radestock
e6a474ead7 fix report store query chaining
We fell victim to variable shadowing here. Each store would be fed the
original list of report keys, instead of only the ones that weren't
found in the previous store. So if a single report was missing from the
in-process cache, we would then fetch all reports from memcache. And if
that in turn was missing a single report we would fetch all reports from
S3.

We chain report stores for a reason - to reduce latency and, in case of
the in-process cache, eliminate decoding costs. So this bug has a huge
impact on query service performance.

To make matters worse, we return *all* the reports - now possibly in
triplicate. Fortunately, the SmartMerger filters these out, so at least
we were not incurring extra merge costs.
2016-08-16 00:55:39 +01:00
Matthias Radestock
d6e5f0a154 allow more accurate reporting of memcache hit ratio
A lot of time could pass between recording the request count and hit
count pertaining to a particular report fetching batch, which skewed
calculations cache hit ratios.

Fix that by defering the request count recording to the end, which is
when we record the hit count.
2016-08-15 16:25:12 +01:00
Matthias Radestock
9cf178f130 fix MemcacheClient.FetchReports miss & leak on corrupt report
Problem: Decoding a corrupt report grows the 'missing' list. Since we
are waiting for 'len(keys)-len(missing)' decoder go-routines, this
results in waiting for fewer go-routines than we should. The surplus
go-routines leak and we ignore their reports. And since the keys of the
ignored reports are not included in 'missing', we won't attempt to fetch
them from S3 either. Oops.

Fix: calculate the number of go-routines once, at the beginning.
2016-08-15 10:44:29 +01:00
Krzesimir Nowak
dcaa7503b8 Fix a typo
The typo is here from March, 2016. It is strange that it wasn't
detected earlier.
2016-08-04 11:36:04 +02:00
Jonathan Lange
a3648f0c89 Inline StoreBytes 2016-07-15 12:58:27 +01:00
Jonathan Lange
bbd75ddd24 Use memcache compression level from config 2016-07-15 11:24:37 +01:00
Jonathan Lange
1fd8a5fb88 Use StoreReport in main AWS routine 2016-07-15 11:24:37 +01:00
Jonathan Lange
46dfeb627d Call it reportKey 2016-07-15 11:24:37 +01:00
Jonathan Lange
0058229687 Extract functions for calculating keys
Not so much for re-use as to help jml understand what's going on
2016-07-15 11:24:36 +01:00
Jonathan Lange
270a55060f Add StoreReport methods to stores
Not sure if we'll use them.
2016-07-15 11:24:36 +01:00
Jonathan Lange
60e14c1dc2 Plumb through an option for compression 2016-07-15 11:24:36 +01:00
Jonathan Lange
2bfd6d7eb7 Parametrize compression level 2016-07-15 11:24:36 +01:00
Jonathan Lange
d83d7318d0 Remove the old metric 2016-07-12 18:16:35 +01:00
Jonathan Lange
d2298aa8f3 Store a histogram of report sizes 2016-07-12 16:37:29 +01:00
Tom Wilkie
3173f6ad75 Use histograms over summaries 2016-07-12 11:15:57 +01:00
Jonathan Lange
49f2e4e40c Count memcache requests even if they time out 2016-07-11 13:01:02 +01:00
Paul Bellamy
8cb1ecdf2c Merge pull request #1642 from weaveworks/refactoring-timing
refactor some timing helpers into a common lib
2016-07-05 13:01:28 +01:00
Paul Bellamy
7736564337 refactor some timing helpers into a common lib 2016-07-05 12:29:00 +01:00
Jonathan Lange
31c88fd62b Instrumentation that we might like to keep 2016-07-04 16:03:50 +01:00
Jonathan Lange
7dd2c6371e Parametrize window rather than assuming default 2016-07-04 13:50:54 +01:00
Jonathan Lange
c1dab17fb3 Make expiration a Duration 2016-07-04 13:30:23 +01:00
Jonathan Lange
f7bdedc149 Config struct for memcache client 2016-07-04 13:25:45 +01:00
Jonathan Lange
96520d7a46 Fixes to memcache support (#1628)
* Fix errors discovered in dev

* Log an error rather than aborting when memcache doesn't resolve
* Initialize map correctly

* Review tweaks
2016-07-04 11:00:11 +01:00
Jonathan Lange
9e0f0c51b9 Configuration type for AWS collector 2016-06-30 17:01:58 +01:00
Jonathan Lange
baacaa8cc5 Rename dynamoCollector to awsCollector 2016-06-30 16:44:43 +01:00
Jonathan Lange
6520f8f5f3 Pass in memcache client 2016-06-30 09:59:55 +01:00
Jonathan Lange
abec257c59 Just pass in the s3 client 2016-06-30 09:57:49 +01:00
Jonathan Lange
d984605de1 Write back to the in-process cache 2016-06-30 09:57:49 +01:00
Jonathan Lange
5ec422c7a3 Fetch all reports at once
Rather than have getReports be responsible for determining keys, instead
call getReportKeys directly and then pass keys to getReports
2016-06-30 09:57:49 +01:00
Jonathan Lange
87da22767e Move s3 logic to separate file 2016-06-30 09:57:49 +01:00