Commit Graph

58 Commits

Author SHA1 Message Date
Bryan Boreham
ae9441bb53 Fix batching to dynamodb 2019-06-08 11:23:48 +00:00
Bryan Boreham
2d588a57a7 Implement pagination on DynamoDB queries 2019-06-06 17:08:22 +00:00
Bryan Boreham
4ce900e466 Fix delete record parsing 2019-06-06 16:48:07 +00:00
Bryan Boreham
e60e25da4a Remove duplicate 'segments' flag 2019-06-06 16:41:45 +00:00
Bryan Boreham
cfe7ca7cea Read a file of org-hour records 2019-06-04 17:07:58 +00:00
Bryan Boreham
18cd43d2bf Refactor: delete from DynamoDB in same goroutine as S3 2019-06-04 16:07:31 +00:00
Bryan Boreham
bc59b03f67 Fix errorf format bug 2019-06-04 15:14:10 +00:00
Bryan Boreham
cdf28708e8 Refactor: extract deleteOneOrgHour() method 2019-06-04 15:13:48 +00:00
Bryan Boreham
388d273e5f Add 'just scan' option 2019-04-09 09:46:19 +00:00
Bryan Boreham
b23f7a7b0d Default stop hour to now 2018-11-20 11:56:50 +00:00
Bryan Boreham
2fe3f46cf7 Handle retries ourselves 2018-11-20 07:53:20 +00:00
Bryan Boreham
a81bd74a4a Use rate-limiters on DynamoDB operations 2018-11-18 22:53:28 +00:00
Bryan Boreham
1ce9b1e53c Refactor to make bigger S3 writes 2018-11-17 13:47:35 +00:00
Bryan Boreham
c6f16436a9 Sample Kubernetes job 2018-11-15 18:19:17 +00:00
Bryan Boreham
ab796c7c6f Fix up logging 2018-11-15 18:18:07 +00:00
Bryan Boreham
7143356fd5 Move to extras directory, add Makefile, Dockerfile 2018-11-15 18:17:27 +00:00
Bryan Boreham
dad34df575 Merge pull request #3267 from weaveworks/pool-gzwriters
Re-use gzip writers in a pool
2018-07-13 16:09:24 +01:00
Bryan Boreham
dac008b9c4 Early return from LatestMap merge loop
If we run out of things to look at in the other map, return quickly.
Also move the equal-key case above the less-than case, since maps with
equal keys are the common case when merging.
2018-07-12 19:54:01 +00:00
Bryan Boreham
c91fd6c67b Optimise merge of LatestMaps with same keys, different timestamps
Swap the two maps so we are merging older into later.
2018-07-12 19:54:01 +00:00
Bryan Boreham
43a0a7e5d3 Don't copy LatestMap if merging a subset 2018-07-12 19:50:02 +00:00
Bryan Boreham
c96611b13d Less verbose string dump of LatestMap
The default time.Time.String() includes nanoseconds, wall-clock adjustment, etc.
Also don't include the key twice when stringizing the map itself.
2018-07-08 16:32:51 +00:00
Bryan Boreham
70cd79307e Re-use gzip writers in a pool
Since they allocate sizeable chunks of memory, this can save on garbage-collection.
2018-07-08 16:30:07 +00:00
Bryan Boreham
06c895267c Reports: streamline report serialization
Move the creation of the buffer and the choice of compression level
(which never changes) into WriteBinary(), to simplify the code.
2018-06-19 22:59:01 +00:00
Bryan Boreham
96f51c47af probe: Eliminate Publisher interface from app_client
Simplification: everything now implements Publish(Report), and we do
away with writing/reading/writing in the MultiAppClient.
2018-06-19 22:51:25 +00:00
Marc Carré
4e58b83024 Add org.opencontainers.image.* labels to Dockerfiles
- This should ultimately help for image-to-code back references.
- `org.label-schema.*` labels are now deprecated, in favour of `org.opencontainers.image.*` labels.
  See also: https://github.com/opencontainers/image-spec/blob/master/annotations.md#back-compatibility-with-label-schema
- Git revision (`git rev-parse HEAD`) is now injected at `docker build` time.
2018-05-14 11:54:33 +01:00
Bryan Boreham
27cf8f3b06 Fix format string to work for all types 2018-04-05 08:18:52 +00:00
Bryan Boreham
f17640646e Blank out value on LatestMap decode insert
Fixing a rare case that came up in a test. In order for this to cause
a problem, the data being decoded has to have entries out of order,
and have a value that is nil or omitted.
2018-02-26 10:34:53 +00:00
Matthias Radestock
e24d3e93fe Add static map of common keys to reduce allocations 2017-12-11 20:26:47 +00:00
Bryan Boreham
d1bf98dfde Remove struct wrapping LatestMap
It isn't necessary, and it prevents the codec from seeing maps as empty
2017-12-06 15:46:31 +00:00
Matthias Radestock
4162b5d734 allocate less memory in LatestMap merging
Most maps we merge have the same keys, or at least one set of keys is
the subset of the other. Therefore, allocate a result slice capable of
holding only the max of number keys, rather than the sum.
2017-12-02 13:13:08 +00:00
cglewis
bbde07d42f MAINTAINER is deprecated, now using LABEL 2017-10-31 10:32:09 -07:00
Bryan Boreham
5acece6e58 Refactor: loop replaced with append() 2017-10-11 09:50:06 +00:00
Bryan Boreham
29f139d424 Re-implement LatestMap as a sorted slice for better performance
both the Python generator and the Go generated code are checked in
2017-10-01 16:18:59 +00:00
Bryan Boreham
ae9d708dbc Reformat Python examples to placate linter 2017-07-13 16:11:59 +00:00
Matthias Radestock
4dbb913ca2 refactor: remove dead Copy() code 2017-07-04 07:35:30 +01:00
Matthias Radestock
430e74a80a refactor: remove report latest map Delete()
It wasn't used, and is problematic in any case since it introduces
non-monotonicity.
2017-07-03 02:06:21 +01:00
Matthias Radestock
9dc50b5202 refactor: hide "empty set" constants
They are an implementation detail.
2017-07-03 01:26:22 +01:00
Bryan Boreham
1898b67e1f Use 127.0.0.1 instead of localhost in case that name resolves to something else 2017-06-05 10:31:27 +00:00
Matthias Radestock
fa0f4a4d59 add copyreport utility
useful for codec and report structure experiments
2017-05-26 15:33:15 +01:00
Alban Crequy
0096688415 shfmt: fix shell formatting
Symptoms:
```
extras/dialer/time-scope-probe-cgroup:31:18: arithmetic expressions must consist of names and numbers
extras/dialer/time-scope-probe-cgroup: run shfmt -i 4 -w extras/dialer/time-scope-probe-cgroup
extras/dialer/time-scope-probe-proc:28:18: arithmetic expressions must consist of names and numbers
extras/dialer/time-scope-probe-proc: run shfmt -i 4 -w extras/dialer/time-scope-probe-proc
extras/in_parallel.sh:29:16: arithmetic expressions must consist of names and numbers
extras/in_parallel.sh: run shfmt -i 4 -w extras/in_parallel.sh
```

Visible on https://circleci.com/gh/kinvolk/scope/980
2017-05-22 17:27:35 +02:00
Bryan Boreham
36ff5ab71a Write psMap out directly
rather than going via intermediate data structure
2017-04-25 10:38:56 +00:00
Alban Crequy
ddf654fdf2 extras/dialer: updates and fixes
* remove randomness as far as possible
* make listener to close connections
* report time in a csv-friendly format
* remove tools from README
* rename time-scope-probe -> time-scope-probe-proc
* add time-scope-probe-cgroup
2017-03-30 07:30:28 +02:00
Bryan Boreham
b3f53a7a81 Call CodecDecodeSelf() instead of Decode()
This avoids a runtime type lookup, so goes a little faster.
Also having less recursion makes it easier to interpret profiles.
2017-03-23 12:25:12 +00:00
Bryan Boreham
de97a72661 Remove indirection via LatestMap
Structs like StringLatestMap now use ps.Map directly, which saves
a memory allocation for LatestEntry.Value to point to.
The values in the ps.Map are now pointers, which saves a memory
allocation indirecting a value type to an interface{}
2017-03-20 11:51:04 +00:00
Alessandro Puccetti
6b2b483cbc extras/in_parallel.sh: remove erroneous quotes
$COMMAND can contain "make RM="

This patch fixes a regression caused by
https://github.com/weaveworks/scope/pull/2068/files#diff-42138a4802cce915f347919ccd707876R28
2017-02-22 11:16:51 +01:00
Alessandro Puccetti
e81f674a02 extra/in_parallel.sh: call the right python version
We want to use the python installed as dependency to use an updated
version of the requests module (>=v2.4.2).

This regression is caused by
https://github.com/weaveworks/build-tools/pull/73
2017-02-21 17:25:39 +01:00
Alessandro Puccetti
340438b3c3 extras/build_on_circle.sh: fix find error
The find command fails on circleci running ubuntu 14.04 with the
following message:

Unix filenames usually don't contain slashes (though pathnames do).
That means that '-name `./*.go'' will probably evaluate to false all the
time on this system.  You might find the '-wholename' test more useful,
or perhaps '-samefile'.  Alternatively, if you are using GNU grep, you
could use 'find ... -print0 | grep -FzZ `./*.go''.'

This fix is compatible with ubuntu 12.04

This regression is caused by
https://github.com/weaveworks/scope/pull/2068/files#diff-3072c24bd91fea6eb71aecc590951d7e
2017-02-21 17:19:19 +01:00
Michael Schubert
c349c165e5 extras/dialer: allow to seed random
Helpful if you want to get the same random numbers in subsequent runs
(e.g. to compare results between branches).
2017-02-08 12:42:32 +01:00
Iago López Galeiras
5cb5c7d42e Fix shfmt issues 2017-01-18 11:51:41 +01:00
Iago López Galeiras
43d9f38c5e Fix various linter issues
Found via shellcheck.
2017-01-18 11:51:41 +01:00