Bryan Boreham
f60c2380fb
Update job definition
2019-07-02 08:39:38 +00:00
Bryan Boreham
a9e9e476b4
Don't tag for quay.io
2019-07-02 08:39:28 +00:00
Bryan Boreham
8b31c7e0fc
Add 'keep' option to delete everything but some orgs
2019-06-13 17:48:44 +00:00
Bryan Boreham
1c11aefe03
Read compressed records file
2019-06-10 07:19:58 +00:00
Bryan Boreham
3456c844ec
Don't run delete batches in parallel
...
Now we are handling multiple org-hour records in parallel, it's best
no to have additional parallelism at the delete level where they may
collide on hash key.
2019-06-09 09:27:04 +00:00
Bryan Boreham
04668590f6
Metric for DynamoDB errors
2019-06-09 09:26:20 +00:00
Bryan Boreham
89121cb509
Wait using the correct batch size
...
Otherwise waits tend to bust the rate-limiter batch size and it
doesn't wait at all.
2019-06-08 14:22:46 +00:00
Bryan Boreham
0368edf612
Metric for records done
2019-06-08 13:50:58 +00:00
Bryan Boreham
f9f28f6d78
Do multiple org-hour records in parallel
2019-06-08 13:50:03 +00:00
Bryan Boreham
dd71191bb6
Check earlier for blank records file
2019-06-08 13:10:19 +00:00
Bryan Boreham
b729c5c3bb
Reduce rate limit defaults
2019-06-08 13:07:51 +00:00
Bryan Boreham
424a18cf2b
Fix batching again
2019-06-08 13:07:33 +00:00
Bryan Boreham
e46514c24a
Advance s3_items_deleted counter more steadily
2019-06-08 11:25:51 +00:00
Bryan Boreham
15546dc690
If no delete orgs file supplied, delete all records
2019-06-08 11:24:35 +00:00
Bryan Boreham
654c26442c
Spacing of debug message
2019-06-08 11:24:11 +00:00
Bryan Boreham
ae9441bb53
Fix batching to dynamodb
2019-06-08 11:23:48 +00:00
Bryan Boreham
2d588a57a7
Implement pagination on DynamoDB queries
2019-06-06 17:08:22 +00:00
Bryan Boreham
4ce900e466
Fix delete record parsing
2019-06-06 16:48:07 +00:00
Bryan Boreham
e60e25da4a
Remove duplicate 'segments' flag
2019-06-06 16:41:45 +00:00
Bryan Boreham
cfe7ca7cea
Read a file of org-hour records
2019-06-04 17:07:58 +00:00
Bryan Boreham
18cd43d2bf
Refactor: delete from DynamoDB in same goroutine as S3
2019-06-04 16:07:31 +00:00
Bryan Boreham
bc59b03f67
Fix errorf format bug
2019-06-04 15:14:10 +00:00
Bryan Boreham
cdf28708e8
Refactor: extract deleteOneOrgHour() method
2019-06-04 15:13:48 +00:00
Bryan Boreham
388d273e5f
Add 'just scan' option
2019-04-09 09:46:19 +00:00
Bryan Boreham
b23f7a7b0d
Default stop hour to now
2018-11-20 11:56:50 +00:00
Bryan Boreham
2fe3f46cf7
Handle retries ourselves
2018-11-20 07:53:20 +00:00
Bryan Boreham
a81bd74a4a
Use rate-limiters on DynamoDB operations
2018-11-18 22:53:28 +00:00
Bryan Boreham
1ce9b1e53c
Refactor to make bigger S3 writes
2018-11-17 13:47:35 +00:00
Bryan Boreham
c6f16436a9
Sample Kubernetes job
2018-11-15 18:19:17 +00:00
Bryan Boreham
ab796c7c6f
Fix up logging
2018-11-15 18:18:07 +00:00
Bryan Boreham
7143356fd5
Move to extras directory, add Makefile, Dockerfile
2018-11-15 18:17:27 +00:00
Bryan Boreham
dad34df575
Merge pull request #3267 from weaveworks/pool-gzwriters
...
Re-use gzip writers in a pool
2018-07-13 16:09:24 +01:00
Bryan Boreham
dac008b9c4
Early return from LatestMap merge loop
...
If we run out of things to look at in the other map, return quickly.
Also move the equal-key case above the less-than case, since maps with
equal keys are the common case when merging.
2018-07-12 19:54:01 +00:00
Bryan Boreham
c91fd6c67b
Optimise merge of LatestMaps with same keys, different timestamps
...
Swap the two maps so we are merging older into later.
2018-07-12 19:54:01 +00:00
Bryan Boreham
43a0a7e5d3
Don't copy LatestMap if merging a subset
2018-07-12 19:50:02 +00:00
Bryan Boreham
c96611b13d
Less verbose string dump of LatestMap
...
The default time.Time.String() includes nanoseconds, wall-clock adjustment, etc.
Also don't include the key twice when stringizing the map itself.
2018-07-08 16:32:51 +00:00
Bryan Boreham
70cd79307e
Re-use gzip writers in a pool
...
Since they allocate sizeable chunks of memory, this can save on garbage-collection.
2018-07-08 16:30:07 +00:00
Bryan Boreham
06c895267c
Reports: streamline report serialization
...
Move the creation of the buffer and the choice of compression level
(which never changes) into WriteBinary(), to simplify the code.
2018-06-19 22:59:01 +00:00
Bryan Boreham
96f51c47af
probe: Eliminate Publisher interface from app_client
...
Simplification: everything now implements Publish(Report), and we do
away with writing/reading/writing in the MultiAppClient.
2018-06-19 22:51:25 +00:00
Marc Carré
4e58b83024
Add org.opencontainers.image.* labels to Dockerfiles
...
- This should ultimately help for image-to-code back references.
- `org.label-schema.*` labels are now deprecated, in favour of `org.opencontainers.image.*` labels.
See also: https://github.com/opencontainers/image-spec/blob/master/annotations.md#back-compatibility-with-label-schema
- Git revision (`git rev-parse HEAD`) is now injected at `docker build` time.
2018-05-14 11:54:33 +01:00
Bryan Boreham
27cf8f3b06
Fix format string to work for all types
2018-04-05 08:18:52 +00:00
Bryan Boreham
f17640646e
Blank out value on LatestMap decode insert
...
Fixing a rare case that came up in a test. In order for this to cause
a problem, the data being decoded has to have entries out of order,
and have a value that is nil or omitted.
2018-02-26 10:34:53 +00:00
Matthias Radestock
e24d3e93fe
Add static map of common keys to reduce allocations
2017-12-11 20:26:47 +00:00
Bryan Boreham
d1bf98dfde
Remove struct wrapping LatestMap
...
It isn't necessary, and it prevents the codec from seeing maps as empty
2017-12-06 15:46:31 +00:00
Matthias Radestock
4162b5d734
allocate less memory in LatestMap merging
...
Most maps we merge have the same keys, or at least one set of keys is
the subset of the other. Therefore, allocate a result slice capable of
holding only the max of number keys, rather than the sum.
2017-12-02 13:13:08 +00:00
cglewis
bbde07d42f
MAINTAINER is deprecated, now using LABEL
2017-10-31 10:32:09 -07:00
Bryan Boreham
5acece6e58
Refactor: loop replaced with append()
2017-10-11 09:50:06 +00:00
Bryan Boreham
29f139d424
Re-implement LatestMap as a sorted slice for better performance
...
both the Python generator and the Go generated code are checked in
2017-10-01 16:18:59 +00:00
Bryan Boreham
ae9d708dbc
Reformat Python examples to placate linter
2017-07-13 16:11:59 +00:00
Matthias Radestock
4dbb913ca2
refactor: remove dead Copy() code
2017-07-04 07:35:30 +01:00
