From fe8d1bcaae9b125124d661acb6cc17d0fe9e9c3c Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Tue, 16 Aug 2016 16:43:06 -0400 Subject: [PATCH 01/23] Consolidate HTTP server and `ocat` docs --- README.md | 256 +++++++++++++++++++++++++++--------------------------- 1 file changed, 127 insertions(+), 129 deletions(-) diff --git a/README.md b/README.md index 45f4857..5c99e21 100644 --- a/README.md +++ b/README.md @@ -18,24 +18,12 @@ We developed the _recorder_ as a one-stop solution to storing location data publ * [Getting started](#getting-started) * [`ot-recorder` options and variables](#ot-recorder-options-and-variables) * [The HTTP Server](#the-http-server) - * [Last position of a particular user](#last-position-of-a-particular-user) - * [Display map with points starting at a particular date](#display-map-with-points-starting-at-a-particular-date) - * [Display a track (a.k.a. linestring)](#display-a-track-aka-linestring) - * [Tabular display](#tabular-display) - * [Live map](#live-map) -* [`ocat`](#ocat) -* [`ocat` examples](#ocat-examples) - * [List users and devices](#list-users-and-devices) - * [Show the last position reported by a user](#show-the-last-position-reported-by-a-user) - * [What were the last 4 positions reported?](#what-were-the-last-4-positions-reported) -* [Design decisions](#design-decisions) -* [Storage](#storage) -* [Configuration file](#configuration-file) -* [Reverse Geo](#reverse-geo) - * [Precision](#precisioin) - * [The geo cache](#the-geo-cache) -* [Monitoring](#monitoring) -* [HTTP server](#http-server) + * [Example functionality](#example-functionality) + * [Last position of a particular user](#last-position-of-a-particular-user) + * [Display map with points starting at a particular date](#display-map-with-points-starting-at-a-particular-date) + * [Display a track (a.k.a. linestring)](#display-a-track-aka-linestring) + * [Tabular display](#tabular-display) + * [Live map](#live-map) * [API](#api) * [`monitor`](#monitor) * [`last`](#last) @@ -45,6 +33,19 @@ We developed the _recorder_ as a one-stop solution to storing location data publ * [`photo`](#photo) * [`kill`](#kill) * [`version`](#version) +* [`ocat`](#ocat) + * [Environment](#environment) + * [Examples](#examples) + * [List users and devices](#list-users-and-devices) + * [Show the last position reported by a user](#show-the-last-position-reported-by-a-user) + * [What were the last 4 positions reported?](#what-were-the-last-4-positions-reported) +* [Design decisions](#design-decisions) +* [Storage](#storage) +* [Configuration file](#configuration-file) +* [Reverse Geo](#reverse-geo) + * [Precision](#precisioin) + * [The geo cache](#the-geo-cache) +* [Monitoring](#monitoring) * [Lua hooks](#lua-hooks) * [`otr_init`](#otr_init) * [`otr_exit`](#otr_exit) @@ -52,7 +53,6 @@ We developed the _recorder_ as a one-stop solution to storing location data publ * [`otr_putrec`](#otr_putrec) * [`otr_httpobject`](#otr_httpobject) * [Hooklets](#hooklets) -* [Environment](#environment) * [Reverse proxy](#reverse-proxy) * [nginx](#nginx) * [Apache](#apache) @@ -199,7 +199,13 @@ This section lists the most important options of the _recorder_ with their long ## The HTTP server -Some examples of what the _recorder_'s built-in HTTP server is capable of, in addition to obtaining OwnTracks app data via HTTP POST to the `/pub` endpoint. +The _recorder_ has a built-in HTTP server with which it servers static files from either the compiled-in default `DOCROOT` directory or that specified at run-time with the `--doc-root` option. Furthermore, it serves JSON data from the API end-point at `/api/0/` and it has a built-in WebSocket server for the live map. + +The API basically serves the same data as _ocat_ is able to produce. The server also accepts OwnTracks app data via HTTP POST to the `/pub` endpoint. + +### Example functionality + +Some examples of what the server can do: #### Last position of a particular user @@ -271,6 +277,98 @@ The _recorder_'s built-in WebSocket server updates a map as it receives publishe ![Live map](assets/demo-live-map.png) +### API + +The _recorder_'s API provides most of the functions that are surfaced by _ocat_. GET and POST requests are supported, and if a username and device are needed, these can be passed in via `X-Limit-User` and `X-Limit-Device` headers alternatively to GET or POST parameters. (From and To dates may also be specified as `X-Limit-From` and `X-Limit-To` +respectively.) + +The API endpoint is at `/api/0` and is followed by the verb. + +#### `monitor` + +Returns the content of the `monitor` file as plain text. + +``` +curl 'http://127.0.0.1:8083/api/0/monitor' +1441962082 owntracks/jjolie/phone +``` + +#### `last` + +Returns a list of last users' positions. (Can be limited by _user_, _device_, and _fields_, a comma-separated list of fields which should be returned instead of the default of all fields.) + +``` +curl http://127.0.0.1:8083/api/0/last [-d user=jjolie [-d device=phone]] +``` + +``` +curl 'http://127.0.0.1:8083/api/0/last?fields=tst,tid,addr,topic,isotst' +``` + +#### `list` + +List users. If _user_ is specified, lists that user's devices. If both _user_ and _device_ are specified, lists that device's `.rec` files. + +#### `locations` + +Here comes the actual data. This lists users' locations and requires both _user_ and _device_. Output format is JSON unless a different _format_ is given (`csv`, `json`, `geojson`, `xml`, and `linestring` are supported). + +In order to limit the number of records returned, use _limit_ which causes a reverse search through the `.rec` files; this can be used to find the last N positions. + +Date/time ranges may be specified as _from_ and _to_ with dates/times specified as described for _ocat_ above. + +``` +curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s +curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s -d limit=1 +curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s -d format=geojson +curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s -d from=2014-08-03 +curl 'http://127.0.0.1:8083/api/0/locations?from=2015-09-01&user=jpm&device=5s&fields=tst,tid,addr,isotst' +``` + +#### `q` + +Query the geo cache for a particular _lat_ and _lon_. + +``` +curl 'http://127.0.0.1:8083/api/0/q?lat=48.85833&lon=2.295' +{ + "cc": "FR", + "addr": "9 Avenue Anatole France, 75007 Paris, France", + "tst": 1441984405 +} +``` + +The reported timestamp was the time at which this cache entry was made. Note that this interface queries only -- it does not populate the cache. + +#### `photo` + +Requires GET method and _user_, and will return the `image/png` 40x40px photograph of a user if available in `STORAGEDIR/photos/` or a transparent 40x40png with a black border otherwise. + +#### `kill` + +If support for this is compiled in, this API endpoint allows a client to remove data from _storage_. (Warning: *any* client can do this, as there is no authentication/authorization in the _recorder_!) + +``` +curl 'http://127.0.0.1:8083/api/0/kill?user=ngin&device=ojo' + +{ + "path": "s0/rec/ngin/ojo", + "status": "OK", + "last": "s0/last/ngin/ojo/ngin-ojo.json", + "killed": [ + "2015-09.rec", + ] +} +``` +The response contains a list of removed `.rec` files, and file system operations are logged to syslog. + +#### `version` + +Returns a JSON object which contains the Recorder's version string, such as + +```json +{ "version": "0.4.7" } +``` ## `ocat` @@ -345,7 +443,15 @@ The `--from` and `--to` options allow you to specify a UTC date and/or timestamp The `--limit` option limits the output to the last specified number of records. This is a bit of an "expensive" operation because we search the `.rec` files backwards (i.e. from end to beginning). When using `--limit` the 6 hours mentioned earlier do not apply. -## `ocat` examples +### Environment + +The following environment variables control _ocat_'s behaviour: + +* `OCAT_FORMAT` can be set to the preferred output format. If unset, JSON is used. The `--format` option overrides this setting. +* `OCAT_USERNAME` can be set to the preferred username. The `--user` option overrides this environment variable. +* `OCAT_DEVICE` can be set to the preferred device name. The `--device` option overrides this environment variable. + +### Examples The _recorder_ has been running for a while, and the OwnTracks apps have published data. Let us have a look at some of this data. @@ -545,105 +651,6 @@ After sending a _pingping_, you can query the REST interface to determine the di OK ot-recorder pingping at http://127.0.0.1:8085: 0 seconds difference ``` -## HTTP server - -The _recorder_ has a built-in HTTP server with which it servers static files from either the compiled-in default `DOCROOT` directory or that specified at run-time with the `--doc-root` option. Furthermore, it serves JSON data from the API end-point at `/api/0/` and it has a built-in WebSocket server for the live map. - -The API basically serves the same data as _ocat_ is able to produce. - -### API - -The _recorder_'s API provides most of the functions that are surfaced by _ocat_. GET and POST requests are supported, and if a username and device are needed, these can be passed in via `X-Limit-User` and `X-Limit-Device` headers alternatively to GET or POST parameters. (From and To dates may also be specified as `X-Limit-From` and `X-Limit-To` -respectively.) - -The API endpoint is at `/api/0` and is followed by the verb. - -#### `monitor` - -Returns the content of the `monitor` file as plain text. - -``` -curl 'http://127.0.0.1:8083/api/0/monitor' -1441962082 owntracks/jjolie/phone -``` - -#### `last` - -Returns a list of last users' positions. (Can be limited by _user_, _device_, and _fields_, a comma-separated list of fields which should be returned instead of the default of all fields.) - -``` -curl http://127.0.0.1:8083/api/0/last [-d user=jjolie [-d device=phone]] -``` - -``` -curl 'http://127.0.0.1:8083/api/0/last?fields=tst,tid,addr,topic,isotst' -``` - -#### `list` - -List users. If _user_ is specified, lists that user's devices. If both _user_ and _device_ are specified, lists that device's `.rec` files. - -#### `locations` - -Here comes the actual data. This lists users' locations and requires both _user_ and _device_. Output format is JSON unless a different _format_ is given (`csv`, `json`, `geojson`, `xml`, and `linestring` are supported). - -In order to limit the number of records returned, use _limit_ which causes a reverse search through the `.rec` files; this can be used to find the last N positions. - -Date/time ranges may be specified as _from_ and _to_ with dates/times specified as described for _ocat_ above. - -``` -curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s -curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s -d limit=1 -curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s -d format=geojson -curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s -d from=2014-08-03 -curl 'http://127.0.0.1:8083/api/0/locations?from=2015-09-01&user=jpm&device=5s&fields=tst,tid,addr,isotst' -``` - -#### `q` - -Query the geo cache for a particular _lat_ and _lon_. - -``` -curl 'http://127.0.0.1:8083/api/0/q?lat=48.85833&lon=2.295' -{ - "cc": "FR", - "addr": "9 Avenue Anatole France, 75007 Paris, France", - "tst": 1441984405 -} -``` - -The reported timestamp was the time at which this cache entry was made. Note that this interface queries only -- it does not populate the cache. - -#### `photo` - -Requires GET method and _user_, and will return the `image/png` 40x40px photograph of a user if available in `STORAGEDIR/photos/` or a transparent 40x40png with a black border otherwise. - -#### `kill` - -If support for this is compiled in, this API endpoint allows a client to remove data from _storage_. (Warning: *any* client can do this, as there is no authentication/authorization in the _recorder_!) - -``` -curl 'http://127.0.0.1:8083/api/0/kill?user=ngin&device=ojo' - -{ - "path": "s0/rec/ngin/ojo", - "status": "OK", - "last": "s0/last/ngin/ojo/ngin-ojo.json", - "killed": [ - "2015-09.rec", - ] -} -``` -The response contains a list of removed `.rec` files, and file system operations are logged to syslog. - -#### `version` - -Returns a JSON object which contains the Recorder's version string, such as - -```json -{ "version": "0.4.7" } -``` - ## Lua hooks If _recorder_ is compiled with Lua support, a Lua script you provide is launched at startup. Lua is _a powerful, fast, lightweight, embeddable scripting language_. You can use this to process location publishes in any way you desire: your imagination (and Lua-scripting knowhow) set the limits. Some examples: @@ -738,15 +745,6 @@ After running `otr_hook()`, the _recorder_ attempts to invoke a Lua function for You define a hooklet function only if you're interested in expressly triggering on a particular JSON element. - -## Environment - -The following environment variables control _ocat_'s behaviour: - -* `OCAT_FORMAT` can be set to the preferred output format. If unset, JSON is used. The `--format` option overrides this setting. -* `OCAT_USERNAME` can be set to the preferred username. The `--user` option overrides this environment variable. -* `OCAT_DEVICE` can be set to the preferred device name. The `--device` option overrides this environment variable. - ## Reverse proxy Running the _recorder_ protected by an _nginx_ or _Apache_ server is possible and is the only recommended method if you want to server data behind _localhost_. The snippets below show how to do it, but you would also add authentication to them. From 0a9dc887e0a94212ea0bcf422ea966178fea3df2 Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Tue, 16 Aug 2016 16:52:23 -0400 Subject: [PATCH 02/23] Consolidate building instructions --- README.md | 77 ++++++++++++++++++++++++++----------------------------- 1 file changed, 36 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index 5c99e21..f02e1fa 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,8 @@ We developed the _recorder_ as a one-stop solution to storing location data publ * [`recorder`](#recorder) * [Installing](#installing) * [Building from source](#building-from-source) + * [Prerequisites](#prerequisites) + * [Building](#building) * [Getting started](#getting-started) * [`ot-recorder` options and variables](#ot-recorder-options-and-variables) * [The HTTP Server](#the-http-server) @@ -70,10 +72,6 @@ We developed the _recorder_ as a one-stop solution to storing location data publ * [`keys`](#keys) * [`friends`](#friends) * [Encryption (*experimental!*)](#encryption-experimental) -* [Prerequisites for building](#prerequisites-for-building) - * [Debian](#debian) - * [CentOS 7](#centos-7) - * [Ubuntu](#ubuntu) * [Packages](#packages) * [Installing on CentOS 7](#installing-on-centos-7) * [Installing on Raspian (Wheezy)](#installing-on-raspian-wheezy) @@ -100,6 +98,8 @@ You will, however, need to acquire and configure apikeys for the maps. (See belo ## Building from source +### Prerequisites + You will require: * [libmosquitto](http://mosquitto.org) unless you disable MQTT during building, but see below for platform instructions @@ -109,6 +109,33 @@ You will require: * Optionally [Lua](http://lua.org) * Optionally [libsodium](https://github.com/jedisct1/libsodium) for secret-key encryption of payloads +You need a current version of libmosquitto (and you probably require the Mosquitto broker as well for OwnTracks). We strongly recommend installing Mosquitto either from [source](http://mosquitto.org/download/) or from a [binary package](http://mosquitto.org/download/), both of which are provided by the [Mosquitto project](http://mosquitto.org/). In particular, older or LTS OS versions profit from this. + +On Debian, you can install the needed packages with: + +``` +apt-get install build-essential linux-headers-$(uname -r) libcurl4-openssl-dev libmosquitto-dev liblua5.2-dev libsodium-dev libconfig-dev +``` + +On CentOS 7: + +``` +yum groupinstall 'Development Tools' +yum install libmosquitto-devel libcurl-devel lua-devel libsodium-devel libconfig-devel +``` + +(libsodium is in epel-stable) + +On Ubuntu: + +``` +sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa +sudo apt-get update +sudo apt-get install libmosquitto-dev libcurl3 libcurl4-openssl-dev libconfig-dev +``` + +### Building + 1. Obtain and download the software, via [our Homebrew Tap](https://github.com/owntracks/homebrew-recorder) on Mac OS X, directly as a clone of the repository, or as a [tar ball](https://github.com/owntracks/recorder/releases) which you unpack. 2. Copy the included `config.mk.in` file to `config.mk` and edit that. You specify the features or tweaks you need. (The file is commented.) Pay particular attention to the installation directory and the value of the _store_ (`STORAGEDEFAULT`): that is where the recorder will store its files. `DOCROOT` is the root of the directory from which the _recorder_'s HTTP server will serve files. 3. Type `make` and watch the fun. @@ -121,11 +148,6 @@ Ensure the LMDB databases are initialized by running the following command which ot-recorder --initialize ``` -Unless already provided by the package you installed, we recommend you create a shell script with which you hence-force launch the _recorder_. Note that you can have it subscribe to multiple topics, and you can launch sundry instances of the recorder (e.g. for distinct brokers) as long as you ensure: - -* that each instance uses a distinct `--storage` -* that each instance uses a distinct `--http-port` (or `0` if you don't wish to provide HTTP support for a particular instance) - ## Getting started The _recorder_ has, like _ocat_, a daunting number of options, most of which you will not require. Running either utility with the `-h` or `--help` switch will summarize their meanings. You can, for example launch with a specific storage directory, disable the HTTP server, change its port, etc. @@ -148,9 +170,13 @@ The location message received by the _recorder_ will be written to storage. In p 2. a directory called `rec/` with several subdirectories and a `.rec` file therein. 3. a directory called `last/` which contains subdirectories and a `.json` file therein. - When the recorder has received a publish or two, visit it with your favorite Web browser by pointing your browser at `http://127.0.0.1:8083` or the address / port configured with the `--http-host` and `--http-port` options respectively. +Unless already provided by the package you installed, we recommend you create a shell script with which you hence-force launch the _recorder_. Note that you can have it subscribe to multiple topics, and you can launch sundry instances of the recorder (e.g. for distinct brokers) as long as you ensure: + +* that each instance uses a distinct `--storage` +* that each instance uses a distinct `--http-port` (or `0` if you don't wish to provide HTTP support for a particular instance) + ### `ot-recorder` options and variables This section lists the most important options of the _recorder_ with their long names; check the usage (`recorder -h`) for the short versions. @@ -1092,37 +1118,6 @@ If compiled with `WITH_ENCRYPT` support (this is the default in our packages), t Upon successful decryption, the Recorder processes the original (device-transmitted) JSON and stores the result in plain (i.e. un-encrypted) form in the store. - - -## Prerequisites for building - -You need a current version of the Mosquitto library (and you probably require the Mosquitto broker as well for OwnTracks). We strongly recommend installing Mosquitto either from [source](http://mosquitto.org/download/) or from a [binary package](http://mosquitto.org/download/), both of which are provided by the [Mosquitto project](http://mosquitto.org/). In particular, older or LTS OS versions profit from this. - -### Debian - -``` -apt-get install build-essential linux-headers-$(uname -r) libcurl4-openssl-dev libmosquitto-dev liblua5.2-dev libsodium-dev libconfig-dev -``` - -### CentOS 7 - -``` -yum groupinstall 'Development Tools' -yum install libmosquitto-devel libcurl-devel lua-devel libsodium-devel libconfig-devel -``` - -libsodium is in epel-stable - -### Ubuntu - -``` -sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa -sudo apt-get update -sudo apt-get install libmosquitto-dev -sudo apt-get install libcurl3 libcurl4-openssl-dev -sudo apt-get install libconfig-dev -``` - [![Build Status](https://travis-ci.org/owntracks/recorder.svg?branch=master)](https://travis-ci.org/owntracks/recorder) ## Packages From f1888c40f9645133bffb3e0ab78e10900176895e Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Tue, 16 Aug 2016 16:55:29 -0400 Subject: [PATCH 03/23] Move Travis badge to the top of README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index f02e1fa..c871cf7 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,8 @@ ![Recorder logo](assets/recorder-logo-192.png) +[![Build Status](https://travis-ci.org/owntracks/recorder.svg?branch=master)](https://travis-ci.org/owntracks/recorder) + The _OwnTracks Recorder_ is a lightweight program for storing and accessing location data published via MQTT (or HTTP) by the [OwnTracks](http://owntracks.org) apps. It is a compiled program which is easily to install and operate even on low-end hardware, and it doesn't require an external database. ![Architecture of the Recorder](assets/ot-recorder.png) @@ -1118,8 +1120,6 @@ If compiled with `WITH_ENCRYPT` support (this is the default in our packages), t Upon successful decryption, the Recorder processes the original (device-transmitted) JSON and stores the result in plain (i.e. un-encrypted) form in the store. -[![Build Status](https://travis-ci.org/owntracks/recorder.svg?branch=master)](https://travis-ci.org/owntracks/recorder) - ## Packages We create packages for releases for a few distributions. Please note that these packages depend on libmosquitto1 from the [Mosquitto project](http://mosquitto.org/downloads). From c573f7374fbace3f2a7c38f2fc328b3fb99b5d45 Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Tue, 16 Aug 2016 17:37:03 -0400 Subject: [PATCH 04/23] Remove duplicated/out-of-place env variable docs --- README.md | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index c871cf7..4978ee1 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ We developed the _recorder_ as a one-stop solution to storing location data publ * [Prerequisites](#prerequisites) * [Building](#building) * [Getting started](#getting-started) -* [`ot-recorder` options and variables](#ot-recorder-options-and-variables) +* [`ot-recorder` options](#ot-recorder-options) * [The HTTP Server](#the-http-server) * [Example functionality](#example-functionality) * [Last position of a particular user](#last-position-of-a-particular-user) @@ -179,7 +179,7 @@ Unless already provided by the package you installed, we recommend you create a * that each instance uses a distinct `--storage` * that each instance uses a distinct `--http-port` (or `0` if you don't wish to provide HTTP support for a particular instance) -### `ot-recorder` options and variables +### `ot-recorder` options This section lists the most important options of the _recorder_ with their long names; check the usage (`recorder -h`) for the short versions. @@ -191,10 +191,6 @@ This section lists the most important options of the _recorder_ with their long `--user` overrides `$OTR_USER` and specifies the username to use in the MQTT connection. -`$OTR_PASS` is the password for the MQTT connection. - -`$OTR_CAFILE` specifies the path to a readable PEM-formatted file containing the CA certificate chain to be used for the MQTT TLS connection. If this environment variable is set, a TLS connection is assumed (and the port number should probably be adjusted accordingly). - `--qos` specifies the MQTT QoS to use; it defaults to 2. `--storagedir` is configured at build time and overrides `$OTR_STORAGEDIR`. @@ -611,7 +607,7 @@ The following configuration settings may be applied (a `Y` in column `$` means a | `OTR_PRECISION` | | `7` | Reverse-geo precision | `OTR_GEOKEY` | | | API key for reverse-geo lookups | `OTR_TOPICS` | | | String containing a space-separated list of topics to subscribe to for MQTT (overriden by command-line arguments) -| `OTR_CAFILE` | Y | | Path to PEM-encoded CA certificate file for MQTT +| `OTR_CAFILE` | Y | | Path to PEM-encoded CA certificate file for MQTT (implicitly enables TLS) Note that options passed to `ot-recorder` override both configuration file settings and environment variables. From 1fbccc40f1cc8aadfdedb66c6a861eb7a4399e02 Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Tue, 16 Aug 2016 17:41:34 -0400 Subject: [PATCH 05/23] s/websocket/WebSocket/g --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4978ee1..d435475 100644 --- a/README.md +++ b/README.md @@ -207,7 +207,7 @@ This section lists the most important options of the _recorder_ with their long `--initialize` creates the a structure within the storage directory and initializes the LMDB database. It is safe to use this even if such a database exists -- the database is not wiped. After initialization, _recorder_ exits. -`--label` specifies a label (default: "Recorder") to be shown in the websocket live map. +`--label` specifies a label (default: "Recorder") to be shown in the WebSocket live map. `--http-host` and `--http-port` define the listen address and port number for the API. If `--http-port` is 0, the Web server is disabled. From 350ac615ee4b30fba9f748c9d5a9b5b005667c37 Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Tue, 16 Aug 2016 17:52:21 -0400 Subject: [PATCH 06/23] Refer to Recorder in a consistent way --- README.md | 110 +++++++++++++++++++++++++++--------------------------- 1 file changed, 55 insertions(+), 55 deletions(-) diff --git a/README.md b/README.md index d435475..3c62db9 100644 --- a/README.md +++ b/README.md @@ -8,9 +8,9 @@ The _OwnTracks Recorder_ is a lightweight program for storing and accessing loca ![Architecture of the Recorder](assets/ot-recorder.png) -There are two main components: the _recorder_ obtains data via MQTT subscribes or HTTP POST, stores the data in plain files and serve it via its built-in REST API, and the _ocat_ command-line utility reads stored data in a variety of formats. +There are two main components: the _Recorder_ obtains data via MQTT subscribes or HTTP POST, stores the data in plain files and serve it via its built-in REST API, and the _ocat_ command-line utility reads stored data in a variety of formats. -We developed the _recorder_ as a one-stop solution to storing location data published by our OwnTracks apps (iOS and Android) and retrieving this data. Our previous offerings (`m2s`, `o2s`/`Pista`) also work of course, but we believe the _recorder_ is best suited to most environments. +We developed the Recorder as a one-stop solution to storing location data published by our OwnTracks apps (iOS and Android) and retrieving this data. Our previous offerings (`m2s`, `o2s`/`Pista`) also work of course, but we believe the Recorder is best suited to most environments. ## Table of Contents @@ -86,10 +86,10 @@ We developed the _recorder_ as a one-stop solution to storing location data publ ## `recorder` -The _recorder_ serves two purposes: +The Recorder serves two purposes: 1. It subscribes to an MQTT broker and reads messages published from the OwnTracks apps, storing these in a particular fashion into what we call the _store_ which is basically a bunch of plain files on the file system. Alternatively the Recorder can listen on HTTP for OwnTracks-type JSON messages POSTed to its HTTP server. -2. It provides a Web server which serves static pages, a REST API you use to request data from the _store_, and a WebSocket server. The distribution comes with a few examples of how to access the data through its HTTP interface (REST API). In particular a _table_ of last locations has been made available as well as a _live map_ which updates via the _recorder_'s WebSocket interface when location publishes are received. In addition we provide maps with last points or tracks using the GeoJSON produced by the _recorder_. +2. It provides a Web server which serves static pages, a REST API you use to request data from the _store_, and a WebSocket server. The distribution comes with a few examples of how to access the data through its HTTP interface (REST API). In particular a _table_ of last locations has been made available as well as a _live map_ which updates via the Recorder's WebSocket interface when location publishes are received. In addition we provide maps with last points or tracks using the GeoJSON produced by the Recorder. ## Installing @@ -139,10 +139,10 @@ sudo apt-get install libmosquitto-dev libcurl3 libcurl4-openssl-dev libconfig-de ### Building 1. Obtain and download the software, via [our Homebrew Tap](https://github.com/owntracks/homebrew-recorder) on Mac OS X, directly as a clone of the repository, or as a [tar ball](https://github.com/owntracks/recorder/releases) which you unpack. -2. Copy the included `config.mk.in` file to `config.mk` and edit that. You specify the features or tweaks you need. (The file is commented.) Pay particular attention to the installation directory and the value of the _store_ (`STORAGEDEFAULT`): that is where the recorder will store its files. `DOCROOT` is the root of the directory from which the _recorder_'s HTTP server will serve files. +2. Copy the included `config.mk.in` file to `config.mk` and edit that. You specify the features or tweaks you need. (The file is commented.) Pay particular attention to the installation directory and the value of the _store_ (`STORAGEDEFAULT`): that is where the Recorder will store its files. `DOCROOT` is the root of the directory from which the Recorder's HTTP server will serve files. 3. Type `make` and watch the fun. -When _make_ finishes, you should have at least two executable programs called `ot-recorder` which is the _recorder_ proper, and `ocat`. If you want you can install these using `make install`, but this is not necessary: the programs will run from whichever directory you like if you add `--doc-root ./docroot` to the _recorder_ options. +When _make_ finishes, you should have at least two executable programs called `ot-recorder` which is the Recorder proper, and `ocat`. If you want you can install these using `make install`, but this is not necessary: the programs will run from whichever directory you like if you add `--doc-root ./docroot` to the Recorder options. Ensure the LMDB databases are initialized by running the following command which is safe to do, also after an upgrade. (This initialization is non-destructive -- it will not delete any data.) @@ -152,11 +152,11 @@ ot-recorder --initialize ## Getting started -The _recorder_ has, like _ocat_, a daunting number of options, most of which you will not require. Running either utility with the `-h` or `--help` switch will summarize their meanings. You can, for example launch with a specific storage directory, disable the HTTP server, change its port, etc. +The Recorder has, like _ocat_, a daunting number of options, most of which you will not require. Running either utility with the `-h` or `--help` switch will summarize their meanings. You can, for example launch with a specific storage directory, disable the HTTP server, change its port, etc. If you require authentication or TLS to connect to your MQTT broker, pay attention to the `$OTR_` environment variables listed in the help. -Launch the recorder: +Launch the Recorder: ``` $ ./ot-recorder 'owntracks/#' @@ -164,24 +164,24 @@ $ ./ot-recorder 'owntracks/#' (In httpmode, you do not have to specify a topic.) -Publish a location from your OwnTracks app and you should see the _recorder_ receive that on the console. If you haven't disabled Geo-lookups, you'll also see the address from which the publish originated. +Publish a location from your OwnTracks app and you should see the Recorder receive that on the console. If you haven't disabled Geo-lookups, you'll also see the address from which the publish originated. -The location message received by the _recorder_ will be written to storage. In particular you should verify that your _storage_ directory contains: +The location message received by the Recorder will be written to storage. In particular you should verify that your _storage_ directory contains: 1. a directory called `ghash/` 2. a directory called `rec/` with several subdirectories and a `.rec` file therein. 3. a directory called `last/` which contains subdirectories and a `.json` file therein. -When the recorder has received a publish or two, visit it with your favorite Web browser by pointing your browser at `http://127.0.0.1:8083` or the address / port configured with the `--http-host` and `--http-port` options respectively. +When the Recorder has received a publish or two, visit it with your favorite Web browser by pointing your browser at `http://127.0.0.1:8083` or the address / port configured with the `--http-host` and `--http-port` options respectively. -Unless already provided by the package you installed, we recommend you create a shell script with which you hence-force launch the _recorder_. Note that you can have it subscribe to multiple topics, and you can launch sundry instances of the recorder (e.g. for distinct brokers) as long as you ensure: +Unless already provided by the package you installed, we recommend you create a shell script with which you hence-force launch the Recorder. Note that you can have it subscribe to multiple topics, and you can launch sundry instances of the Recorder (e.g. for distinct brokers) as long as you ensure: * that each instance uses a distinct `--storage` * that each instance uses a distinct `--http-port` (or `0` if you don't wish to provide HTTP support for a particular instance) ### `ot-recorder` options -This section lists the most important options of the _recorder_ with their long names; check the usage (`recorder -h`) for the short versions. +This section lists the most important options of the Recorder with their long names; check the usage (`recorder -h`) for the short versions. `--clientid` specifies the MQTT client identifier to use upon connecting to the broker, thus overriding a constructed default. @@ -205,7 +205,7 @@ This section lists the most important options of the _recorder_ with their long `--quiet` disables printing of messages to _stdout_. -`--initialize` creates the a structure within the storage directory and initializes the LMDB database. It is safe to use this even if such a database exists -- the database is not wiped. After initialization, _recorder_ exits. +`--initialize` creates the a structure within the storage directory and initializes the LMDB database. It is safe to use this even if such a database exists -- the database is not wiped. After initialization, Recorder exits. `--label` specifies a label (default: "Recorder") to be shown in the WebSocket live map. @@ -223,7 +223,7 @@ This section lists the most important options of the _recorder_ with their long ## The HTTP server -The _recorder_ has a built-in HTTP server with which it servers static files from either the compiled-in default `DOCROOT` directory or that specified at run-time with the `--doc-root` option. Furthermore, it serves JSON data from the API end-point at `/api/0/` and it has a built-in WebSocket server for the live map. +The Recorder has a built-in HTTP server with which it servers static files from either the compiled-in default `DOCROOT` directory or that specified at run-time with the `--doc-root` option. Furthermore, it serves JSON data from the API end-point at `/api/0/` and it has a built-in WebSocket server for the live map. The API basically serves the same data as _ocat_ is able to produce. The server also accepts OwnTracks app data via HTTP POST to the `/pub` endpoint. @@ -291,19 +291,19 @@ If we change the `format` parameter of the previous URL to `linestring`, the res #### Tabular display -The _recorder_'s Web server also provides a tabular display which shows the last position of devices, their address, country, etc. Some of the columns are sortable, you can search for users/devices and click on the address to have a map opened at the device's last location. +The Recorder's Web server also provides a tabular display which shows the last position of devices, their address, country, etc. Some of the columns are sortable, you can search for users/devices and click on the address to have a map opened at the device's last location. ![Table](assets/demo-table.png) #### Live map -The _recorder_'s built-in WebSocket server updates a map as it receives publishes from the OwnTracks devices. Here's an example: +The Recorder's built-in WebSocket server updates a map as it receives publishes from the OwnTracks devices. Here's an example: ![Live map](assets/demo-live-map.png) ### API -The _recorder_'s API provides most of the functions that are surfaced by _ocat_. GET and POST requests are supported, and if a username and device are needed, these can be passed in via `X-Limit-User` and `X-Limit-Device` headers alternatively to GET or POST parameters. (From and To dates may also be specified as `X-Limit-From` and `X-Limit-To` +The Recorder's API provides most of the functions that are surfaced by _ocat_. GET and POST requests are supported, and if a username and device are needed, these can be passed in via `X-Limit-User` and `X-Limit-Device` headers alternatively to GET or POST parameters. (From and To dates may also be specified as `X-Limit-From` and `X-Limit-To` respectively.) The API endpoint is at `/api/0` and is followed by the verb. @@ -370,7 +370,7 @@ Requires GET method and _user_, and will return the `image/png` 40x40px photogra #### `kill` -If support for this is compiled in, this API endpoint allows a client to remove data from _storage_. (Warning: *any* client can do this, as there is no authentication/authorization in the _recorder_!) +If support for this is compiled in, this API endpoint allows a client to remove data from _storage_. (Warning: *any* client can do this, as there is no authentication/authorization in the Recorder!) ``` curl 'http://127.0.0.1:8083/api/0/kill?user=ngin&device=ojo' @@ -396,7 +396,7 @@ Returns a JSON object which contains the Recorder's version string, such as ## `ocat` -_ocat_ is a CLI query program for data stored by _recorder_: it prints data from storage in a variety of output formats: +_ocat_ is a CLI query program for data stored by Recorder: it prints data from storage in a variety of output formats: * JSON * GeoJSON (points) @@ -407,7 +407,7 @@ _ocat_ is a CLI query program for data stored by _recorder_: it prints data from * raw (the lines contained in the REC file with ISO timestamp) * payload (basically just the payload part from RAW) -The _ocat_ utility accesses _storage_ directly — it doesn’t use the _recorder_’s REST interface. _ocat_ has a daunting number of options, some combinations of which make no sense at all. +The _ocat_ utility accesses _storage_ directly — it doesn’t use the Recorder’s REST interface. _ocat_ has a daunting number of options, some combinations of which make no sense at all. Some example uses we consider useful: @@ -477,7 +477,7 @@ The following environment variables control _ocat_'s behaviour: ### Examples -The _recorder_ has been running for a while, and the OwnTracks apps have published data. Let us have a look at some of this data. +The Recorder has been running for a while, and the OwnTracks apps have published data. Let us have a look at some of this data. #### List users and devices @@ -537,7 +537,7 @@ $ ocat --last --user demo --device iphone Several things worth mentioning: * The returned data structure is an array of JSON objects; had we omitted specifying a particular device or even a particular user we would have obtained the last position of all this user's devices or all users' devices respectively. -* If you are familiar with the [JSON data reported by the OwnTracks apps](http://owntracks.org/booklet/tech/json/) you'll notice that this JSON contains more information: this is provided on the fly by _ocat_ and the REST API, e.g. from the reverse-geo cache the _recorder_ maintains. +* If you are familiar with the [JSON data reported by the OwnTracks apps](http://owntracks.org/booklet/tech/json/) you'll notice that this JSON contains more information: this is provided on the fly by _ocat_ and the REST API, e.g. from the reverse-geo cache the Recorder maintains. #### What were the last 4 positions reported? @@ -555,35 +555,35 @@ isotst,vel,addr ## Design decisions -We took a number of decisions when designing the _recorder_ and its utilities: +We took a number of decisions when designing the Recorder and its utilities: -* Flat files. The filesystem is the database. Period. That's were everything is stored. It makes incremental backups, purging old data, manipulation via the Unix toolset easy. (Admittedly, for fast geo-lookups we employ LMDB as a cache, but the final word is in the filesystem.) We considered all manner of databases and decided to keep this as simple and lightweight as possible. You can however have the _recorder_ send data to a database of your choosing, in addition to the file system it uses, by utilizing our embedded Lua hook. +* Flat files. The filesystem is the database. Period. That's were everything is stored. It makes incremental backups, purging old data, manipulation via the Unix toolset easy. (Admittedly, for fast geo-lookups we employ LMDB as a cache, but the final word is in the filesystem.) We considered all manner of databases and decided to keep this as simple and lightweight as possible. You can however have the Recorder send data to a database of your choosing, in addition to the file system it uses, by utilizing our embedded Lua hook. * We wanted to store received data in the format it's published in. As this format is JSON, we store this raw payload in the `.rec` files. If we add an attribute to the JSON published by our apps, you have it right there. There's one slight exception: the monthly logs (the `.rec` files) have a leading timestamp and a relative topic; see below. (In the particular case of the OwnTracks firmware for Greenwich devices which can publish in CSV mode, we convert the CSV into OwnTracks JSON for storage.) * File names are lower case. A user called `JaNe` with a device named `myPHONe` will be found in a file named `jane/myphone`. * All times are UTC (a.k.a. Zulu or GMT). We got sick and tired of converting stuff back and forth. It is up to the consumer of the data to convert to localtime if need be. -* The _recorder_ does not provide authentication or authorization. Nothing at all. Zilch. Nada. Think about this before making it available on a publicly-accessible IP address. Or rather: don't think about it; just don't do it. You can of course place a HTTP proxy in front of the `recorder` to control access to it. Or use views (see below). -* `ocat`, the _cat_ program for the _recorder_ uses the same back-end which is used by the API though it accesses it directly (i.e. without resorting to HTTP). -* The _recorder_ supports 3-level MQTT topics only, in the typical OwnTracks format: `"owntracks//"`, optionally with a leading slash. (The first part of the topic need not be "owntracks".) Publishes via HTTP POST construct a ficticious topic internally using the provided user (`u`) and device (`d`) parameters. +* The Recorder does not provide authentication or authorization. Nothing at all. Zilch. Nada. Think about this before making it available on a publicly-accessible IP address. Or rather: don't think about it; just don't do it. You can of course place a HTTP proxy in front of the Recorder to control access to it. Or use views (see below). +* `ocat`, the _cat_ program for the Recorder uses the same back-end which is used by the API though it accesses it directly (i.e. without resorting to HTTP). +* The Recorder supports 3-level MQTT topics only, in the typical OwnTracks format: `"owntracks//"`, optionally with a leading slash. (The first part of the topic need not be "owntracks".) Publishes via HTTP POST construct a ficticious topic internally using the provided user (`u`) and device (`d`) parameters. ## Storage -As mentioned earlier, data is stored in files, and these files are relative to `STORAGEDIR` (compiled into the programs or specified as an option). In particular, the following directory structure can exist, whereby directories are created as needed by the _recorder_: +As mentioned earlier, data is stored in files, and these files are relative to `STORAGEDIR` (compiled into the programs or specified as an option). In particular, the following directory structure can exist, whereby directories are created as needed by the Recorder: -* `cards/`, optional, may contains user cards. This card is then stored here and used with, e.g., `ocat --last` to show a user's name and optional avatar. User cards are typically stored in a subdirectory called _username_, and therein a JSON file _username_.json. When reading cards, the recorder will first attempt to open _username_/_device_/_username_.json and then _username_/_username_.json. +* `cards/`, optional, may contains user cards. This card is then stored here and used with, e.g., `ocat --last` to show a user's name and optional avatar. User cards are typically stored in a subdirectory called _username_, and therein a JSON file _username_.json. When reading cards, the Recorder will first attempt to open _username_/_device_/_username_.json and then _username_/_username_.json. * `config/`, optional, contains the JSON of a [device configuration](http://owntracks.org/booklet/features/remoteconfig/) (`.otrc`) which was requested remotely via a [dump command](http://owntracks.org/booklet/tech/json/#_typecmd). Note that this will contain sensitive data. You can use this `.otrc` file to restore the OwnTracks configuration on your device by copying to the device and opening it in OwnTracks. * `ghash/`, unless disabled, reverse Geo data (using a Google service) is collected into an LMDB database located in this directory. This LMDB database also contains named databases which are used by your optional Lua hooks, as well as a `topic2tid` database which can be used for TID re-mapping. -* `last/` contains the last location published by devices. E.g. Jane's last publish from her iPhone would be in `last/jjolie/iphone/jjolie-iphone.json`. The JSON payload contained therein is enhanced with the fields `user`, `device`, `topic`, and `ghash`. If a device's `last/` directory contains a file called `extra.json` (i.e. matching the example, this would be `last/jjolie/iphone/extra.json`), the content of this file is merged into the existing JSON for this user and returned by the API. Note, that you cannot overwrite existing values. So, an `extra.json` containing `{ "tst" : 11 }` will do nothing because the `tst` element we obtain from location data overrules, but adding `{ "beverage" : "water" }` will do what you want. If _recorder_ is built with support for our Greenwich firmware, this directory might contain `batt.json`, `ext.json`, and/or `status.json` each of which hold an array of the last 100 reports for internal battery voltage, external voltage, and status respectively. These values are returned via the API in the LAST object. A file `http.json` which should contain either a single JSON object or an array of JSON objects is returned to clients in HTTPmode. +* `last/` contains the last location published by devices. E.g. Jane's last publish from her iPhone would be in `last/jjolie/iphone/jjolie-iphone.json`. The JSON payload contained therein is enhanced with the fields `user`, `device`, `topic`, and `ghash`. If a device's `last/` directory contains a file called `extra.json` (i.e. matching the example, this would be `last/jjolie/iphone/extra.json`), the content of this file is merged into the existing JSON for this user and returned by the API. Note, that you cannot overwrite existing values. So, an `extra.json` containing `{ "tst" : 11 }` will do nothing because the `tst` element we obtain from location data overrules, but adding `{ "beverage" : "water" }` will do what you want. If Recorder is built with support for our Greenwich firmware, this directory might contain `batt.json`, `ext.json`, and/or `status.json` each of which hold an array of the last 100 reports for internal battery voltage, external voltage, and status respectively. These values are returned via the API in the LAST object. A file `http.json` which should contain either a single JSON object or an array of JSON objects is returned to clients in HTTPmode. * `monitor` a file which contains a timestamp and the last received topic (see Monitoring below). * `msg/` contains messages received by the Messaging system. * `photos/` optional; contains the binary photos from a _card_. -* `rec/` the recorder data proper. One subdirectory per user, one subdirectory therein per device. Data files are named `YYYY-MM.rec` (e.g. `2015-08.rec` for the data accumulated during the month of August 2015. +* `rec/` the Recorder data proper. One subdirectory per user, one subdirectory therein per device. Data files are named `YYYY-MM.rec` (e.g. `2015-08.rec` for the data accumulated during the month of August 2015. * `waypoints/` contains a directory per user and device. Therein are individual files named by a timestamp with the JSON payload of published (i.e. shared) waypoints. The file names are timestamps because the `tst` of a waypoint is its key. If a user publishes all waypoints from a device (Publish Waypoints), the payload is stored in this directory as `username-device.otrw`. (Note, that this is the JSON [waypoints import format](http://owntracks.org/booklet/tech/json/#_typewaypoints).) You can use this `.otrw` file to restore the waypoints on your device by copying to the device and opening it in OwnTracks. -You should definitely **not** modify or touch these files: they remain under the control of the _recorder_. You can of course, remove old `.rec` files if they consume too much space. +You should definitely **not** modify or touch these files: they remain under the control of the Recorder. You can of course, remove old `.rec` files if they consume too much space. ## Configuration file -The recorder attempts to read its startup configuration from a configuration file; the path to this is compiled into the Recorder (typically `/etc/defaults/ot-recorder`, and `ocat -v` will display the compiled-in default). The format of this file approximates that of a shell script with variables to be exported (the intention is so that it can be sourced by a shell script). Lines beginning with an octothorp (`#`) are ignored as are blank lines. Configuration settings proper are set as follows (note that some older versions of _libconfig_ require a trailing semicolon (`;`) at the end of a variable assignment): +The Recorder attempts to read its startup configuration from a configuration file; the path to this is compiled into the Recorder (typically `/etc/defaults/ot-recorder`, and `ocat -v` will display the compiled-in default). The format of this file approximates that of a shell script with variables to be exported (the intention is so that it can be sourced by a shell script). Lines beginning with an octothorp (`#`) are ignored as are blank lines. Configuration settings proper are set as follows (note that some older versions of _libconfig_ require a trailing semicolon (`;`) at the end of a variable assignment): ``` OTR_STORAGEDIR="/var/spool/owntracks/recorder/store" @@ -615,7 +615,7 @@ Note that options passed to `ot-recorder` override both configuration file setti ## Reverse Geo -If not disabled with option `--norevgeo`, the _recorder_ will attempt to perform a reverse-geo lookup on the location coordinates it obtains and store them in an LMDB database. If a lookup is not possible, for example because you're over quota, the service isn't available, etc., _recorder_ keeps tracks of the coordinates which could *not* be resolved in a file named `missing`: +If not disabled with option `--norevgeo`, the Recorder will attempt to perform a reverse-geo lookup on the location coordinates it obtains and store them in an LMDB database. If a lookup is not possible, for example because you're over quota, the service isn't available, etc., Recorder keeps tracks of the coordinates which could *not* be resolved in a file named `missing`: ``` $ cat store/ghash/missing @@ -626,11 +626,11 @@ u0m97hc 46.652733 7.868803 This can be used to subsequently obtain missed lookups. -We recommend you keep reverse-geo lookups enabled, this data (country code `cc`, and the locations address `addr`) is used by the example Web apps provided by the _recorder_ to show where a particular device is. In addition, this cached data is used the the API (also _ocat_) when printing location data. +We recommend you keep reverse-geo lookups enabled, this data (country code `cc`, and the locations address `addr`) is used by the example Web apps provided by the Recorder to show where a particular device is. In addition, this cached data is used the the API (also _ocat_) when printing location data. ### Precision -The precision with which reverse-geo lookups are performed is controlled with the `--precison` option to _recorder_ (and with the `--precision` option to _ocat_ when you query for data). The default precision is compiled into the code (from `config.mk`). The higher the number, the more frequently lookups are performed; conversely, the lower the number, the fewer lookups are performed. For example, a precision of 1 means that points within an area of approximately 5000 km^2 would resolve to a single address, whereas a precision of 7 means that points within an area of approximately 150 m^2 resolve to one address. The _recorder_ obtains a location publish, extracts the latitude and longitude, and then calculates the [geohash](https://en.wikipedia.org/wiki/Geohash) string and truncates it to _precision_. If the calculated geohash string can be found in our local LMDB cache, we consider the point cached; otherwise an actual reverse geo lookup (via HTTP) is performed and the result is cached in LMDB at the key of the geohash. +The precision with which reverse-geo lookups are performed is controlled with the `--precison` option to Recorder (and with the `--precision` option to _ocat_ when you query for data). The default precision is compiled into the code (from `config.mk`). The higher the number, the more frequently lookups are performed; conversely, the lower the number, the fewer lookups are performed. For example, a precision of 1 means that points within an area of approximately 5000 km^2 would resolve to a single address, whereas a precision of 7 means that points within an area of approximately 150 m^2 resolve to one address. The Recorder obtains a location publish, extracts the latitude and longitude, and then calculates the [geohash](https://en.wikipedia.org/wiki/Geohash) string and truncates it to _precision_. If the calculated geohash string can be found in our local LMDB cache, we consider the point cached; otherwise an actual reverse geo lookup (via HTTP) is performed and the result is cached in LMDB at the key of the geohash. As an example, let's assume Jane's device is at position (lat, lon) `48.879840, 2.323522`, which resolves to a geohash string of length 7 `u09whf7`. We can [visualize this](http://www.movable-type.co.uk/scripts/geohash.html) and show what this looks like. (See also: [visualizing geohash](http://www.bigdatamodeling.org/2013/01/intuitive-geohash.html).) @@ -644,7 +644,7 @@ and a precision of 2 would mean that a very large part of France resolves to a s ![geohash2](assets/geohash-2.png) -The bottom line: if you run the _recorder_ with just a few devices and want to know quite exactly where you've been, use a high precision (7 is probably good). If you, on the other hand, run _recorder_ with many devices and are only interested in where a device was approximately, lower the precision; this also has the effect that fewer reverse-geo lookups will be performed in the Google infrastructure. (Also: respect their quotas!) +The bottom line: if you run the Recorder with just a few devices and want to know quite exactly where you've been, use a high precision (7 is probably good). If you, on the other hand, run Recorder with many devices and are only interested in where a device was approximately, lower the precision; this also has the effect that fewer reverse-geo lookups will be performed in the Google infrastructure. (Also: respect their quotas!) ### The geo cache @@ -660,14 +660,14 @@ The key to this data is the geohash string (here with an example of precision 2) ## Monitoring -In order to monitor the _recorder_, whenever an MQTT message is received, a `monitor` file located relative to STORAGEDEFAULT is maintained. It contains a single line of text: the epoch timestamp and the last received topic separated from each other by a space. +In order to monitor the Recorder, whenever an MQTT message is received, a `monitor` file located relative to STORAGEDEFAULT is maintained. It contains a single line of text: the epoch timestamp and the last received topic separated from each other by a space. ``` 1439738692 owntracks/jjolie/ipad ``` -If _recorder_ is built with `WITH_PING` (default), a location publish to `owntracks/ping/ping` (i.e. username is `ping` and device is `ping`) can be used to round-trip-test the recorder. For this particular username/device combination, _recorder_ will store LAST position, but it will not keep a `.REC` file for it. This can be used to verify, say, via your favorite monitoring system, that the _recorder_ is still operational. +If Recorder is built with `WITH_PING` (default), a location publish to `owntracks/ping/ping` (i.e. username is `ping` and device is `ping`) can be used to round-trip-test the ecorder. For this particular username/device combination, Recorder will store LAST position, but it will not keep a `.REC` file for it. This can be used to verify, say, via your favorite monitoring system, that the Recorder is still operational. After sending a _pingping_, you can query the REST interface to determine the difference in time. The `contrib/` directory has an example Python program (`ot-ping.py`) which you can adapt as needed for use by Icinga or Nagios. @@ -677,18 +677,18 @@ OK ot-recorder pingping at http://127.0.0.1:8085: 0 seconds difference ## Lua hooks -If _recorder_ is compiled with Lua support, a Lua script you provide is launched at startup. Lua is _a powerful, fast, lightweight, embeddable scripting language_. You can use this to process location publishes in any way you desire: your imagination (and Lua-scripting knowhow) set the limits. Some examples: +If Recorder is compiled with Lua support, a Lua script you provide is launched at startup. Lua is _a powerful, fast, lightweight, embeddable scripting language_. You can use this to process location publishes in any way you desire: your imagination (and Lua-scripting knowhow) set the limits. Some examples: * insert publishes into a database of your choice * switch on the coffee machine when your OwnTracks device reports you're entering home (but see also [mqttwarn](http://jpmens.net/2014/02/17/introducing-mqttwarn-a-pluggable-mqtt-notifier/)) * write a file with data in a format of your choice (see `etc/example.lua`) -Run the _recorder_ with the path to your Lua script specified in its `--lua-script` option (there is no default). If the script cannot be loaded (e.g. because it cannot be read or contains syntax errors), the _recorder_ unloads Lua and continues *without* your script. +Run the Recorder with the path to your Lua script specified in its `--lua-script` option (there is no default). If the script cannot be loaded (e.g. because it cannot be read or contains syntax errors), the Recorder unloads Lua and continues *without* your script. If the Lua script can be loaded, it is automatically provided with a table variable called `otr` which contains the following members: -* `otr.version` is a read-only string with the _recorder_ version (example: `"0.3.2"`) -* `otr.log(s)` is a function which takes a string `s` which is logged to syslog at the _recorder_'s facility and log level INFO. +* `otr.version` is a read-only string with the Recorder version (example: `"0.3.2"`) +* `otr.log(s)` is a function which takes a string `s` which is logged to syslog at the Recorder's facility and log level INFO. * `otr.strftime(fmt, t)` is a function which takes a format string `fmt` (see `strftime(3)`) and an integer number of seconds `t` and returns a string with the formatted UTC time. If `t` is 0 or negative, the current system time is used. * `otr.putdb(key, value)` is a function which takes two strings `k` and `v` and stores them in the named LMDB database called `luadb`. This can be viewed with * `otr.getdb(key)` is a function which takes a single string `key` and returns the database value associated with that key or `nil` if the key isn't stored. @@ -701,20 +701,20 @@ Your Lua script *must* provide the following functions: ### `otr_init` -This is invoked at start of _recorder_. If the function returns a non-zero value, _recorder_ unloads Lua and disables its processing; i.e. the `hook()` will *not* be invoked on location publishes. +This is invoked at start of Recorder. If the function returns a non-zero value, Recorder unloads Lua and disables its processing; i.e. the `hook()` will *not* be invoked on location publishes. ### `otr_exit` -This is invoked when the _recorder_ stops, which it doesn't really do unless you CTRL-C it or send it a SIGTERM signal. +This is invoked when the Recorder stops, which it doesn't really do unless you CTRL-C it or send it a SIGTERM signal. ### `otr_hook` -This function is invoked at every location publish processed by the _recorder_. Your function is passed three arguments: +This function is invoked at every location publish processed by the Recorder. Your function is passed three arguments: 1. _topic_ is the topic published to (e.g. `owntracks/jane/phone`) 2. _type_ is the type of MQTT message. This is the `_type` in our JSON messages (e.g. `location`, `cmd`, `transition`, ...) or `"unknown"`. -3. _location_ is a [Lua table](http://www.lua.org/pil/2.5.html) (associative array) with all the elements obtained in the JSON message. In the case of _type_ being `location`, we also add country code (`cc`) and the location's address (`addr`) unless reverse-geo lookups have been disabled in _recorder_. +3. _location_ is a [Lua table](http://www.lua.org/pil/2.5.html) (associative array) with all the elements obtained in the JSON message. In the case of _type_ being `location`, we also add country code (`cc`) and the location's address (`addr`) unless reverse-geo lookups have been disabled in Recorder. Assume the following small example Lua script in `example.lua`: @@ -737,7 +737,7 @@ function otr_exit() end ``` -When _recorder_ is launched with `--lua-script example.lua` it invokes `otr_init()` which opens a file. Then, for each location received, it calls `otr_hook()` which updates the file. +When Recorder is launched with `--lua-script example.lua` it invokes `otr_init()` which opens a file. Then, for each location received, it calls `otr_hook()` which updates the file. Assuming an OwnTracks device publishes this payload @@ -757,7 +757,7 @@ It is 14:10:01 in the year 2015 owntracks/jane/phone lat=48.858339 Avenue Anatol An optional function you provide is called `otr_putrec(u, d, s)`. If it exists, it is called with the current user in `u`, the device in `d` and the payload (which for OwnTracks apps is JSON but for, eg Greenwich devices might not be) in the string `s`. If your function returns a -non-zero value, the _recorder_ will *not* write the REC file for this publish. +non-zero value, the Recorder will *not* write the REC file for this publish. ### `otr_httpobject` @@ -765,13 +765,13 @@ An optional function you provide is called `otr_httpobject(u, d, t, data)` where ### Hooklets -After running `otr_hook()`, the _recorder_ attempts to invoke a Lua function for each of the elements in the extended JSON. If, say, your Lua script contains a function called `hooklet_lat`, it will be invoked every time a `lat` is received as part of the JSON payload. Similarly with `hooklet_addr`, `hooklet_cc`, `hooklet_tst`, etc. These _hooklets_ are invoked with the same parameters as `otr_hook()`. +After running `otr_hook()`, the Recorder attempts to invoke a Lua function for each of the elements in the extended JSON. If, say, your Lua script contains a function called `hooklet_lat`, it will be invoked every time a `lat` is received as part of the JSON payload. Similarly with `hooklet_addr`, `hooklet_cc`, `hooklet_tst`, etc. These _hooklets_ are invoked with the same parameters as `otr_hook()`. You define a hooklet function only if you're interested in expressly triggering on a particular JSON element. ## Reverse proxy -Running the _recorder_ protected by an _nginx_ or _Apache_ server is possible and is the only recommended method if you want to server data behind _localhost_. The snippets below show how to do it, but you would also add authentication to them. +Running the Recorder protected by an _nginx_ or _Apache_ server is possible and is the only recommended method if you want to server data behind _localhost_. The snippets below show how to do it, but you would also add authentication to them. ### nginx @@ -808,7 +808,7 @@ server { ### Apache -This will hand URIs which begin with `/otr/` to the _recorder_. +This will hand URIs which begin with `/otr/` to the Recorder. ``` @@ -1090,7 +1090,7 @@ This named lmdb database is keyed on topic name (`owntracks/jane/phone`). If the #### `keys` -If the _recorder_ was built with encryption support (see below), this named database contains the secret decryption keys for users/device pairs. The LMDB key is the username followed by a dash followed by the device name, all lower case, with spaces translated to a single dash. For example, if user Jjolie with device iPhone needs a secret entered, the database key will be `jjolie-iphone`. This can be entered into the database as follows: +If the Recorder was built with encryption support (see below), this named database contains the secret decryption keys for users/device pairs. The LMDB key is the username followed by a dash followed by the device name, all lower case, with spaces translated to a single dash. For example, if user Jjolie with device iPhone needs a secret entered, the database key will be `jjolie-iphone`. This can be entered into the database as follows: ```bash echo "jjolie-iphone s3cr1t" | ocat --load=keys @@ -1164,7 +1164,7 @@ The packages we provide have a systemd unit file in `/usr/share/doc/ot-recorder/ ## Docker -We also have a Docker image to create containers which integrate a [Mosquitto broker](http://mosquitto.org) with the _Recorder_. The Docker image is [available from the Docker hub](https://hub.docker.com/r/owntracks/recorderd/) (e.g. `docker pull owntracks/recorderd`), and it's [usage is documented in the Booklet](http://owntracks.org/booklet/clients/recorder/). +We also have a Docker image to create containers which integrate a [Mosquitto broker](http://mosquitto.org) with the Recorder. The Docker image is [available from the Docker hub](https://hub.docker.com/r/owntracks/recorderd/) (e.g. `docker pull owntracks/recorderd`), and it's [usage is documented in the Booklet](http://owntracks.org/booklet/clients/recorder/). ## Tips and Tricks From c96d4bdfbf517d7ba063536cc31f77f74aee06b0 Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Tue, 16 Aug 2016 17:53:47 -0400 Subject: [PATCH 07/23] Fix typo --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 3c62db9..3822f22 100644 --- a/README.md +++ b/README.md @@ -667,7 +667,7 @@ In order to monitor the Recorder, whenever an MQTT message is received, a `monit ``` -If Recorder is built with `WITH_PING` (default), a location publish to `owntracks/ping/ping` (i.e. username is `ping` and device is `ping`) can be used to round-trip-test the ecorder. For this particular username/device combination, Recorder will store LAST position, but it will not keep a `.REC` file for it. This can be used to verify, say, via your favorite monitoring system, that the Recorder is still operational. +If Recorder is built with `WITH_PING` (default), a location publish to `owntracks/ping/ping` (i.e. username is `ping` and device is `ping`) can be used to round-trip-test the Recorder. For this particular username/device combination, Recorder will store LAST position, but it will not keep a `.REC` file for it. This can be used to verify, say, via your favorite monitoring system, that the Recorder is still operational. After sending a _pingping_, you can query the REST interface to determine the difference in time. The `contrib/` directory has an example Python program (`ot-ping.py`) which you can adapt as needed for use by Icinga or Nagios. @@ -1112,7 +1112,7 @@ The user/device separator in the array's strings may be a slash (`/`), a dash (` ## Encryption (*experimental!*) -If compiled with `WITH_ENCRYPT` support (this is the default in our packages), the recorder will handle messages from OwnTracks [devices which support payload encryption](http://owntracks.org/booklet/features/encrypt/). Each user / device requires a secret key which is configured on the device and which must be configured on the Recorder host in order for the Recorder to be able to decrypt the payloads. +If compiled with `WITH_ENCRYPT` support (this is the default in our packages), the Recorder will handle messages from OwnTracks [devices which support payload encryption](http://owntracks.org/booklet/features/encrypt/). Each user / device requires a secret key which is configured on the device and which must be configured on the Recorder host in order for the Recorder to be able to decrypt the payloads. Upon successful decryption, the Recorder processes the original (device-transmitted) JSON and stores the result in plain (i.e. un-encrypted) form in the store. From 6f361a9dc73e6ae688ac9a78ac339f0bfa50dcbe Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Tue, 16 Aug 2016 18:02:15 -0400 Subject: [PATCH 08/23] Consolidate installation instructions --- README.md | 128 +++++++++++++++++++++++++++--------------------------- 1 file changed, 65 insertions(+), 63 deletions(-) diff --git a/README.md b/README.md index 3822f22..463c42d 100644 --- a/README.md +++ b/README.md @@ -16,9 +16,15 @@ We developed the Recorder as a one-stop solution to storing location data publis * [`recorder`](#recorder) * [Installing](#installing) -* [Building from source](#building-from-source) - * [Prerequisites](#prerequisites) - * [Building](#building) + * [Packages](#packages) + * [Installing on CentOS 7](#installing-on-centos-7) + * [Installing on Raspian (Wheezy)](#installing-on-raspian-wheezy) + * [Installing on Debian 8 (Jessie)](#installing-on-debian-8-jessie) + * [systemd service](#systemd-service) + * [Docker](#docker) + * [Building from source](#building-from-source) + * [Prerequisites](#prerequisites) + * [Building](#building) * [Getting started](#getting-started) * [`ot-recorder` options](#ot-recorder-options) * [The HTTP Server](#the-http-server) @@ -74,12 +80,6 @@ We developed the Recorder as a one-stop solution to storing location data publis * [`keys`](#keys) * [`friends`](#friends) * [Encryption (*experimental!*)](#encryption-experimental) -* [Packages](#packages) - * [Installing on CentOS 7](#installing-on-centos-7) - * [Installing on Raspian (Wheezy)](#installing-on-raspian-wheezy) - * [Installing on Debian 8 (Jessie)](#installing-on-debian-8-jessie) - * [systemd service](#systemd-service) -* [Docker](#docker) * [Tips and Tricks](#tips-and-tricks) * [Gatewaying HTTP to MQTT](#gatewaying-http-to-mqtt) * [Override reverse-geo precision](#override-reverse-geo-precision) @@ -96,11 +96,61 @@ The Recorder serves two purposes: We provide a ready-to-run packages for a limited number of platforms on our [package repository](http://repo.owntracks.org/README.txt), and we provide a Docker image which bundles the Recorder and a Mosquitto broker [directly from the Docker hub](https://hub.docker.com/r/owntracks/recorderd/). -You will, however, need to acquire and configure apikeys for the maps. (See below.) +If those don't work for you, you can build from source. -## Building from source +### Packages -### Prerequisites +We create packages for releases for a few distributions. Please note that these packages depend on libmosquitto1 from the [Mosquitto project](http://mosquitto.org/downloads). + +Binaries (`ocat`, `ot-recorder`) from these packages run setuid to user `owntracks` so that they work for all users of the system. Note that, say, certificate files you provide must therefore also be readable by the user `owntracks`. + +#### Installing on CentOS 7 + +``` +curl -o /etc/yum.repos.d/mosquitto.repo http://download.opensuse.org/repositories/home:/oojah:/mqtt/CentOS_CentOS-7/home:oojah:mqtt.repo + +curl -o /etc/yum.repos.d/owntracks.repo http://repo.owntracks.org/centos/owntracks.repo + +yum install ot-recorder +``` + +#### Installing on Raspian (Wheezy) + +``` +wget http://repo.owntracks.org/repo.owntracks.org.gpg.key +apt-key add repo.owntracks.org.gpg.key +echo "deb http://repo.owntracks.org/debian wheezy main" > /etc/apt/sources.list.d/owntracks.list +apt-get update +apt-get install ot-recorder +``` + +#### Installing on Debian 8 (Jessie) + +``` +wget http://repo.owntracks.org/repo.owntracks.org.gpg.key +apt-key add repo.owntracks.org.gpg.key +echo "deb http://repo.owntracks.org/debian jessie main" > /etc/apt/sources.list.d/owntracks.list +apt-get update +apt-get install ot-recorder +``` + +#### systemd service + +The packages we provide have a systemd unit file in `/usr/share/doc/ot-recorder/ot-recorder.service` which you can use to have the Recorder started automatically: + +1. Ensure you have a configuration file with the settings you require +2. `install -m444 /usr/share/doc/ot-recorder/ot-recorder.service /etc/systemd/system/ot-recorder.service` +3. Enable the service to run at startup: `systemctl enable ot-recorder` +4. Launch the service `systemctl start ot-recorder` + + +### Docker + +We also have a Docker image to create containers which integrate a [Mosquitto broker](http://mosquitto.org) with the Recorder. The Docker image is [available from the Docker hub](https://hub.docker.com/r/owntracks/recorderd/) (e.g. `docker pull owntracks/recorderd`), and it's [usage is documented in the Booklet](http://owntracks.org/booklet/clients/recorder/). + +### Building from source + +#### Prerequisites You will require: @@ -136,7 +186,7 @@ sudo apt-get update sudo apt-get install libmosquitto-dev libcurl3 libcurl4-openssl-dev libconfig-dev ``` -### Building +#### Building 1. Obtain and download the software, via [our Homebrew Tap](https://github.com/owntracks/homebrew-recorder) on Mac OS X, directly as a clone of the repository, or as a [tar ball](https://github.com/owntracks/recorder/releases) which you unpack. 2. Copy the included `config.mk.in` file to `config.mk` and edit that. You specify the features or tweaks you need. (The file is commented.) Pay particular attention to the installation directory and the value of the _store_ (`STORAGEDEFAULT`): that is where the Recorder will store its files. `DOCROOT` is the root of the directory from which the Recorder's HTTP server will serve files. @@ -179,6 +229,8 @@ Unless already provided by the package you installed, we recommend you create a * that each instance uses a distinct `--storage` * that each instance uses a distinct `--http-port` (or `0` if you don't wish to provide HTTP support for a particular instance) +You also need to provide API keys for the maps. + ### `ot-recorder` options This section lists the most important options of the Recorder with their long names; check the usage (`recorder -h`) for the short versions. @@ -1116,56 +1168,6 @@ If compiled with `WITH_ENCRYPT` support (this is the default in our packages), t Upon successful decryption, the Recorder processes the original (device-transmitted) JSON and stores the result in plain (i.e. un-encrypted) form in the store. -## Packages - -We create packages for releases for a few distributions. Please note that these packages depend on libmosquitto1 from the [Mosquitto project](http://mosquitto.org/downloads). - -Binaries (`ocat`, `ot-recorder`) from these packages run setuid to user `owntracks` so that they work for all users of the system. Note that, say, certificate files you provide must therefore also be readable by the user `owntracks`. - -### Installing on CentOS 7 - -``` -curl -o /etc/yum.repos.d/mosquitto.repo http://download.opensuse.org/repositories/home:/oojah:/mqtt/CentOS_CentOS-7/home:oojah:mqtt.repo - -curl -o /etc/yum.repos.d/owntracks.repo http://repo.owntracks.org/centos/owntracks.repo - -yum install ot-recorder -``` - -### Installing on Raspian (Wheezy) - -``` -wget http://repo.owntracks.org/repo.owntracks.org.gpg.key -apt-key add repo.owntracks.org.gpg.key -echo "deb http://repo.owntracks.org/debian wheezy main" > /etc/apt/sources.list.d/owntracks.list -apt-get update -apt-get install ot-recorder -``` - -### Installing on Debian 8 (Jessie) - -``` -wget http://repo.owntracks.org/repo.owntracks.org.gpg.key -apt-key add repo.owntracks.org.gpg.key -echo "deb http://repo.owntracks.org/debian jessie main" > /etc/apt/sources.list.d/owntracks.list -apt-get update -apt-get install ot-recorder -``` - -### systemd service - -The packages we provide have a systemd unit file in `/usr/share/doc/ot-recorder/ot-recorder.service` which you can use to have the Recorder started automatically: - -1. Ensure you have a configuration file with the settings you require -2. `install -m444 /usr/share/doc/ot-recorder/ot-recorder.service /etc/systemd/system/ot-recorder.service` -3. Enable the service to run at startup: `systemctl enable ot-recorder` -4. Launch the service `systemctl start ot-recorder` - - -## Docker - -We also have a Docker image to create containers which integrate a [Mosquitto broker](http://mosquitto.org) with the Recorder. The Docker image is [available from the Docker hub](https://hub.docker.com/r/owntracks/recorderd/) (e.g. `docker pull owntracks/recorderd`), and it's [usage is documented in the Booklet](http://owntracks.org/booklet/clients/recorder/). - ## Tips and Tricks ### Gatewaying HTTP to MQTT From 830a0aaeee13fce7af8c4999275f1c8d1fd7d563 Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Tue, 16 Aug 2016 18:06:56 -0400 Subject: [PATCH 09/23] Remove package installation distro headers --- README.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 463c42d..1ab4bbc 100644 --- a/README.md +++ b/README.md @@ -17,9 +17,6 @@ We developed the Recorder as a one-stop solution to storing location data publis * [`recorder`](#recorder) * [Installing](#installing) * [Packages](#packages) - * [Installing on CentOS 7](#installing-on-centos-7) - * [Installing on Raspian (Wheezy)](#installing-on-raspian-wheezy) - * [Installing on Debian 8 (Jessie)](#installing-on-debian-8-jessie) * [systemd service](#systemd-service) * [Docker](#docker) * [Building from source](#building-from-source) @@ -104,7 +101,7 @@ We create packages for releases for a few distributions. Please note that these Binaries (`ocat`, `ot-recorder`) from these packages run setuid to user `owntracks` so that they work for all users of the system. Note that, say, certificate files you provide must therefore also be readable by the user `owntracks`. -#### Installing on CentOS 7 +If you're on CentOS 7, you can install with: ``` curl -o /etc/yum.repos.d/mosquitto.repo http://download.opensuse.org/repositories/home:/oojah:/mqtt/CentOS_CentOS-7/home:oojah:mqtt.repo @@ -114,7 +111,7 @@ curl -o /etc/yum.repos.d/owntracks.repo http://repo.owntracks.org/centos/owntrac yum install ot-recorder ``` -#### Installing on Raspian (Wheezy) +Debian 7 "Wheezy" (including Rasbian): ``` wget http://repo.owntracks.org/repo.owntracks.org.gpg.key @@ -124,7 +121,7 @@ apt-get update apt-get install ot-recorder ``` -#### Installing on Debian 8 (Jessie) +Debian 8 "Jessie": ``` wget http://repo.owntracks.org/repo.owntracks.org.gpg.key From 3e25dbc3a5be82330cfa393dad9f6a45931911f7 Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Tue, 16 Aug 2016 18:10:48 -0400 Subject: [PATCH 10/23] Improve package installation commands Most users will be using `sudo` and an unprivileged account anyway, and the `>` shell operator doesn't work in this scenario. So, optimize for the majority and give commands that can be blindly copied and pasted. Running things directly as root isn't a great idea anyway. --- README.md | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 1ab4bbc..4b16c7c 100644 --- a/README.md +++ b/README.md @@ -104,31 +104,29 @@ Binaries (`ocat`, `ot-recorder`) from these packages run setuid to user `owntrac If you're on CentOS 7, you can install with: ``` -curl -o /etc/yum.repos.d/mosquitto.repo http://download.opensuse.org/repositories/home:/oojah:/mqtt/CentOS_CentOS-7/home:oojah:mqtt.repo +sudo curl -o /etc/yum.repos.d/mosquitto.repo http://download.opensuse.org/repositories/home:/oojah:/mqtt/CentOS_CentOS-7/home:oojah:mqtt.repo -curl -o /etc/yum.repos.d/owntracks.repo http://repo.owntracks.org/centos/owntracks.repo +sudo curl -o /etc/yum.repos.d/owntracks.repo http://repo.owntracks.org/centos/owntracks.repo -yum install ot-recorder +sudo yum install ot-recorder ``` Debian 7 "Wheezy" (including Rasbian): ``` -wget http://repo.owntracks.org/repo.owntracks.org.gpg.key -apt-key add repo.owntracks.org.gpg.key -echo "deb http://repo.owntracks.org/debian wheezy main" > /etc/apt/sources.list.d/owntracks.list -apt-get update -apt-get install ot-recorder +curl http://repo.owntracks.org/repo.owntracks.org.gpg.key | sudo apt-key add - +echo "deb http://repo.owntracks.org/debian wheezy main" | sudo tee /etc/apt/sources.list.d/owntracks.list > /dev/null +sudo apt-get update +sudo apt-get install ot-recorder ``` Debian 8 "Jessie": ``` -wget http://repo.owntracks.org/repo.owntracks.org.gpg.key -apt-key add repo.owntracks.org.gpg.key -echo "deb http://repo.owntracks.org/debian jessie main" > /etc/apt/sources.list.d/owntracks.list -apt-get update -apt-get install ot-recorder +curl http://repo.owntracks.org/repo.owntracks.org.gpg.key | sudo apt-key add - +echo "deb http://repo.owntracks.org/debian jessie main" | sudo tee /etc/apt/sources.list.d/owntracks.list > /dev/null +sudo apt-get update +sudo apt-get install ot-recorder ``` #### systemd service From 793f2a75b3062081eba254e378f08a388a581c25 Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Tue, 16 Aug 2016 19:06:35 -0400 Subject: [PATCH 11/23] Split out API docs into API.md --- API.md | 92 +++++++++++++++++++++++++++++++++++++++++++++++ README.md | 104 +----------------------------------------------------- 2 files changed, 93 insertions(+), 103 deletions(-) create mode 100644 API.md diff --git a/API.md b/API.md new file mode 100644 index 0000000..592b989 --- /dev/null +++ b/API.md @@ -0,0 +1,92 @@ +# API + +The Recorder's API provides most of the functions that are surfaced by _ocat_. GET and POST requests are supported, and if a username and device are needed, these can be passed in via `X-Limit-User` and `X-Limit-Device` headers alternatively to GET or POST parameters. (From and To dates may also be specified as `X-Limit-From` and `X-Limit-To` respectively.) + +The API endpoint is at `/api/0` and is followed by the verb. + +## `monitor` + +Returns the content of the `monitor` file as plain text. + +``` +curl 'http://127.0.0.1:8083/api/0/monitor' +1441962082 owntracks/jjolie/phone +``` + +## `last` + +Returns a list of last users' positions. (Can be limited by _user_, _device_, and _fields_, a comma-separated list of fields which should be returned instead of the default of all fields.) + +``` +curl http://127.0.0.1:8083/api/0/last [-d user=jjolie [-d device=phone]] +``` + +``` +curl 'http://127.0.0.1:8083/api/0/last?fields=tst,tid,addr,topic,isotst' +``` + +## `list` + +List users. If _user_ is specified, lists that user's devices. If both _user_ and _device_ are specified, lists that device's `.rec` files. + +## `locations` + +Here comes the actual data. This lists users' locations and requires both _user_ and _device_. Output format is JSON unless a different _format_ is given (`csv`, `json`, `geojson`, `xml`, and `linestring` are supported). + +In order to limit the number of records returned, use _limit_ which causes a reverse search through the `.rec` files; this can be used to find the last N positions. + +Date/time ranges may be specified as _from_ and _to_ with dates/times specified as described for _ocat_ above. + +``` +curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s +curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s -d limit=1 +curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s -d format=geojson +curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s -d from=2014-08-03 +curl 'http://127.0.0.1:8083/api/0/locations?from=2015-09-01&user=jpm&device=5s&fields=tst,tid,addr,isotst' +``` + +## `q` + +Query the geo cache for a particular _lat_ and _lon_. + +``` +curl 'http://127.0.0.1:8083/api/0/q?lat=48.85833&lon=2.295' +{ + "cc": "FR", + "addr": "9 Avenue Anatole France, 75007 Paris, France", + "tst": 1441984405 +} +``` + +The reported timestamp was the time at which this cache entry was made. Note that this interface queries only -- it does not populate the cache. + +## `photo` + +Requires GET method and _user_, and will return the `image/png` 40x40px photograph of a user if available in `STORAGEDIR/photos/` or a transparent 40x40png with a black border otherwise. + +## `kill` + +If support for this is compiled in, this API endpoint allows a client to remove data from _storage_. (Warning: *any* client can do this, as there is no authentication/authorization in the Recorder!) + +``` +curl 'http://127.0.0.1:8083/api/0/kill?user=ngin&device=ojo' + +{ + "path": "s0/rec/ngin/ojo", + "status": "OK", + "last": "s0/last/ngin/ojo/ngin-ojo.json", + "killed": [ + "2015-09.rec", + ] +} +``` +The response contains a list of removed `.rec` files, and file system operations are logged to syslog. + +## `version` + +Returns a JSON object which contains the Recorder's version string, such as + +```json +{ "version": "0.4.7" } +``` + diff --git a/README.md b/README.md index 4b16c7c..97fc190 100644 --- a/README.md +++ b/README.md @@ -31,15 +31,6 @@ We developed the Recorder as a one-stop solution to storing location data publis * [Display a track (a.k.a. linestring)](#display-a-track-aka-linestring) * [Tabular display](#tabular-display) * [Live map](#live-map) - * [API](#api) - * [`monitor`](#monitor) - * [`last`](#last) - * [`list`](#list) - * [`locations`](#locations) - * [`q`](#q) - * [`photo`](#photo) - * [`kill`](#kill) - * [`version`](#version) * [`ocat`](#ocat) * [Environment](#environment) * [Examples](#examples) @@ -272,7 +263,7 @@ This section lists the most important options of the Recorder with their long na The Recorder has a built-in HTTP server with which it servers static files from either the compiled-in default `DOCROOT` directory or that specified at run-time with the `--doc-root` option. Furthermore, it serves JSON data from the API end-point at `/api/0/` and it has a built-in WebSocket server for the live map. -The API basically serves the same data as _ocat_ is able to produce. The server also accepts OwnTracks app data via HTTP POST to the `/pub` endpoint. +The API basically serves the same data as _ocat_ is able to produce - see [API.md](https://github.com/owntracks/recorder/blob/master/API.md). The server also accepts OwnTracks app data via HTTP POST to the `/pub` endpoint. ### Example functionality @@ -348,99 +339,6 @@ The Recorder's built-in WebSocket server updates a map as it receives publishes ![Live map](assets/demo-live-map.png) -### API - -The Recorder's API provides most of the functions that are surfaced by _ocat_. GET and POST requests are supported, and if a username and device are needed, these can be passed in via `X-Limit-User` and `X-Limit-Device` headers alternatively to GET or POST parameters. (From and To dates may also be specified as `X-Limit-From` and `X-Limit-To` -respectively.) - -The API endpoint is at `/api/0` and is followed by the verb. - -#### `monitor` - -Returns the content of the `monitor` file as plain text. - -``` -curl 'http://127.0.0.1:8083/api/0/monitor' -1441962082 owntracks/jjolie/phone -``` - -#### `last` - -Returns a list of last users' positions. (Can be limited by _user_, _device_, and _fields_, a comma-separated list of fields which should be returned instead of the default of all fields.) - -``` -curl http://127.0.0.1:8083/api/0/last [-d user=jjolie [-d device=phone]] -``` - -``` -curl 'http://127.0.0.1:8083/api/0/last?fields=tst,tid,addr,topic,isotst' -``` - -#### `list` - -List users. If _user_ is specified, lists that user's devices. If both _user_ and _device_ are specified, lists that device's `.rec` files. - -#### `locations` - -Here comes the actual data. This lists users' locations and requires both _user_ and _device_. Output format is JSON unless a different _format_ is given (`csv`, `json`, `geojson`, `xml`, and `linestring` are supported). - -In order to limit the number of records returned, use _limit_ which causes a reverse search through the `.rec` files; this can be used to find the last N positions. - -Date/time ranges may be specified as _from_ and _to_ with dates/times specified as described for _ocat_ above. - -``` -curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s -curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s -d limit=1 -curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s -d format=geojson -curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s -d from=2014-08-03 -curl 'http://127.0.0.1:8083/api/0/locations?from=2015-09-01&user=jpm&device=5s&fields=tst,tid,addr,isotst' -``` - -#### `q` - -Query the geo cache for a particular _lat_ and _lon_. - -``` -curl 'http://127.0.0.1:8083/api/0/q?lat=48.85833&lon=2.295' -{ - "cc": "FR", - "addr": "9 Avenue Anatole France, 75007 Paris, France", - "tst": 1441984405 -} -``` - -The reported timestamp was the time at which this cache entry was made. Note that this interface queries only -- it does not populate the cache. - -#### `photo` - -Requires GET method and _user_, and will return the `image/png` 40x40px photograph of a user if available in `STORAGEDIR/photos/` or a transparent 40x40png with a black border otherwise. - -#### `kill` - -If support for this is compiled in, this API endpoint allows a client to remove data from _storage_. (Warning: *any* client can do this, as there is no authentication/authorization in the Recorder!) - -``` -curl 'http://127.0.0.1:8083/api/0/kill?user=ngin&device=ojo' - -{ - "path": "s0/rec/ngin/ojo", - "status": "OK", - "last": "s0/last/ngin/ojo/ngin-ojo.json", - "killed": [ - "2015-09.rec", - ] -} -``` -The response contains a list of removed `.rec` files, and file system operations are logged to syslog. - -#### `version` - -Returns a JSON object which contains the Recorder's version string, such as - -```json -{ "version": "0.4.7" } -``` - ## `ocat` _ocat_ is a CLI query program for data stored by Recorder: it prints data from storage in a variety of output formats: From c914f57352aa4e323a735f9e9c790f822cabc0fb Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Tue, 16 Aug 2016 19:14:49 -0400 Subject: [PATCH 12/23] Remove note about MQTT support in the HTTP section It's out of place and irrelevant. --- README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/README.md b/README.md index 97fc190..3373acd 100644 --- a/README.md +++ b/README.md @@ -947,8 +947,6 @@ curl -H 'X-Limit-U: jane' -H 'X-Limit-D: 3s' --data "${payload}" 'http://127.0.0 The content of the request is used by the Recorder as though it had arrived as an MQTT message; Lua hooks and WebSocket pushes are handled accordingly. -If the Recorder is compiled without specifying `WITH_MQTT` at build time, support for MQTT is disabled completely. - ### Friends in HTTP mode When a device posts a location request in HTTP mode, the endpoint may return a JSON array of OwnTracks objects of which `_type`s `cmd`, `location` and `card` may be supported by the device. This allows the device to see, say, friends. The Recorder has built-in support for this with the named "friends" lmdb database. From 7b54006839560587e3a846f4810ba43578f5070f Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Tue, 16 Aug 2016 19:18:07 -0400 Subject: [PATCH 13/23] Move reverse proxy and configuration file sections The top is a much more sensible location for them. --- README.md | 177 +++++++++++++++++++++++++++--------------------------- 1 file changed, 88 insertions(+), 89 deletions(-) diff --git a/README.md b/README.md index 3373acd..743d0ff 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,10 @@ We developed the Recorder as a one-stop solution to storing location data publis * [Building](#building) * [Getting started](#getting-started) * [`ot-recorder` options](#ot-recorder-options) +* [Configuration file](#configuration-file) +* [Reverse proxy](#reverse-proxy) + * [nginx](#nginx) + * [Apache](#apache) * [The HTTP Server](#the-http-server) * [Example functionality](#example-functionality) * [Last position of a particular user](#last-position-of-a-particular-user) @@ -39,7 +43,6 @@ We developed the Recorder as a one-stop solution to storing location data publis * [What were the last 4 positions reported?](#what-were-the-last-4-positions-reported) * [Design decisions](#design-decisions) * [Storage](#storage) -* [Configuration file](#configuration-file) * [Reverse Geo](#reverse-geo) * [Precision](#precisioin) * [The geo cache](#the-geo-cache) @@ -51,9 +54,6 @@ We developed the Recorder as a one-stop solution to storing location data publis * [`otr_putrec`](#otr_putrec) * [`otr_httpobject`](#otr_httpobject) * [Hooklets](#hooklets) -* [Reverse proxy](#reverse-proxy) - * [nginx](#nginx) - * [Apache](#apache) * [Views](#views) * [view JSON](#view-json) * [Authentication](#authentication) @@ -259,6 +259,90 @@ This section lists the most important options of the Recorder with their long na `--debug` enables a bit of additional debugging on stderr. +## Configuration file + +The Recorder attempts to read its startup configuration from a configuration file; the path to this is compiled into the Recorder (typically `/etc/defaults/ot-recorder`, and `ocat -v` will display the compiled-in default). The format of this file approximates that of a shell script with variables to be exported (the intention is so that it can be sourced by a shell script). Lines beginning with an octothorp (`#`) are ignored as are blank lines. Configuration settings proper are set as follows (note that some older versions of _libconfig_ require a trailing semicolon (`;`) at the end of a variable assignment): + +``` +OTR_STORAGEDIR="/var/spool/owntracks/recorder/store" +``` + +The following configuration settings may be applied (a `Y` in column `$` means an environment variable of the same name overrides a setting in the config file): + +| Variable | $ | Default | Usage +| --------------------- | :--- | :------------ | --------------- +| `OTR_STORAGEDIR` | Y | compiled in | Pathname to the storage directory +| `OTR_HOST` | Y | `localhost` | MQTT hostname/address to connect to +| `OTR_PORT` | Y | `1883` | MQTT port number to connect to +| `OTR_USER` | Y | | MQTT username +| `OTR_PASS` | Y | | MQTT password +| `OTR_QOS` | | `2` | MQTT QoS +| `OTR_CLIENTID` | | hostname+pid | MQTT ClientID (override with -i) +| `OTR_HTTPHOST` | | `localhost` | Address for the HTTP module to bind to +| `OTR_HTTPPORT` | | `8083` | Port number of the HTTP module to bind to +| `OTR_HTTPLOGDIR` | | | Directory in which to store access.log. Override with --http-logdir +| `OTR_LUASCRIPT` | | | Path to the Lua script +| `OTR_PRECISION` | | `7` | Reverse-geo precision +| `OTR_GEOKEY` | | | API key for reverse-geo lookups +| `OTR_TOPICS` | | | String containing a space-separated list of topics to subscribe to for MQTT (overriden by command-line arguments) +| `OTR_CAFILE` | Y | | Path to PEM-encoded CA certificate file for MQTT (implicitly enables TLS) + + +Note that options passed to `ot-recorder` override both configuration file settings and environment variables. + +## Reverse proxy + +Running the Recorder protected by an _nginx_ or _Apache_ server is possible and is the only recommended method if you want to server data behind _localhost_. The snippets below show how to do it, but you would also add authentication to them. + +### nginx + +``` +server { + listen 8080; + server_name 192.168.1.130; + + location / { + root html; + index index.html index.htm; + } + + # Proxy and upgrade WebSocket connection + location /otr/ws { + rewrite ^/otr/(.*) /$1 break; + proxy_pass http://127.0.0.1:8083; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + } + + location /otr/ { + proxy_pass http://127.0.0.1:8083/; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Real-IP $remote_addr; + } +} +``` + +### Apache + +This will hand URIs which begin with `/otr/` to the Recorder. + +``` + +# WebSocket URL endpoint +# a2enmod proxy_wstunnel +ProxyPass /otr/ws ws://127.0.0.1:8083/ws keepalive=on retry=60 +ProxyPassReverse /otr/ws ws://127.0.0.1:8083/ws keepalive=on + +# Static files +ProxyPass /otr http://127.0.0.1:8083/ +ProxyPassReverse /otr http://127.0.0.1:8083/ +``` + ## The HTTP server The Recorder has a built-in HTTP server with which it servers static files from either the compiled-in default `DOCROOT` directory or that specified at run-time with the `--doc-root` option. Furthermore, it serves JSON data from the API end-point at `/api/0/` and it has a built-in WebSocket server for the live map. @@ -526,38 +610,6 @@ As mentioned earlier, data is stored in files, and these files are relative to ` You should definitely **not** modify or touch these files: they remain under the control of the Recorder. You can of course, remove old `.rec` files if they consume too much space. -## Configuration file - -The Recorder attempts to read its startup configuration from a configuration file; the path to this is compiled into the Recorder (typically `/etc/defaults/ot-recorder`, and `ocat -v` will display the compiled-in default). The format of this file approximates that of a shell script with variables to be exported (the intention is so that it can be sourced by a shell script). Lines beginning with an octothorp (`#`) are ignored as are blank lines. Configuration settings proper are set as follows (note that some older versions of _libconfig_ require a trailing semicolon (`;`) at the end of a variable assignment): - -``` -OTR_STORAGEDIR="/var/spool/owntracks/recorder/store" -``` - -The following configuration settings may be applied (a `Y` in column `$` means an environment variable of the same name overrides a setting in the config file): - -| Variable | $ | Default | Usage -| --------------------- | :--- | :------------ | --------------- -| `OTR_STORAGEDIR` | Y | compiled in | Pathname to the storage directory -| `OTR_HOST` | Y | `localhost` | MQTT hostname/address to connect to -| `OTR_PORT` | Y | `1883` | MQTT port number to connect to -| `OTR_USER` | Y | | MQTT username -| `OTR_PASS` | Y | | MQTT password -| `OTR_QOS` | | `2` | MQTT QoS -| `OTR_CLIENTID` | | hostname+pid | MQTT ClientID (override with -i) -| `OTR_HTTPHOST` | | `localhost` | Address for the HTTP module to bind to -| `OTR_HTTPPORT` | | `8083` | Port number of the HTTP module to bind to -| `OTR_HTTPLOGDIR` | | | Directory in which to store access.log. Override with --http-logdir -| `OTR_LUASCRIPT` | | | Path to the Lua script -| `OTR_PRECISION` | | `7` | Reverse-geo precision -| `OTR_GEOKEY` | | | API key for reverse-geo lookups -| `OTR_TOPICS` | | | String containing a space-separated list of topics to subscribe to for MQTT (overriden by command-line arguments) -| `OTR_CAFILE` | Y | | Path to PEM-encoded CA certificate file for MQTT (implicitly enables TLS) - - -Note that options passed to `ot-recorder` override both configuration file settings and environment variables. - - ## Reverse Geo If not disabled with option `--norevgeo`, the Recorder will attempt to perform a reverse-geo lookup on the location coordinates it obtains and store them in an LMDB database. If a lookup is not possible, for example because you're over quota, the service isn't available, etc., Recorder keeps tracks of the coordinates which could *not* be resolved in a file named `missing`: @@ -714,59 +766,6 @@ After running `otr_hook()`, the Recorder attempts to invoke a Lua function for e You define a hooklet function only if you're interested in expressly triggering on a particular JSON element. -## Reverse proxy - -Running the Recorder protected by an _nginx_ or _Apache_ server is possible and is the only recommended method if you want to server data behind _localhost_. The snippets below show how to do it, but you would also add authentication to them. - -### nginx - -``` -server { - listen 8080; - server_name 192.168.1.130; - - location / { - root html; - index index.html index.htm; - } - - # Proxy and upgrade WebSocket connection - location /otr/ws { - rewrite ^/otr/(.*) /$1 break; - proxy_pass http://127.0.0.1:8083; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection "upgrade"; - proxy_set_header Host $host; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - } - - location /otr/ { - proxy_pass http://127.0.0.1:8083/; - proxy_http_version 1.1; - proxy_set_header Host $host; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Real-IP $remote_addr; - } -} -``` - -### Apache - -This will hand URIs which begin with `/otr/` to the Recorder. - -``` - -# WebSocket URL endpoint -# a2enmod proxy_wstunnel -ProxyPass /otr/ws ws://127.0.0.1:8083/ws keepalive=on retry=60 -ProxyPassReverse /otr/ws ws://127.0.0.1:8083/ws keepalive=on - -# Static files -ProxyPass /otr http://127.0.0.1:8083/ -ProxyPassReverse /otr http://127.0.0.1:8083/ -``` - ## Views A view is a sort of sandboxed look at data provided by the Recorder. Assume you host several devices, be they your own or those of some of your friends, and assume you want to allow somebody else to see where you are or have been during a specific time frame: with the Recorder's default Web server you cannot limit a visitor to see specific data only; once they reach the Recorder's Web interface, they have access to all your data. (We warned you about that earlier.) Using a HTTP proxy, you can provide an insight into certain portions of your data only. From 5a9b9b4023dbb461e70b2d7ccdf2950af0633a56 Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Tue, 16 Aug 2016 21:38:41 -0400 Subject: [PATCH 14/23] Split Lua hooks docs into their own file --- HOOKS.md | 93 ++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 104 +++--------------------------------------------------- 2 files changed, 97 insertions(+), 100 deletions(-) create mode 100644 HOOKS.md diff --git a/HOOKS.md b/HOOKS.md new file mode 100644 index 0000000..9e589cd --- /dev/null +++ b/HOOKS.md @@ -0,0 +1,93 @@ +# Lua hooks + +If Recorder is compiled with Lua support, a Lua script you provide is launched at startup. Lua is _a powerful, fast, lightweight, embeddable scripting language_. You can use this to process location publishes in any way you desire: your imagination (and Lua-scripting knowhow) set the limits. Some examples: + +* insert publishes into a database of your choice +* switch on the coffee machine when your OwnTracks device reports you're entering home (but see also [mqttwarn](http://jpmens.net/2014/02/17/introducing-mqttwarn-a-pluggable-mqtt-notifier/)) +* write a file with data in a format of your choice (see `etc/example.lua`) + +Run the Recorder with the path to your Lua script specified in its `--lua-script` option (there is no default). If the script cannot be loaded (e.g. because it cannot be read or contains syntax errors), the Recorder unloads Lua and continues *without* your script. + +If the Lua script can be loaded, it is automatically provided with a table variable called `otr` which contains the following members: + +* `otr.version` is a read-only string with the Recorder version (example: `"0.3.2"`) +* `otr.log(s)` is a function which takes a string `s` which is logged to syslog at the Recorder's facility and log level INFO. +* `otr.strftime(fmt, t)` is a function which takes a format string `fmt` (see `strftime(3)`) and an integer number of seconds `t` and returns a string with the formatted UTC time. If `t` is 0 or negative, the current system time is used. +* `otr.putdb(key, value)` is a function which takes two strings `k` and `v` and stores them in the named LMDB database called `luadb`. This can be viewed with +* `otr.getdb(key)` is a function which takes a single string `key` and returns the database value associated with that key or `nil` if the key isn't stored. + +``` +ocat --dump=luadb +``` + +Your Lua script *must* provide the following functions: + +## `otr_init` + +This is invoked at start of Recorder. If the function returns a non-zero value, Recorder unloads Lua and disables its processing; i.e. the `hook()` will *not* be invoked on location publishes. + +## `otr_exit` + +This is invoked when the Recorder stops, which it doesn't really do unless you CTRL-C it or send it a SIGTERM signal. + + +## `otr_hook` + +This function is invoked at every location publish processed by the Recorder. Your function is passed three arguments: + +1. _topic_ is the topic published to (e.g. `owntracks/jane/phone`) +2. _type_ is the type of MQTT message. This is the `_type` in our JSON messages (e.g. `location`, `cmd`, `transition`, ...) or `"unknown"`. +3. _location_ is a [Lua table](http://www.lua.org/pil/2.5.html) (associative array) with all the elements obtained in the JSON message. In the case of _type_ being `location`, we also add country code (`cc`) and the location's address (`addr`) unless reverse-geo lookups have been disabled in Recorder. + +Assume the following small example Lua script in `example.lua`: + +```lua +local file + +function otr_init() + otr.log("example.lua starting; writing to /tmp/lua.out") + file = io.open("/tmp/lua.out", "a") + file:write("written by OwnTracks Recorder version " .. otr.version .. "\n") +end + +function otr_hook(topic, _type, data) + local timestr = otr.strftime("It is %T in the year %Y", 0) + print("L: " .. topic .. " -> " .. _type) + file:write(timestr .. " " .. topic .. " lat=" .. data['lat'] .. data['addr'] .. "\n") +end + +function otr_exit() +end +``` + +When Recorder is launched with `--lua-script example.lua` it invokes `otr_init()` which opens a file. Then, for each location received, it calls `otr_hook()` which updates the file. + +Assuming an OwnTracks device publishes this payload + +```json +{"cog":-1,"batt":-1,"lon":2.29513,"acc":5,"vel":-1,"vac":-1,"lat":48.85833,"t":"u","tst":1441984413,"alt":0,"_type":"location","tid":"JJ"} +``` + +the file `/tmp/lua.out` would contain + +```txt +written by OwnTracks Recorder version 0.3.0 +It is 14:10:01 in the year 2015 owntracks/jane/phone lat=48.858339 Avenue Anatole France, 75007 Paris, France +``` + +## `otr_putrec` + +An optional function you provide is called `otr_putrec(u, d, s)`. If it exists, +it is called with the current user in `u`, the device in `d` and the payload +(which for OwnTracks apps is JSON but for, eg Greenwich devices might not be) in the string `s`. If your function returns a +non-zero value, the Recorder will *not* write the REC file for this publish. + +## `otr_httpobject` + +An optional function you provide is called `otr_httpobject(u, d, t, data)` where `u` is the username used by the client (`?u=`), `d` is the device name (`&d=` in the URI), `t` is the OwnTracks JSON `_type` and `data` a Lua table built from the OwnTracks JSON payload of `_type`. If it exists, this function is called whenever a POST is received in httpmode and the Recorder is gathering data to return to the client app. The function *must* return a Lua table containing any number of string, number, or boolean values which are converted to a JSON object and appended to the JSON array returned to the client. An [example](etc/example.lua) shows how, say, a transition event can be used to open the Featured content tab in the app. + +## Hooklets + +After running `otr_hook()`, the Recorder attempts to invoke a Lua function for each of the elements in the extended JSON. If, say, your Lua script contains a function called `hooklet_lat`, it will be invoked every time a `lat` is received as part of the JSON payload. Similarly with `hooklet_addr`, `hooklet_cc`, `hooklet_tst`, etc. These _hooklets_ are invoked with the same parameters as `otr_hook()`. + +You define a hooklet function only if you're interested in expressly triggering on a particular JSON element. diff --git a/README.md b/README.md index 743d0ff..3a88bda 100644 --- a/README.md +++ b/README.md @@ -48,12 +48,6 @@ We developed the Recorder as a one-stop solution to storing location data publis * [The geo cache](#the-geo-cache) * [Monitoring](#monitoring) * [Lua hooks](#lua-hooks) - * [`otr_init`](#otr_init) - * [`otr_exit`](#otr_exit) - * [`otr_hook`](#otr_hook) - * [`otr_putrec`](#otr_putrec) - * [`otr_httpobject`](#otr_httpobject) - * [Hooklets](#hooklets) * [Views](#views) * [view JSON](#view-json) * [Authentication](#authentication) @@ -655,6 +649,10 @@ u09ey1r {"cc":"FR","addr":"D83, 91590 La Ferté-Alais, France","tst":1445435679, The key to this data is the geohash string (here with an example of precision 2). +## Lua hooks + +You can customize Recorder's behavior with Lua hooks. See [HOOKS.md](https://github.com/owntracks/recorder/blob/master/HOOKS.md). + ## Monitoring In order to monitor the Recorder, whenever an MQTT message is received, a `monitor` file located relative to STORAGEDEFAULT is maintained. It contains a single line of text: the epoch timestamp and the last received topic separated from each other by a space. @@ -672,100 +670,6 @@ After sending a _pingping_, you can query the REST interface to determine the di OK ot-recorder pingping at http://127.0.0.1:8085: 0 seconds difference ``` -## Lua hooks - -If Recorder is compiled with Lua support, a Lua script you provide is launched at startup. Lua is _a powerful, fast, lightweight, embeddable scripting language_. You can use this to process location publishes in any way you desire: your imagination (and Lua-scripting knowhow) set the limits. Some examples: - -* insert publishes into a database of your choice -* switch on the coffee machine when your OwnTracks device reports you're entering home (but see also [mqttwarn](http://jpmens.net/2014/02/17/introducing-mqttwarn-a-pluggable-mqtt-notifier/)) -* write a file with data in a format of your choice (see `etc/example.lua`) - -Run the Recorder with the path to your Lua script specified in its `--lua-script` option (there is no default). If the script cannot be loaded (e.g. because it cannot be read or contains syntax errors), the Recorder unloads Lua and continues *without* your script. - -If the Lua script can be loaded, it is automatically provided with a table variable called `otr` which contains the following members: - -* `otr.version` is a read-only string with the Recorder version (example: `"0.3.2"`) -* `otr.log(s)` is a function which takes a string `s` which is logged to syslog at the Recorder's facility and log level INFO. -* `otr.strftime(fmt, t)` is a function which takes a format string `fmt` (see `strftime(3)`) and an integer number of seconds `t` and returns a string with the formatted UTC time. If `t` is 0 or negative, the current system time is used. -* `otr.putdb(key, value)` is a function which takes two strings `k` and `v` and stores them in the named LMDB database called `luadb`. This can be viewed with -* `otr.getdb(key)` is a function which takes a single string `key` and returns the database value associated with that key or `nil` if the key isn't stored. - -``` -ocat --dump=luadb -``` - -Your Lua script *must* provide the following functions: - -### `otr_init` - -This is invoked at start of Recorder. If the function returns a non-zero value, Recorder unloads Lua and disables its processing; i.e. the `hook()` will *not* be invoked on location publishes. - -### `otr_exit` - -This is invoked when the Recorder stops, which it doesn't really do unless you CTRL-C it or send it a SIGTERM signal. - - -### `otr_hook` - -This function is invoked at every location publish processed by the Recorder. Your function is passed three arguments: - -1. _topic_ is the topic published to (e.g. `owntracks/jane/phone`) -2. _type_ is the type of MQTT message. This is the `_type` in our JSON messages (e.g. `location`, `cmd`, `transition`, ...) or `"unknown"`. -3. _location_ is a [Lua table](http://www.lua.org/pil/2.5.html) (associative array) with all the elements obtained in the JSON message. In the case of _type_ being `location`, we also add country code (`cc`) and the location's address (`addr`) unless reverse-geo lookups have been disabled in Recorder. - -Assume the following small example Lua script in `example.lua`: - -```lua -local file - -function otr_init() - otr.log("example.lua starting; writing to /tmp/lua.out") - file = io.open("/tmp/lua.out", "a") - file:write("written by OwnTracks Recorder version " .. otr.version .. "\n") -end - -function otr_hook(topic, _type, data) - local timestr = otr.strftime("It is %T in the year %Y", 0) - print("L: " .. topic .. " -> " .. _type) - file:write(timestr .. " " .. topic .. " lat=" .. data['lat'] .. data['addr'] .. "\n") -end - -function otr_exit() -end -``` - -When Recorder is launched with `--lua-script example.lua` it invokes `otr_init()` which opens a file. Then, for each location received, it calls `otr_hook()` which updates the file. - -Assuming an OwnTracks device publishes this payload - -```json -{"cog":-1,"batt":-1,"lon":2.29513,"acc":5,"vel":-1,"vac":-1,"lat":48.85833,"t":"u","tst":1441984413,"alt":0,"_type":"location","tid":"JJ"} -``` - -the file `/tmp/lua.out` would contain - -```txt -written by OwnTracks Recorder version 0.3.0 -It is 14:10:01 in the year 2015 owntracks/jane/phone lat=48.858339 Avenue Anatole France, 75007 Paris, France -``` - -### `otr_putrec` - -An optional function you provide is called `otr_putrec(u, d, s)`. If it exists, -it is called with the current user in `u`, the device in `d` and the payload -(which for OwnTracks apps is JSON but for, eg Greenwich devices might not be) in the string `s`. If your function returns a -non-zero value, the Recorder will *not* write the REC file for this publish. - -### `otr_httpobject` - -An optional function you provide is called `otr_httpobject(u, d, t, data)` where `u` is the username used by the client (`?u=`), `d` is the device name (`&d=` in the URI), `t` is the OwnTracks JSON `_type` and `data` a Lua table built from the OwnTracks JSON payload of `_type`. If it exists, this function is called whenever a POST is received in httpmode and the Recorder is gathering data to return to the client app. The function *must* return a Lua table containing any number of string, number, or boolean values which are converted to a JSON object and appended to the JSON array returned to the client. An [example](etc/example.lua) shows how, say, a transition event can be used to open the Featured content tab in the app. - -### Hooklets - -After running `otr_hook()`, the Recorder attempts to invoke a Lua function for each of the elements in the extended JSON. If, say, your Lua script contains a function called `hooklet_lat`, it will be invoked every time a `lat` is received as part of the JSON payload. Similarly with `hooklet_addr`, `hooklet_cc`, `hooklet_tst`, etc. These _hooklets_ are invoked with the same parameters as `otr_hook()`. - -You define a hooklet function only if you're interested in expressly triggering on a particular JSON element. - ## Views A view is a sort of sandboxed look at data provided by the Recorder. Assume you host several devices, be they your own or those of some of your friends, and assume you want to allow somebody else to see where you are or have been during a specific time frame: with the Recorder's default Web server you cannot limit a visitor to see specific data only; once they reach the Recorder's Web interface, they have access to all your data. (We warned you about that earlier.) Using a HTTP proxy, you can provide an insight into certain portions of your data only. From 668722a37711d32807d8f85a9a9b0bc3a71564be Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Wed, 17 Aug 2016 10:18:35 -0400 Subject: [PATCH 15/23] Consolidate the reverse proxy docs some more --- README.md | 64 ++++++++++++++++++++++++++----------------------------- 1 file changed, 30 insertions(+), 34 deletions(-) diff --git a/README.md b/README.md index 3a88bda..c0cb5fc 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,6 @@ We developed the Recorder as a one-stop solution to storing location data publis * [Views](#views) * [view JSON](#view-json) * [Authentication](#authentication) - * [HTTP proxy](#http-proxy) * [HTTP mode](#http-mode) * [Friends in HTTP mode](#friends-in-http-mode) * [Authentication](#authentication-1) @@ -301,8 +300,8 @@ server { } # Proxy and upgrade WebSocket connection - location /otr/ws { - rewrite ^/otr/(.*) /$1 break; + location /owntracks/ws { + rewrite ^/owntracks/(.*) /$1 break; proxy_pass http://127.0.0.1:8083; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; @@ -311,30 +310,50 @@ server { proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } - location /otr/ { + location /owntracks/ { proxy_pass http://127.0.0.1:8083/; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; } + + # OwnTracks Recorder Views + location /owntracks/view/ { + proxy_buffering off; # Chrome + proxy_pass http://127.0.0.1:8085/view/; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Real-IP $remote_addr; + } + location /owntracks/static/ { + proxy_pass http://127.0.0.1:8085/static/; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Real-IP $remote_addr; + } + } ``` ### Apache -This will hand URIs which begin with `/otr/` to the Recorder. +This will hand URIs which begin with `/owntracks/` to the Recorder. ``` # WebSocket URL endpoint # a2enmod proxy_wstunnel -ProxyPass /otr/ws ws://127.0.0.1:8083/ws keepalive=on retry=60 -ProxyPassReverse /otr/ws ws://127.0.0.1:8083/ws keepalive=on +ProxyPass /owntracks/ws ws://127.0.0.1:8083/ws keepalive=on retry=60 +ProxyPassReverse /owntracks/ws ws://127.0.0.1:8083/ws keepalive=on # Static files -ProxyPass /otr http://127.0.0.1:8083/ -ProxyPassReverse /otr http://127.0.0.1:8083/ +ProxyPass /owntracks http://127.0.0.1:8083/ +ProxyPassReverse /owntracks http://127.0.0.1:8083/ + +# TODO: add views ``` ## The HTTP server @@ -698,6 +717,8 @@ Jane's friends can now visit the URL `/view/loire` (note the missing `.json` ext ![Jane's vacation](assets/view-map.png) +It's recommended that you configure your reverse proxy to show views. You can find an example of this for nginx in the [reverse proxy](#reverse-proxy) section. + ### view JSON The JSON in the view file (called `view.json` here) contains mandatory and optional elements: @@ -814,31 +835,6 @@ Re-enter password: ``` -### HTTP proxy - -We recommend you have the Recorder listening to a loopback interface (e.g. 127.0.0.1) as it does by default, and set up a reverse proxy to its views. Using _nginx_ the following configuration shows how we proxy the `view/` and the required `static/` URIs into the Recorder: - -``` -# OwnTracks Recorder Views -location /owntracks/view/ { - proxy_buffering off; # Chrome - proxy_pass http://127.0.0.1:8085/view/; - proxy_http_version 1.1; - proxy_set_header Host $host; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Real-IP $remote_addr; -} -location /owntracks/static/ { - proxy_pass http://127.0.0.1:8085/static/; - proxy_http_version 1.1; - proxy_set_header Host $host; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Real-IP $remote_addr; -} -``` - -You would then visit `http://example.com/owntracks/view/loire` to see the `loire` view, assuming `example.com` is your proxy. - ## HTTP mode If enabled at compile time (`WITH_HTTP`), the Recorder will accept OwnTracks-type JSON payloads via HTTP at the URL endpoint `/pub&u=username&d=device`. You specify the username with the `u` parameter and the device name with the `d` parameter. (Alternatively you can provide `X-Limit-U` and `X-Limit-D` as headers with the username and device name respectively.) If unspecified, the username defaults to `owntracks` and the device to `phone`. For example: From b6c45992152ffd5bdad8d2eb78367a096b3bf210 Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Wed, 17 Aug 2016 12:17:07 -0400 Subject: [PATCH 16/23] Move HTTP mode auth docs into "reverse proxy" --- README.md | 44 +++++++++++++++++--------------------------- 1 file changed, 17 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index c0cb5fc..db995f9 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,6 @@ We developed the Recorder as a one-stop solution to storing location data publis * [Authentication](#authentication) * [HTTP mode](#http-mode) * [Friends in HTTP mode](#friends-in-http-mode) - * [Authentication](#authentication-1) * [Advanced topics](#advanced-topics) * [Browser API keys](#browser-api-keys) * [The LMDB database](#the-lmdb-database) @@ -285,7 +284,7 @@ Note that options passed to `ot-recorder` override both configuration file setti ## Reverse proxy -Running the Recorder protected by an _nginx_ or _Apache_ server is possible and is the only recommended method if you want to server data behind _localhost_. The snippets below show how to do it, but you would also add authentication to them. +Running the Recorder protected by an _nginx_ or _Apache_ server is possible and is the only recommended method if you want to server data behind _localhost_. The snippets below show how to do it, but you would also add authentication to them - or at least, to everything but the views. The snippet for HTTP mode shows an example of how to do this. ### nginx @@ -335,6 +334,22 @@ server { proxy_set_header X-Real-IP $remote_addr; } + # HTTP Mode + location /owntracks/pub { + auth_basic "OwnTracks pub"; + auth_basic_user_file /usr/local/etc/nginx/owntracks.htpasswd; + proxy_pass http://127.0.0.1:8083/pub; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Real-IP $remote_addr; + + # Optionally force Recorder to use username from Basic + # authentication user. Whether or not client sets + # X-Limit-U and/or uses ?u= parameter, the user will + # be set to $remote_user. + proxy_set_header X-Limit-U $remote_user; + } } ``` @@ -878,31 +893,6 @@ when user `jane` and device `phone` POST a new location via HTTP, the Recorder w Note, that Jane's user/device tuple should also be returned in order to display Jane on the map or list of friends in the apps. -### Authentication - -In HTTP mode, the Recorder provides no form of authentication; anybody who "stumbles" over the correct endpoint will be able to post location data to your Recorder! You do not want this to happen. - -Install, say, an _nginx_ proxy before it and ensure it's configured for HTTP basic authentication: - -``` -# - Recorder PUB ----------------------------------------------------------- -location /owntracks/pub { - auth_basic "OwnTracks pub"; - auth_basic_user_file /usr/local/etc/nginx/owntracks.htpasswd; - proxy_pass http://127.0.0.1:8083/pub; - proxy_http_version 1.1; - proxy_set_header Host $host; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Real-IP $remote_addr; - - # Optionally force Recorder to use username from Basic - # authentication user. Whether or not client sets - # X-Limit-U and/or uses ?u= parameter, the user will - # be set to $remote_user. - proxy_set_header X-Limit-U $remote_user; -} -``` - ## Advanced topics ### Browser API keys From 57da88d9ca642d44d841bb90b1d629ed4793673c Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Wed, 17 Aug 2016 12:18:20 -0400 Subject: [PATCH 17/23] Normalize Nginx snippet whitespace --- README.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index db995f9..9f3a457 100644 --- a/README.md +++ b/README.md @@ -300,21 +300,21 @@ server { # Proxy and upgrade WebSocket connection location /owntracks/ws { - rewrite ^/owntracks/(.*) /$1 break; - proxy_pass http://127.0.0.1:8083; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection "upgrade"; - proxy_set_header Host $host; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + rewrite ^/owntracks/(.*) /$1 break; + proxy_pass http://127.0.0.1:8083; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } location /owntracks/ { - proxy_pass http://127.0.0.1:8083/; - proxy_http_version 1.1; - proxy_set_header Host $host; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Real-IP $remote_addr; + proxy_pass http://127.0.0.1:8083/; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Real-IP $remote_addr; } # OwnTracks Recorder Views From 4cf32905c3dc71b2c309380fde2666e9ca7dd7eb Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Wed, 17 Aug 2016 12:41:14 -0400 Subject: [PATCH 18/23] Fix excessive use of italics in README.md --- README.md | 72 +++++++++++++++++++++++++++---------------------------- 1 file changed, 36 insertions(+), 36 deletions(-) diff --git a/README.md b/README.md index 9f3a457..13a3125 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ The _OwnTracks Recorder_ is a lightweight program for storing and accessing loca ![Architecture of the Recorder](assets/ot-recorder.png) -There are two main components: the _Recorder_ obtains data via MQTT subscribes or HTTP POST, stores the data in plain files and serve it via its built-in REST API, and the _ocat_ command-line utility reads stored data in a variety of formats. +There are two main components: the _Recorder_ obtains data via MQTT subscribes or HTTP POST, stores the data in plain files and serve it via its built-in REST API, and the `ocat` command-line utility reads stored data in a variety of formats. We developed the Recorder as a one-stop solution to storing location data published by our OwnTracks apps (iOS and Android) and retrieving this data. Our previous offerings (`m2s`, `o2s`/`Pista`) also work of course, but we believe the Recorder is best suited to most environments. @@ -69,7 +69,7 @@ We developed the Recorder as a one-stop solution to storing location data publis The Recorder serves two purposes: 1. It subscribes to an MQTT broker and reads messages published from the OwnTracks apps, storing these in a particular fashion into what we call the _store_ which is basically a bunch of plain files on the file system. Alternatively the Recorder can listen on HTTP for OwnTracks-type JSON messages POSTed to its HTTP server. -2. It provides a Web server which serves static pages, a REST API you use to request data from the _store_, and a WebSocket server. The distribution comes with a few examples of how to access the data through its HTTP interface (REST API). In particular a _table_ of last locations has been made available as well as a _live map_ which updates via the Recorder's WebSocket interface when location publishes are received. In addition we provide maps with last points or tracks using the GeoJSON produced by the Recorder. +2. It provides a Web server which serves static pages, a REST API you use to request data from the store, and a WebSocket server. The distribution comes with a few examples of how to access the data through its HTTP interface (REST API). In particular a table of last locations has been made available as well as a live map which updates via the Recorder's WebSocket interface when location publishes are received. In addition we provide maps with last points or tracks using the GeoJSON produced by the Recorder. ## Installing @@ -167,10 +167,10 @@ sudo apt-get install libmosquitto-dev libcurl3 libcurl4-openssl-dev libconfig-de #### Building 1. Obtain and download the software, via [our Homebrew Tap](https://github.com/owntracks/homebrew-recorder) on Mac OS X, directly as a clone of the repository, or as a [tar ball](https://github.com/owntracks/recorder/releases) which you unpack. -2. Copy the included `config.mk.in` file to `config.mk` and edit that. You specify the features or tweaks you need. (The file is commented.) Pay particular attention to the installation directory and the value of the _store_ (`STORAGEDEFAULT`): that is where the Recorder will store its files. `DOCROOT` is the root of the directory from which the Recorder's HTTP server will serve files. +2. Copy the included `config.mk.in` file to `config.mk` and edit that. You specify the features or tweaks you need. (The file is commented.) Pay particular attention to the installation directory and the value of the store (`STORAGEDEFAULT`): that is where the Recorder will store its files. `DOCROOT` is the root of the directory from which the Recorder's HTTP server will serve files. 3. Type `make` and watch the fun. -When _make_ finishes, you should have at least two executable programs called `ot-recorder` which is the Recorder proper, and `ocat`. If you want you can install these using `make install`, but this is not necessary: the programs will run from whichever directory you like if you add `--doc-root ./docroot` to the Recorder options. +When `make` finishes, you should have at least two executable programs called `ot-recorder` which is the Recorder proper, and `ocat`. If you want you can install these using `make install`, but this is not necessary: the programs will run from whichever directory you like if you add `--doc-root ./docroot` to the Recorder options. Ensure the LMDB databases are initialized by running the following command which is safe to do, also after an upgrade. (This initialization is non-destructive -- it will not delete any data.) @@ -180,7 +180,7 @@ ot-recorder --initialize ## Getting started -The Recorder has, like _ocat_, a daunting number of options, most of which you will not require. Running either utility with the `-h` or `--help` switch will summarize their meanings. You can, for example launch with a specific storage directory, disable the HTTP server, change its port, etc. +The Recorder has, like `ocat`, a daunting number of options, most of which you will not require. Running either utility with the `-h` or `--help` switch will summarize their meanings. You can, for example launch with a specific storage directory, disable the HTTP server, change its port, etc. If you require authentication or TLS to connect to your MQTT broker, pay attention to the `$OTR_` environment variables listed in the help. @@ -194,7 +194,7 @@ $ ./ot-recorder 'owntracks/#' Publish a location from your OwnTracks app and you should see the Recorder receive that on the console. If you haven't disabled Geo-lookups, you'll also see the address from which the publish originated. -The location message received by the Recorder will be written to storage. In particular you should verify that your _storage_ directory contains: +The location message received by the Recorder will be written to storage. In particular you should verify that your storage directory contains: 1. a directory called `ghash/` 2. a directory called `rec/` with several subdirectories and a `.rec` file therein. @@ -229,11 +229,11 @@ This section lists the most important options of the Recorder with their long na `--norec` disables writing of REC files, so no location history or other similar publishes are stored, and the Lua `otr_putrec()` function is not invoked even if it exists. What is stored are CARDS and PHOTOS, as well as the LAST location of a device. As such, the API's `/locations` endpoint becomes useless. -`--norevgeo` suppresses reverse geo lookups, but this means that historic data will not show addresses (e.g. with the API or with _ocat_). See below for information on Reverse Geo lookups. +`--norevgeo` suppresses reverse geo lookups, but this means that historic data will not show addresses (e.g. with the API or with `ocat`). See below for information on Reverse Geo lookups. `--logfacility` is the syslog facility to use (default is `LOCAL0`). -`--quiet` disables printing of messages to _stdout_. +`--quiet` disables printing of messages to stdout. `--initialize` creates the a structure within the storage directory and initializes the LMDB database. It is safe to use this even if such a database exists -- the database is not wiped. After initialization, Recorder exits. @@ -253,7 +253,7 @@ This section lists the most important options of the Recorder with their long na ## Configuration file -The Recorder attempts to read its startup configuration from a configuration file; the path to this is compiled into the Recorder (typically `/etc/defaults/ot-recorder`, and `ocat -v` will display the compiled-in default). The format of this file approximates that of a shell script with variables to be exported (the intention is so that it can be sourced by a shell script). Lines beginning with an octothorp (`#`) are ignored as are blank lines. Configuration settings proper are set as follows (note that some older versions of _libconfig_ require a trailing semicolon (`;`) at the end of a variable assignment): +The Recorder attempts to read its startup configuration from a configuration file; the path to this is compiled into the Recorder (typically `/etc/defaults/ot-recorder`, and `ocat -v` will display the compiled-in default). The format of this file approximates that of a shell script with variables to be exported (the intention is so that it can be sourced by a shell script). Lines beginning with an octothorp (`#`) are ignored as are blank lines. Configuration settings proper are set as follows (note that some older versions of libconfig require a trailing semicolon (`;`) at the end of a variable assignment): ``` OTR_STORAGEDIR="/var/spool/owntracks/recorder/store" @@ -284,7 +284,7 @@ Note that options passed to `ot-recorder` override both configuration file setti ## Reverse proxy -Running the Recorder protected by an _nginx_ or _Apache_ server is possible and is the only recommended method if you want to server data behind _localhost_. The snippets below show how to do it, but you would also add authentication to them - or at least, to everything but the views. The snippet for HTTP mode shows an example of how to do this. +Running the Recorder protected by an Nginx or Apache server is possible and is the only recommended method if you want to serve data behind localhost. The snippets below show how to do it, but you would also add authentication to them - or at least, to everything but the views. The snippet for HTTP mode shows an example of how to do this. ### nginx @@ -375,7 +375,7 @@ ProxyPassReverse /owntracks http://127.0.0.1:8083/ The Recorder has a built-in HTTP server with which it servers static files from either the compiled-in default `DOCROOT` directory or that specified at run-time with the `--doc-root` option. Furthermore, it serves JSON data from the API end-point at `/api/0/` and it has a built-in WebSocket server for the live map. -The API basically serves the same data as _ocat_ is able to produce - see [API.md](https://github.com/owntracks/recorder/blob/master/API.md). The server also accepts OwnTracks app data via HTTP POST to the `/pub` endpoint. +The API basically serves the same data as `ocat` is able to produce - see [API.md](https://github.com/owntracks/recorder/blob/master/API.md). The server also accepts OwnTracks app data via HTTP POST to the `/pub` endpoint. ### Example functionality @@ -453,7 +453,7 @@ The Recorder's built-in WebSocket server updates a map as it receives publishes ## `ocat` -_ocat_ is a CLI query program for data stored by Recorder: it prints data from storage in a variety of output formats: +`ocat` is a CLI query program for data stored by Recorder: it prints data from storage in a variety of output formats: * JSON * GeoJSON (points) @@ -464,12 +464,12 @@ _ocat_ is a CLI query program for data stored by Recorder: it prints data from s * raw (the lines contained in the REC file with ISO timestamp) * payload (basically just the payload part from RAW) -The _ocat_ utility accesses _storage_ directly — it doesn’t use the Recorder’s REST interface. _ocat_ has a daunting number of options, some combinations of which make no sense at all. +The `ocat` utility accesses the store directly — it doesn’t use the Recorder’s REST interface. `ocat` has a daunting number of options, some combinations of which make no sense at all. Some example uses we consider useful: * `ocat --list` - show which uers are in _storage_. + show which uers are in the store. * `ocat --list --user jjolie` show devices for the specified user * `ocat --user jjolie --device ipad` @@ -510,9 +510,9 @@ Some example uses we consider useful: * `ocat ... --limit 10` prints data for the current month, starting now and going backwards; only 10 locations will be printed. Generally, the `--limit` option reads the storage back to front which makes no sense in some combinations. -Specifying `--fields lat,tid,lon` will request just those JSON elements from _storage_. (Note that doing so with output GPX or GEOJSON could render those formats useless if, say, `lat` is missing in the list of fields.) +Specifying `--fields lat,tid,lon` will request just those JSON elements from the store. (Note that doing so with output GPX or GEOJSON could render those formats useless if, say, `lat` is missing in the list of fields.) -The `--from` and `--to` options allow you to specify a UTC date and/or timestamp from which respectively until which data will be read. By default, the last 6 hours of data are produced. If `--from` is not specified, it therefore defaults to _now minus 6 hours_. If `--to` is not specified it defaults to _now_. Dates and times must be specified as strings, and the following formats are recognized: +The `--from` and `--to` options allow you to specify a UTC date and/or timestamp from which respectively until which data will be read. By default, the last 6 hours of data are produced. If `--from` is not specified, it therefore defaults to "now minus 6 hours". If `--to` is not specified it defaults to "now". Dates and times must be specified as strings, and the following formats are recognized: ``` %Y-%m-%dT%H:%M:%S @@ -526,7 +526,7 @@ The `--limit` option limits the output to the last specified number of records. ### Environment -The following environment variables control _ocat_'s behaviour: +The following environment variables control `ocat`'s behaviour: * `OCAT_FORMAT` can be set to the preferred output format. If unset, JSON is used. The `--format` option overrides this setting. * `OCAT_USERNAME` can be set to the preferred username. The `--user` option overrides this environment variable. @@ -538,7 +538,7 @@ The Recorder has been running for a while, and the OwnTracks apps have published #### List users and devices -We obtain a list of users from the _store_: +We obtain a list of users from the store: ``` $ ocat --list @@ -549,7 +549,7 @@ $ ocat --list } ``` -From which devices has user _demo_ published data? +From which devices has user `demo` published data? ``` $ ocat --list --user demo @@ -562,7 +562,7 @@ $ ocat --list --user demo #### Show the last position reported by a user -Where was _demo_'s _iphone_ last seen? (Omit `--user` and `--device` to get LAST for all users and devices.) +Where was `demo`'s iPhone last seen? (Omit `--user` and `--device` to get LAST for all users and devices.) ``` $ ocat --last --user demo --device iphone @@ -594,7 +594,7 @@ $ ocat --last --user demo --device iphone Several things worth mentioning: * The returned data structure is an array of JSON objects; had we omitted specifying a particular device or even a particular user we would have obtained the last position of all this user's devices or all users' devices respectively. -* If you are familiar with the [JSON data reported by the OwnTracks apps](http://owntracks.org/booklet/tech/json/) you'll notice that this JSON contains more information: this is provided on the fly by _ocat_ and the REST API, e.g. from the reverse-geo cache the Recorder maintains. +* If you are familiar with the [JSON data reported by the OwnTracks apps](http://owntracks.org/booklet/tech/json/) you'll notice that this JSON contains more information: this is provided on the fly by `ocat` and the REST API, e.g. from the reverse-geo cache the Recorder maintains. #### What were the last 4 positions reported? @@ -619,20 +619,20 @@ We took a number of decisions when designing the Recorder and its utilities: * File names are lower case. A user called `JaNe` with a device named `myPHONe` will be found in a file named `jane/myphone`. * All times are UTC (a.k.a. Zulu or GMT). We got sick and tired of converting stuff back and forth. It is up to the consumer of the data to convert to localtime if need be. * The Recorder does not provide authentication or authorization. Nothing at all. Zilch. Nada. Think about this before making it available on a publicly-accessible IP address. Or rather: don't think about it; just don't do it. You can of course place a HTTP proxy in front of the Recorder to control access to it. Or use views (see below). -* `ocat`, the _cat_ program for the Recorder uses the same back-end which is used by the API though it accesses it directly (i.e. without resorting to HTTP). +* `ocat`, the `cat` program for the Recorder, uses the same back-end which is used by the API though it accesses it directly (i.e. without resorting to HTTP). * The Recorder supports 3-level MQTT topics only, in the typical OwnTracks format: `"owntracks//"`, optionally with a leading slash. (The first part of the topic need not be "owntracks".) Publishes via HTTP POST construct a ficticious topic internally using the provided user (`u`) and device (`d`) parameters. ## Storage As mentioned earlier, data is stored in files, and these files are relative to `STORAGEDIR` (compiled into the programs or specified as an option). In particular, the following directory structure can exist, whereby directories are created as needed by the Recorder: -* `cards/`, optional, may contains user cards. This card is then stored here and used with, e.g., `ocat --last` to show a user's name and optional avatar. User cards are typically stored in a subdirectory called _username_, and therein a JSON file _username_.json. When reading cards, the Recorder will first attempt to open _username_/_device_/_username_.json and then _username_/_username_.json. +* `cards/`, optional, may contains user cards. This card is then stored here and used with, e.g., `ocat --last` to show a user's name and optional avatar. User cards are typically stored in a subdirectory called `username`, and therein a JSON file `[username].json`. When reading cards, the Recorder will first attempt to open `[username]/[device]/[username].json` and then `[username]/[username].json`. * `config/`, optional, contains the JSON of a [device configuration](http://owntracks.org/booklet/features/remoteconfig/) (`.otrc`) which was requested remotely via a [dump command](http://owntracks.org/booklet/tech/json/#_typecmd). Note that this will contain sensitive data. You can use this `.otrc` file to restore the OwnTracks configuration on your device by copying to the device and opening it in OwnTracks. * `ghash/`, unless disabled, reverse Geo data (using a Google service) is collected into an LMDB database located in this directory. This LMDB database also contains named databases which are used by your optional Lua hooks, as well as a `topic2tid` database which can be used for TID re-mapping. * `last/` contains the last location published by devices. E.g. Jane's last publish from her iPhone would be in `last/jjolie/iphone/jjolie-iphone.json`. The JSON payload contained therein is enhanced with the fields `user`, `device`, `topic`, and `ghash`. If a device's `last/` directory contains a file called `extra.json` (i.e. matching the example, this would be `last/jjolie/iphone/extra.json`), the content of this file is merged into the existing JSON for this user and returned by the API. Note, that you cannot overwrite existing values. So, an `extra.json` containing `{ "tst" : 11 }` will do nothing because the `tst` element we obtain from location data overrules, but adding `{ "beverage" : "water" }` will do what you want. If Recorder is built with support for our Greenwich firmware, this directory might contain `batt.json`, `ext.json`, and/or `status.json` each of which hold an array of the last 100 reports for internal battery voltage, external voltage, and status respectively. These values are returned via the API in the LAST object. A file `http.json` which should contain either a single JSON object or an array of JSON objects is returned to clients in HTTPmode. * `monitor` a file which contains a timestamp and the last received topic (see Monitoring below). * `msg/` contains messages received by the Messaging system. -* `photos/` optional; contains the binary photos from a _card_. +* `photos/` optional; contains the binary photos from a card. * `rec/` the Recorder data proper. One subdirectory per user, one subdirectory therein per device. Data files are named `YYYY-MM.rec` (e.g. `2015-08.rec` for the data accumulated during the month of August 2015. * `waypoints/` contains a directory per user and device. Therein are individual files named by a timestamp with the JSON payload of published (i.e. shared) waypoints. The file names are timestamps because the `tst` of a waypoint is its key. If a user publishes all waypoints from a device (Publish Waypoints), the payload is stored in this directory as `username-device.otrw`. (Note, that this is the JSON [waypoints import format](http://owntracks.org/booklet/tech/json/#_typewaypoints).) You can use this `.otrw` file to restore the waypoints on your device by copying to the device and opening it in OwnTracks. @@ -651,11 +651,11 @@ u0m97hc 46.652733 7.868803 This can be used to subsequently obtain missed lookups. -We recommend you keep reverse-geo lookups enabled, this data (country code `cc`, and the locations address `addr`) is used by the example Web apps provided by the Recorder to show where a particular device is. In addition, this cached data is used the the API (also _ocat_) when printing location data. +We recommend you keep reverse-geo lookups enabled, this data (country code `cc`, and the locations address `addr`) is used by the example Web apps provided by the Recorder to show where a particular device is. In addition, this cached data is used the the API (also `ocat`) when printing location data. ### Precision -The precision with which reverse-geo lookups are performed is controlled with the `--precison` option to Recorder (and with the `--precision` option to _ocat_ when you query for data). The default precision is compiled into the code (from `config.mk`). The higher the number, the more frequently lookups are performed; conversely, the lower the number, the fewer lookups are performed. For example, a precision of 1 means that points within an area of approximately 5000 km^2 would resolve to a single address, whereas a precision of 7 means that points within an area of approximately 150 m^2 resolve to one address. The Recorder obtains a location publish, extracts the latitude and longitude, and then calculates the [geohash](https://en.wikipedia.org/wiki/Geohash) string and truncates it to _precision_. If the calculated geohash string can be found in our local LMDB cache, we consider the point cached; otherwise an actual reverse geo lookup (via HTTP) is performed and the result is cached in LMDB at the key of the geohash. +The precision with which reverse-geo lookups are performed is controlled with the `--precison` option to Recorder (and with the `--precision` option to `ocat` when you query for data). The default precision is compiled into the code (from `config.mk`). The higher the number, the more frequently lookups are performed; conversely, the lower the number, the fewer lookups are performed. For example, a precision of 1 means that points within an area of approximately 5000 km^2 would resolve to a single address, whereas a precision of 7 means that points within an area of approximately 150 m^2 resolve to one address. The Recorder obtains a location publish, extracts the latitude and longitude, and then calculates the [geohash](https://en.wikipedia.org/wiki/Geohash) string and truncates it to `precision`. If the calculated geohash string can be found in our local LMDB cache, we consider the point cached; otherwise an actual reverse geo lookup (via HTTP) is performed and the result is cached in LMDB at the key of the geohash. As an example, let's assume Jane's device is at position (lat, lon) `48.879840, 2.323522`, which resolves to a geohash string of length 7 `u09whf7`. We can [visualize this](http://www.movable-type.co.uk/scripts/geohash.html) and show what this looks like. (See also: [visualizing geohash](http://www.bigdatamodeling.org/2013/01/intuitive-geohash.html).) @@ -698,7 +698,7 @@ In order to monitor the Recorder, whenever an MQTT message is received, a `monit If Recorder is built with `WITH_PING` (default), a location publish to `owntracks/ping/ping` (i.e. username is `ping` and device is `ping`) can be used to round-trip-test the Recorder. For this particular username/device combination, Recorder will store LAST position, but it will not keep a `.REC` file for it. This can be used to verify, say, via your favorite monitoring system, that the Recorder is still operational. -After sending a _pingping_, you can query the REST interface to determine the difference in time. The `contrib/` directory has an example Python program (`ot-ping.py`) which you can adapt as needed for use by Icinga or Nagios. +After sending a pingping, you can query the REST interface to determine the difference in time. The `contrib/` directory has an example Python program (`ot-ping.py`) which you can adapt as needed for use by Icinga or Nagios. ``` OK ot-recorder pingping at http://127.0.0.1:8085: 0 seconds difference @@ -708,7 +708,7 @@ OK ot-recorder pingping at http://127.0.0.1:8085: 0 seconds difference A view is a sort of sandboxed look at data provided by the Recorder. Assume you host several devices, be they your own or those of some of your friends, and assume you want to allow somebody else to see where you are or have been during a specific time frame: with the Recorder's default Web server you cannot limit a visitor to see specific data only; once they reach the Recorder's Web interface, they have access to all your data. (We warned you about that earlier.) Using a HTTP proxy, you can provide an insight into certain portions of your data only. -You configure a view by creating a small JSON file of an arbitrary name which defines which user / device combination of data the view should display. Say you are recording data for `owntracks/jjolie/phone`, the _user_ would be `jjolie` and the _device_ is `phone`. You can also create a specific HTML page for this view or just use the default `vmap.html` we provide. +You configure a view by creating a small JSON file of an arbitrary name which defines which user / device combination of data the view should display. Say you are recording data for `owntracks/jjolie/phone`, the user would be `jjolie` and the device is `phone`. You can also create a specific HTML page for this view or just use the default `vmap.html` we provide. The view then provides three URLs: @@ -728,7 +728,7 @@ Suppose Jane wishes to have her acqaintances see where she is whilst on vacation } ``` -Jane's friends can now visit the URL `/view/loire` (note the missing `.json` extension) to be served a map showing Jane's progress along the Loire valley (if that is where she's actually travelling through). Jane can keep that view up even after she returns because the view will not serve data after the 15th of July, in other words, her location at any other time before or after the _from_ / _to_ dates is hidden. +Jane's friends can now visit the URL `/view/loire` (note the missing `.json` extension) to be served a map showing Jane's progress along the Loire valley (if that is where she's actually travelling through). Jane can keep that view up even after she returns because the view will not serve data after the 15th of July, in other words, her location at any other time before or after the from/to dates is hidden. ![Jane's vacation](assets/view-map.png) @@ -753,12 +753,12 @@ The JSON in the view file (called `view.json` here) contains mandatory and optio -The _page_ is a single HTML file which must be located in the `views/` directory of the Recorder's document root. Trivial (primitive actually) text substitution is done for the following two tokens: +The `page` is a single HTML file which must be located in the `views/` directory of the Recorder's document root. Trivial (primitive actually) text substitution is done for the following two tokens: * `@@@LASTPOS@@@` is converted to a URI on which the Recorder will serve the last position data * `@@@GEO@@@` is converted to a URI on which the Recorder will serve GeoJSON data from its storage. -The default _page_ we provide is called `vmap.html`; by default it refreshes the last position every 60 seconds, and clicking on _Load track_ loads the GeoJSON track for the time frame specified by `from` and `to`. +The default `page` we provide is called `vmap.html`; by default it refreshes the last position every 60 seconds, and clicking on "Load track" loads the GeoJSON track for the time frame specified by `from` and `to`. ![Jane's vacation track](assets/view-track.png) @@ -780,7 +780,7 @@ A little bit more complex view would look like this: } ``` -All JSON elements are copied into the _lastpos_ data which is returned to the caller. Using the above view configuration, a user requesting `http://localhost:8083/view/loire?lastpos=1` would obtain +All JSON elements are copied into the `lastpos` data which is returned to the caller. Using the above view configuration, a user requesting `http://localhost:8083/view/loire?lastpos=1` would obtain ```json { @@ -816,7 +816,7 @@ All JSON elements are copied into the _lastpos_ data which is returned to the ca } ``` -Note how `pathname` and `port` have been copied into the object. These values can be used by the _page_ served in the view. +Note how `pathname` and `port` have been copied into the object. These values can be used by the `page` served in the view. ### Authentication @@ -828,7 +828,7 @@ If `view.json` contains an element called `auth`, it is assumed to be an array o Each user/password digest combination will be able to access the view. -You create these strings with, say, the _htdigest_ program or `contrib/new-view-auth.py`: +You create these strings with, say, the `htdigest` program or `contrib/new-view-auth.py`: ```bash htdigest -c /tmp/dd owntracks-recorder jjolie @@ -897,7 +897,7 @@ Note, that Jane's user/device tuple should also be returned in order to display ### Browser API keys -In order to use the Recorder's maps and views, you have to obtain a [Google API "Browser key"](https://developers.google.com/maps/documentation/javascript/get-api-key). You then add this key to your _docroot_ directory so that the Recorder can find it. (You should already have a file called `apikey.js.sample` in that directory; copy or rename the file to `apikey.js` and ensure its content is valid JavaScript: +In order to use the Recorder's maps and views, you have to obtain a [Google API "Browser key"](https://developers.google.com/maps/documentation/javascript/get-api-key). You then add this key to your docroot directory so that the Recorder can find it. (You should already have a file called `apikey.js.sample` in that directory; copy or rename the file to `apikey.js` and ensure its content is valid JavaScript: ```bash $ cat .../static/apikey.js @@ -907,7 +907,7 @@ var apiKey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"; ### The LMDB database -`ocat --load` and `ocat --dump` can be use to load and dump the lmdb database respectively. There is some support for loading/dumping named databases using `--load=xx` or `--dump=xx` to specify the name. Use the mdb utilities to actually perform backups of these. _load_ expects key/value strings in pairs, separated by exactly one space. If the value is the string `DELETE`, the key is deleted from the database, which allows us to, say, remove a whole bunch of geohash prefixes in one go (but be careful doing this): +`ocat --load` and `ocat --dump` can be use to load and dump the lmdb database respectively. There is some support for loading/dumping named databases using `--load=xx` or `--dump=xx` to specify the name. Use the mdb utilities to actually perform backups of these. `--load` expects key/value strings in pairs, separated by exactly one space. If the value is the string `DELETE`, the key is deleted from the database, which allows us to, say, remove a whole bunch of geohash prefixes in one go (but be careful doing this): ```bash ocat --dump | From b505ea1477cb8b08251c37ce80e84ea6e0f30809 Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Wed, 17 Aug 2016 13:02:18 -0400 Subject: [PATCH 19/23] Fix spelling errors --- README.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 13a3125..13bff19 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ We developed the Recorder as a one-stop solution to storing location data publis * [Design decisions](#design-decisions) * [Storage](#storage) * [Reverse Geo](#reverse-geo) - * [Precision](#precisioin) + * [Precision](#precision) * [The geo cache](#the-geo-cache) * [Monitoring](#monitoring) * [Lua hooks](#lua-hooks) @@ -190,7 +190,7 @@ Launch the Recorder: $ ./ot-recorder 'owntracks/#' ``` -(In httpmode, you do not have to specify a topic.) +(In HTTP mode, you do not have to specify a topic.) Publish a location from your OwnTracks app and you should see the Recorder receive that on the console. If you haven't disabled Geo-lookups, you'll also see the address from which the publish originated. @@ -276,7 +276,7 @@ The following configuration settings may be applied (a `Y` in column `$` means a | `OTR_LUASCRIPT` | | | Path to the Lua script | `OTR_PRECISION` | | `7` | Reverse-geo precision | `OTR_GEOKEY` | | | API key for reverse-geo lookups -| `OTR_TOPICS` | | | String containing a space-separated list of topics to subscribe to for MQTT (overriden by command-line arguments) +| `OTR_TOPICS` | | | String containing a space-separated list of topics to subscribe to for MQTT (overridden by command-line arguments) | `OTR_CAFILE` | Y | | Path to PEM-encoded CA certificate file for MQTT (implicitly enables TLS) @@ -464,12 +464,12 @@ The Recorder's built-in WebSocket server updates a map as it receives publishes * raw (the lines contained in the REC file with ISO timestamp) * payload (basically just the payload part from RAW) -The `ocat` utility accesses the store directly — it doesn’t use the Recorder’s REST interface. `ocat` has a daunting number of options, some combinations of which make no sense at all. +The `ocat` utility accesses the store directly — it doesn't use the Recorder’s REST interface. `ocat` has a daunting number of options, some combinations of which make no sense at all. Some example uses we consider useful: * `ocat --list` - show which uers are in the store. + show which users are in the store. * `ocat --list --user jjolie` show devices for the specified user * `ocat --user jjolie --device ipad` @@ -526,7 +526,7 @@ The `--limit` option limits the output to the last specified number of records. ### Environment -The following environment variables control `ocat`'s behaviour: +The following environment variables control `ocat`'s behavior: * `OCAT_FORMAT` can be set to the preferred output format. If unset, JSON is used. The `--format` option overrides this setting. * `OCAT_USERNAME` can be set to the preferred username. The `--user` option overrides this environment variable. @@ -617,10 +617,10 @@ We took a number of decisions when designing the Recorder and its utilities: * Flat files. The filesystem is the database. Period. That's were everything is stored. It makes incremental backups, purging old data, manipulation via the Unix toolset easy. (Admittedly, for fast geo-lookups we employ LMDB as a cache, but the final word is in the filesystem.) We considered all manner of databases and decided to keep this as simple and lightweight as possible. You can however have the Recorder send data to a database of your choosing, in addition to the file system it uses, by utilizing our embedded Lua hook. * We wanted to store received data in the format it's published in. As this format is JSON, we store this raw payload in the `.rec` files. If we add an attribute to the JSON published by our apps, you have it right there. There's one slight exception: the monthly logs (the `.rec` files) have a leading timestamp and a relative topic; see below. (In the particular case of the OwnTracks firmware for Greenwich devices which can publish in CSV mode, we convert the CSV into OwnTracks JSON for storage.) * File names are lower case. A user called `JaNe` with a device named `myPHONe` will be found in a file named `jane/myphone`. -* All times are UTC (a.k.a. Zulu or GMT). We got sick and tired of converting stuff back and forth. It is up to the consumer of the data to convert to localtime if need be. +* All times are UTC (a.k.a. Zulu or GMT). We got sick and tired of converting stuff back and forth. It is up to the consumer of the data to convert to local time if need be. * The Recorder does not provide authentication or authorization. Nothing at all. Zilch. Nada. Think about this before making it available on a publicly-accessible IP address. Or rather: don't think about it; just don't do it. You can of course place a HTTP proxy in front of the Recorder to control access to it. Or use views (see below). * `ocat`, the `cat` program for the Recorder, uses the same back-end which is used by the API though it accesses it directly (i.e. without resorting to HTTP). -* The Recorder supports 3-level MQTT topics only, in the typical OwnTracks format: `"owntracks//"`, optionally with a leading slash. (The first part of the topic need not be "owntracks".) Publishes via HTTP POST construct a ficticious topic internally using the provided user (`u`) and device (`d`) parameters. +* The Recorder supports 3-level MQTT topics only, in the typical OwnTracks format: `"owntracks//"`, optionally with a leading slash. (The first part of the topic need not be "owntracks".) Publishes via HTTP POST construct a fictitious topic internally using the provided user (`u`) and device (`d`) parameters. ## Storage @@ -629,7 +629,7 @@ As mentioned earlier, data is stored in files, and these files are relative to ` * `cards/`, optional, may contains user cards. This card is then stored here and used with, e.g., `ocat --last` to show a user's name and optional avatar. User cards are typically stored in a subdirectory called `username`, and therein a JSON file `[username].json`. When reading cards, the Recorder will first attempt to open `[username]/[device]/[username].json` and then `[username]/[username].json`. * `config/`, optional, contains the JSON of a [device configuration](http://owntracks.org/booklet/features/remoteconfig/) (`.otrc`) which was requested remotely via a [dump command](http://owntracks.org/booklet/tech/json/#_typecmd). Note that this will contain sensitive data. You can use this `.otrc` file to restore the OwnTracks configuration on your device by copying to the device and opening it in OwnTracks. * `ghash/`, unless disabled, reverse Geo data (using a Google service) is collected into an LMDB database located in this directory. This LMDB database also contains named databases which are used by your optional Lua hooks, as well as a `topic2tid` database which can be used for TID re-mapping. -* `last/` contains the last location published by devices. E.g. Jane's last publish from her iPhone would be in `last/jjolie/iphone/jjolie-iphone.json`. The JSON payload contained therein is enhanced with the fields `user`, `device`, `topic`, and `ghash`. If a device's `last/` directory contains a file called `extra.json` (i.e. matching the example, this would be `last/jjolie/iphone/extra.json`), the content of this file is merged into the existing JSON for this user and returned by the API. Note, that you cannot overwrite existing values. So, an `extra.json` containing `{ "tst" : 11 }` will do nothing because the `tst` element we obtain from location data overrules, but adding `{ "beverage" : "water" }` will do what you want. If Recorder is built with support for our Greenwich firmware, this directory might contain `batt.json`, `ext.json`, and/or `status.json` each of which hold an array of the last 100 reports for internal battery voltage, external voltage, and status respectively. These values are returned via the API in the LAST object. A file `http.json` which should contain either a single JSON object or an array of JSON objects is returned to clients in HTTPmode. +* `last/` contains the last location published by devices. E.g. Jane's last publish from her iPhone would be in `last/jjolie/iphone/jjolie-iphone.json`. The JSON payload contained therein is enhanced with the fields `user`, `device`, `topic`, and `ghash`. If a device's `last/` directory contains a file called `extra.json` (i.e. matching the example, this would be `last/jjolie/iphone/extra.json`), the content of this file is merged into the existing JSON for this user and returned by the API. Note, that you cannot overwrite existing values. So, an `extra.json` containing `{ "tst" : 11 }` will do nothing because the `tst` element we obtain from location data overrules, but adding `{ "beverage" : "water" }` will do what you want. If Recorder is built with support for our Greenwich firmware, this directory might contain `batt.json`, `ext.json`, and/or `status.json` each of which hold an array of the last 100 reports for internal battery voltage, external voltage, and status respectively. These values are returned via the API in the LAST object. A file `http.json` which should contain either a single JSON object or an array of JSON objects is returned to clients in HTTP mode. * `monitor` a file which contains a timestamp and the last received topic (see Monitoring below). * `msg/` contains messages received by the Messaging system. * `photos/` optional; contains the binary photos from a card. @@ -655,7 +655,7 @@ We recommend you keep reverse-geo lookups enabled, this data (country code `cc`, ### Precision -The precision with which reverse-geo lookups are performed is controlled with the `--precison` option to Recorder (and with the `--precision` option to `ocat` when you query for data). The default precision is compiled into the code (from `config.mk`). The higher the number, the more frequently lookups are performed; conversely, the lower the number, the fewer lookups are performed. For example, a precision of 1 means that points within an area of approximately 5000 km^2 would resolve to a single address, whereas a precision of 7 means that points within an area of approximately 150 m^2 resolve to one address. The Recorder obtains a location publish, extracts the latitude and longitude, and then calculates the [geohash](https://en.wikipedia.org/wiki/Geohash) string and truncates it to `precision`. If the calculated geohash string can be found in our local LMDB cache, we consider the point cached; otherwise an actual reverse geo lookup (via HTTP) is performed and the result is cached in LMDB at the key of the geohash. +The precision with which reverse-geo lookups are performed is controlled with the `--precision` option to Recorder (and with the `--precision` option to `ocat` when you query for data). The default precision is compiled into the code (from `config.mk`). The higher the number, the more frequently lookups are performed; conversely, the lower the number, the fewer lookups are performed. For example, a precision of 1 means that points within an area of approximately 5000 km^2 would resolve to a single address, whereas a precision of 7 means that points within an area of approximately 150 m^2 resolve to one address. The Recorder obtains a location publish, extracts the latitude and longitude, and then calculates the [geohash](https://en.wikipedia.org/wiki/Geohash) string and truncates it to `precision`. If the calculated geohash string can be found in our local LMDB cache, we consider the point cached; otherwise an actual reverse geo lookup (via HTTP) is performed and the result is cached in LMDB at the key of the geohash. As an example, let's assume Jane's device is at position (lat, lon) `48.879840, 2.323522`, which resolves to a geohash string of length 7 `u09whf7`. We can [visualize this](http://www.movable-type.co.uk/scripts/geohash.html) and show what this looks like. (See also: [visualizing geohash](http://www.bigdatamodeling.org/2013/01/intuitive-geohash.html).) @@ -716,7 +716,7 @@ The view then provides three URLs: * `views/viewname?lastpos=1` serves a JSON array of objects with the last position recorded * `views/viewname?geodata=1` serves a GeoJSON object containing recorded track data -Suppose Jane wishes to have her acqaintances see where she is whilst on vacation. Jane knows she'll be en-route between 2015-06-29 and 2015-07-15. She creates a file called, say, `loire.json` in the `views/` directory of the Recorder's document root: +Suppose Jane wishes to have her acquaintances see where she is while on vacation. Jane knows she'll be en-route between 2015-06-29 and 2015-07-15. She creates a file called, say, `loire.json` in the `views/` directory of the Recorder's document root: ```json { @@ -728,7 +728,7 @@ Suppose Jane wishes to have her acqaintances see where she is whilst on vacation } ``` -Jane's friends can now visit the URL `/view/loire` (note the missing `.json` extension) to be served a map showing Jane's progress along the Loire valley (if that is where she's actually travelling through). Jane can keep that view up even after she returns because the view will not serve data after the 15th of July, in other words, her location at any other time before or after the from/to dates is hidden. +Jane's friends can now visit the URL `/view/loire` (note the missing `.json` extension) to be served a map showing Jane's progress along the Loire valley (if that is where she's actually traveling through). Jane can keep that view up even after she returns because the view will not serve data after the 15th of July, in other words, her location at any other time before or after the from/to dates is hidden. ![Jane's vacation](assets/view-map.png) @@ -956,5 +956,5 @@ It actually is possible to gateway location publishes arriving via HTTP into MQT ### Override reverse-geo precision -If a payload is received with an element called `_geoprec` it contains an overide for the Recorder's configured reverse-geo precision. So, for example, if Recorder is running with precision 7, say, and the received payload contains `"_geoprec" : 2` the 2 will be used for this particular publish. This is not used in the OwnTracks apps, but it can be used with payloads you generate otherwise. If `_geoprec` is negative, new reverse geo lookups will not be performed, but cached entries of `abs(_geoprec)` will be used. +If a payload is received with an element called `_geoprec` it contains an override for the Recorder's configured reverse-geo precision. So, for example, if Recorder is running with precision 7, say, and the received payload contains `"_geoprec" : 2` the 2 will be used for this particular publish. This is not used in the OwnTracks apps, but it can be used with payloads you generate otherwise. If `_geoprec` is negative, new reverse geo lookups will not be performed, but cached entries of `abs(_geoprec)` will be used. From 71b2dd4cfae8edec015258dc768f5892dba6fc6b Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Wed, 17 Aug 2016 13:07:08 -0400 Subject: [PATCH 20/23] Move HOOKS.md into a doc/ directory --- README.md | 2 +- HOOKS.md => doc/HOOKS.md | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename HOOKS.md => doc/HOOKS.md (100%) diff --git a/README.md b/README.md index 13bff19..67cfe8d 100644 --- a/README.md +++ b/README.md @@ -685,7 +685,7 @@ The key to this data is the geohash string (here with an example of precision 2) ## Lua hooks -You can customize Recorder's behavior with Lua hooks. See [HOOKS.md](https://github.com/owntracks/recorder/blob/master/HOOKS.md). +You can customize Recorder's behavior with Lua hooks. See [HOOKS.md](https://github.com/owntracks/recorder/blob/master/doc/HOOKS.md). ## Monitoring diff --git a/HOOKS.md b/doc/HOOKS.md similarity index 100% rename from HOOKS.md rename to doc/HOOKS.md From b4777290f67d98e2d85077369d924f8732335172 Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Wed, 17 Aug 2016 13:10:06 -0400 Subject: [PATCH 21/23] Split out "Storage" and "Design decisions" These are useful but don't really need to be in the README, which is long enough as it is. --- README.md | 31 ------------------------------- doc/DESIGN.md | 11 +++++++++++ doc/STORE.md | 15 +++++++++++++++ 3 files changed, 26 insertions(+), 31 deletions(-) create mode 100644 doc/DESIGN.md create mode 100644 doc/STORE.md diff --git a/README.md b/README.md index 67cfe8d..375ccf5 100644 --- a/README.md +++ b/README.md @@ -41,8 +41,6 @@ We developed the Recorder as a one-stop solution to storing location data publis * [List users and devices](#list-users-and-devices) * [Show the last position reported by a user](#show-the-last-position-reported-by-a-user) * [What were the last 4 positions reported?](#what-were-the-last-4-positions-reported) -* [Design decisions](#design-decisions) -* [Storage](#storage) * [Reverse Geo](#reverse-geo) * [Precision](#precision) * [The geo cache](#the-geo-cache) @@ -609,35 +607,6 @@ isotst,vel,addr 2015-08-24T08:24:59Z,40,"A14, 04741 Roßwein, Germany" ``` - -## Design decisions - -We took a number of decisions when designing the Recorder and its utilities: - -* Flat files. The filesystem is the database. Period. That's were everything is stored. It makes incremental backups, purging old data, manipulation via the Unix toolset easy. (Admittedly, for fast geo-lookups we employ LMDB as a cache, but the final word is in the filesystem.) We considered all manner of databases and decided to keep this as simple and lightweight as possible. You can however have the Recorder send data to a database of your choosing, in addition to the file system it uses, by utilizing our embedded Lua hook. -* We wanted to store received data in the format it's published in. As this format is JSON, we store this raw payload in the `.rec` files. If we add an attribute to the JSON published by our apps, you have it right there. There's one slight exception: the monthly logs (the `.rec` files) have a leading timestamp and a relative topic; see below. (In the particular case of the OwnTracks firmware for Greenwich devices which can publish in CSV mode, we convert the CSV into OwnTracks JSON for storage.) -* File names are lower case. A user called `JaNe` with a device named `myPHONe` will be found in a file named `jane/myphone`. -* All times are UTC (a.k.a. Zulu or GMT). We got sick and tired of converting stuff back and forth. It is up to the consumer of the data to convert to local time if need be. -* The Recorder does not provide authentication or authorization. Nothing at all. Zilch. Nada. Think about this before making it available on a publicly-accessible IP address. Or rather: don't think about it; just don't do it. You can of course place a HTTP proxy in front of the Recorder to control access to it. Or use views (see below). -* `ocat`, the `cat` program for the Recorder, uses the same back-end which is used by the API though it accesses it directly (i.e. without resorting to HTTP). -* The Recorder supports 3-level MQTT topics only, in the typical OwnTracks format: `"owntracks//"`, optionally with a leading slash. (The first part of the topic need not be "owntracks".) Publishes via HTTP POST construct a fictitious topic internally using the provided user (`u`) and device (`d`) parameters. - -## Storage - -As mentioned earlier, data is stored in files, and these files are relative to `STORAGEDIR` (compiled into the programs or specified as an option). In particular, the following directory structure can exist, whereby directories are created as needed by the Recorder: - -* `cards/`, optional, may contains user cards. This card is then stored here and used with, e.g., `ocat --last` to show a user's name and optional avatar. User cards are typically stored in a subdirectory called `username`, and therein a JSON file `[username].json`. When reading cards, the Recorder will first attempt to open `[username]/[device]/[username].json` and then `[username]/[username].json`. -* `config/`, optional, contains the JSON of a [device configuration](http://owntracks.org/booklet/features/remoteconfig/) (`.otrc`) which was requested remotely via a [dump command](http://owntracks.org/booklet/tech/json/#_typecmd). Note that this will contain sensitive data. You can use this `.otrc` file to restore the OwnTracks configuration on your device by copying to the device and opening it in OwnTracks. -* `ghash/`, unless disabled, reverse Geo data (using a Google service) is collected into an LMDB database located in this directory. This LMDB database also contains named databases which are used by your optional Lua hooks, as well as a `topic2tid` database which can be used for TID re-mapping. -* `last/` contains the last location published by devices. E.g. Jane's last publish from her iPhone would be in `last/jjolie/iphone/jjolie-iphone.json`. The JSON payload contained therein is enhanced with the fields `user`, `device`, `topic`, and `ghash`. If a device's `last/` directory contains a file called `extra.json` (i.e. matching the example, this would be `last/jjolie/iphone/extra.json`), the content of this file is merged into the existing JSON for this user and returned by the API. Note, that you cannot overwrite existing values. So, an `extra.json` containing `{ "tst" : 11 }` will do nothing because the `tst` element we obtain from location data overrules, but adding `{ "beverage" : "water" }` will do what you want. If Recorder is built with support for our Greenwich firmware, this directory might contain `batt.json`, `ext.json`, and/or `status.json` each of which hold an array of the last 100 reports for internal battery voltage, external voltage, and status respectively. These values are returned via the API in the LAST object. A file `http.json` which should contain either a single JSON object or an array of JSON objects is returned to clients in HTTP mode. -* `monitor` a file which contains a timestamp and the last received topic (see Monitoring below). -* `msg/` contains messages received by the Messaging system. -* `photos/` optional; contains the binary photos from a card. -* `rec/` the Recorder data proper. One subdirectory per user, one subdirectory therein per device. Data files are named `YYYY-MM.rec` (e.g. `2015-08.rec` for the data accumulated during the month of August 2015. -* `waypoints/` contains a directory per user and device. Therein are individual files named by a timestamp with the JSON payload of published (i.e. shared) waypoints. The file names are timestamps because the `tst` of a waypoint is its key. If a user publishes all waypoints from a device (Publish Waypoints), the payload is stored in this directory as `username-device.otrw`. (Note, that this is the JSON [waypoints import format](http://owntracks.org/booklet/tech/json/#_typewaypoints).) You can use this `.otrw` file to restore the waypoints on your device by copying to the device and opening it in OwnTracks. - -You should definitely **not** modify or touch these files: they remain under the control of the Recorder. You can of course, remove old `.rec` files if they consume too much space. - ## Reverse Geo If not disabled with option `--norevgeo`, the Recorder will attempt to perform a reverse-geo lookup on the location coordinates it obtains and store them in an LMDB database. If a lookup is not possible, for example because you're over quota, the service isn't available, etc., Recorder keeps tracks of the coordinates which could *not* be resolved in a file named `missing`: diff --git a/doc/DESIGN.md b/doc/DESIGN.md new file mode 100644 index 0000000..185ba16 --- /dev/null +++ b/doc/DESIGN.md @@ -0,0 +1,11 @@ +# Design decisions + +We took a number of decisions when designing the Recorder and its utilities: + +* Flat files. The filesystem is the database. Period. That's were everything is stored. It makes incremental backups, purging old data, manipulation via the Unix toolset easy. (Admittedly, for fast geo-lookups we employ LMDB as a cache, but the final word is in the filesystem.) We considered all manner of databases and decided to keep this as simple and lightweight as possible. You can however have the Recorder send data to a database of your choosing, in addition to the file system it uses, by utilizing our embedded Lua hook. +* We wanted to store received data in the format it's published in. As this format is JSON, we store this raw payload in the `.rec` files. If we add an attribute to the JSON published by our apps, you have it right there. There's one slight exception: the monthly logs (the `.rec` files) have a leading timestamp and a relative topic; see below. (In the particular case of the OwnTracks firmware for Greenwich devices which can publish in CSV mode, we convert the CSV into OwnTracks JSON for storage.) +* File names are lower case. A user called `JaNe` with a device named `myPHONe` will be found in a file named `jane/myphone`. +* All times are UTC (a.k.a. Zulu or GMT). We got sick and tired of converting stuff back and forth. It is up to the consumer of the data to convert to local time if need be. +* The Recorder does not provide authentication or authorization. Nothing at all. Zilch. Nada. Think about this before making it available on a publicly-accessible IP address. Or rather: don't think about it; just don't do it. You can of course place a HTTP proxy in front of the Recorder to control access to it. Or use views (see below). +* `ocat`, the `cat` program for the Recorder, uses the same back-end which is used by the API though it accesses it directly (i.e. without resorting to HTTP). +* The Recorder supports 3-level MQTT topics only, in the typical OwnTracks format: `"owntracks//"`, optionally with a leading slash. (The first part of the topic need not be "owntracks".) Publishes via HTTP POST construct a fictitious topic internally using the provided user (`u`) and device (`d`) parameters. diff --git a/doc/STORE.md b/doc/STORE.md new file mode 100644 index 0000000..fe42a20 --- /dev/null +++ b/doc/STORE.md @@ -0,0 +1,15 @@ +# Storage + +As mentioned earlier, data is stored in files, and these files are relative to `STORAGEDIR` (compiled into the programs or specified as an option). In particular, the following directory structure can exist, whereby directories are created as needed by the Recorder: + +* `cards/`, optional, may contains user cards. This card is then stored here and used with, e.g., `ocat --last` to show a user's name and optional avatar. User cards are typically stored in a subdirectory called `username`, and therein a JSON file `[username].json`. When reading cards, the Recorder will first attempt to open `[username]/[device]/[username].json` and then `[username]/[username].json`. +* `config/`, optional, contains the JSON of a [device configuration](http://owntracks.org/booklet/features/remoteconfig/) (`.otrc`) which was requested remotely via a [dump command](http://owntracks.org/booklet/tech/json/#_typecmd). Note that this will contain sensitive data. You can use this `.otrc` file to restore the OwnTracks configuration on your device by copying to the device and opening it in OwnTracks. +* `ghash/`, unless disabled, reverse Geo data (using a Google service) is collected into an LMDB database located in this directory. This LMDB database also contains named databases which are used by your optional Lua hooks, as well as a `topic2tid` database which can be used for TID re-mapping. +* `last/` contains the last location published by devices. E.g. Jane's last publish from her iPhone would be in `last/jjolie/iphone/jjolie-iphone.json`. The JSON payload contained therein is enhanced with the fields `user`, `device`, `topic`, and `ghash`. If a device's `last/` directory contains a file called `extra.json` (i.e. matching the example, this would be `last/jjolie/iphone/extra.json`), the content of this file is merged into the existing JSON for this user and returned by the API. Note, that you cannot overwrite existing values. So, an `extra.json` containing `{ "tst" : 11 }` will do nothing because the `tst` element we obtain from location data overrules, but adding `{ "beverage" : "water" }` will do what you want. If Recorder is built with support for our Greenwich firmware, this directory might contain `batt.json`, `ext.json`, and/or `status.json` each of which hold an array of the last 100 reports for internal battery voltage, external voltage, and status respectively. These values are returned via the API in the LAST object. A file `http.json` which should contain either a single JSON object or an array of JSON objects is returned to clients in HTTP mode. +* `monitor` a file which contains a timestamp and the last received topic (see Monitoring below). +* `msg/` contains messages received by the Messaging system. +* `photos/` optional; contains the binary photos from a card. +* `rec/` the Recorder data proper. One subdirectory per user, one subdirectory therein per device. Data files are named `YYYY-MM.rec` (e.g. `2015-08.rec` for the data accumulated during the month of August 2015. +* `waypoints/` contains a directory per user and device. Therein are individual files named by a timestamp with the JSON payload of published (i.e. shared) waypoints. The file names are timestamps because the `tst` of a waypoint is its key. If a user publishes all waypoints from a device (Publish Waypoints), the payload is stored in this directory as `username-device.otrw`. (Note, that this is the JSON [waypoints import format](http://owntracks.org/booklet/tech/json/#_typewaypoints).) You can use this `.otrw` file to restore the waypoints on your device by copying to the device and opening it in OwnTracks. + +You should definitely **not** modify or touch these files: they remain under the control of the Recorder. You can of course, remove old `.rec` files if they consume too much space. From 507ec602c80c0741952fd477b63d8e8f0035898e Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Wed, 17 Aug 2016 13:16:51 -0400 Subject: [PATCH 22/23] Fix typo --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 375ccf5..4bba6ce 100644 --- a/README.md +++ b/README.md @@ -876,7 +876,7 @@ var apiKey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"; ### The LMDB database -`ocat --load` and `ocat --dump` can be use to load and dump the lmdb database respectively. There is some support for loading/dumping named databases using `--load=xx` or `--dump=xx` to specify the name. Use the mdb utilities to actually perform backups of these. `--load` expects key/value strings in pairs, separated by exactly one space. If the value is the string `DELETE`, the key is deleted from the database, which allows us to, say, remove a whole bunch of geohash prefixes in one go (but be careful doing this): +`ocat --load` and `ocat --dump` can be use to load and dump the LMDB database respectively. There is some support for loading/dumping named databases using `--load=xx` or `--dump=xx` to specify the name. Use the mdb utilities to actually perform backups of these. `--load` expects key/value strings in pairs, separated by exactly one space. If the value is the string `DELETE`, the key is deleted from the database, which allows us to, say, remove a whole bunch of geohash prefixes in one go (but be careful doing this): ```bash ocat --dump | From 11aaffd9c880a068d6bc07e085f874c51b35ba08 Mon Sep 17 00:00:00 2001 From: Alex Jordan Date: Wed, 17 Aug 2016 13:17:04 -0400 Subject: [PATCH 23/23] Link HOOKS.md, DESIGN.md, and STORE.md in README --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 4bba6ce..4710e5d 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,8 @@ There are two main components: the _Recorder_ obtains data via MQTT subscribes o We developed the Recorder as a one-stop solution to storing location data published by our OwnTracks apps (iOS and Android) and retrieving this data. Our previous offerings (`m2s`, `o2s`/`Pista`) also work of course, but we believe the Recorder is best suited to most environments. +See also [HOOKS.md](https://github.com/owntracks/recorder/blob/master/doc/HOOKS.md), [DESIGN.md](https://github.com/owntracks/recorder/blob/master/doc/DESIGN.md), and [STORE.md](https://github.com/owntracks/recorder/blob/master/doc/STORE.md) for more information on Lua hooks, program design, and storage layout, respectively. + ## Table of Contents * [`recorder`](#recorder)