mirror of
https://github.com/owntracks/recorder.git
synced 2026-08-23 11:26:16 +00:00
Merge pull request #138 from strugee/reorganize-readme
Reorganize README
This commit is contained in:
JP Mens
GitHub
@@ -0,0 +1,92 @@
|
||||
# API
|
||||
|
||||
The Recorder's API provides most of the functions that are surfaced by _ocat_. GET and POST requests are supported, and if a username and device are needed, these can be passed in via `X-Limit-User` and `X-Limit-Device` headers alternatively to GET or POST parameters. (From and To dates may also be specified as `X-Limit-From` and `X-Limit-To` respectively.)
|
||||
|
||||
The API endpoint is at `/api/0` and is followed by the verb.
|
||||
|
||||
## `monitor`
|
||||
|
||||
Returns the content of the `monitor` file as plain text.
|
||||
|
||||
```
|
||||
curl 'http://127.0.0.1:8083/api/0/monitor'
|
||||
1441962082 owntracks/jjolie/phone
|
||||
```
|
||||
|
||||
## `last`
|
||||
|
||||
Returns a list of last users' positions. (Can be limited by _user_, _device_, and _fields_, a comma-separated list of fields which should be returned instead of the default of all fields.)
|
||||
|
||||
```
|
||||
curl http://127.0.0.1:8083/api/0/last [-d user=jjolie [-d device=phone]]
|
||||
```
|
||||
|
||||
```
|
||||
curl 'http://127.0.0.1:8083/api/0/last?fields=tst,tid,addr,topic,isotst'
|
||||
```
|
||||
|
||||
## `list`
|
||||
|
||||
List users. If _user_ is specified, lists that user's devices. If both _user_ and _device_ are specified, lists that device's `.rec` files.
|
||||
|
||||
## `locations`
|
||||
|
||||
Here comes the actual data. This lists users' locations and requires both _user_ and _device_. Output format is JSON unless a different _format_ is given (`csv`, `json`, `geojson`, `xml`, and `linestring` are supported).
|
||||
|
||||
In order to limit the number of records returned, use _limit_ which causes a reverse search through the `.rec` files; this can be used to find the last N positions.
|
||||
|
||||
Date/time ranges may be specified as _from_ and _to_ with dates/times specified as described for _ocat_ above.
|
||||
|
||||
```
|
||||
curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s
|
||||
curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s -d limit=1
|
||||
curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s -d format=geojson
|
||||
curl http://127.0.0.1:8083/api/0/locations -d user=jpm -d device=5s -d from=2014-08-03
|
||||
curl 'http://127.0.0.1:8083/api/0/locations?from=2015-09-01&user=jpm&device=5s&fields=tst,tid,addr,isotst'
|
||||
```
|
||||
|
||||
## `q`
|
||||
|
||||
Query the geo cache for a particular _lat_ and _lon_.
|
||||
|
||||
```
|
||||
curl 'http://127.0.0.1:8083/api/0/q?lat=48.85833&lon=2.295'
|
||||
{
|
||||
"cc": "FR",
|
||||
"addr": "9 Avenue Anatole France, 75007 Paris, France",
|
||||
"tst": 1441984405
|
||||
}
|
||||
```
|
||||
|
||||
The reported timestamp was the time at which this cache entry was made. Note that this interface queries only -- it does not populate the cache.
|
||||
|
||||
## `photo`
|
||||
|
||||
Requires GET method and _user_, and will return the `image/png` 40x40px photograph of a user if available in `STORAGEDIR/photos/` or a transparent 40x40png with a black border otherwise.
|
||||
|
||||
## `kill`
|
||||
|
||||
If support for this is compiled in, this API endpoint allows a client to remove data from _storage_. (Warning: *any* client can do this, as there is no authentication/authorization in the Recorder!)
|
||||
|
||||
```
|
||||
curl 'http://127.0.0.1:8083/api/0/kill?user=ngin&device=ojo'
|
||||
|
||||
{
|
||||
"path": "s0/rec/ngin/ojo",
|
||||
"status": "OK",
|
||||
"last": "s0/last/ngin/ojo/ngin-ojo.json",
|
||||
"killed": [
|
||||
"2015-09.rec",
|
||||
]
|
||||
}
|
||||
```
|
||||
The response contains a list of removed `.rec` files, and file system operations are logged to syslog.
|
||||
|
||||
## `version`
|
||||
|
||||
Returns a JSON object which contains the Recorder's version string, such as
|
||||
|
||||
```json
|
||||
{ "version": "0.4.7" }
|
||||
```
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
# Design decisions
|
||||
|
||||
We took a number of decisions when designing the Recorder and its utilities:
|
||||
|
||||
* Flat files. The filesystem is the database. Period. That's were everything is stored. It makes incremental backups, purging old data, manipulation via the Unix toolset easy. (Admittedly, for fast geo-lookups we employ LMDB as a cache, but the final word is in the filesystem.) We considered all manner of databases and decided to keep this as simple and lightweight as possible. You can however have the Recorder send data to a database of your choosing, in addition to the file system it uses, by utilizing our embedded Lua hook.
|
||||
* We wanted to store received data in the format it's published in. As this format is JSON, we store this raw payload in the `.rec` files. If we add an attribute to the JSON published by our apps, you have it right there. There's one slight exception: the monthly logs (the `.rec` files) have a leading timestamp and a relative topic; see below. (In the particular case of the OwnTracks firmware for Greenwich devices which can publish in CSV mode, we convert the CSV into OwnTracks JSON for storage.)
|
||||
* File names are lower case. A user called `JaNe` with a device named `myPHONe` will be found in a file named `jane/myphone`.
|
||||
* All times are UTC (a.k.a. Zulu or GMT). We got sick and tired of converting stuff back and forth. It is up to the consumer of the data to convert to local time if need be.
|
||||
* The Recorder does not provide authentication or authorization. Nothing at all. Zilch. Nada. Think about this before making it available on a publicly-accessible IP address. Or rather: don't think about it; just don't do it. You can of course place a HTTP proxy in front of the Recorder to control access to it. Or use views (see below).
|
||||
* `ocat`, the `cat` program for the Recorder, uses the same back-end which is used by the API though it accesses it directly (i.e. without resorting to HTTP).
|
||||
* The Recorder supports 3-level MQTT topics only, in the typical OwnTracks format: `"owntracks/<username>/<devicename>"`, optionally with a leading slash. (The first part of the topic need not be "owntracks".) Publishes via HTTP POST construct a fictitious topic internally using the provided user (`u`) and device (`d`) parameters.
|
||||
@@ -0,0 +1,93 @@
|
||||
# Lua hooks
|
||||
|
||||
If Recorder is compiled with Lua support, a Lua script you provide is launched at startup. Lua is _a powerful, fast, lightweight, embeddable scripting language_. You can use this to process location publishes in any way you desire: your imagination (and Lua-scripting knowhow) set the limits. Some examples:
|
||||
|
||||
* insert publishes into a database of your choice
|
||||
* switch on the coffee machine when your OwnTracks device reports you're entering home (but see also [mqttwarn](http://jpmens.net/2014/02/17/introducing-mqttwarn-a-pluggable-mqtt-notifier/))
|
||||
* write a file with data in a format of your choice (see `etc/example.lua`)
|
||||
|
||||
Run the Recorder with the path to your Lua script specified in its `--lua-script` option (there is no default). If the script cannot be loaded (e.g. because it cannot be read or contains syntax errors), the Recorder unloads Lua and continues *without* your script.
|
||||
|
||||
If the Lua script can be loaded, it is automatically provided with a table variable called `otr` which contains the following members:
|
||||
|
||||
* `otr.version` is a read-only string with the Recorder version (example: `"0.3.2"`)
|
||||
* `otr.log(s)` is a function which takes a string `s` which is logged to syslog at the Recorder's facility and log level INFO.
|
||||
* `otr.strftime(fmt, t)` is a function which takes a format string `fmt` (see `strftime(3)`) and an integer number of seconds `t` and returns a string with the formatted UTC time. If `t` is 0 or negative, the current system time is used.
|
||||
* `otr.putdb(key, value)` is a function which takes two strings `k` and `v` and stores them in the named LMDB database called `luadb`. This can be viewed with
|
||||
* `otr.getdb(key)` is a function which takes a single string `key` and returns the database value associated with that key or `nil` if the key isn't stored.
|
||||
|
||||
```
|
||||
ocat --dump=luadb
|
||||
```
|
||||
|
||||
Your Lua script *must* provide the following functions:
|
||||
|
||||
## `otr_init`
|
||||
|
||||
This is invoked at start of Recorder. If the function returns a non-zero value, Recorder unloads Lua and disables its processing; i.e. the `hook()` will *not* be invoked on location publishes.
|
||||
|
||||
## `otr_exit`
|
||||
|
||||
This is invoked when the Recorder stops, which it doesn't really do unless you CTRL-C it or send it a SIGTERM signal.
|
||||
|
||||
|
||||
## `otr_hook`
|
||||
|
||||
This function is invoked at every location publish processed by the Recorder. Your function is passed three arguments:
|
||||
|
||||
1. _topic_ is the topic published to (e.g. `owntracks/jane/phone`)
|
||||
2. _type_ is the type of MQTT message. This is the `_type` in our JSON messages (e.g. `location`, `cmd`, `transition`, ...) or `"unknown"`.
|
||||
3. _location_ is a [Lua table](http://www.lua.org/pil/2.5.html) (associative array) with all the elements obtained in the JSON message. In the case of _type_ being `location`, we also add country code (`cc`) and the location's address (`addr`) unless reverse-geo lookups have been disabled in Recorder.
|
||||
|
||||
Assume the following small example Lua script in `example.lua`:
|
||||
|
||||
```lua
|
||||
local file
|
||||
|
||||
function otr_init()
|
||||
otr.log("example.lua starting; writing to /tmp/lua.out")
|
||||
file = io.open("/tmp/lua.out", "a")
|
||||
file:write("written by OwnTracks Recorder version " .. otr.version .. "\n")
|
||||
end
|
||||
|
||||
function otr_hook(topic, _type, data)
|
||||
local timestr = otr.strftime("It is %T in the year %Y", 0)
|
||||
print("L: " .. topic .. " -> " .. _type)
|
||||
file:write(timestr .. " " .. topic .. " lat=" .. data['lat'] .. data['addr'] .. "\n")
|
||||
end
|
||||
|
||||
function otr_exit()
|
||||
end
|
||||
```
|
||||
|
||||
When Recorder is launched with `--lua-script example.lua` it invokes `otr_init()` which opens a file. Then, for each location received, it calls `otr_hook()` which updates the file.
|
||||
|
||||
Assuming an OwnTracks device publishes this payload
|
||||
|
||||
```json
|
||||
{"cog":-1,"batt":-1,"lon":2.29513,"acc":5,"vel":-1,"vac":-1,"lat":48.85833,"t":"u","tst":1441984413,"alt":0,"_type":"location","tid":"JJ"}
|
||||
```
|
||||
|
||||
the file `/tmp/lua.out` would contain
|
||||
|
||||
```txt
|
||||
written by OwnTracks Recorder version 0.3.0
|
||||
It is 14:10:01 in the year 2015 owntracks/jane/phone lat=48.858339 Avenue Anatole France, 75007 Paris, France
|
||||
```
|
||||
|
||||
## `otr_putrec`
|
||||
|
||||
An optional function you provide is called `otr_putrec(u, d, s)`. If it exists,
|
||||
it is called with the current user in `u`, the device in `d` and the payload
|
||||
(which for OwnTracks apps is JSON but for, eg Greenwich devices might not be) in the string `s`. If your function returns a
|
||||
non-zero value, the Recorder will *not* write the REC file for this publish.
|
||||
|
||||
## `otr_httpobject`
|
||||
|
||||
An optional function you provide is called `otr_httpobject(u, d, t, data)` where `u` is the username used by the client (`?u=`), `d` is the device name (`&d=` in the URI), `t` is the OwnTracks JSON `_type` and `data` a Lua table built from the OwnTracks JSON payload of `_type`. If it exists, this function is called whenever a POST is received in httpmode and the Recorder is gathering data to return to the client app. The function *must* return a Lua table containing any number of string, number, or boolean values which are converted to a JSON object and appended to the JSON array returned to the client. An [example](etc/example.lua) shows how, say, a transition event can be used to open the Featured content tab in the app.
|
||||
|
||||
## Hooklets
|
||||
|
||||
After running `otr_hook()`, the Recorder attempts to invoke a Lua function for each of the elements in the extended JSON. If, say, your Lua script contains a function called `hooklet_lat`, it will be invoked every time a `lat` is received as part of the JSON payload. Similarly with `hooklet_addr`, `hooklet_cc`, `hooklet_tst`, etc. These _hooklets_ are invoked with the same parameters as `otr_hook()`.
|
||||
|
||||
You define a hooklet function only if you're interested in expressly triggering on a particular JSON element.
|
||||
@@ -0,0 +1,15 @@
|
||||
# Storage
|
||||
|
||||
As mentioned earlier, data is stored in files, and these files are relative to `STORAGEDIR` (compiled into the programs or specified as an option). In particular, the following directory structure can exist, whereby directories are created as needed by the Recorder:
|
||||
|
||||
* `cards/`, optional, may contains user cards. This card is then stored here and used with, e.g., `ocat --last` to show a user's name and optional avatar. User cards are typically stored in a subdirectory called `username`, and therein a JSON file `[username].json`. When reading cards, the Recorder will first attempt to open `[username]/[device]/[username].json` and then `[username]/[username].json`.
|
||||
* `config/`, optional, contains the JSON of a [device configuration](http://owntracks.org/booklet/features/remoteconfig/) (`.otrc`) which was requested remotely via a [dump command](http://owntracks.org/booklet/tech/json/#_typecmd). Note that this will contain sensitive data. You can use this `.otrc` file to restore the OwnTracks configuration on your device by copying to the device and opening it in OwnTracks.
|
||||
* `ghash/`, unless disabled, reverse Geo data (using a Google service) is collected into an LMDB database located in this directory. This LMDB database also contains named databases which are used by your optional Lua hooks, as well as a `topic2tid` database which can be used for TID re-mapping.
|
||||
* `last/` contains the last location published by devices. E.g. Jane's last publish from her iPhone would be in `last/jjolie/iphone/jjolie-iphone.json`. The JSON payload contained therein is enhanced with the fields `user`, `device`, `topic`, and `ghash`. If a device's `last/` directory contains a file called `extra.json` (i.e. matching the example, this would be `last/jjolie/iphone/extra.json`), the content of this file is merged into the existing JSON for this user and returned by the API. Note, that you cannot overwrite existing values. So, an `extra.json` containing `{ "tst" : 11 }` will do nothing because the `tst` element we obtain from location data overrules, but adding `{ "beverage" : "water" }` will do what you want. If Recorder is built with support for our Greenwich firmware, this directory might contain `batt.json`, `ext.json`, and/or `status.json` each of which hold an array of the last 100 reports for internal battery voltage, external voltage, and status respectively. These values are returned via the API in the LAST object. A file `http.json` which should contain either a single JSON object or an array of JSON objects is returned to clients in HTTP mode.
|
||||
* `monitor` a file which contains a timestamp and the last received topic (see Monitoring below).
|
||||
* `msg/` contains messages received by the Messaging system.
|
||||
* `photos/` optional; contains the binary photos from a card.
|
||||
* `rec/` the Recorder data proper. One subdirectory per user, one subdirectory therein per device. Data files are named `YYYY-MM.rec` (e.g. `2015-08.rec` for the data accumulated during the month of August 2015.
|
||||
* `waypoints/` contains a directory per user and device. Therein are individual files named by a timestamp with the JSON payload of published (i.e. shared) waypoints. The file names are timestamps because the `tst` of a waypoint is its key. If a user publishes all waypoints from a device (Publish Waypoints), the payload is stored in this directory as `username-device.otrw`. (Note, that this is the JSON [waypoints import format](http://owntracks.org/booklet/tech/json/#_typewaypoints).) You can use this `.otrw` file to restore the waypoints on your device by copying to the device and opening it in OwnTracks.
|
||||
|
||||
You should definitely **not** modify or touch these files: they remain under the control of the Recorder. You can of course, remove old `.rec` files if they consume too much space.
|
||||
Reference in New Issue
Block a user