From a81c66d220ee09c1f26d09c7fa6bdac6a692c039 Mon Sep 17 00:00:00 2001 From: Steffen Vogel Date: Sat, 14 May 2022 10:15:01 +0200 Subject: [PATCH 01/12] cleanup README --- README.md | 63 ++++++++++++++++++++++++++++++------------------------- 1 file changed, 34 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index 87b549e..70d6d72 100644 --- a/README.md +++ b/README.md @@ -6,12 +6,13 @@ Dockerfile for the [Recorder](https://github.com/owntracks/recorder) of the OwnTracks project. The image is [owntracks/recorder](https://hub.docker.com/r/owntracks/recorder). ## Quickstart + ```bash -$ docker volume create recorder_store -$ docker run -d -p 8083:8083 -v recorder_store:/store -e OTR_HOST=mqtt_broker owntracks/recorder +docker volume create recorder_store +docker run -d -p 8083:8083 -v recorder_store:/store -e OTR_HOST=mqtt_broker owntracks/recorder ``` -Recorder is now accessible at `http://localhost:8083`. +Recorder is now accessible at `http://localhost:8083`. `-p 8083:8083` makes the container reachable at port 8083. `-d` detaches the container into the background. The volume `recorder_store` is mounted at @@ -21,6 +22,7 @@ environment variables. Multiple `-e` parameters can be used for multiple environment variables. ## Configuration + The Recorder can be configured using two methods, environment variables and via the a `recorder.conf` file in the `/config` volume of the container. @@ -29,31 +31,33 @@ via the a `recorder.conf` file in the `/config` volume of the container. Can be passed to the container with the `-e` parameter. Example: ```bash -$ docker run -d -p 8083:8083 \ +docker run -d -p 8083:8083 \ -e OTR_HOST=mqtt_broker \ - -e OTR_PORT=1883 \ - -e OTR_USER=user \ - -e OTR_PASS=pass \ - owntracks/recorder + -e OTR_PORT=1883 \ + -e OTR_USER=user \ + -e OTR_PASS=pass \ + owntracks/recorder ``` The complete list of parameters can be found in the [recorder documentation](https://github.com/owntracks/recorder/blob/master/README.md#configuration-file). ### Configuration file + One can also use a configuration file. The container reads a `recorder.conf` file from the `/config` folder. To use this, create a folder e.g. `./config` and mount it into you docker container at `/config`. ```bash -$ mkdir config -$ docker run -d -p 8083:8083 -v recorder_store:/store -v ./config:/config owntracks/recorder +mkdir config +docker run -d -p 8083:8083 -v recorder_store:/store -v ./config:/config owntracks/recorder ``` Up on starting the recorder, a default `recorder.conf` file will be created if none exists. Possible options are documented [here](https://github.com/owntracks/recorder/blob/master/README.md#configuration-file). **Notes:** + - The value of `OTR_HOST` is as seen from the container. Thus `localhost` refers to the container not the host and should likely not be used. - Environment variables, overwrite the `recorder.conf` file options. @@ -66,14 +70,15 @@ The `/store` volume of the container is used for persistent storage of location data. The volume needs to be created explicitly. ```bash -$ docker volume create recorder_storage -$ docker run -d -p 8083:8083 -v recorder_store:/store owntracks/recorder +docker volume create recorder_storage +docker run -d -p 8083:8083 -v recorder_store:/store owntracks/recorder ``` + It is also possible to use a local folder instead of an static docker volume. ```bash -$ mkdir store -$ docker run -d -p 8083:8083 -v ./store:/store owntracks/recorder +mkdir store +docker run -d -p 8083:8083 -v ./store:/store owntracks/recorder ``` If nothing is mounted at `/store`, docker will create a unique volume @@ -81,13 +86,14 @@ automatically. However up on recreation of the docker container, this process will be repeated and another unique volume will be created. As a result, the container will have forgotten about previous tracks. -## TLS between MQTT broker and Recorder +## TLS between MQTT Broker and Recorder + The `OTR_CAPATH` of the container defaults to the `/config` volume. Thus certificates and key files belong into the `/config` volume. `OTR_CAFILE` must be configured for TLS. `OTR_CERTFILE` defaults to `cert.pem` and `OTR_KEYFILE` to `key.pem`. These files are optional and the options are ignored if the files don't exist. -## TLS encryption via reverse proxy +## TLS encryption via Reverse Proxy The Recorder has no encryption module by it self. Instead use a reverse proxy setup. See https://github.com/jwilder/nginx-proxy for how to do this in a semi @@ -97,17 +103,17 @@ details. ## Healthcheck -The Recorder container performs a Docker-style HEALTHCHECK on itself by periodically +The Recorder container performs a Docker-style `HEALTHCHECK` on itself by periodically running `recorder-health.sh` on itself. This program POSTS a `_type: location` JSON message to itself over HTTP to the ping-ping endpoint and verifies via the HTTP API whether the message was received. - ## Docker compose files -Save a file with the name [docker-compose.yml](docker-compose.yml) and following content. Run with -`docker-compose up` from the same folder. -``` yaml +Save a file with the name [docker-compose.yml](docker-compose.yml) and following content. +Run with `docker-compose up` from the same folder. + +```yaml version: '3' services: @@ -124,7 +130,6 @@ services: volumes: store: config: - ``` This [docker-compose.yml](docker-compose.yml) file creates `store` and `config` volumes. It is @@ -137,7 +142,7 @@ variables see An example might look like: -``` yaml +```yaml version: '3' services: @@ -156,7 +161,6 @@ services: volumes: store: - ``` ### With MQTT broker @@ -164,7 +168,7 @@ volumes: If you need to set up an MQTT broker, you can easily use, say, Mosquitto. There are ready to use containers available on docker hub. To use `eclipse-mosquitto` add something like [the following](docker-compose-mqtt.yml) to your `docker-compose.yml` file. -``` yaml +```yaml version: '3' services: @@ -196,6 +200,7 @@ volumes: mosquitto-logs: mosquitto-conf: ``` + See [here](https://hub.docker.com/_/eclipse-mosquitto) for info on the eclipse-mosquitto image and how to configure it. ### All in one solution with reverse proxy and Let's Encrypt @@ -222,7 +227,7 @@ There are some caveats people seem to step into: the virtual host, e.g. `owntrack.domain.com` in the **folder** `/etc/nginx/htpasswd` -``` yaml +```yaml version: '2' @@ -313,9 +318,10 @@ networks: external: name: nginx-proxy ``` + a minimal mosquitto.conf which can act as a start: -``` +``` allow_anonymous false password_file /etc/mosquitto/passwd #use mosquitto_passwd inside container to populate the passwd file @@ -338,8 +344,7 @@ cafile /etc/letsencrypt/live/mqtt.domain.com/chain.pem keyfile /etc/letsencrypt/live/mqtt.domain.com/key.pem ``` - -# Possible enhancements +## Possible enhancements - Maybe put the most common Mosquitto options in the section which uses an MQTT broker in the docker-compose file From 8e3b734a1d48e067e6219a90f88d4f39860ab622 Mon Sep 17 00:00:00 2001 From: Steffen Vogel Date: Sat, 14 May 2022 10:15:41 +0200 Subject: [PATCH 02/12] use multi-stage Docker builds --- Dockerfile | 71 +++++++++++++++++++++++++++--------------------------- 1 file changed, 36 insertions(+), 35 deletions(-) diff --git a/Dockerfile b/Dockerfile index 974e990..4c3cbf5 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,46 +1,47 @@ -FROM alpine:3.13 as builder -LABEL version="1.0" description="OwnTracks Recorder" -LABEL authors="Jan-Piet Mens , Giovanni Angoli , Amy Nagle , Malte Deiseroth " -MAINTAINER Malte Deiseroth +FROM alpine:3.15 AS builder -# build with `docker build --build-arg recorder_version=x.y.z ' -ARG recorder_version=0.8.8 +ARG RECORDER_VERSION=0.8.8 +# ARG RECORDER_VERSION=master -COPY entrypoint.sh /entrypoint.sh -COPY config.mk /config.mk -COPY recorder.conf /etc/default/recorder.conf -COPY recorder-health.sh /usr/local/sbin/recorder-health.sh +RUN apk add \ + make \ + gcc \ + git \ + shadow \ + musl-dev \ + curl-dev \ + libconfig-dev \ + mosquitto-dev \ + lmdb-dev \ + libsodium-dev \ + lua5.2-dev -ENV VERSION=$recorder_version -ENV EUID=9999 +RUN git clone --branch=${RECORDER_VERSION} https://github.com/owntracks/recorder +WORKDIR recorder -RUN apk add --no-cache --virtual .build-deps \ - curl-dev libconfig-dev make \ - gcc musl-dev mosquitto-dev shadow wget \ - && apk add --no-cache \ - libcurl libconfig-dev mosquitto-dev lmdb-dev libsodium-dev lua5.2-dev \ - && groupadd -g $EUID appuser \ - && useradd -r -u $EUID -s "/bin/sh" -g appuser appuser \ - && mkdir -p /usr/local/source \ - && cd /usr/local/source \ - && wget https://github.com/owntracks/recorder/archive/$VERSION.tar.gz \ - && tar xzf $VERSION.tar.gz \ - && cd recorder-$VERSION \ - && mv /config.mk ./ \ - && make \ - && make install \ - && cd / \ - && chmod 755 /entrypoint.sh \ - && rm -rf /usr/local/source \ - && chmod 755 /usr/local/sbin/recorder-health.sh \ - && apk del .build-deps -RUN apk add --no-cache \ - curl jq +COPY config.mk . +RUN make -j $(nprocs) +RUN make install DESTDIR=/app + +FROM alpine:3.15 VOLUME ["/store", "/config"] +RUN apk add \ + curl \ + jq \ + libcurl \ + libconfig \ + mosquitto \ + lmdb \ + libsodium \ + lua5.2 + COPY recorder.conf /config/recorder.conf COPY JSON.lua /config/JSON.lua +COPY --from=builder /app / + +COPY recorder-health.sh /usr/local/sbin/recorder-health.sh # If you absolutely need health-checking, enable the option below. Keep in # mind that until https://github.com/systemd/systemd/issues/6432 is resolved, @@ -50,4 +51,4 @@ COPY JSON.lua /config/JSON.lua EXPOSE 8083 -ENTRYPOINT ["/entrypoint.sh"] +ENTRYPOINT ["/usr/sbin/ot-recorder"] From 406cd6ecc2b394648e9a0826bd994fe6e0940c0f Mon Sep 17 00:00:00 2001 From: Steffen Vogel Date: Sat, 14 May 2022 10:15:52 +0200 Subject: [PATCH 03/12] enable IPV6 in Mongoose --- config.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config.mk b/config.mk index 34f4125..e7946f8 100644 --- a/config.mk +++ b/config.mk @@ -1,4 +1,4 @@ -CFLAGS += -g +CFLAGS += -g -DNS_ENABLE_IPV6 INSTALLDIR = /usr From b6da99b4ae40952dc33956266356d46e4104bb22 Mon Sep 17 00:00:00 2001 From: Steffen Vogel Date: Sat, 14 May 2022 10:16:02 +0200 Subject: [PATCH 04/12] cleanup health check script --- recorder-health.sh | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/recorder-health.sh b/recorder-health.sh index 02e65ef..9015488 100644 --- a/recorder-health.sh +++ b/recorder-health.sh @@ -1,30 +1,29 @@ #!/bin/sh -addr=`hostname` -port=8083 +ADDR=`hostname` +PORT=8083 -epoch=$(date +%s) +EPOCH=$(date +%s) -location=$(cat < /dev/null +curl -sSL --data "${LOCATION}" "http://${ADDR}:${PORT}/pub?u=ping&d=ping" > /dev/null # obtain tst of ping/ping's last location -ret_epoch=$(curl -sSL http://${addr}:${port}/api/0/last --data "user=ping&device=ping" | - env jq -r '.[0].tst' ) +RET_EPOCH=$(curl -sSL http://${ADDR}:${PORT}/api/0/last --data "user=ping&device=ping" | env jq -r '.[0].tst' ) -if [ $epoch -ne $ret_epoch ]; then - echo PANIC $epoch $ret_epoch +if [ ${EPOCH} -ne ${RET_EPOCH} ]; then + echo PANIC ${EPOCH} ${RET_EPOCH} exit 1 else echo OK From 86627d5237f87fd83c300157561527e5fb360a74 Mon Sep 17 00:00:00 2001 From: Steffen Vogel Date: Sat, 14 May 2022 10:16:14 +0200 Subject: [PATCH 05/12] remove entrypoint script --- entrypoint.sh | 16 ---------------- 1 file changed, 16 deletions(-) delete mode 100644 entrypoint.sh diff --git a/entrypoint.sh b/entrypoint.sh deleted file mode 100644 index e3b3f78..0000000 --- a/entrypoint.sh +++ /dev/null @@ -1,16 +0,0 @@ -#!/bin/sh - -# If running as root (first invocation), fix mountpoint permissions -# and re-run this script as appuser. -if [[ $(id -u) -eq 0 ]]; then - chown -R appuser:appuser /store /config - exec su appuser -- "$0" "$@" -fi - -# Load Default recorder.conf if not available -if [ ! -f /config/recorder.conf ]; then - cp /etc/default/recorder.conf /config/recorder.conf -fi - -ot-recorder --initialize -ot-recorder "$@" From 1925429c0168540e5eab7f28ea0baffa1b40e2f0 Mon Sep 17 00:00:00 2001 From: Steffen Vogel Date: Mon, 16 May 2022 09:51:14 +0200 Subject: [PATCH 06/12] re-add --no-cache option to apk --- Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Dockerfile b/Dockerfile index 4c3cbf5..6a9eb00 100644 --- a/Dockerfile +++ b/Dockerfile @@ -3,7 +3,7 @@ FROM alpine:3.15 AS builder ARG RECORDER_VERSION=0.8.8 # ARG RECORDER_VERSION=master -RUN apk add \ +RUN apk add --no-cache \ make \ gcc \ git \ @@ -27,7 +27,7 @@ FROM alpine:3.15 VOLUME ["/store", "/config"] -RUN apk add \ +RUN apk add --no-cache \ curl \ jq \ libcurl \ From 59db28f3c967a7ffc1267feb55007307c90f958d Mon Sep 17 00:00:00 2001 From: Steffen Vogel Date: Mon, 16 May 2022 10:04:08 +0200 Subject: [PATCH 07/12] re-add entrypoint.sh --- Dockerfile | 11 ++++++++--- entrypoint.sh | 7 +++++++ 2 files changed, 15 insertions(+), 3 deletions(-) create mode 100644 entrypoint.sh diff --git a/Dockerfile b/Dockerfile index 6a9eb00..8d7bc91 100644 --- a/Dockerfile +++ b/Dockerfile @@ -41,14 +41,19 @@ COPY recorder.conf /config/recorder.conf COPY JSON.lua /config/JSON.lua COPY --from=builder /app / -COPY recorder-health.sh /usr/local/sbin/recorder-health.sh +COPY recorder-health.sh /usr/sbin/recorder-health.sh +COPY entrypoint.sh /usr/sbin/entrypoint.sh # If you absolutely need health-checking, enable the option below. Keep in # mind that until https://github.com/systemd/systemd/issues/6432 is resolved, # using the HEALTHCHECK feature will cause systemd to generate a significant # amount of spam in the system logs. -# HEALTHCHECK CMD /usr/local/sbin/recorder-health.sh +# HEALTHCHECK CMD /usr/sbin/recorder-health.sh EXPOSE 8083 -ENTRYPOINT ["/usr/sbin/ot-recorder"] +ENV OTR_CAFILE=/etc/ssl/cert.pem +ENV OTR_STORAGEDIR=/store +ENV OTR_TOPIC="owntracks/#" + +ENTRYPOINT ["/usr/sbin/entrypoint.sh"] diff --git a/entrypoint.sh b/entrypoint.sh new file mode 100644 index 0000000..294aaae --- /dev/null +++ b/entrypoint.sh @@ -0,0 +1,7 @@ +#!/bin/sh + +if ! [ -f ${OTR_STORAGEDIR}/ghash/data.mdb ]; then + ot-recorder --initialize +fi + +ot-recorder ${OTR_TOPIC} From 28553c96a862d8126cce9da9eea42719c6dc9715 Mon Sep 17 00:00:00 2001 From: Steffen Vogel Date: Mon, 16 May 2022 10:04:15 +0200 Subject: [PATCH 08/12] use absolute workdir --- Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Dockerfile b/Dockerfile index 8d7bc91..fd02c96 100644 --- a/Dockerfile +++ b/Dockerfile @@ -16,8 +16,8 @@ RUN apk add --no-cache \ libsodium-dev \ lua5.2-dev -RUN git clone --branch=${RECORDER_VERSION} https://github.com/owntracks/recorder -WORKDIR recorder +RUN git clone --branch=${RECORDER_VERSION} https://github.com/owntracks/recorder /src/recorder +WORKDIR /src/recorder COPY config.mk . RUN make -j $(nprocs) From 47d93467b1e97cd87494ac4a53bb1b90bb730021 Mon Sep 17 00:00:00 2001 From: Steffen Vogel Date: Mon, 16 May 2022 10:04:28 +0200 Subject: [PATCH 09/12] listen to both IPv4 and IPv6 by default --- recorder.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/recorder.conf b/recorder.conf index 9671e63..a94062e 100644 --- a/recorder.conf +++ b/recorder.conf @@ -4,7 +4,7 @@ # and its associated utilities to override compiled-in defaults. OTR_TOPICS = "owntracks/#" -OTR_HTTPHOST = "0.0.0.0" +OTR_HTTPHOST = "[::]" # OTR_CAPATH = "/config" # OTR_CAFILE = "ca.pem" From 1a34096ac7f84e0121749f79cea5062d23f3fca4 Mon Sep 17 00:00:00 2001 From: Steffen Vogel Date: Mon, 16 May 2022 10:09:54 +0200 Subject: [PATCH 10/12] use lowercase nouns in README --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 70d6d72..fc3380f 100644 --- a/README.md +++ b/README.md @@ -86,14 +86,14 @@ automatically. However up on recreation of the docker container, this process will be repeated and another unique volume will be created. As a result, the container will have forgotten about previous tracks. -## TLS between MQTT Broker and Recorder +## TLS between MQTT broker and recorder The `OTR_CAPATH` of the container defaults to the `/config` volume. Thus certificates and key files belong into the `/config` volume. `OTR_CAFILE` must be configured for TLS. `OTR_CERTFILE` defaults to `cert.pem` and `OTR_KEYFILE` to `key.pem`. These files are optional and the options are ignored if the files don't exist. -## TLS encryption via Reverse Proxy +## TLS encryption via reverse proxy The Recorder has no encryption module by it self. Instead use a reverse proxy setup. See https://github.com/jwilder/nginx-proxy for how to do this in a semi From 97fef2deb798f3003850d016319f8e3e3fd1ce11 Mon Sep 17 00:00:00 2001 From: Steffen Vogel Date: Mon, 16 May 2022 10:33:22 +0200 Subject: [PATCH 11/12] clarify IPv4/IPv6 semantics of OTR_HTTPHOST --- recorder.conf | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/recorder.conf b/recorder.conf index a94062e..d8c448a 100644 --- a/recorder.conf +++ b/recorder.conf @@ -4,9 +4,16 @@ # and its associated utilities to override compiled-in defaults. OTR_TOPICS = "owntracks/#" + +# Binding on 0.0.0.0 will listen on IPv4 only +# Binding on [::] will listen on IPv4 and IPv6 +# OTR_HTTPHOST = "0.0.0.0" OTR_HTTPHOST = "[::]" + +# CA data for MQTT client # OTR_CAPATH = "/config" # OTR_CAFILE = "ca.pem" +# Server Certificate for builtin HTTPS server # OTR_CERTFILE = "cert.pem" # OTR_KEYFILE = "key.pem" From 5faedd1fa5928461f1b78c85399ed940f1927a75 Mon Sep 17 00:00:00 2001 From: Steffen Vogel Date: Mon, 16 May 2022 10:36:50 +0200 Subject: [PATCH 12/12] make shell scripts executable --- Dockerfile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Dockerfile b/Dockerfile index fd02c96..db6e265 100644 --- a/Dockerfile +++ b/Dockerfile @@ -44,6 +44,8 @@ COPY --from=builder /app / COPY recorder-health.sh /usr/sbin/recorder-health.sh COPY entrypoint.sh /usr/sbin/entrypoint.sh +RUN chmod +x /usr/sbin/*.sh + # If you absolutely need health-checking, enable the option below. Keep in # mind that until https://github.com/systemd/systemd/issues/6432 is resolved, # using the HEALTHCHECK feature will cause systemd to generate a significant