mirror of
https://github.com/prymitive/karma
synced 2026-05-07 03:26:52 +00:00
1005 lines
29 KiB
Markdown
1005 lines
29 KiB
Markdown
# Configuration options
|
|
|
|
## Config file
|
|
|
|
By default karma will try to read configuration file named `karma.yaml` from
|
|
current directory. Configuration file uses [YAML](http://yaml.org/) format and
|
|
it needs to have `.yaml` extension.
|
|
Custom filename and directory can be passed via command line flags or
|
|
environment variables:
|
|
|
|
- `--config.file` flag or `CONFIG_FILE` env variable - path to the config file
|
|
|
|
Example with flags:
|
|
|
|
```shell
|
|
karma --config.file docs/example.yaml
|
|
```
|
|
|
|
Example with environment variables:
|
|
|
|
```shell
|
|
CONFIG_FILE="docs/example.yaml"
|
|
```
|
|
|
|
### Alertmanagers
|
|
|
|
`alertmanager` section allows setting Alertmanager servers that should be
|
|
queried for alerts.
|
|
You can configure one or more Alertmanager servers, alerts
|
|
with identical label set will be deduplicated and labeled with each Alertmanager
|
|
server they were observed at. This allows using karma to collect alerts from a
|
|
pair of Alertmanager instances running in
|
|
[HA mode](https://prometheus.io/docs/alerting/alertmanager/#high-availability).
|
|
Syntax:
|
|
|
|
```YAML
|
|
alertmanager:
|
|
interval: duration
|
|
servers:
|
|
- name: string
|
|
uri: string
|
|
external_uri: string
|
|
timeout: duration
|
|
proxy: bool
|
|
tls:
|
|
ca: string
|
|
cert: string
|
|
key: string
|
|
insecureSkipVerify: bool
|
|
headers:
|
|
any: string
|
|
```
|
|
|
|
- `interval` - how often alerts should be refreshed, a string in
|
|
[time.Duration](https://golang.org/pkg/time/#ParseDuration) format. If set to
|
|
`1m` karma will query every Alertmanager server once a minute. This is global
|
|
setting applied to every Alertmanager server. All instances will be queried
|
|
in parallel.
|
|
Note that the maximum value for this option is `15m`.
|
|
The UI has a watchdog that tracks the timestamp of the last pull. If the UI
|
|
does not receive updates for more than 15 minutes it will print an error and
|
|
reload the page.
|
|
- `name` - name of this Alertmanager server, will be used as a label added to
|
|
every alert in the UI and for filtering alerts using `@alertmanager=NAME`
|
|
filter
|
|
- `uri` - base URI of this Alertmanager server. Supported URI schemes are
|
|
`http://` and `https://`.
|
|
If URI contains basic auth info
|
|
(`https://user:password@alertmanager.example.com`) and you don't want it to
|
|
be visible to users then ensure `proxy: true` is also set in order to avoid
|
|
leaking auth information to the browser.
|
|
Note: if URI contains username and password and proxy option is NOT enabled
|
|
(see below), then the username & password information will be stripped from
|
|
the URI and `Authorization` header using Basic Auth will be set for all
|
|
in browser requests.
|
|
- `external_uri` - this option allows to override base URI of this Alertmanager
|
|
used for browser links and also silence requests (but only when proxy mode is
|
|
not enabled).
|
|
- `timeout` - timeout for requests send to this Alertmanager server, a string in
|
|
[time.Duration](https://golang.org/pkg/time/#ParseDuration) format.
|
|
- `proxy` - if enabled requests from user browsers to this Alertmanager will be
|
|
proxied via karma. This applies to requests made when managing silences via
|
|
karma (creating or expiring silences).
|
|
THis option cannot be used when `external_uri` is set.
|
|
- `tls:ca` - path to CA certificate used to establish TLS connection to this
|
|
Alertmanager instance (for URIs using `https://` scheme). If unset or empty
|
|
string is set then Go will try to find system CA certificates using well known
|
|
paths.
|
|
- `tls:cert` - path to a TLS client certificate file to use when establishing
|
|
TLS connections to this Alertmanager instance if it requires a TLS client
|
|
authentication.
|
|
Note that this option requires `tls:key` to be also set.
|
|
- `tls:key` - path to a TLS client key file to use when establishing
|
|
TLS connections to this Alertmanager instance if it requires a TLS client
|
|
authentication.
|
|
Note that this option requires `tls:cert` to be also set.
|
|
- `tls:insecureSkipVerify` - disable server certificate validation, can be set
|
|
to allow using self-signed certs, use at your own risk
|
|
- `headers` - a map with a list of key: values which are header: value.
|
|
These custom headers will be sent with every request to the alert manager
|
|
instance.
|
|
|
|
Note: there are multiple supported combination of URI settings which result in
|
|
a slightly different behavior. Settings that control it are:
|
|
|
|
- `uri` - this option tells karma backend the URI that should be used to collect
|
|
all alerts and silence data from given Alertmanager instance. This setting is
|
|
required.
|
|
- `proxy` - this option when set to true enables karma backend to proxy all
|
|
silence management requests (creating, editing or deleting silences via karma
|
|
UI), so when the user creates a silence via karma UI the browser makes a
|
|
request to karma backend, the backend then forwards this request to the
|
|
Alertmanager using the value of `uri` option as the URI.
|
|
When this option is set to `false` all browser requests will use `uri` value.
|
|
This setting is optional, default value for it is `false`.
|
|
- `external_uri` - this option tells karma how the browser should connect to
|
|
given Alertmanager instance, it can be used for silence management requests
|
|
(creating, editing or deleting silences via karma UI) and how to generate
|
|
links to silences in Alertmanager web UI. Behavior of this option depends on
|
|
the value of `proxy` setting.
|
|
When proxy mode is enabled:
|
|
- silence management requests will use karma backend URI
|
|
- silence links to Alertmanager web UI will use `external_uri` value as base
|
|
URI
|
|
When proxy mode is disabled:
|
|
- silence management requests will use `external_uri` value as base URI
|
|
- silence links to Alertmanager web UI will use `external_uri` value as base
|
|
URI
|
|
|
|
Breakdown of all combination of settings:
|
|
|
|
1. Only `uri` is set:
|
|
|
|
```YAML
|
|
uri: http://localhost:123
|
|
```
|
|
|
|
Karma would use those URIs for:
|
|
|
|
| Backend | Silence management | Silence links |
|
|
|-|-|-|
|
|
| `http://localhost:123` | `http://localhost:123` | `http://localhost:123` |
|
|
|
|
1. Proxy mode is enabled:
|
|
|
|
```YAML
|
|
uri: http://localhost:123
|
|
proxy: true
|
|
```
|
|
|
|
Karma would use those URIs for:
|
|
|
|
| Backend | Silence management | Silence links |
|
|
|-|-|-|
|
|
| `http://localhost:123` | Karma internal URI | `http://localhost:123` |
|
|
|
|
1. `external_uri` is set, but proxy mode is disabled:
|
|
|
|
```YAML
|
|
uri: http://localhost:123
|
|
external_uri: http://example.com
|
|
```
|
|
|
|
Karma would use those URIs for:
|
|
|
|
| Backend | Silence management | Silence links |
|
|
|-|-|-|
|
|
| `http://localhost:123` | `http://example.com` | `http://example.com` |
|
|
|
|
1. Proxy mode is enabled and `external_uri` is set:
|
|
|
|
```YAML
|
|
uri: http://localhost:123
|
|
proxy: true
|
|
external_uri: http://example.com
|
|
```
|
|
|
|
Karma would use those URIs for:
|
|
|
|
| Backend | Silence management | Silence links |
|
|
|-|-|-|
|
|
| `http://localhost:123` | Karma internal URI | `http://example.com` |
|
|
|
|
Example with two production Alertmanager instances running in HA mode and a
|
|
staging instance that is also proxied and requires a custom auth header:
|
|
|
|
```YAML
|
|
alertmanager:
|
|
interval: 1m
|
|
servers:
|
|
- name: production1
|
|
uri: https://alertmanager1.prod.example.com
|
|
timeout: 20s
|
|
proxy: false
|
|
- name: production2
|
|
uri: https://alertmanager2.prod.example.com
|
|
timeout: 20s
|
|
proxy: false
|
|
- name: staging
|
|
uri: https://alertmanager.staging.example.com
|
|
timeout: 30s
|
|
proxy: true
|
|
tls:
|
|
ca: /etc/ssl/staging-ca.crt
|
|
headers:
|
|
X-Auth-Token: aValidToken
|
|
- name: protected
|
|
uri: https://alertmanager-auth.prod.example.com
|
|
timeout: 20s
|
|
tls:
|
|
cert: /etc/ssl/client.pem
|
|
key: /etc/ssl/client.key
|
|
- name: self-signed
|
|
uri: https://test.example.com
|
|
tls:
|
|
insecureSkipVerify: true
|
|
```
|
|
|
|
Defaults:
|
|
|
|
```YAML
|
|
alertmanager:
|
|
interval: 1m
|
|
servers: []
|
|
```
|
|
|
|
There is no default for `alertmanager.servers` and it's a required option for
|
|
setting multiple Alertmanager servers. For cases where only a single server
|
|
needs to be configured without a config file see
|
|
[Simplified Configuration](#simplified-configuration).
|
|
|
|
### Alert acknowledgement
|
|
|
|
Prometheus Alertmanager allows alerts to be in 3 states:
|
|
|
|
- `active` - when alert is firing
|
|
- `suppressed` - when alert is either silenced by a
|
|
[silence rule](https://prometheus.io/docs/alerting/alertmanager/#silences) or
|
|
inhibited by another alert using
|
|
[inhibition rules](https://prometheus.io/docs/alerting/alertmanager/#inhibition)
|
|
- `unprocessed` - initial state for new alerts before they are checked against
|
|
all silence rules so Alertmanager doesn't yet know if the alert should be
|
|
`active` or `supported`
|
|
|
|
A silence rule can be used to mark an alert as acknowledged and being worked on.
|
|
To simplify creating of such silences karma provides a one click button that
|
|
will create a silence matching alert group it was clicked for.
|
|
`alertAcknowledgement` allows to enable this feature and customize it's
|
|
configuration.
|
|
Syntax:
|
|
|
|
```YAML
|
|
alertAcknowledgement:
|
|
enabled: bool
|
|
duration: duration
|
|
author: string
|
|
commentPrefix: string
|
|
```
|
|
|
|
- `enabled` - setting it to true will enable creation of short lived
|
|
acknowledgement silences.
|
|
- `duration` - duration for acknowledgement silences, value is a string in
|
|
[time.Duration](https://golang.org/pkg/time/#ParseDuration) format.
|
|
- `author` - default author for acknowledgement silences. If user set the
|
|
author field on the silence form then that value will be used instead.
|
|
- `commentPrefix` - a string that will be added as a prefix to autogenerated
|
|
silence comment (optional).
|
|
|
|
Defaults:
|
|
|
|
```YAML
|
|
alertAcknowledgement:
|
|
enabled: false
|
|
duration: 15m0s
|
|
author: karma
|
|
commentPrefix: ACK!
|
|
```
|
|
|
|
A common problem is setting a correct duration for the silence.
|
|
If set for too short it can expire before the issue is resolved, and will
|
|
require re-silencing all the alerts.
|
|
If set for too long it mask the same problem reoccurring in the future. This
|
|
requires user to expire the silence once the issue is resolved.
|
|
|
|
[kthxbye](https://github.com/prymitive/kthxbye) is a tiny daemon that can help
|
|
with managing short lived acknowledged silences. It will continuously extend
|
|
short lived acknowledgement silences if there are alerts firing against those
|
|
silences, which means that the user doesn't need to worry about setting proper
|
|
duration for such silences.
|
|
To use it run an instance of kthxbye with every alertmanager instance or
|
|
cluster and configure it to use the same comment prefix as `commentPrefix`.
|
|
With this setup when user clicks to acknowledge an alert karma will create
|
|
a short lived silence and kthxbye will keep that silence in Alertmanager
|
|
until there are no alerts matching it, meaning that the issue was resolved.
|
|
|
|
### Annotations
|
|
|
|
`annotations` section allows configuring how alert annotation are displayed in
|
|
the UI.
|
|
Syntax:
|
|
|
|
```YAML
|
|
annotations:
|
|
default:
|
|
hidden: bool
|
|
hidden: list of strings
|
|
visible: list of strings
|
|
keep: list of strings
|
|
strip: list of strings
|
|
```
|
|
|
|
- `default:hidden` - bool, true if all annotations should be hidden by default.
|
|
- `hidden` - list of annotations that should be hidden by default.
|
|
- `visible` - list of annotations that should be visible by default when
|
|
`default:hidden` is set to `true`.
|
|
- `keep` - list of allowed annotations, if empty all annotations are allowed.
|
|
- `strip` - list of ignored annotations.
|
|
|
|
The difference between `hidden`/`visible` and `keep`/`strip` is that hidden
|
|
annotations are still accessible, but they are shown in the UI collapsed by
|
|
default (only name is visible, value is shown after clicking), while stripped
|
|
annotations are removed entirely and never presented to the user.
|
|
|
|
Example where all annotations except `summary` are hidden by default. If there
|
|
are additional annotation keys user will need to click on the `+` icon to see
|
|
them.
|
|
|
|
```YAML
|
|
annotations:
|
|
default:
|
|
hidden: true
|
|
hidden: []
|
|
visible:
|
|
- summary
|
|
keep: []
|
|
strip:
|
|
- help
|
|
- verylong
|
|
```
|
|
|
|
Example where all annotations except `details` are visible by default. If
|
|
`details` annotation is present on any alert user will need to click on the `+`
|
|
icon to see it. Additionally `secret` annotation is stripped and never shown
|
|
in the UI.
|
|
|
|
```YAML
|
|
annotations:
|
|
default:
|
|
hidden: false
|
|
hidden:
|
|
- details
|
|
visible: []
|
|
keep: []
|
|
strip:
|
|
- secret
|
|
```
|
|
|
|
Defaults:
|
|
|
|
```YAML
|
|
annotations:
|
|
default:
|
|
hidden: false
|
|
hidden: []
|
|
visible: []
|
|
```
|
|
|
|
### Filters
|
|
|
|
`filters` section allows configuring default set of filters used in the UI.
|
|
|
|
Syntax:
|
|
|
|
```YAML
|
|
filters:
|
|
default: list of strings
|
|
```
|
|
|
|
- `default` - list of filters to use by default when user navigates to karma
|
|
web UI. Visit `/help` page in karma for details on available filters.
|
|
Note that if a string starts with `@` YAML requires to wrap it in quotes.
|
|
|
|
Example:
|
|
|
|
```YAML
|
|
filters:
|
|
default:
|
|
- "@state=active"
|
|
- severity=critical
|
|
```
|
|
|
|
Defaults:
|
|
|
|
```YAML
|
|
filters:
|
|
default: []
|
|
```
|
|
|
|
### Grid
|
|
|
|
`grid` section allows customizing how alert grid is rendered in the UI.
|
|
Sorting configuration can be overridden by each user via UI settings.
|
|
Syntax:
|
|
|
|
```YAML
|
|
grid:
|
|
sorting:
|
|
order: string
|
|
reverse: bool
|
|
label: string
|
|
customValues:
|
|
labels: dict
|
|
```
|
|
|
|
- `sorting:order` - default sort order for alert grid, valid values are:
|
|
- `disabled` - no sorting, alert groups are rendered in the order they are
|
|
returned by the API
|
|
- `startsAt` - sort by alert timestamps, most recent alert in each group will
|
|
be used when comparing each group
|
|
- `label` - sort by labels, if the label used for sorting is not shared by
|
|
all alerts in a group then the first alert in the group will be queried for
|
|
it
|
|
- `sorting:reverse` - default value for reversed sort order
|
|
- `sorting:label` - label name for sorting when `grid:sorting:order` is set
|
|
to `label`. Labels can be assigned custom values used only by sorting via
|
|
`sorting:customValues:labels`.
|
|
- `sorting:customValues:labels` - when sorting using alert labels values are
|
|
compared as strings, which work for labels like `cluster=A`, `cluster=B` &
|
|
`cluster=C`, but not for `cluster=prod`, `cluster=staging` & `cluster=dev`.
|
|
Alphabetic sort would order the second case as follows: `dev`, `prod`,
|
|
`staging`. To allow for more natural sorting `sorting:valueMapping` can be
|
|
used to map label values to integer values which will be used for sorting
|
|
instead of original string values.
|
|
Note: this option is not available via environment variables, you can only set
|
|
it via the config file.
|
|
|
|
Defaults:
|
|
|
|
```YAML
|
|
grid:
|
|
sorting:
|
|
order: startsAt
|
|
reverse: true
|
|
label: alertname
|
|
customValues:
|
|
labels: {}
|
|
```
|
|
|
|
Example with sorting using `severity` label and value mappings for it:
|
|
|
|
```YAML
|
|
grid:
|
|
sorting:
|
|
order: label
|
|
reverse: false
|
|
label: severity
|
|
customValues:
|
|
labels:
|
|
severity:
|
|
critical: 1
|
|
warning: 2
|
|
info: 3
|
|
```
|
|
|
|
### Karma
|
|
|
|
`karma` section allows configuring miscellaneous internal options.
|
|
|
|
Syntax:
|
|
|
|
```YAML
|
|
karma:
|
|
name: string
|
|
```
|
|
|
|
- `name` - name of given karma instance, this is currently used for the browser
|
|
tab title.
|
|
|
|
Defaults:
|
|
|
|
```YAML
|
|
karma:
|
|
name: karma
|
|
```
|
|
|
|
### Labels
|
|
|
|
`labels` section allows configuring how alert labels will be rendered in the
|
|
UI.
|
|
All labels will be parsed when collecting alerts from Alertmanager API and
|
|
used when deduplicating alerts, but some labels aren't useful to users and so
|
|
can be removed from the UI, this is controlled by `keep` and `strip` options.
|
|
`colors` section allows configuring which labels should have colors applied
|
|
to label background in the UI. Colors can help visually identify alerts
|
|
with shared labels, for example coloring hostname label will allow to quickly
|
|
spot all alerts for the same host.
|
|
Syntax:
|
|
|
|
```YAML
|
|
labels:
|
|
color:
|
|
static: list of strings
|
|
unique: list of strings
|
|
custom:
|
|
foo:
|
|
- value: string
|
|
value_re: string
|
|
color: string
|
|
keep: list of strings
|
|
strip: list of strings
|
|
```
|
|
|
|
- `color:static` - list of label names that will all have the same color applied
|
|
(different than the default label color). This allows to quickly spot a
|
|
specific label that can have high range of values, but it's important when
|
|
reading the dashboard. For example coloring the instance label allows to
|
|
quickly learn which instance is affected by given alert.
|
|
- `color:unique` - list of label names that should have unique colors generated
|
|
in the UI.
|
|
- `color:custom` - nested map of label names and value with colors - this allows
|
|
to configure a set of labels with custom predefined colors applied to them
|
|
rather than generated. Value is a mapping with `label name` ->
|
|
`list of dicts`, each dict object allows setting:
|
|
|
|
- `value` - the exact value of the label to match against
|
|
- `value_re` - Go compatible
|
|
[regular expression](https://golang.org/pkg/regexp/) to match against
|
|
- `color`: color to apply if either `value` or `value_re` matches
|
|
|
|
Either `value` or `value_re` is required, both can be set in which case
|
|
`value` with be tested first. Entries are evaluated in the order they appear
|
|
in the config file.
|
|
Note: this option is not available via environment variables, you can only set
|
|
it via the config file.
|
|
|
|
- `keep` - list of allowed labels, if empty all labels are allowed.
|
|
- `strip` - list of ignored labels.
|
|
|
|
Example with static color for the `job` label (every `job` label will have the
|
|
same color regardless of the value) and unique color for the `@receiver` label
|
|
(every `@receiver` label will have color unique for each value).
|
|
|
|
```YAML
|
|
labels:
|
|
color:
|
|
static:
|
|
- job
|
|
unique:
|
|
- "@receiver"
|
|
```
|
|
|
|
Example where `task_id` label is ignored by karma:
|
|
|
|
```YAML
|
|
labels:
|
|
keep: []
|
|
strip:
|
|
- task_id
|
|
```
|
|
|
|
Example where all but `instance` and `alertname` labels are allowed:
|
|
|
|
```YAML
|
|
labels:
|
|
keep:
|
|
- alertname
|
|
- instance
|
|
strip: []
|
|
```
|
|
|
|
Example where `severity` label will have a red color for `critical`, yellow
|
|
for `warning` and blue for `info`:
|
|
|
|
```YAML
|
|
labels:
|
|
color:
|
|
custom:
|
|
"@alertmanager":
|
|
- value: prod
|
|
color: "#e6e"
|
|
severity:
|
|
- value: info
|
|
color: "#87c4e0"
|
|
- value: warning
|
|
color: "#ffae42"
|
|
- value: critical
|
|
color: "#ff220c"
|
|
```
|
|
|
|
Example with a regex value, `info`, `warning` and `critical` will get colors
|
|
as below, but any value not matching those 3 values will use the color from
|
|
`.*`:
|
|
|
|
```YAML
|
|
labels:
|
|
color:
|
|
custom:
|
|
severity:
|
|
- value: info
|
|
color: "#87c4e0"
|
|
- value: warning
|
|
color: "#ffae42"
|
|
- value: critical
|
|
color: "#ff220c"
|
|
- value_re: ".*"
|
|
color: "#736598"
|
|
```
|
|
|
|
Note: be sure to set fallback values at the end of the list, so they're only
|
|
evaluated if there's no exact value match
|
|
|
|
Defaults:
|
|
|
|
```YAML
|
|
labels:
|
|
color:
|
|
static: []
|
|
unique: []
|
|
custom: {}
|
|
keep: []
|
|
strip: []
|
|
```
|
|
|
|
### Listen
|
|
|
|
`listen` section allows configuring karma web server behavior.
|
|
Syntax:
|
|
|
|
```YAML
|
|
listen:
|
|
address: string
|
|
port: integer
|
|
prefix: string
|
|
```
|
|
|
|
- `address` - Hostname or IP to listen on.
|
|
- `port` - HTTP port to listen on.
|
|
- `prefix` - URL root for karma, you can use to if you wish to serve it from
|
|
location other than `/`. This option is mostly useful when using karma behind
|
|
reverse proxy with other services on the same IP but different URL root.
|
|
|
|
Example where karma would listen for HTTP requests on `http://1.2.3.4:80/karma/`
|
|
|
|
```YAML
|
|
listen:
|
|
address: 1.2.3.4
|
|
port: 80
|
|
prefix: /karma/
|
|
```
|
|
|
|
Defaults:
|
|
|
|
```YAML
|
|
listen:
|
|
address: "0.0.0.0"
|
|
port: 8080
|
|
prefix: /
|
|
```
|
|
|
|
### Log
|
|
|
|
`log` section allows configuring logging subsystem.
|
|
Syntax:
|
|
|
|
```YAML
|
|
log:
|
|
config: bool
|
|
level: string
|
|
format: string
|
|
```
|
|
|
|
- `config` - if set to `true` karma will log used configuration on startup
|
|
- `level` - log level to set for karma, possible values are `debug`, `info`,
|
|
`warning`, `error`, `fatal` and `panic`.
|
|
- `format` - controls how log messages are formatted, possible values are
|
|
`text` and `json`. If set to `json` each log will be a JSON object.
|
|
|
|
Defaults:
|
|
|
|
```YAML
|
|
log:
|
|
config: true
|
|
level: info
|
|
format: text
|
|
```
|
|
|
|
### Silences
|
|
|
|
`silences` section allows specifying to configure silence post post-processing.
|
|
Syntax:
|
|
|
|
```YAML
|
|
silences:
|
|
comments:
|
|
linkDetect:
|
|
rules: list of link detection rules
|
|
```
|
|
|
|
- `comments:linkDetect:rules` allows to specify a list of rules to detect links
|
|
inside silence comments. It's intended to find ticket system ID strings and
|
|
turn them into links.
|
|
Each rule must specify:
|
|
- `regex` - regular expression that matches ticket system IDs. Each regex must
|
|
contain at least one capture group `(regex)`.
|
|
- `uriTemplate` - template string that will be used to generate a link.
|
|
Each template must include `$1` which will be replaced with text matched
|
|
by the `regex`.
|
|
|
|
Example where a string `DEVOPS-123` inside a comment would be rendered as a link
|
|
to a JIRA ticket `https://jira.example.com/browse/DEVOPS-123`.
|
|
|
|
```YAML
|
|
silences:
|
|
comments:
|
|
linkDetect:
|
|
rules:
|
|
- regex: "(DEVOPS-[0-9]+)"
|
|
uriTemplate: https://jira.example.com/browse/$1
|
|
```
|
|
|
|
Defaults:
|
|
|
|
```YAML
|
|
jira: []
|
|
```
|
|
|
|
### Receivers
|
|
|
|
`receivers` section allows configuring how alerts from different receivers are
|
|
handled by karma. If alerts are routed to multiple receivers they can be
|
|
duplicated in the UI, each instance will have different value for `@receiver`.
|
|
Syntax:
|
|
|
|
```YAML
|
|
receivers:
|
|
keep: list of strings
|
|
strip: list of strings
|
|
```
|
|
|
|
- `keep` - list of receivers name that are allowed, if empty all receivers are
|
|
allowed.
|
|
- `strip` - list of receiver names that will not be shown in the UI.
|
|
|
|
Example where alerts that are routed to the `alertmanage2es` receiver are
|
|
ignored by karma.
|
|
|
|
```YAML
|
|
receivers:
|
|
strip:
|
|
- alertmanage2es
|
|
```
|
|
|
|
Defaults:
|
|
|
|
```YAML
|
|
receivers:
|
|
strip: []
|
|
```
|
|
|
|
### Sentry
|
|
|
|
`sentry` section allows configuring [Sentry](https://sentry.io) integration. See
|
|
[Sentry documentation](https://docs.sentry.io/quickstart/#configure-the-dsn) for
|
|
details.
|
|
Syntax:
|
|
|
|
```YAML
|
|
sentry:
|
|
private: string
|
|
public: string
|
|
```
|
|
|
|
- `private` - Sentry DSN for Go exceptions, this value is only used by karma
|
|
binary and never exposed to the user.
|
|
- `public` - Sentry DSN for JavaScript exceptions, this value will be exposed
|
|
to the user browser.
|
|
|
|
Example:
|
|
|
|
```YAML
|
|
sentry:
|
|
private: https://<key>:<secret>@sentry.io/<project>
|
|
public: https://<key>:<secret>@sentry.io/<project>
|
|
```
|
|
|
|
## Silence form
|
|
|
|
`silenceForm` section allows customizing silence form behavior.
|
|
`author:populate_from_header` subsection allows to configure fetching of author
|
|
name used on the silence form from the request header. It can be used with
|
|
setups where karma is deployed behind authentication proxy that adds some extra
|
|
headers with username for all requests received by karma.
|
|
|
|
Syntax:
|
|
|
|
```YAML
|
|
silenceForm:
|
|
author:
|
|
populate_from_header:
|
|
header: string
|
|
value_re: string
|
|
strip:
|
|
labels: list of strings
|
|
```
|
|
|
|
- `author:populate_from_header:header` - name of the header to read the username
|
|
from
|
|
- `author:populate_from_header:value_re` -
|
|
[regex](https://golang.org/s/re2syntax) used to extract the username from the
|
|
request header. It must include one numbered capturing group, whatever is
|
|
matched by that group will be used as the silence form author field. Both
|
|
`header` and `value_re` must be set for this feature to work.
|
|
- `strip:labels` - list of labels to ignore when populating silence form from
|
|
individual alerts or group of alerts. This allows to create silences matching
|
|
only unique labels, like `instance` or `host`, ignoring any common labels like
|
|
`job`.
|
|
|
|
Example where `job` label won't be auto populated onto the silence form and
|
|
where the `X-Auth` header with value `User foobar` will set the default silence
|
|
author to `foobar`.
|
|
|
|
```YAML
|
|
silenceForm:
|
|
author:
|
|
populate_from_header:
|
|
header: X-Auth
|
|
value_re: ^User (.+)$
|
|
strip:
|
|
labels:
|
|
- job
|
|
```
|
|
|
|
## UI defaults
|
|
|
|
`ui` section allows configuring default values for UI settings controled via the
|
|
configuration modal. Those defaults can be overwritten by use via UI controls.
|
|
|
|
Syntax:
|
|
|
|
```YAML
|
|
ui:
|
|
refresh: duration
|
|
hideFiltersWhenIdle: bool
|
|
colorTitlebar: bool
|
|
darkTheme: bool
|
|
minimalGroupWidth: integer
|
|
alertsPerGroup: integer
|
|
collapseGroups: string
|
|
```
|
|
|
|
- `refresh` - default refresh interval, this tells the UI how often karma API
|
|
should be queried
|
|
- `hideFiltersWhenIdle` - if enabled filter bar will be hidden after some
|
|
user inactivity
|
|
- `colorTitlebar` - if enabled alert group title bar color will be set to follow
|
|
alerts in that group
|
|
- `darkTheme` - if enabled dark mode will be enabled.
|
|
Note: dark mode is *experimental* and might be buggy.
|
|
- `minimalGroupWidth` - minimal width (in pixels) for each alert group rendered
|
|
on the grid. This value is used to calculate the number of columns rendered on
|
|
the grid.
|
|
- `alertsPerGroup` - default number of alerts to show for each group
|
|
- `collapseGroups` - controls if alert groups will default to being rendered
|
|
expanded or collapsed (only title bar is visible). Valid values:
|
|
- expanded - groups are always expanded
|
|
- collapsed - groups are always collapsed
|
|
- collapsedOnMobile - groups are expanded on desktop and collapsed on mobile
|
|
browsers
|
|
|
|
Defaults:
|
|
|
|
```YAML
|
|
ui:
|
|
refresh: 30s
|
|
hideFiltersWhenIdle: true
|
|
colorTitlebar: false
|
|
darkTheme: false
|
|
minimalGroupWidth: 420
|
|
alertsPerGroup: 5
|
|
collapseGroups: collapsedOnMobile
|
|
```
|
|
|
|
## Customizing karma
|
|
|
|
In order to keep the core code simple karma doesn't support any way of extending
|
|
provided functionality. There is however possibility to inject custom CSS &
|
|
JavaScript code, which can be used to either override built in CSS styles
|
|
or integrate with extra services, for example using error handlers other than
|
|
Sentry.
|
|
|
|
```YAML
|
|
custom:
|
|
css: string
|
|
js: string
|
|
```
|
|
|
|
- `css` - path to a CSS file
|
|
- `js` - path to JavaScript file
|
|
|
|
Example:
|
|
|
|
```YAML
|
|
custom:
|
|
css: /theme/custom.css
|
|
js: /assets/custom.js
|
|
```
|
|
|
|
Use at your own risk and be aware that used CSS class names might change without
|
|
warning. This feature is provided as is without any guarantees.
|
|
|
|
## Command line flags
|
|
|
|
Config file options are mapped to command line flags, so `alertmanager:interval`
|
|
config file key is accessible as `--alertmanager.interval` flag, run
|
|
`karma --help` to see a full list.
|
|
Exceptions for passing flags:
|
|
|
|
- `jira` - this option is a list of maps and it's only available when using
|
|
config file.
|
|
|
|
There's no support for configuring multiple Alertmanager servers using
|
|
flags, but it's possible to configure a single Alertmanager instance this way,
|
|
see the [Simplified Configuration](#simplified-configuration) section.
|
|
|
|
## Environment variables
|
|
|
|
Environment variables are mapped in a similar way as command line flags,
|
|
`alertmanager:interval` is accessible as `ALERTMANAGER_INTERVAL` env.
|
|
Exceptions for passing flags:
|
|
|
|
- `HOST` - used by gin webserver, same effect as setting `listen:address` config
|
|
option
|
|
- `PORT` - used by gin webserver, same effect as setting `listen:port` config
|
|
option
|
|
- `SENTRY_DSN` - is used by Sentry itself, same effect as passing value to
|
|
`sentry:private` config option.
|
|
|
|
There's no support for configuring multiple alertmanager servers using
|
|
environment variables, but it's possible to configure a single Alertmanager
|
|
instance this way, see the [Simplified Configuration](#simplified-configuration)
|
|
section.
|
|
|
|
## Simplified Configuration
|
|
|
|
To configure multiple Alertmanager instances karma requires a config file, but
|
|
for a single Alertmanager instance cases it's possible to configure all
|
|
Alertmanager server options that are set for `alertmanager.servers` config
|
|
section using only flags or environment variables.
|
|
|
|
### Alertmanager URI
|
|
|
|
To set the `uri` key from `alertmanager.servers` map `ALERTMANAGER_URI` env or
|
|
`--alertmanager.uri` flag can be used.
|
|
Examples:
|
|
|
|
```shell
|
|
ALERTMANAGER_URI=https://alertmanager.example.com karma
|
|
karma --alertmanager.uri https://alertmanager.example.com
|
|
```
|
|
|
|
### Alertmanager external URI
|
|
|
|
To set the `external_uri` key from `alertmanager.servers` map
|
|
`ALERTMANAGER_EXTERNAL_URI` env or `--alertmanager.external_uri` flag can be
|
|
used.
|
|
Examples:
|
|
|
|
```shell
|
|
ALERTMANAGER_EXTERNAL_URI=https://alertmanager.example.com karma
|
|
karma --alertmanager.external_uri https://alertmanager.example.com
|
|
```
|
|
|
|
### Alertmanager name
|
|
|
|
To set the `name` key from `alertmanager.servers` map `ALERTMANAGER_NAME` env or
|
|
`--alertmanager.name` flag can be used.
|
|
Examples:
|
|
|
|
```shell
|
|
ALERTMANAGER_NAME=single karma
|
|
karma --alertmanager.name single
|
|
```
|
|
|
|
### Alertmanager timeout
|
|
|
|
To set the `timeout` key from `alertmanager.servers` map `ALERTMANAGER_TIMEOUT`
|
|
env or `--alertmanager.timeout` flag can be used.
|
|
Examples:
|
|
|
|
```shell
|
|
ALERTMANAGER_TIMEOUT=10s karma
|
|
karma --alertmanager.timeout 10s
|
|
```
|
|
|
|
### Alertmanager request proxy
|
|
|
|
To set the `proxy` key from `alertmanager.servers` map `ALERTMANAGER_PROXY`
|
|
env or `--alertmanager.proxy` flag can be used.
|
|
Examples:
|
|
|
|
```shell
|
|
ALERTMANAGER_PROXY=true karma
|
|
karma --alertmanager.proxy
|
|
```
|