mirror of
https://github.com/prymitive/karma
synced 2026-05-07 03:26:52 +00:00
feat(ui): allow acknowledging alerts using short lived silences
This commit is contained in:
@@ -54,6 +54,13 @@ all current alerts.
|
||||
|
||||

|
||||
|
||||
Starting with `v0.50` karma can create short lived silences to acknowledge
|
||||
alerts with a single button click. To create silences that will resolve itself
|
||||
only after all alerts are resolved you can use
|
||||
[kthxbye](https://github.com/prymitive/kthxbye).
|
||||
See [configuration docs](/docs/CONFIGURATION.md#alert-acknowledgement) for
|
||||
details.
|
||||
|
||||
[Online demo](https://karma-demo.herokuapp.com/)
|
||||
|
||||
To get notifications about new karma releases go to
|
||||
|
||||
@@ -266,7 +266,7 @@ class SilencedAlert(AlertGenerator):
|
||||
[newMatcher("alertname", self.name, False)],
|
||||
"{}Z".format(now.isoformat()),
|
||||
"{}Z".format((now + datetime.timedelta(
|
||||
minutes=random.randint(0, 60))).isoformat()),
|
||||
minutes=random.randint(1, 60))).isoformat()),
|
||||
"me@example.com",
|
||||
"This alert is always silenced and the silence comment is very "
|
||||
"long to test the UI. Lorem ipsum dolor sit amet, consectetur "
|
||||
@@ -304,7 +304,7 @@ class MixedAlerts(AlertGenerator):
|
||||
newMatcher("instance", "server(1|3|5|7)", True)],
|
||||
"{}Z".format(now.isoformat()),
|
||||
"{}Z".format((now + datetime.timedelta(
|
||||
minutes=random.randint(0, 30))).isoformat()),
|
||||
minutes=random.randint(1, 30))).isoformat()),
|
||||
"me@example.com",
|
||||
"Silence '{}''".format(self.name)
|
||||
)
|
||||
@@ -421,7 +421,7 @@ class PaginationTest(AlertGenerator):
|
||||
],
|
||||
"{}Z".format(now.isoformat()),
|
||||
"{}Z".format((now + datetime.timedelta(
|
||||
minutes=random.randint(0, 30))).isoformat()),
|
||||
minutes=random.randint(1, 30))).isoformat()),
|
||||
"me@example.com",
|
||||
"DEVOPS-123 Pagination Test alert silenced with a long text "
|
||||
"to see if it gets truncated properly. It only matches first "
|
||||
|
||||
@@ -9,6 +9,11 @@ alertmanager:
|
||||
uri: "http://localhost:9094"
|
||||
timeout: 10s
|
||||
proxy: true
|
||||
alertAcknowledgement:
|
||||
enabled: true
|
||||
duration: 15m0s
|
||||
author: karma-ack
|
||||
commentPrefix: ACK!
|
||||
annotations:
|
||||
hidden:
|
||||
- help
|
||||
|
||||
@@ -229,6 +229,70 @@ setting multiple Alertmanager servers. For cases where only a single server
|
||||
needs to be configured without a config file see
|
||||
[Simplified Configuration](#simplified-configuration).
|
||||
|
||||
### Alert acknowledgement
|
||||
|
||||
Prometheus Alertmanager allows alerts to be in 3 states:
|
||||
|
||||
- `active` - when alert is firing
|
||||
- `suppressed` - when alert is either silenced by a
|
||||
[silence rule](https://prometheus.io/docs/alerting/alertmanager/#silences) or
|
||||
inhibited by another alert using
|
||||
[inhibition rules](https://prometheus.io/docs/alerting/alertmanager/#inhibition)
|
||||
- `unprocessed` - initial state for new alerts before they are checked against
|
||||
all silence rules so Alertmanager doesn't yet know if the alert should be
|
||||
`active` or `supported`
|
||||
|
||||
A silence rule can be used to mark an alert as acknowledged and being worked on.
|
||||
To simplify creating of such silences karma provides a one click button that
|
||||
will create a silence matching alert group it was clicked for.
|
||||
`alertAcknowledgement` allows to enable this feature and customize it's
|
||||
configuration.
|
||||
Syntax:
|
||||
|
||||
```YAML
|
||||
alertAcknowledgement:
|
||||
enabled: bool
|
||||
duration: duration
|
||||
author: string
|
||||
commentPrefix: string
|
||||
```
|
||||
|
||||
- `enabled` - setting it to true will enable creation of short lived
|
||||
acknowledgement silences.
|
||||
- `duration` - duration for acknowledgement silences, value is a string in
|
||||
[time.Duration](https://golang.org/pkg/time/#ParseDuration) format.
|
||||
- `author` - default author for acknowledgement silences. If user set the
|
||||
author field on the silence form then that value will be used instead.
|
||||
- `commentPrefix` - a string that will be added as a prefix to autogenerated
|
||||
silence comment (optional).
|
||||
|
||||
Defaults:
|
||||
|
||||
```YAML
|
||||
alertAcknowledgement:
|
||||
enabled: false
|
||||
duration: 15m0s
|
||||
author: karma
|
||||
commentPrefix: ACK!
|
||||
```
|
||||
|
||||
A common problem is setting a correct duration for the silence.
|
||||
If set for too short it can expire before the issue is resolved, and will
|
||||
require re-silencing all the alerts.
|
||||
If set for too long it mask the same problem reoccurring in the future. This
|
||||
requires user to expire the silence once the issue is resolved.
|
||||
|
||||
[kthxbye](https://github.com/prymitive/kthxbye) is a tiny daemon that can help
|
||||
with managing short lived acknowledged silences. It will continuously extend
|
||||
short lived acknowledgement silences if there are alerts firing against those
|
||||
silences, which means that the user doesn't need to worry about setting proper
|
||||
duration for such silences.
|
||||
To use it run an instance of kthxbye with every alertmanager instance or
|
||||
cluster and configure it to use the same comment prefix as `commentPrefix`.
|
||||
With this setup when user clicks to acknowledge an alert karma will create
|
||||
a short lived silence and kthxbye will keep that silence in Alertmanager
|
||||
until there are no alerts matching it, meaning that the issue was resolved.
|
||||
|
||||
### Annotations
|
||||
|
||||
`annotations` section allows configuring how alert annotation are displayed in
|
||||
|
||||
255
ui/src/Components/AlertAck/index.js
Normal file
255
ui/src/Components/AlertAck/index.js
Normal file
@@ -0,0 +1,255 @@
|
||||
import React, { Component } from "react";
|
||||
import PropTypes from "prop-types";
|
||||
|
||||
import { observable, action, computed, toJS } from "mobx";
|
||||
import { observer } from "mobx-react";
|
||||
|
||||
import moment from "moment";
|
||||
|
||||
import semver from "semver";
|
||||
|
||||
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
|
||||
import { faCheck } from "@fortawesome/free-solid-svg-icons/faCheck";
|
||||
import { faCheckCircle } from "@fortawesome/free-solid-svg-icons/faCheckCircle";
|
||||
import { faSpinner } from "@fortawesome/free-solid-svg-icons/faSpinner";
|
||||
import { faExclamationCircle } from "@fortawesome/free-solid-svg-icons/faExclamationCircle";
|
||||
|
||||
import { APIGroup } from "Models/API";
|
||||
import { AlertStore } from "Stores/AlertStore";
|
||||
import {
|
||||
SilenceFormStore,
|
||||
MatchersFromGroup,
|
||||
GenerateAlertmanagerSilenceData
|
||||
} from "Stores/SilenceFormStore";
|
||||
import { FetchWithCredentials } from "Common/Fetch";
|
||||
import { TooltipWrapper } from "Components/TooltipWrapper";
|
||||
|
||||
const SubmitState = Object.freeze({
|
||||
Idle: "Idle",
|
||||
InProgress: "InProgress",
|
||||
Done: "Done",
|
||||
Failed: "Failed"
|
||||
});
|
||||
|
||||
const newPendingSilence = (
|
||||
group,
|
||||
members,
|
||||
durationSeconds,
|
||||
author,
|
||||
commentPrefix
|
||||
) => ({
|
||||
payload: GenerateAlertmanagerSilenceData(
|
||||
moment.utc(),
|
||||
moment.utc().add(durationSeconds, "seconds"),
|
||||
MatchersFromGroup(group, []),
|
||||
author,
|
||||
`${
|
||||
commentPrefix ? commentPrefix + " " : ""
|
||||
}This alert was acknowledged using karma on ${moment.utc().toString()}`
|
||||
),
|
||||
membersToTry: members,
|
||||
submitState: SubmitState.Idle,
|
||||
submitResult: null,
|
||||
isDone: false,
|
||||
isFailed: false,
|
||||
fetch: null
|
||||
});
|
||||
|
||||
const AlertAck = observer(
|
||||
class AlertAck extends Component {
|
||||
static propTypes = {
|
||||
alertStore: PropTypes.instanceOf(AlertStore).isRequired,
|
||||
silenceFormStore: PropTypes.instanceOf(SilenceFormStore).isRequired,
|
||||
group: APIGroup.isRequired
|
||||
};
|
||||
|
||||
constructor(props) {
|
||||
super(props);
|
||||
this.submitState = observable(
|
||||
{
|
||||
silencesByCluster: {},
|
||||
reset() {
|
||||
this.silencesByCluster = {};
|
||||
},
|
||||
pushSilence(cluster, silence) {
|
||||
this.silencesByCluster[cluster] = silence;
|
||||
},
|
||||
markDone(cluster) {
|
||||
this.silencesByCluster[cluster].isDone = true;
|
||||
},
|
||||
markFailed(cluster) {
|
||||
this.silencesByCluster[cluster].isDone = true;
|
||||
this.silencesByCluster[cluster].isFailed = true;
|
||||
},
|
||||
get isIdle() {
|
||||
return Object.keys(this.silencesByCluster).length === 0;
|
||||
},
|
||||
get isInprogress() {
|
||||
return (
|
||||
Object.values(this.silencesByCluster).filter(
|
||||
pendingSilence => pendingSilence.isDone === false
|
||||
).length > 0
|
||||
);
|
||||
},
|
||||
get isDone() {
|
||||
return (
|
||||
Object.values(this.silencesByCluster).filter(
|
||||
pendingSilence => pendingSilence.isDone === true
|
||||
).length > 0
|
||||
);
|
||||
},
|
||||
get isFailed() {
|
||||
return (
|
||||
Object.values(this.silencesByCluster).filter(
|
||||
pendingSilence => pendingSilence.isFailed === true
|
||||
).length > 0
|
||||
);
|
||||
}
|
||||
},
|
||||
{
|
||||
reset: action.bound,
|
||||
pushSilence: action.bound,
|
||||
markDone: action.bound,
|
||||
markFailed: action.bound,
|
||||
isIdle: computed,
|
||||
isInprogress: computed,
|
||||
isDone: computed,
|
||||
isFailed: computed
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
maybeTryAgainAfterError = cluster => {
|
||||
if (this.submitState.silencesByCluster[cluster].membersToTry.length) {
|
||||
this.handleAlertmanagerRequest(cluster);
|
||||
} else {
|
||||
this.submitState.markFailed(cluster);
|
||||
}
|
||||
};
|
||||
|
||||
handleAlertmanagerRequest = cluster => {
|
||||
const { alertStore } = this.props;
|
||||
|
||||
const member = this.submitState.silencesByCluster[
|
||||
cluster
|
||||
].membersToTry.pop();
|
||||
|
||||
const am = alertStore.data.getAlertmanagerByName(member);
|
||||
if (am === undefined) {
|
||||
const err = `Alertmanager instance "${member} not found`;
|
||||
console.error(err);
|
||||
this.maybeTryAgainAfterError(cluster);
|
||||
return;
|
||||
}
|
||||
|
||||
const isOpenAPI = semver.satisfies(am.version, ">=0.16.0");
|
||||
|
||||
const uri = isOpenAPI
|
||||
? `${am.uri}/api/v2/silences`
|
||||
: `${am.uri}/api/v1/silences`;
|
||||
|
||||
this.submitState.silencesByCluster[cluster].fetch = FetchWithCredentials(
|
||||
uri,
|
||||
{
|
||||
method: "POST",
|
||||
body: JSON.stringify(
|
||||
this.submitState.silencesByCluster[cluster].payload
|
||||
),
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
...am.headers
|
||||
}
|
||||
}
|
||||
)
|
||||
.then(result => {
|
||||
if (isOpenAPI) {
|
||||
if (result.ok) {
|
||||
return result
|
||||
.json()
|
||||
.then(r => this.submitState.markDone(cluster));
|
||||
} else {
|
||||
this.maybeTryAgainAfterError(cluster);
|
||||
}
|
||||
} else {
|
||||
return result
|
||||
.json()
|
||||
.then(r =>
|
||||
r.status === "success"
|
||||
? this.submitState.markDone(cluster)
|
||||
: this.maybeTryAgainAfterError(cluster)
|
||||
);
|
||||
}
|
||||
})
|
||||
.catch(() => {
|
||||
this.maybeTryAgainAfterError(cluster);
|
||||
});
|
||||
};
|
||||
|
||||
onACK = () => {
|
||||
const { group, alertStore, silenceFormStore } = this.props;
|
||||
|
||||
if (this.submitState.isInprogress || this.submitState.isDone) {
|
||||
return;
|
||||
}
|
||||
|
||||
const alertmanagers = Object.entries(group.alertmanagerCount)
|
||||
.filter(([amName, alertCount]) => alertCount > 0)
|
||||
.map(([amName, _]) => amName);
|
||||
const clusters = Object.entries(
|
||||
alertStore.data.upstreams.clusters
|
||||
).filter(([clusterName, clusterMembers]) =>
|
||||
alertmanagers.some(m => clusterMembers.includes(m))
|
||||
);
|
||||
|
||||
this.submitState.reset();
|
||||
for (const [clusterName, clusterMembers] of clusters) {
|
||||
const pendingSilence = newPendingSilence(
|
||||
toJS(group),
|
||||
toJS(clusterMembers),
|
||||
toJS(alertStore.settings.values.alertAcknowledgement.durationSeconds),
|
||||
silenceFormStore.data.author !== ""
|
||||
? toJS(silenceFormStore.data.author)
|
||||
: toJS(alertStore.settings.values.alertAcknowledgement.author),
|
||||
toJS(alertStore.settings.values.alertAcknowledgement.commentPrefix)
|
||||
);
|
||||
this.submitState.pushSilence(clusterName, pendingSilence);
|
||||
this.handleAlertmanagerRequest(clusterName);
|
||||
}
|
||||
};
|
||||
|
||||
render() {
|
||||
const { alertStore } = this.props;
|
||||
|
||||
if (alertStore.settings.values.alertAcknowledgement.enabled === false) {
|
||||
return null;
|
||||
}
|
||||
|
||||
return (
|
||||
<TooltipWrapper title="Acknowlage this alert with a short lived silence">
|
||||
<span
|
||||
className={`badge badge-pill components-label components-label-with-hover px-2 ${
|
||||
this.submitState.isFailed
|
||||
? "badge-warning"
|
||||
: this.submitState.isDone
|
||||
? "badge-success"
|
||||
: "badge-secondary"
|
||||
}`}
|
||||
onClick={this.onACK}
|
||||
>
|
||||
{this.submitState.isIdle ? (
|
||||
<FontAwesomeIcon icon={faCheck} fixedWidth />
|
||||
) : this.submitState.isInprogress ? (
|
||||
<FontAwesomeIcon icon={faSpinner} fixedWidth spin />
|
||||
) : this.submitState.isFailed ? (
|
||||
<FontAwesomeIcon icon={faExclamationCircle} fixedWidth />
|
||||
) : (
|
||||
<FontAwesomeIcon icon={faCheckCircle} fixedWidth />
|
||||
)}
|
||||
</span>
|
||||
</TooltipWrapper>
|
||||
);
|
||||
}
|
||||
}
|
||||
);
|
||||
|
||||
export { AlertAck };
|
||||
343
ui/src/Components/AlertAck/index.test.js
Normal file
343
ui/src/Components/AlertAck/index.test.js
Normal file
@@ -0,0 +1,343 @@
|
||||
import React from "react";
|
||||
|
||||
import { mount } from "enzyme";
|
||||
|
||||
import { advanceTo, clear } from "jest-date-mock";
|
||||
|
||||
import { MockAlertGroup, MockAlert } from "__mocks__/Alerts.js";
|
||||
import { AlertStore } from "Stores/AlertStore";
|
||||
import { SilenceFormStore } from "Stores/SilenceFormStore";
|
||||
import { AlertAck } from ".";
|
||||
|
||||
let alertStore;
|
||||
let silenceFormStore;
|
||||
let alerts;
|
||||
let group;
|
||||
|
||||
beforeEach(() => {
|
||||
advanceTo(new Date(Date.UTC(2000, 1, 1, 0, 0, 0)));
|
||||
|
||||
alertStore = new AlertStore([]);
|
||||
silenceFormStore = new SilenceFormStore();
|
||||
|
||||
alertStore.settings.values.alertAcknowledgement = {
|
||||
enabled: true,
|
||||
durationSeconds: 123,
|
||||
author: "default author",
|
||||
commentPrefix: "PREFIX"
|
||||
};
|
||||
alertStore.data.upstreams = {
|
||||
clusters: { default: ["default"] },
|
||||
instances: [
|
||||
{
|
||||
name: "default",
|
||||
uri: "http://localhost",
|
||||
publicURI: "http://example.com",
|
||||
headers: { foo: "bar" },
|
||||
error: "",
|
||||
version: "0.15.0",
|
||||
cluster: "default",
|
||||
clusterMembers: ["default"]
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
alerts = [
|
||||
MockAlert([], { foo: "bar" }, "active"),
|
||||
MockAlert([], { foo: "baz" }, "suppressed")
|
||||
];
|
||||
group = MockAlertGroup({ alertname: "Fake Alert" }, alerts, [], {}, {});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
fetch.resetMocks();
|
||||
jest.clearAllTimers();
|
||||
jest.clearAllMocks();
|
||||
jest.restoreAllMocks();
|
||||
clear();
|
||||
});
|
||||
|
||||
const MountedAlertAck = () => {
|
||||
return mount(
|
||||
<AlertAck
|
||||
alertStore={alertStore}
|
||||
silenceFormStore={silenceFormStore}
|
||||
group={group}
|
||||
/>
|
||||
);
|
||||
};
|
||||
|
||||
const MountAndClick = async () => {
|
||||
const tree = MountedAlertAck();
|
||||
const button = tree.find("span.badge");
|
||||
button.simulate("click");
|
||||
await expect(
|
||||
tree.instance().submitState.silencesByCluster["default"].fetch
|
||||
).resolves.toBeUndefined();
|
||||
};
|
||||
|
||||
describe("<AlertAck />", () => {
|
||||
it("is null when acks are disabled", () => {
|
||||
alertStore.settings.values.alertAcknowledgement.enabled = false;
|
||||
const tree = MountedAlertAck();
|
||||
expect(tree.html()).toBeNull();
|
||||
});
|
||||
|
||||
it("uses faCheck icon when idle", () => {
|
||||
const tree = MountedAlertAck();
|
||||
expect(tree.html()).toMatch(/fa-check/);
|
||||
});
|
||||
|
||||
it("uses faExclamationCircle after failed fetch", async () => {
|
||||
fetch.mockResponse("error message", { status: 500 });
|
||||
const tree = MountedAlertAck();
|
||||
const button = tree.find("span.badge");
|
||||
button.simulate("click");
|
||||
await expect(
|
||||
tree.instance().submitState.silencesByCluster["default"].fetch
|
||||
).resolves.toBeUndefined();
|
||||
expect(tree.html()).toMatch(/fa-exclamation-circle/);
|
||||
});
|
||||
|
||||
it("[v1] uses faCheckCircle after successful fetch", async () => {
|
||||
fetch.mockResponse(
|
||||
JSON.stringify({ status: "success", data: { silenceId: "123456789" } })
|
||||
);
|
||||
const tree = MountedAlertAck();
|
||||
const button = tree.find("span.badge");
|
||||
button.simulate("click");
|
||||
await expect(
|
||||
tree.instance().submitState.silencesByCluster["default"].fetch
|
||||
).resolves.toBeUndefined();
|
||||
expect(tree.html()).toMatch(/fa-check-circle/);
|
||||
});
|
||||
|
||||
it("[v2] uses faCheckCircle after successful fetch", async () => {
|
||||
fetch.mockResponse(JSON.stringify({ silenceID: "123" }));
|
||||
alertStore.data.upstreams.instances[0].version = "0.16.2";
|
||||
const tree = MountedAlertAck();
|
||||
const button = tree.find("span.badge");
|
||||
button.simulate("click");
|
||||
await expect(
|
||||
tree.instance().submitState.silencesByCluster["default"].fetch
|
||||
).resolves.toBeUndefined();
|
||||
expect(tree.html()).toMatch(/fa-check-circle/);
|
||||
});
|
||||
|
||||
it("sends a request on click", () => {
|
||||
MountAndClick();
|
||||
expect(fetch.mock.calls).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("doesn't send any request on click when already in progress", async () => {
|
||||
const tree = MountedAlertAck();
|
||||
const button = tree.find("span.badge");
|
||||
button.simulate("click");
|
||||
button.simulate("click");
|
||||
await expect(
|
||||
tree.instance().submitState.silencesByCluster["default"].fetch
|
||||
).resolves.toBeUndefined();
|
||||
expect(fetch.mock.calls).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("doesn't send any request on click when already done", async () => {
|
||||
const tree = MountedAlertAck();
|
||||
const button = tree.find("span.badge");
|
||||
|
||||
button.simulate("click");
|
||||
await expect(
|
||||
tree.instance().submitState.silencesByCluster["default"].fetch
|
||||
).resolves.toBeUndefined();
|
||||
expect(fetch.mock.calls).toHaveLength(1);
|
||||
|
||||
button.simulate("click");
|
||||
await expect(
|
||||
tree.instance().submitState.silencesByCluster["default"].fetch
|
||||
).resolves.toBeUndefined();
|
||||
expect(fetch.mock.calls).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("sends POST requests", () => {
|
||||
MountAndClick();
|
||||
expect(fetch.mock.calls[0][1].method).toBe("POST");
|
||||
});
|
||||
|
||||
it("sends correct payload", () => {
|
||||
fetch.mockResponse(
|
||||
JSON.stringify({ status: "success", data: { silenceId: "123456789" } })
|
||||
);
|
||||
|
||||
silenceFormStore.data.author = "karma/ui";
|
||||
MountAndClick();
|
||||
expect(JSON.parse(fetch.mock.calls[0][1].body)).toEqual({
|
||||
comment:
|
||||
"PREFIX This alert was acknowledged using karma on Tue Feb 01 2000 00:00:00 GMT+0000",
|
||||
createdBy: "karma/ui",
|
||||
endsAt: "2000-02-01T00:02:03.000Z",
|
||||
matchers: [
|
||||
{ isRegex: false, name: "alertname", value: "Fake Alert" },
|
||||
{ isRegex: true, name: "foo", value: "(bar|baz)" }
|
||||
],
|
||||
startsAt: "2000-02-01T00:00:00.000Z"
|
||||
});
|
||||
});
|
||||
|
||||
it("uses settings when generating payload", () => {
|
||||
alertStore.settings.values.alertAcknowledgement.durationSeconds = 237;
|
||||
alertStore.settings.values.alertAcknowledgement.author = "me";
|
||||
alertStore.settings.values.alertAcknowledgement.commentPrefix = "";
|
||||
MountAndClick();
|
||||
expect(JSON.parse(fetch.mock.calls[0][1].body)).toEqual({
|
||||
comment:
|
||||
"This alert was acknowledged using karma on Tue Feb 01 2000 00:00:00 GMT+0000",
|
||||
createdBy: "me",
|
||||
endsAt: "2000-02-01T00:03:57.000Z",
|
||||
matchers: [
|
||||
{ isRegex: false, name: "alertname", value: "Fake Alert" },
|
||||
{ isRegex: true, name: "foo", value: "(bar|baz)" }
|
||||
],
|
||||
startsAt: "2000-02-01T00:00:00.000Z"
|
||||
});
|
||||
});
|
||||
|
||||
it("[v1] sends POST request to /api/v1/silences", () => {
|
||||
MountAndClick();
|
||||
const uri = fetch.mock.calls[0][0];
|
||||
expect(uri).toBe("http://localhost/api/v1/silences");
|
||||
});
|
||||
|
||||
it("[v2] sends POST request to /api/v2/silences", () => {
|
||||
alertStore.data.upstreams.instances[0].version = "0.16.2";
|
||||
MountAndClick();
|
||||
const uri = fetch.mock.calls[0][0];
|
||||
expect(uri).toBe("http://localhost/api/v2/silences");
|
||||
});
|
||||
|
||||
it("[v1] will retry on another cluster member after fetch failure", async () => {
|
||||
fetch
|
||||
.mockResponseOnce(JSON.stringify({ status: "error" }))
|
||||
.mockResponseOnce(
|
||||
JSON.stringify({ status: "success", data: { silenceId: "123456789" } })
|
||||
);
|
||||
alertStore.data.upstreams = {
|
||||
clusters: { default: ["default", "fallback"] },
|
||||
instances: [
|
||||
{
|
||||
name: "default",
|
||||
uri: "http://am1.example.com",
|
||||
publicURI: "http://am1.example.com",
|
||||
headers: {},
|
||||
error: "",
|
||||
version: "0.15.0",
|
||||
cluster: "default",
|
||||
clusterMembers: ["default", "fallback"]
|
||||
},
|
||||
{
|
||||
name: "fallback",
|
||||
uri: "http://am2.example.com",
|
||||
publicURI: "http://am2.example.com",
|
||||
headers: {},
|
||||
error: "",
|
||||
version: "0.15.0",
|
||||
cluster: "default",
|
||||
clusterMembers: ["default", "fallback"]
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
const tree = MountedAlertAck();
|
||||
const button = tree.find("span.badge");
|
||||
button.simulate("click");
|
||||
await expect(
|
||||
tree.instance().submitState.silencesByCluster["default"].fetch
|
||||
).resolves.toBeUndefined();
|
||||
expect(fetch.mock.calls[0][0]).toBe(
|
||||
"http://am2.example.com/api/v1/silences"
|
||||
);
|
||||
await expect(
|
||||
tree.instance().submitState.silencesByCluster["default"].fetch
|
||||
).resolves.toBeUndefined();
|
||||
expect(fetch.mock.calls[1][0]).toBe(
|
||||
"http://am1.example.com/api/v1/silences"
|
||||
);
|
||||
});
|
||||
|
||||
it("[v2] will retry on another cluster member after fetch failure", async () => {
|
||||
fetch
|
||||
.mockResponseOnce("error message", { status: 500 })
|
||||
.mockResponseOnce(JSON.stringify({ silenceID: "123" }));
|
||||
alertStore.data.upstreams = {
|
||||
clusters: { default: ["default", "fallback"] },
|
||||
instances: [
|
||||
{
|
||||
name: "default",
|
||||
uri: "http://am1.example.com",
|
||||
publicURI: "http://am1.example.com",
|
||||
headers: {},
|
||||
error: "",
|
||||
version: "0.16.2",
|
||||
cluster: "default",
|
||||
clusterMembers: ["default", "fallback"]
|
||||
},
|
||||
{
|
||||
name: "fallback",
|
||||
uri: "http://am2.example.com",
|
||||
publicURI: "http://am2.example.com",
|
||||
headers: {},
|
||||
error: "",
|
||||
version: "0.16.2",
|
||||
cluster: "default",
|
||||
clusterMembers: ["default", "fallback"]
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
const tree = MountedAlertAck();
|
||||
const button = tree.find("span.badge");
|
||||
button.simulate("click");
|
||||
await expect(
|
||||
tree.instance().submitState.silencesByCluster["default"].fetch
|
||||
).resolves.toBeUndefined();
|
||||
expect(fetch.mock.calls[0][0]).toBe(
|
||||
"http://am2.example.com/api/v2/silences"
|
||||
);
|
||||
await expect(
|
||||
tree.instance().submitState.silencesByCluster["default"].fetch
|
||||
).resolves.toBeUndefined();
|
||||
expect(fetch.mock.calls[1][0]).toBe(
|
||||
"http://am1.example.com/api/v2/silences"
|
||||
);
|
||||
});
|
||||
|
||||
it("will log an error if Alertmanager instance is missing from instances and try the next one", async () => {
|
||||
const consoleSpy = jest
|
||||
.spyOn(console, "error")
|
||||
.mockImplementation(() => {});
|
||||
alertStore.data.upstreams = {
|
||||
clusters: { default: ["default", "fallback"] },
|
||||
instances: [
|
||||
{
|
||||
name: "default",
|
||||
uri: "http://am1.example.com",
|
||||
publicURI: "http://am1.example.com",
|
||||
headers: {},
|
||||
error: "",
|
||||
version: "0.15.0",
|
||||
cluster: "default",
|
||||
clusterMembers: ["default", "fallback"]
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
const tree = MountedAlertAck();
|
||||
const button = tree.find("span.badge");
|
||||
button.simulate("click");
|
||||
await expect(
|
||||
tree.instance().submitState.silencesByCluster["default"].fetch
|
||||
).resolves.toBeUndefined();
|
||||
expect(fetch.mock.calls[0][0]).toBe(
|
||||
"http://am1.example.com/api/v1/silences"
|
||||
);
|
||||
expect(consoleSpy).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
@@ -75,7 +75,11 @@ const Alert = observer(
|
||||
}
|
||||
|
||||
return (
|
||||
<li className={classNames.join(" ")}>
|
||||
<li
|
||||
className={classNames.join(" ")}
|
||||
onMouseEnter={this.toggleHover}
|
||||
onMouseLeave={this.toggleHover}
|
||||
>
|
||||
<div>
|
||||
{alert.annotations
|
||||
.filter(a => a.isLink === false)
|
||||
|
||||
@@ -13,6 +13,7 @@ import { SilenceFormStore } from "Stores/SilenceFormStore";
|
||||
import { FilteringLabel } from "Components/Labels/FilteringLabel";
|
||||
import { FilteringCounterBadge } from "Components/Labels/FilteringCounterBadge";
|
||||
import { TooltipWrapper } from "Components/TooltipWrapper";
|
||||
import { AlertAck } from "Components/AlertAck";
|
||||
import { GroupMenu } from "./GroupMenu";
|
||||
|
||||
const GroupHeader = observer(
|
||||
@@ -65,6 +66,13 @@ const GroupHeader = observer(
|
||||
))}
|
||||
</span>
|
||||
<span className="flex-shrink-0 flex-grow-0 ml-auto pl-1">
|
||||
{group.stateCount.active > 0 && (
|
||||
<AlertAck
|
||||
alertStore={alertStore}
|
||||
silenceFormStore={silenceFormStore}
|
||||
group={group}
|
||||
/>
|
||||
)}
|
||||
<FilteringCounterBadge
|
||||
name="@state"
|
||||
value="unprocessed"
|
||||
|
||||
@@ -200,6 +200,12 @@ class AlertStore {
|
||||
strip: {
|
||||
labels: []
|
||||
}
|
||||
},
|
||||
alertAcknowledgement: {
|
||||
enabled: false,
|
||||
durationSeconds: 900,
|
||||
author: "karma / author missing",
|
||||
commentPrefix: ""
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
@@ -36,6 +36,80 @@ const SilenceTabNames = Object.freeze({
|
||||
Browser: "browser"
|
||||
});
|
||||
|
||||
const MatchersFromGroup = (group, stripLabels, alerts) => {
|
||||
let matchers = [];
|
||||
|
||||
// add matchers for all shared labels in this group
|
||||
for (const [key, value] of Object.entries(
|
||||
Object.assign({}, group.labels, group.shared.labels)
|
||||
)) {
|
||||
if (!stripLabels.includes(key)) {
|
||||
const matcher = NewEmptyMatcher();
|
||||
matcher.name = key;
|
||||
matcher.values = [MatcherValueToObject(value)];
|
||||
matchers.push(matcher);
|
||||
}
|
||||
}
|
||||
|
||||
// add matchers for all unique labels in this group
|
||||
let labels = {};
|
||||
const allAlerts = alerts ? alerts : group.alerts;
|
||||
for (const alert of allAlerts) {
|
||||
for (const [key, value] of Object.entries(alert.labels)) {
|
||||
if (!stripLabels.includes(key)) {
|
||||
if (!labels[key]) {
|
||||
labels[key] = new Set();
|
||||
}
|
||||
labels[key].add(value);
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const [key, values] of Object.entries(labels)) {
|
||||
matchers.push({
|
||||
id: uniqueId(),
|
||||
name: key,
|
||||
values: [...values].sort().map(value => MatcherValueToObject(value)),
|
||||
suggestions: {
|
||||
names: [],
|
||||
values: []
|
||||
},
|
||||
isRegex: values.size > 1
|
||||
});
|
||||
}
|
||||
|
||||
return matchers;
|
||||
};
|
||||
|
||||
const GenerateAlertmanagerSilenceData = (
|
||||
startsAt,
|
||||
endsAt,
|
||||
matchers,
|
||||
author,
|
||||
comment,
|
||||
silenceID
|
||||
) => {
|
||||
const payload = {
|
||||
matchers: matchers.map(m => ({
|
||||
name: m.name,
|
||||
value:
|
||||
m.values.length > 1
|
||||
? `(${m.values.map(v => v.value).join("|")})`
|
||||
: m.values.length === 1
|
||||
? m.values[0].value
|
||||
: "",
|
||||
isRegex: m.isRegex
|
||||
})),
|
||||
startsAt: startsAt.toISOString(),
|
||||
endsAt: endsAt.toISOString(),
|
||||
createdBy: author,
|
||||
comment: comment
|
||||
};
|
||||
if (silenceID !== null) {
|
||||
payload.id = silenceID;
|
||||
}
|
||||
return payload;
|
||||
};
|
||||
|
||||
class SilenceFormStore {
|
||||
// this is used to store modal visibility toggle
|
||||
toggle = observable(
|
||||
@@ -141,49 +215,7 @@ class SilenceFormStore {
|
||||
|
||||
// if alerts argument is not passed all group alerts will be used
|
||||
fillMatchersFromGroup(group, stripLabels, alerts) {
|
||||
let matchers = [];
|
||||
|
||||
// add matchers for all shared labels in this group
|
||||
for (const [key, value] of Object.entries(
|
||||
Object.assign({}, group.labels, group.shared.labels)
|
||||
)) {
|
||||
if (!stripLabels.includes(key)) {
|
||||
const matcher = NewEmptyMatcher();
|
||||
matcher.name = key;
|
||||
matcher.values = [MatcherValueToObject(value)];
|
||||
matchers.push(matcher);
|
||||
}
|
||||
}
|
||||
|
||||
// add matchers for all unique labels in this group
|
||||
let labels = {};
|
||||
const allAlerts = alerts ? alerts : group.alerts;
|
||||
for (const alert of allAlerts) {
|
||||
for (const [key, value] of Object.entries(alert.labels)) {
|
||||
if (!stripLabels.includes(key)) {
|
||||
if (!labels[key]) {
|
||||
labels[key] = new Set();
|
||||
}
|
||||
labels[key].add(value);
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const [key, values] of Object.entries(labels)) {
|
||||
matchers.push({
|
||||
id: uniqueId(),
|
||||
name: key,
|
||||
values: [...values]
|
||||
.sort()
|
||||
.map(value => MatcherValueToObject(value)),
|
||||
suggestions: {
|
||||
names: [],
|
||||
values: []
|
||||
},
|
||||
isRegex: values.size > 1
|
||||
});
|
||||
}
|
||||
|
||||
this.matchers = matchers;
|
||||
this.matchers = MatchersFromGroup(group, stripLabels, alerts);
|
||||
// ensure that silenceID is nulled, since it's used to edit silences
|
||||
// and this is used to silence groups
|
||||
this.silenceID = null;
|
||||
@@ -241,32 +273,14 @@ class SilenceFormStore {
|
||||
},
|
||||
|
||||
get toAlertmanagerPayload() {
|
||||
const payload = {
|
||||
matchers: this.matchers.map(m => ({
|
||||
name: m.name,
|
||||
value:
|
||||
m.values.length > 1
|
||||
? `(${m.values.map(v => v.value).join("|")})`
|
||||
: m.values.length === 1
|
||||
? m.values[0].value
|
||||
: "",
|
||||
isRegex: m.isRegex
|
||||
})),
|
||||
startsAt: this.startsAt
|
||||
.second(0)
|
||||
.millisecond(0)
|
||||
.toISOString(),
|
||||
endsAt: this.endsAt
|
||||
.second(0)
|
||||
.millisecond(0)
|
||||
.toISOString(),
|
||||
createdBy: this.author,
|
||||
comment: this.comment
|
||||
};
|
||||
if (this.silenceID !== null) {
|
||||
payload.id = this.silenceID;
|
||||
}
|
||||
return payload;
|
||||
return GenerateAlertmanagerSilenceData(
|
||||
this.startsAt.second(0).millisecond(0),
|
||||
this.endsAt.second(0).millisecond(0),
|
||||
this.matchers,
|
||||
this.author,
|
||||
this.comment,
|
||||
this.silenceID
|
||||
);
|
||||
},
|
||||
|
||||
get toDuration() {
|
||||
@@ -306,5 +320,7 @@ export {
|
||||
NewEmptyMatcher,
|
||||
MatcherValueToObject,
|
||||
AlertmanagerClustersToOption,
|
||||
SilenceTabNames
|
||||
SilenceTabNames,
|
||||
MatchersFromGroup,
|
||||
GenerateAlertmanagerSilenceData
|
||||
};
|
||||
|
||||
@@ -48,6 +48,12 @@ const EmptyAPIResponse = () => ({
|
||||
labels: []
|
||||
}
|
||||
},
|
||||
alertAcknowledgement: {
|
||||
enabled: false,
|
||||
durationSeconds: 900,
|
||||
author: "karma / author missing",
|
||||
commentPrefix: ""
|
||||
},
|
||||
staticColorLabels: ["job"],
|
||||
annotationsDefaultHidden: false,
|
||||
annotationsHidden: [],
|
||||
|
||||
Reference in New Issue
Block a user