init

2026-08-27 12:37:20 +00:00 · 2021-07-02 10:01:38 +02:00
commit 4d783f28c2
26 changed files with 625 additions and 0 deletions
@@ -0,0 +1,3 @@
+[submodule "themes/book"]
+	path = themes/book
+	url = https://github.com/alex-shpak/hugo-book
@@ -0,0 +1,202 @@
+Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "{}"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright {yyyy} {name of copyright owner}
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
@@ -0,0 +1,6 @@
+---
+title: "{{ replace .Name "-" " " | title }}"
+date: {{ .Date }}
+draft: true
+---
+
@@ -0,0 +1,16 @@
+baseURL: "https://runbooks.thaum.xyz"
+languageCode: "en-us"
+title: "kube-prometheus runbooks"
+
+menu:
+  # before: []
+  after:
+    - name: "Add runbook"
+      url: "https://github.com/paulfantom/runbooks/issues/new?assignees=&labels=kind%2Fbug&template=bug.md"
+      weight: 10
+
+params:
+  BookRepo: "https://github.com/paulfantom/runbooks"
+  BookEditPath: "edit/main"
+  BookSearch: true
+
@@ -0,0 +1,41 @@
+---
+title: Introduction
+type: docs
+---
+
+# Acerbo datus maxime
+
+{{< columns >}}
+## Astris ipse furtiva
+
+Est in vagis et Pittheus tu arge accipiter regia iram vocatur nurus. Omnes ut
+olivae sensit **arma sorori** deducit, inesset **crudus**, ego vetuere aliis,
+modo arsit? Utinam rapta fiducia valuere litora _adicit cursu_, ad facies
+
+<--->
+
+## Suis quot vota
+
+Ea _furtique_ risere fratres edidit terrae magis. Colla tam mihi tenebat:
+miseram excita suadent es pecudes iam. Concilio _quam_ velatus posset ait quod
+nunc! Fragosis suae dextra geruntur functus vulgata.
+{{< /columns >}}
+
+
+## Tempora nisi nunc
+
+Lorem **markdownum** emicat gestu. Cannis sol pressit ducta. **Est** Idaei,
+tremens ausim se tutaeque, illi ulnis hausit, sed, lumina cutem. Quae avis
+sequens!
+
+    var panel = ram_design;
+    if (backup + system) {
+        file.readPoint = network_native;
+        sidebar_engine_device(cell_tftp_raster,
+                dual_login_paper.adf_vci.application_reader_design(
+                graphicsNvramCdma, lpi_footer_snmp, integer_model));
+    }
+
+## Locis suis novi cum suoque decidit eadem
+
+Idmoniae ripis, at aves, ali missa adest, ut _et autem_, et ab?
@@ -0,0 +1,3 @@
+---
+bookHidden: true
+---
@@ -0,0 +1,35 @@
+---
+title: Alertmanager Members Inconsistent
+weight: 20
+---
+
+# AlertmanagerMembersInconsistent
+
+## Meaning
+
+At least one of alertmanager cluster members cannot be found.
+
+## Impact
+
+## Diagnosis
+
+Check if IP addresses discovered by alertmanager cluster are the same ones as in alertmanager Service. Following example show possible inconsistency in Endpoint IP addresses:
+
+```bash
+$ kubectl describe svc alertmanager-main
+
+Name:              alertmanager-main
+Namespace:         monitoring
+...
+Endpoints:         10.128.2.3:9095,10.129.2.5:9095,10.131.0.44:9095
+
+$ kubectl get pod -o wide | grep alertmanager-main
+
+alertmanager-main-0                            5/5     Running   0          11d     10.129.2.6
+alertmanager-main-1                            5/5     Running   0          2d16h   10.131.0.44     
+alertmanager-main-2                            5/5     Running   0          6d      10.128.2.3  
+```
+
+## Mitigation
+
+Deleting an incorrect Endpoint should trigger its recreation with a correct IP address.
@@ -0,0 +1,7 @@
+---
+title: Alertmanager
+bookCollapseSection: true
+weight: 10
+bookFlatSection: true
+---
+
@@ -0,0 +1,19 @@
+# TargetDown
+
+## Meaning
+
+The alert means that one or more prometheus scrape targets are down. It fires when at least 10% of scrape targets in a Service are unreachable.
+
+## Impact
+
+Metrics from a particular target cannot be scraped as such there is no data for this target in Prometheus and alerting can be hindered.
+
+## Diagnosis
+
+`/targets` page in Prometheus UI can be used to check the scrape error for the particular target.
+
+`up == 0` query can be used to check the trend over time.
+
+## Mitigation
+
+Mitigation depends on the error reported by prometheus and there is no generic one.
@@ -0,0 +1,7 @@
+---
+title: General
+bookCollapseSection: true
+bookFlatSection: true
+weight: 10
+---
+
@@ -0,0 +1,91 @@
+# Impact
+The overall availability of your Kubernetes cluster isn't guaranteed anymore.  
+There may be **too many errors** returned by the APIServer and/or **responses take too long** for guarantee proper reconciliation.
+
+**This is always important; the only deciding factor is how urgent it is at the current rate**
+
+## Critical
+
+First check the labels `long` and `short`.
+
+* `long: 1h` and `short: 5m`: less than **~2 days** -- You should fix the problem as soon as possible!  
+* `long: 6h` and `short: 30m`: less than **~5 days** -- Track this down now but no immediate fix required.
+
+## Warning
+
+First check the labels `long` and `short`.
+
+* `long: 1d` and `short: 2h`: less than **~10 days** -- This is problematic in the long run. You should take a look in the next 24-48 hours.
+* `long: 3d` and `short: 6h`: less than **~30 days** -- (the entire window of the error budget) at this rate. This means that at the end of the next 30 days there won't be any error budget left at this rate. It's fine to leave this over the weekend and have someone take a look in the coming days at working hours.  
+
+_Example: If you have a 99% availability target this means that at the end of 30 days you're going to be below 99% at this rate._
+
+## Runbook
+
+1. Take a look at the APIServer Grafana dashboard.
+    1. At the very top check your current availability and how much percent of error budget is left. This should indicate the severity too.
+    1. Do you see an elevated error rate in reads or writes?
+    1. Do you see too many slow requests in reads or writes?
+1. Run debugging queries in Prometheus or Grafana Explore to dig deeper.
+    1. If you don't see anything obvious with the error rates, it might be too many slow requests. [Check the queries below!](#example-queries-for-slow-requests)
+1. Maybe it's some dependency of the APIServer? etcd?
+
+### Example Queries for slow requests:
+
+Change the rate window according to your `long` label from the alert.  
+Make sure to update the alert threshold too, like `> 0.01` to `> 14.4 * 0.01` for exmaple.
+#### Slow Read Requests:
+
+If you don't get any results back then there aren't too many slow requests - that's good.
+If you get results than you know what type of requests are too slow.
+
+Cluster scoped:
+```
+(
+sum(rate(apiserver_request_duration_seconds_bucket{job="apiserver",le="40",scope="cluster",verb=~"LIST|GET"}[3d]))
+- 
+sum(rate(apiserver_request_duration_seconds_count{job="apiserver",verb=~"LIST|GET"}[3d]))
+)
+/
+sum(rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d]))
+> 0.01
+```
+Namespace scoped:
+```
+(
+sum(rate(apiserver_request_duration_seconds_bucket{job="apiserver",le="5",scope="namespace",verb=~"LIST|GET"}[3d]))
+- 
+sum(rate(apiserver_request_duration_seconds_count{job="apiserver",verb=~"LIST|GET"}[3d]))
+)
+/
+sum(rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d]))
+> 0.01
+```
+
+Resource scoped:
+```
+(
+sum(rate(apiserver_request_duration_seconds_bucket{job="apiserver",le="1",scope=~"resource|",verb=~"LIST|GET"}[3d])) or vector(0)
+- 
+sum(rate(apiserver_request_duration_seconds_count{job="apiserver",verb=~"LIST|GET"}[3d]))
+)
+/
+sum(rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d]))
+> 0.01
+```
+
+#### Slow Write Requests
+
+```
+(
+sum(rate(apiserver_request_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d]))
+-
+sum(rate(apiserver_request_duration_seconds_bucket{job="apiserver",le="1",verb=~"POST|PUT|PATCH|DELETE"}[3d]))
+)
+/
+sum(rate(apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d]))
+> 0.01
+```
+
+--- 
+Learn more about Multiple Burn Rate Alerts in the [SRE Workbook Chapter 5](https://sre.google/workbook/alerting-on-slos/#recommended_time_windows_and_burn_rates_f).
@@ -0,0 +1,49 @@
+There can be various reasons why a volume is filling up. This runbook does not cover application specific reasons, only mitigations for volumes that are legitimately filling.
+
+## Volume resizing
+
+If volume resizing is available, it's easiest to increase the capacity of the volume.
+
+To check if volume expansion is available, run this with your namespace and PVC-name replaced.
+
+```bash
+$ kubectl get storageclass `kubectl -n <my-namespace> get pvc <my-pvc> -ojson | jq -r '.spec.storageClassName'`       
+NAME                 PROVISIONER            RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
+standard (default)   kubernetes.io/gce-pd   Delete          Immediate           true                   28d
+```
+
+In this case `ALLOWVOLUMEEXPANSION` is true, so we can make use of the feature.
+
+To resize the volume run:
+
+```bash
+$ kubectl -n <my-namespace> edit pvc <my-pvc>
+```
+
+And edit `.spec.resources.requests.storage` to the new desired storage size. Eventually the PVC status will say "Waiting for user to (re-)start a pod to finish file system resize of volume on node."
+
+You can check this with:
+
+```bash
+$ kubectl -n <my-namespace> get pvc <my-pvc>
+```
+
+Once the PVC status says to restart the respective pod, run this to restart it (this automatically finds the pod that mounts the PVC and deletes it, if you know the pod name, you can also just simply delete that pod):
+
+```bash
+$ kubectl -n <my-namespace> delete pod `kubectl -n <my-namespace> get pod -ojson | jq -r '.items[] | select(.spec.volumes[] .persistentVolumeClaim.claimName=="<my-pvc>") | .metadata.name'`
+```
+
+## Migrate data to a new, larger volume
+
+When resizing is not available and the data is not safe to be deleted, then the only way is to create a larger volume and migrate the data.
+
+TODO
+
+## Purge volume
+
+When the data is ephemeral and volume expansion is not available, it may be best to purge the volume.
+
+WARNING/DANGER: This will permanently delete the data on the volume. Performing these steps is your responsibility.
+
+TODO
@@ -0,0 +1 @@
+Runbook available at https://coreos.com/tectonic/docs/latest/troubleshooting/controller-recovery.html#recovering-a-scheduler
@@ -0,0 +1,3 @@
+The Kubelet in a Kubernetes cluster is the agent that ensures Pods are running on that host.
+
+Kubelet's have a configuration that limits how many Pods they can run. The default value of this is 110 Pods per Kubelet, but it is configurable (and this alert takes that configuration into account with the `kube_node_status_capacity_pods` metric). The alert fires when a Kubelet reaches 95% of its capacity. This alert warns about the likelihood that the cluster is close to running out of capacity to run Pods on the cluster. Either the cluster must be increased in its node count, or the number of Pods must be reduced.
@@ -0,0 +1,7 @@
+---
+title: Kubernetes
+bookCollapseSection: true
+bookFlatSection: true
+weight: 10
+---
+
@@ -0,0 +1,7 @@
+---
+title: Node
+bookCollapseSection: true
+bookFlatSection: true
+weight: 10
+---
+
@@ -0,0 +1,17 @@
+# PrometheusBadConfig
+
+## Meaning
+
+Alert fires when Prometheus cannot successfully reload the configuration file due to the file having incorrect content.
+
+## Impact
+
+Configuration cannot be reloaded and prometheus operates with last known good configuration. Configuration changes in any of Prometheus, Probe, PodMonitor, or ServiceMonitor objects may not be picked up by prometheus server.
+
+## Diagnosis
+
+Check prometheus container logs for an explanation of which part of the configuration is problematic. Usually this can occur when ServiceMonitors or PodMonitors share the same job label.
+
+## Mitigation
+
+Remove conflicting configuration option.
@@ -0,0 +1,23 @@
+# PrometheusDuplicateTimestamps
+
+Find the Prometheus Pod that concerns this.
+
+```bash
+$ kubectl -n <namespace> get pod
+prometheus-k8s-0                       2/2     Running   1          122m
+prometheus-k8s-1                       2/2     Running   1          122m
+```
+
+Look at the logs of each of them, there should be a log line such as:
+
+```bash
+$ kubectl -n <namespace> logs prometheus-k8s-0
+level=warn ts=2021-01-04T15:08:55.613Z caller=scrape.go:1372 component="scrape manager" scrape_pool=default/main-ingress-nginx-controller/0 target=http://10.0.7.3:10254/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=16
+```
+
+Now there is a judgement call to make, this could be the result of:
+
+* Faulty configuration, which could be resolved by removing the offending `ServiceMonitor` or `PodMonitor` object, which can be identified through the `scrape_pool` label in the log line, which is in the format of `<namespace>/<service-monitor-name>/<endpoint-id>`.
+* The target is reporting faulty data, sometimes this can be resolved by restarting the target, or it might need to be fixed in code of the offending application.
+
+Further reading: https://www.robustperception.io/debugging-out-of-order-samples
@@ -0,0 +1,3 @@
+# PrometheusOutOfOrderTimestamps
+
+More information in https://www.robustperception.io/debugging-out-of-order-samples
@@ -0,0 +1,3 @@
+Your best starting point is the rules page of the Prometheus UI (:9090/rules). It will show the error. 
+
+You can also evaluate the rule expression yourself, using the UI, or maybe using PromLens to help debug expression issues.
@@ -0,0 +1,7 @@
+---
+title: Prometheus
+bookCollapseSection: true
+bookFlatSection: true
+weight: 10
+---
+
@@ -0,0 +1,31 @@
+---
+title: New Runbook
+---
+
+
+
+Runbook example based on a NodeFilesystemSpaceFillingUp (thanks to @beorn7):
+
+```
+# NodeFilesystemSpaceFillingUp
+
+## Meaning
+
+This alert is based on an extrapolation of the space used in a file system. It fires if both the current usage is above a certain threshold _and_ the extrapolation predicts to run out of space in a certain time. This is a warning-level alert if that time is less than 24h. It's a critical alert if that time is less than 4h.
+
+## Impact
+
+A filesystem running completely full is obviously very bad for any process in need to write to the filesystem. But even before a filesystem runs completely full, performance is usually degrading.
+
+## Diagnosis
+
+Study the recent trends of filesystem usage on a dashboard. Sometimes a periodic pattern of writing and cleaning up can trick the linear prediction into a false alert.
+
+Use the usual OS tools to investigate what directories are the worst and/or recent offenders.
+
+Is this some irregular condition, e.g. a process fails to clean up behind itself, or is this organic growth?
+
+## Mitigation
+
+<Insert site specific measures, for example to grow a persistent volume.>
+```
@@ -0,0 +1,41 @@
+<!DOCTYPE html>
+<html lang="{{ .Site.Language.Lang }}">
+
+  <head>
+    {{ partial "docs/html-head" . }}
+    {{ partial "docs/inject/head" . }}
+
+    <style>
+      .not-found {
+        text-align: center;
+      }
+      .not-found h1 {
+        margin: .25em 0 0 0;
+        opacity: .25;
+        font-size: 40vmin;
+      }
+    </style>
+  </head>
+
+  <body>
+    <main class="flex justify-center not-found">
+      <div>
+        <h1>404</h1>
+        <h2>Page Not Found. Don't panic!</h2>
+        <h4>If you were directed here by a link from an alert, we sadly don't have a runbook for it... yet!</h4>
+        <p>All runbooks on this site are created in the open and maintained by a team of passionate people.</p>
+        <p>We would like to have runbooks for all alerts shipped with kube-prometheus, but we sadly don't have time to write them all.</p>
+        <p>If you would like to help us, please consider opening a pull request. Thank you!</p>
+         <h3>
+          <a href="{{ .Site.Params.BookRepo }}/issues/new?template=new-runbook.md">Add runbook</a>
+        </h3>      
+        <h3>
+          <a href="{{ .Site.Home.RelPermalink }}">Back to main page</a>
+        </h3>
+      </div>
+    </main>
+
+    {{ partial "docs/inject/body" . }}
+  </body>
+
+</html>
@@ -0,0 +1 @@
+{"Target":"book.min.075e66da82f619e274b9c5dfc25e99a977e9a49494ffda671e046410f99b9455.css","MediaType":"text/css","Data":{"Integrity":"sha256-B15m2oL2GeJ0ucXfwl6ZqXfppJSU/9pnHgRkEPmblFU="}}
				`@@ -0,0 +1 @@`
				`Runbook available at https://coreos.com/tectonic/docs/latest/troubleshooting/controller-recovery.html#recovering-a-scheduler`
				`@@ -0,0 +1 @@`
				`{"Target":"book.min.075e66da82f619e274b9c5dfc25e99a977e9a49494ffda671e046410f99b9455.css","MediaType":"text/css","Data":{"Integrity":"sha256-B15m2oL2GeJ0ucXfwl6ZqXfppJSU/9pnHgRkEPmblFU="}}`