github/node-problem-detector
1
0
Fork 0
mirror of https://github.com/kubernetes/node-problem-detector.git synced 2026-08-19 20:26:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,334 Commits 12 Branches 59 Tags
bcac1a6db91ce311f7203b50fa7f6f4b2e6b9b31
Commit Graph
95 Commits
Author SHA1 Message Date
Owen Strain 556ff0cc7a Update log pattern to match kubelet service name from kubelet .deb package 2025-12-26 23:10:06 +00:00
Adrian MoiseyGitHubDan Winship
 		bc3150aa62 Update config/plugin/dns_problem.sh 
Co-authored-by: Dan Winship <danwinship@redhat.com>
 2025-09-22 15:11:07 +02:00
Adrian Moisey 3f139c4165 Add DNS lookup plugin 2025-07-21 21:38:49 +02:00
Jian WenandJian Wen 5562632053 Add UEFI Common Platform Error Record (CPER) support 
CPER is the format used to describe platform hardware error by various
tables, such as ERST, BERT and HEST etc.

The event severity message is printed here:
https://github.com/torvalds/linux/blob/v6.7/drivers/firmware/efi/cper.c#L639

Examples are as below.

Corrected error:
kernel: {37}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 162
kernel: {37}[Hardware Error]: It has been corrected by h/w and requires no further action
kernel: {37}[Hardware Error]: event severity: corrected
kernel: {37}[Hardware Error]:  Error 0, type: corrected
kernel: {37}[Hardware Error]:   section_type: memory error
kernel: {37}[Hardware Error]:   error_status: 0x0000000000000400
kernel: {37}[Hardware Error]:   physical_address: 0x000000b50c68ce80
kernel: {37}[Hardware Error]:   node: 1 card: 4 module: 0 rank: 0 bank: 1 device: 14 row: 58165 column: 816
kernel: {37}[Hardware Error]:   error_type: 2, single-bit ECC
kernel: {37}[Hardware Error]:   DIMM location: CPU 2 DIMM 30

Recoverable error:
kernel: {3}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4
kernel: {3}[Hardware Error]: event severity: recoverable
kernel: {3}[Hardware Error]:  Error 0, type: recoverable
kernel: {3}[Hardware Error]:  fru_text: B1
kernel: {3}[Hardware Error]:   section_type: memory error
kernel: {3}[Hardware Error]:   error_status: 0x0000000000000400
kernel: {3}[Hardware Error]:   physical_address: 0x000000393cfe5040
kernel: {3}[Hardware Error]:   node: 2 card: 0 module: 0 rank: 0 bank: 3 device: 0 row: 34719 column: 320
kernel: {3}[Hardware Error]:   DIMM location: not present. DMI handle: 0x0000

Fatal error:
kernel: BERT: Error records from previous boot:
kernel: [Hardware Error]: event severity: fatal
kernel: [Hardware Error]:  Error 0, type: fatal
kernel: [Hardware Error]:  fru_text: DIMM B5
kernel: [Hardware Error]:   section_type: memory error
kernel: [Hardware Error]:   error_status: 0x0000000000000400
kernel: [Hardware Error]:   physical_address: 0x000000393d7e4040
kernel: [Hardware Error]:   node: 2 card: 4 module: 0 rank: 0 bank: 3 device: 0 row: 34743 column: 256

Steps to test the new metrics.

$ echo "kernel: {37}[Hardware Error]: event severity: corrected" | sudo tee /dev/kmsg
$ echo "kernel: {3}[Hardware Error]: event severity: recoverable" | sudo tee /dev/kmsg
$ echo "kernel: [Hardware Error]: event severity: fatal" | sudo tee /dev/kmsg

Expected metrics are as below:
$ curl localhost:20257/metrics
problem_counter{reason="CperHardwareErrorCorrected"} 1
problem_counter{reason="CperHardwareErrorFatal"} 1
problem_counter{reason="CperHardwareErrorRecoverable"} 1
...
problem_gauge{reason="CperHardwareErrorFatal",type="CperHardwareErrorFatal"} 1

Signed-off-by: Jian Wen <wenjianhn@gmail.com>
 2025-03-12 11:00:50 +08:00
Jian WenandJian Wen 2e15606dda Monitor XFS shutdown 
Related kernel error messages are as below.

kernel: XFS (dm-4): Internal error xfs_iunlink_remove at line 2038 of file fs/xfs/xfs_inode.c.  Caller xfs_ifree+0x33/0x130 [xfs]
kernel: XFS (dm-4): Corruption detected. Unmount and run xfs_repair
kernel: XFS (dm-4): xfs_inactive_ifree: xfs_ifree returned error -117
kernel: XFS (dm-4): xfs_do_force_shutdown(0x1) called from line 1788 of file fs/xfs/xfs_inode.c.  Return address = 000000009d022bf1
kernel: XFS (dm-4): I/O Error Detected. Shutting down filesystem
kernel: XFS (dm-4): Please umount the filesystem and rectify the problem(s)

Signed-off-by: Jian Wen <wenjianhn@gmail.com>
 2024-11-14 15:46:00 +08:00
Kubernetes Prow RobotandGitHub 0f4d8b96c5 Merge pull request #961 from smileusd/upstream_add_black_list_in_log_watcher 
add black list to aviod take too much efforts to translate in file log watcher
 2024-10-16 16:09:03 +01:00
tashen 3a386a659e add skip list to aviod take too much efforts to translate in file log watcher 2024-10-16 10:56:15 +08:00
Veer Singh ee955f9170 Move ReadonlyFilesystem to separate config file 
Moved the ReadonlyFilesystem Node Condition to a separate plugin
configuration file and updated NPD to contain the appropiate new flags.
 2024-10-09 00:20:49 -07:00
baihongruandGitHub daf4f4da3e Update abrt-adaptor.json 
Indicates the unified name of KernelOops
 2024-08-16 16:40:05 +08:00
Eric Lin ce82f2a81b Update config/systemd-monitor.json to match all systemd StatusUnitFormat 2024-01-18 17:16:05 +00:00
Eric Lin c225435bea Use --revert-pattern to discount proactive restarts 2024-01-17 18:24:24 +00:00
Eric Lin 0fba03ef7a Make pattern match all systemd StatusUnitFormat 2024-01-14 20:02:13 +00:00
Antonio Ojea 552b530e0b custom plugin to monitor iptables versions rules 
iptables has two kernel backends, legacy and nft.

Quoting https://developers.redhat.com/blog/2020/08/18/iptables-the-two-variants-and-their-relationship-with-nftables

> It is also important to note that while iptables-nft
> can supplant iptables-legacy, you should never use them simultaneously.

However, we don't want to block the node operations because of this
reason, as there is no enough evidence this is causing big issues in the
wild, so we just signal and warn about this situation.

Once we have more information we can revisit this decision and
keep it as is or move it to permanent.
 2023-12-21 09:34:04 +00:00
Jarkko Sonninen 07900633cb Add disk and memory percent_used 2023-10-28 16:03:48 +03:00
Yordis Prieto Lazo 0842910049 chore: fix misspelling 2022-12-18 22:58:07 -05:00
Mike Miranda 1471f74d98 Add ExcludeInterfaceRegexp to Net Dev monitor 2022-06-15 23:22:38 +00:00
Julie Qi fe09e416bd remove aufs hung check 2021-07-30 13:53:25 -07:00
Kubernetes Prow RobotandGitHub e349323507 Merge pull request #539 from smileusd/health_check 
improvement health-checker
 2021-06-25 09:48:45 -07:00
Jeremy Edwards d52844ae67 Add HCS empty layer error reporting. 2021-06-22 17:06:42 +00:00
michelletandya caf2bad7b6 config/windows-defender-monitor.json 2021-05-24 20:08:47 +00:00
tashen a3b928467e add loopbacktime to reduce time of journalctl call 2021-05-19 13:55:55 +08:00
Kubernetes Prow RobotandGitHub 9c541692ee Merge pull request #557 from vteratipally/adfad 
Make sure the path to known-modules.json is relative
 2021-05-14 14:39:59 -07:00
Varsha Teratipally a79b87ce7e Make sure the path to known-modules.json is relative to the 
system-stats-monitor.json file
 2021-05-14 21:14:55 +00:00
michelletandya 01fa5b3afd Add windows defender problem detection custom plugin 2021-05-12 20:28:33 +00:00
Jeremy Edwards d4933875ed Add support for basic system metrics for Windows. 2021-05-10 21:58:38 +00:00
michelletandya 01cd8dd08c Add healthChecker functionality for kube-proxy service 2021-05-05 17:27:58 +00:00
michelletandya da15eb9afe Detect containerD errors and failures. 2021-04-29 23:47:04 +00:00
michelletandya c4e5400ed6 separate linux/windows health checker files. 2021-04-26 21:45:05 +00:00
michelletandya 344daabaa7 Update windows containerd config file to run without errors 2021-03-30 23:26:06 +00:00
Jeremy Edwards 4181ece888 Windows Support: Fix Build Regressions, Tests Pass 2021-03-14 10:24:45 -07:00
Kubernetes Prow RobotandGitHub 06b5503348 Merge pull request #530 from goushicui/master 
add memory read error
 2021-02-18 07:46:51 -08:00
goushicui 7ecb76f31a add memory read error 2021-02-09 14:08:18 +08:00
Karan Goel 8648fe265a add metric for per-cpu, per-stage timing 2021-01-29 08:46:39 -08:00
Kubernetes Prow RobotandGitHub e34e2763cf Merge pull request #519 from Random-Liu/fix-indention 
Fix system-stats-monitor config indention.
 2021-01-28 23:47:41 -08:00
Lantao Liu 144fad7706 Fix system-stats-monitor config indention. 2021-01-28 22:59:47 -08:00
Lantao Liu c2ad21a380 Add containerd health checker config. 2021-01-28 22:46:55 -08:00
Karan Goel 2a2bab3d28 Add network interface stats 
We do not have to collect these often, so for now set the collection
interval to 120s (even though the Stackdriver exporter is still set to
export every 60s).
 2021-01-20 08:56:34 -08:00
Kubernetes Prow RobotandGitHub c2d7a7be62 Merge pull request #513 from karan/cpu_activity_metrics 
add metrics for process stats
 2021-01-19 18:38:07 -08:00
Jeremy Edwards adc587f222 Support filelog watching in Windows. 2021-01-13 17:16:46 +00:00
Karan Goel 71098097c0 add metrics for process stats 
Tested on a COS VM:

```
$ curl -s localhost:20257/metrics | grep "^system_"
system_interrupts_total{kernel_version="5.4.49+",os_version="cos 85-13310.1041.24"} 8.759236e+07
system_processes_total{kernel_version="5.4.49+",os_version="cos 85-13310.1041.24"} 692506
system_procs_blocked{kernel_version="5.4.49+",os_version="cos 85-13310.1041.24"} 0
system_procs_running{kernel_version="5.4.49+",os_version="cos 85-13310.1041.24"} 2
```
 2021-01-13 09:14:08 -08:00
varsha teratipally f89f620909 added new line in the known_modules.json 2021-01-08 23:25:02 +00:00
varsha teratipally eb38b4b598 added a new metric to retrieve os features like unknown modules 2021-01-08 21:52:16 +00:00
Kubernetes Prow RobotandGitHub 59536256e3 Merge pull request #475 from vteratipally/boot_size_disk 
catching hung task with pattern like "tasks airflow scheduler: *"
 2020-11-18 14:42:50 -08:00
vteratipallyandGitHub 0c258bb704 Update kernel-monitor.json 2020-11-17 13:38:07 -08:00
Kubernetes Prow RobotandGitHub cff4a54d6a Merge pull request #488 from vteratipally/io_errors 
Add Detectection logic for  I/O errors
 2020-11-16 14:06:36 -08:00
Kubernetes Prow RobotandGitHub 2d53c0a2a6 Merge pull request #481 from tosi3k/oom-regex-fix 
Adapt OOMKilling pattern to old and new Linux kernels
 2020-11-16 14:06:20 -08:00
Karan Goel 925ea7393c Collect CPU load averages in a separate metric 2020-11-09 09:41:52 -08:00
varsha teratipally f01b5e5cfe Detect I/O errors 2020-11-06 03:48:33 +00:00
Antoni Zawodny 6b650e785e Adapt OOMKilling pattern to old and new Linux kernels 2020-10-22 15:12:26 +02:00
varsha teratipally f984abbe2e catching hung task with pattern like taks airflow scheduler: some of the events related to hungtask is not identified 2020-10-08 23:04:15 +00:00