config/disk-log-message-filelog.json contained trailing commas and no
trailing newline, so it could not be parsed by encoding/json and NPD
would fail to start when configured with it.
- Fix the invalid JSON
- Add TestShippedConfigs, which verifies that every shipped JSON config
under config/ (top level and one directory down) parses (catching
trailing commas, UTF-8 BOMs and other syntax errors) and is in the
canonical format (2-space indent, trailing newline), showing a diff on
failure
- Add `make fmt-configs` to rewrite nonconforming files in place
(UPDATE_EXPECTED=y)
- Normalize the existing config files to the canonical format
(whitespace-only changes)
The restart path closes the failed parser before retrying. If stopping
is signaled during the retry wait, watchLoop's deferred cleanup closed
the same parser again, logging a spurious 'file already closed' error
at shutdown. Clear the reference after closing and nil-check the defer.
The log monitor stops draining logCh before calling watcher.Stop(), so
with a full channel (e.g. a kmsg burst at shutdown) watchLoop blocked
on the send forever, never called tomb.Done(), and Stop() hung.
Select on tomb.Stopping() alongside the send.
If the restarted parser's channel closes again right away (reads keep
failing on the reopened /dev/kmsg), the watcher restarts in a tight
loop with no delay, spinning a CPU core and flooding the logs
(measured 37k restarts in 200ms). Delay the first attempt when the
previous restart was less than retryDelay ago.
Closing the parser in Stop() triggers kmsgparser's read goroutine to
close its output channel, which made watchLoop take the restart path
(added in #1192) during intentional shutdown. Leave parser cleanup to
watchLoop's defer, which is the single owner of the parser lifecycle.
If you're using some monitoring solution that aggregates events from
your Kubernetes cluster, having the underlying reason why a condition
triggered could be very useful, especially if you are using custom
plugin monitors.
Co-authored-by: Micah Norman <micnorman@paypal.com>
Signed-off-by: Ryan Eschinger <reschinger@paypal.com>
Ciprian Hacman
Kubernetes Prow RobotandGitHub
Arjun Raman
Arjun RamanandGitHub
Sergey Kanzhelev
googs1025
tashen
Manuel RügerandCiprian Hacman
Fan Shang Xiang
Kubernetes Prow RobotandGitHub
guoguangwu
Fan Shang Xiang
Michael Grosser
Yordis Prieto Lazo
whwreflux
Jeremy Edwards
yuzhiquan
Lantao Liu
wenjun93
Andrew Stribblehill
Zhen Wang
Xuewei Zhang
AdamDangandGitHub