The restart path closes the failed parser before retrying. If stopping
is signaled during the retry wait, watchLoop's deferred cleanup closed
the same parser again, logging a spurious 'file already closed' error
at shutdown. Clear the reference after closing and nil-check the defer.
The log monitor stops draining logCh before calling watcher.Stop(), so
with a full channel (e.g. a kmsg burst at shutdown) watchLoop blocked
on the send forever, never called tomb.Done(), and Stop() hung.
Select on tomb.Stopping() alongside the send.
If the restarted parser's channel closes again right away (reads keep
failing on the reopened /dev/kmsg), the watcher restarts in a tight
loop with no delay, spinning a CPU core and flooding the logs
(measured 37k restarts in 200ms). Delay the first attempt when the
previous restart was less than retryDelay ago.
Closing the parser in Stop() triggers kmsgparser's read goroutine to
close its output channel, which made watchLoop take the restart path
(added in #1192) during intentional shutdown. Leave parser cleanup to
watchLoop's defer, which is the single owner of the parser lifecycle.
This adds a logwatcher which is able to parse kernel messages directly
from the /dev/kmsg interface. This supports any modern linux distro,
while also avoiding any dependency on libraries (e.g. as journald
needs).
Ciprian Hacman
Kubernetes Prow RobotandGitHub
Arjun Raman
Arjun RamanandGitHub
Sergey Kanzhelev
Manuel RügerandCiprian Hacman
Jeremy Edwards
Lantao Liu
Zhen Wang
Andy Xie
Euan Kemp