The restart path closes the failed parser before retrying. If stopping
is signaled during the retry wait, watchLoop's deferred cleanup closed
the same parser again, logging a spurious 'file already closed' error
at shutdown. Clear the reference after closing and nil-check the defer.
The log monitor stops draining logCh before calling watcher.Stop(), so
with a full channel (e.g. a kmsg burst at shutdown) watchLoop blocked
on the send forever, never called tomb.Done(), and Stop() hung.
Select on tomb.Stopping() alongside the send.
If the restarted parser's channel closes again right away (reads keep
failing on the reopened /dev/kmsg), the watcher restarts in a tight
loop with no delay, spinning a CPU core and flooding the logs
(measured 37k restarts in 200ms). Delay the first attempt when the
previous restart was less than retryDelay ago.
Closing the parser in Stop() triggers kmsgparser's read goroutine to
close its output channel, which made watchLoop take the restart path
(added in #1192) during intentional shutdown. Leave parser cleanup to
watchLoop's defer, which is the single owner of the parser lifecycle.
Usage of `localhost` is family agnostic and will work regardless if
cluster is IPv4 or IPv6. The current value of `127.0.0.1` only works
for IPv4 clusters.
BREAKING CHANGE: It may break in rare cases where `localhost` does not
resolve as `127.0.0.1` (if OS config does not follow RFC5735 and
RFC6761).
Both outputs are currently hardcoded to being enabled, this allows disabling one or the other. Defaults to both enabled to retain current behavior.
Larger clusters can save some etcd I/O by skipping one of these outputs if they aren't being consumed. In our case we aren't consuming the Events so writing them just creates more churn.
see https://pkg.go.dev/net/http/pprof
> By default, all the profiles listed in runtime/pprof.Profile are
available (via Handler), in addition to the Cmdline, Profile, Symbol,
and Trace profiles defined in this package. If you are not using
DefaultServeMux, you will have to register handlers with the mux you are
using.
Signed-off-by: j4ckstraw <j4ckstraw@foxmail.com>
Ciprian Hacman
Kubernetes Prow RobotandGitHub
Arjun Raman
Arjun RamanandGitHub
novahe
Sergey Kanzhelev
marinoborgesandGitHub
Johanan Liebermann
Marek Chodor
Chris Henzie
Nick Parker
googs1025
tashen
Sudheer Nedlumane
Zhen Wang
Henry Beberman
Eric Lin
Chotiwat ChawannakulandGitHub
StanislavandStanislav Egorkin
Kubernetes Prow RobotandGitHub
Jarkko Sonninen
j4ckstraw