[Bug]: Child node service reports as active, but seems hung and is stale in Netdata Cloud #16809
Comments
|
I experience the very same issue with a netdata parent setup. Some logs: |
|
@shodanshok stable version too? I think the problem is fixed in the latest. |
|
Yes, I am using the latest stable version (1.44) |
|
FYI, since I posted the issue, I've tried playing around with a few options/plugins, and it seems disabling the |
|
@asteinlein Are you still having this issue after updating to v1.44.2? |
I'm still running v1.44.1. Shouldn't it automatically update? ( |
|
Check your auto updates Apt-get update just fetches changes in the repo, not upgrading any piece of software. |
|
I've upgraded and enabled the |
Bug description
After a few hours of running the client, the agent it seems to somehow die/stop collecting and reporting data, even though
systemctl status netdatareports the state as active.The log seems to indicate that a child was killed somehow, and that collectors/replication have stopped, but I have no idea why nor why the agent seems to still run. When I finally tried to stop the client with
systemctl stop netplan, it hung for a long time, before it was eventually automatically killed.Expected behavior
It should continue to run, collect metrics and have the metrics show up in Netdata Cloud.
Steps to reproduce
Installation method
kickstart.sh
System info
Netdata build info
Packaging: Netdata Version ____________________________________________ : v1.44.1 Installation Type __________________________________________ : binpkg-deb Package Architecture _______________________________________ : x86_64 Package Distro _____________________________________________ : Configure Options __________________________________________ : '--build=x86_64-linux-gnu' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--prefix=/usr' '--sysconfdir=/etc' '--localstatedir=/var' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' '--with-user=netdata' '--with-math' '--with-zlib' '--with-webdir=/var/lib/netdata/www' '--disable-dependency-tracking' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fdebug-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fdebug-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security' Default Directories: User Configurations ________________________________________ : /etc/netdata Stock Configurations _______________________________________ : /usr/lib/netdata/conf.d Ephemeral Databases (metrics data, metadata) _______________ : /var/cache/netdata Permanent Databases ________________________________________ : /var/lib/netdata Plugins ____________________________________________________ : /usr/libexec/netdata/plugins.d Static Web Files ___________________________________________ : /var/lib/netdata/www Log Files __________________________________________________ : /var/log/netdata Lock Files _________________________________________________ : /var/lib/netdata/lock Home _______________________________________________________ : /var/lib/netdata Operating System: Kernel _____________________________________________________ : Linux Kernel Version _____________________________________________ : 5.4.0-131-generic Operating System ___________________________________________ : Ubuntu Operating System ID ________________________________________ : ubuntu Operating System ID Like ___________________________________ : debian Operating System Version ___________________________________ : 20.04.6 LTS (Focal Fossa) Operating System Version ID ________________________________ : none Detection __________________________________________________ : /etc/os-release Hardware: CPU Cores __________________________________________________ : 32 CPU Frequency ______________________________________________ : 3000000000 RAM Bytes __________________________________________________ : 67240386560 Disk Capacity ______________________________________________ : 1920394248192 CPU Architecture ___________________________________________ : x86_64 Virtualization Technology __________________________________ : none Virtualization Detection ___________________________________ : systemd-detect-virt Container: Container __________________________________________________ : none Container Detection ________________________________________ : systemd-detect-virt Container Orchestrator _____________________________________ : none Container Operating System _________________________________ : none Container Operating System ID ______________________________ : none Container Operating System ID Like _________________________ : none Container Operating System Version _________________________ : none Container Operating System Version ID ______________________ : none Container Operating System Detection _______________________ : none Features: Built For __________________________________________________ : Linux Netdata Cloud ______________________________________________ : YES Health (trigger alerts and send notifications) _____________ : YES Streaming (stream metrics to parent Netdata servers) _______ : YES Back-filling (of higher database tiers) ____________________ : YES Replication (fill the gaps of parent Netdata servers) ______ : YES Streaming and Replication Compression ______________________ : YES (zstd lz4 gzip) Contexts (index all active and archived metrics) ___________ : YES Tiering (multiple dbs with different metrics resolution) ___ : YES (5) Machine Learning ___________________________________________ : YES Database Engines: dbengine ___________________________________________________ : YES alloc ______________________________________________________ : YES ram ________________________________________________________ : YES map ________________________________________________________ : YES save _______________________________________________________ : YES none _______________________________________________________ : YES Connectivity Capabilities: ACLK (Agent-Cloud Link: MQTT over WebSockets over TLS) _____ : YES static (Netdata internal web server) _______________________ : YES h2o (web server) ___________________________________________ : YES WebRTC (experimental) ______________________________________ : NO Native HTTPS (TLS Support) _________________________________ : YES TLS Host Verification ______________________________________ : YES Libraries: LZ4 (extremely fast lossless compression algorithm) ________ : YES ZSTD (fast, lossless compression algorithm) ________________ : YES zlib (lossless data-compression library) ___________________ : YES Judy (high-performance dynamic arrays and hashtables) ______ : YES (bundled) dlib (robust machine learning toolkit) _____________________ : YES (bundled) protobuf (platform-neutral data serialization protocol) ____ : YES (system) OpenSSL (cryptography) _____________________________________ : YES libdatachannel (stand-alone WebRTC data channels) __________ : NO JSON-C (lightweight JSON manipulation) _____________________ : YES libcap (Linux capabilities system operations) ______________ : NO libcrypto (cryptographic functions) ________________________ : YES libm (mathematical functions) ______________________________ : YES jemalloc ___________________________________________________ : NO TCMalloc ___________________________________________________ : NO Plugins: apps (monitor processes) ___________________________________ : YES cgroups (monitor containers and VMs) _______________________ : YES cgroup-network (associate interfaces to CGROUPS) ___________ : YES proc (monitor Linux systems) _______________________________ : YES tc (monitor Linux network QoS) _____________________________ : YES diskspace (monitor Linux mount points) _____________________ : YES freebsd (monitor FreeBSD systems) __________________________ : NO macos (monitor MacOS systems) ______________________________ : NO statsd (collect custom application metrics) ________________ : YES timex (check system clock synchronization) _________________ : YES idlejitter (check system latency and jitter) _______________ : YES bash (support shell data collection jobs - charts.d) _______ : YES debugfs (kernel debugging metrics) _________________________ : YES cups (monitor printers and print jobs) _____________________ : YES ebpf (monitor system calls) ________________________________ : YES freeipmi (monitor enterprise server H/W) ___________________ : YES nfacct (gather netfilter accounting) _______________________ : YES perf (collect kernel performance events) ___________________ : YES slabinfo (monitor kernel object caching) ___________________ : YES Xen ________________________________________________________ : NO Xen VBD Error Tracking _____________________________________ : NO Logs Management ____________________________________________ : YES Exporters: AWS Kinesis ________________________________________________ : NO GCP PubSub _________________________________________________ : NO MongoDB ____________________________________________________ : YES Prometheus (OpenMetrics) Exporter __________________________ : YES Prometheus Remote Write ____________________________________ : YES Graphite ___________________________________________________ : YES Graphite HTTP / HTTPS ______________________________________ : YES JSON _______________________________________________________ : YES JSON HTTP / HTTPS __________________________________________ : YES OpenTSDB ___________________________________________________ : YES OpenTSDB HTTP / HTTPS ______________________________________ : YES All Metrics API ____________________________________________ : YES Shell (use metrics in shell scripts) _______________________ : YES Debug/Developer Features: Trace All Netdata Allocations (with charts) ________________ : NO Developer Mode (more runtime checks, slower) _______________ : NOAdditional info
systemctl status netdatawhen the agent had hung/stopped reporting:I then tried inspecting logs from Netdata, but I'm a bit confused why I'm not seeing the messages reported above. Instead, all I'm seeing with
journalctl -u netdatais:Then tried to stop Netdata with
systemctl stop netdatawhich hung, but was eventually killed:The text was updated successfully, but these errors were encountered: