Hey team, I have container running older version of netdata/netdata:v2.2.0 it's showing my logs correctly. in my docker config, I did mount /var/log/journald:/var/log/journald:ro to netdata container and all other services container I am pointing them to use log_driver: journald
#journald stopped working starting 2.2.0
woeful frost
was there any change missed ? cause that other server running container 2.0.2 I can still see logs and stream them from logs tab. I am planning to move to >2.2.0 cause there is security updates. just want to make sure if I am missing something regarding logging
woeful frost
Intersting, tho both are running same netdata but different debian version on the one that's not working
time=2025-04-04T20:47:00.561+02:00 comm=systemd-journal.plugin source=collector level=info errno="2, No such file or directory" tid=4055140 thread=sd-jrnl.plugin msg="unable to locate journal data directories. Exiting..."
time=2025-04-04T20:47:00.562+02:00 comm=netdata source=daemon level=info tid=4055128 thread=PD[systemd-jour module=plugins.d[systemd-journal.plugin] node=server_data src_transport=pluginsd request='DISABLE' msg="PLUGINSD: plugin called DISABLE. Disabling it."
time=2025-04-04T20:47:00.562+02:00 comm=netdata source=collector level=warning tid=4055128 thread=PD[systemd-jour module=plugins.d[systemd-journal.plugin] node=server_data src_transport=pluginsd msg="PLUGINSD: 'host:server_data', '/usr/libexec/netdata/plugins.d/systemd-journal.plugin' (pid 4055140) disconnected after 0 successful data collections."
time=2025-04-04T20:47:00.564+02:00 comm=netdata source=daemon level=info errno="3, No such process" tid=4055128 thread=PD[systemd-jour module=plugins.d[systemd-journal.plugin] node=server_data src_transport=pluginsd msg="PLUGINSD: 'host:server_data', '/usr/libexec/netdata/plugins.d/systemd-journal.plugin' (pid 4055140) does not generate useful output but it reports success (exits with 0). Will not start it again - it is now disabled.."
similar to this https://community.netdata.cloud/t/no-systemd-log-files/5316 but it's not related to my current setup with docker containers
the server that is working is using docker engine 24.0 and one is not working 27.3.1
I am using ansible to provision netdata with docker plugin on those instances. the question is what could be the cuplrit
woeful frost
logs showing warning not error. Tho I am not sure why this instance logs are not psuhed to cloud
obsidian palm
Hi. Your logs contain
unable to locate journal data directories
Your mount is wrong, should be (from https://learn.netdata.cloud/docs/netdata-agent/installation/docker#recommended-way)
-v /var/log:/host/var/log:ro \
or if you want to be specific
/var/log/journald:/host/var/log/journald:ro
Hi @obsidian palm
netdata_volumes:
- "{{ ansible_env.HOME }}/netdata:/etc/netdata"
- /:/host/root:ro,rslave
- /etc/passwd:/host/etc/passwd:ro
- /etc/group:/host/etc/group:ro
- /etc/localtime:/etc/localtime:ro
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /etc/os-release:/host/etc/os-release:ro
- /var/log:/host/var/log:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- /run/dbus:/run/dbus:ro
even with this it's not working
my bad I should've update my post 🙂
it's not about the mount. this config is the same for both debian servers.
when I run docker logs netdata | grep "journal" i don't see any error.
latest logs
time=2025-04-05T12:22:57.394+02:00 comm=systemd-journal.plugin source=collector level=info tid=991225 thread=sd-jrnl.plugin msg="Using host prefix directory '/host'"
time=2025-04-05T12:22:57.396+02:00 comm=systemd-journal.plugin source=collector level=info errno="2, No such file or directory" tid=991225 thread=sd-jrnl.plugin msg="unable to locate journal data directories. Exiting..."
time=2025-04-05T12:22:57.478+02:00 comm=netdata source=daemon level=info tid=991161 thread=PD[systemd-jour module=plugins.d[systemd-journal.plugin] node=server_data src_transport=pluginsd request='DISABLE' msg="PLUGINSD: plugin called DISABLE. Disabling it."
time=2025-04-05T12:22:57.482+02:00 comm=netdata source=daemon level=warning errno="32, Broken pipe" tid=991161 thread=PD[systemd-jour module=plugins.d[systemd-journal.plugin] node=server_data src_transport=pluginsd msg="PLUGINSD: cannot send command to plugin (fd = 90, sent bytes = -1 out of 4)"
time=2025-04-05T12:22:57.482+02:00 comm=netdata source=collector level=warning tid=991161 thread=PD[systemd-jour module=plugins.d[systemd-journal.plugin] node=server_data src_transport=pluginsd msg="PLUGINSD: 'host:server_data', '/usr/libexec/netdata/plugins.d/systemd-journal.plugin' (pid 991225) disconnected after 0 successful data collections."
time=2025-04-05T12:22:57.679+02:00 comm=netdata source=daemon level=info errno="3, No such process" tid=991161 thread=PD[systemd-jour module=plugins.d[systemd-journal.plugin] node=server_data src_transport=pluginsd msg="PLUGINSD: 'host:server_data', '/usr/libexec/netdata/plugins.d/systemd-journal.plugin' (pid 991225) does not generate useful output but it reports success (exits with 0). Will not start it again - it is now disabled.."
time=2025-04-05T13:51:38.686+02:00 comm=netdata source=daemon level=info tid=990938 thread=UV_WORKER[1] msg="DBENGINE: created journal file \"/var/cache/netdata/dbengine/journalfile-1-0000000009.njf\"."
I understand that it's saying unable to locate journal data directories is there any known issue for debian 12 on armbian ?
obsidian palm
There is no issue. Check that directory inside dicker container. It won’t have journald logs.
woeful frost
@obsidian palm it's there /host/var/log
drwxr-xr-x 8 root root 4.0K Mar 28 14:18 .
drwxr-xr-x 3 root root 4.0K Apr 5 12:22 ..
lrwxrwxrwx 1 root root 39 Nov 6 03:50 README -> ../../usr/share/doc/systemd/README.logs
-rw-r--r-- 1 root root 0 Apr 6 00:00 alternatives.log
drwxr-xr-x 2 root root 4.0K Apr 6 00:00 apt
-rw-r--r-- 1 root root 0 Apr 6 00:00 armbian-hardware-monitor.log
-rw-r--r-- 1 root root 0 Apr 6 00:00 armbian-install.log
-rw-r----- 1 root adm 25K Apr 6 18:03 auth.log
-rw-r--r-- 1 root root 0 Apr 6 00:00 bootstrap.log
-rw-rw---- 1 root utmp 0 Apr 6 00:00 btmp
-rw-r----- 1 root adm 11K Apr 6 18:00 cron.log
-rw-r--r-- 1 root root 0 Apr 6 00:00 dpkg.log
-rw-r--r-- 1 root root 0 Apr 6 00:00 faillog
lrwxrwxrwx 1 root 999 20 Mar 28 14:17 journal -> /var/log.hdd/journal
drwxr-xr-x 2 root root 4.0K Mar 28 14:18 journald
-rw-r----- 1 root adm 52K Apr 6 03:32 kern.log
-rw-rw-r-- 1 root utmp 290K Apr 6 18:03 lastlog
drwx------ 2 root root 16K Mar 28 14:17 lost+found
drwx------ 2 root root 4.0K Nov 6 03:50 private
drwxr-xr-x 3 root root 4.0K Nov 6 03:51 runit
-rw-r----- 1 root adm 6.6M Apr 6 18:04 syslog
drwxr-x--- 2 root adm 4.0K Nov 25 18:54 unattended-upgrades
-rw-r----- 1 root adm 6.2M Apr 6 18:04 user.log
-rw-rw-r-- 1 root utmp 108K Apr 6 18:03 wtmp
Unfortunately, I can't reproduce the problem.
This is journal directories check function (Netdata logs "unable to locate journal data directories" if it fails)
It follows symlinks (uses stat).
These are default directories
/host/var/log/journal
/host/run/log/journal
You get unable to locate journal data directories if both of these directories don't exist or exist but not a directory.
Are your logs in /host/var/log/journald? If so, this won't work, as the plugin assumes that the logs are in
- /host/var/log/journal
- /host/run/log/journal
You can add a new directory using dyncfg
woeful frost
@obsidian palm I don't have this logs tabs in my node config
obsidian palm
Right. The only way to make it working is to mount to /host/var/log/journal not /host/var/log/journald. You will need to change this symlink /host/var/log/journal -> /var/log.hdd/journal.