#docker overview, messed up

1 messages · Page 1 of 1 (latest)

ashen cipher
#

hey guys.
i am seeing super weird data over some docker vm's i have.

Screenshot_2022-11-16_at_10.39.55_AM.png
#

first this, the docker.healthy_containers
what are the difference between red & green?
isn't red classified as unhealthy ?

#

and then this, after restart of the agent..

Screenshot_2022-11-16_at_10.49.56_AM.png
#

and this, why do i see exited when i do a docker ps all are in healthy state, none is exited.

Screenshot_2022-11-16_at_10.50.46_AM.png
subtle tulip
#

I think it's due to the single node view being replaced, the team is already on it. There's a quick fix for it in that topic

ashen cipher
#

no idea, haven't looked on the node info actually

ashen cipher
humble trout
#

@ashen cipherI think the mess comes from the fact that we have docker collector (gathers data using docker engine API (https://docs.docker.com/engine/api/v1.41/#)) written in python and go. And some nodes have python version running, some - go version.

Docker Engine API v1.41 Reference

Reference documentation and Swagger (OpenAPI) specification for the vv1.41 version of the API served by Docker Engine.

ashen cipher
#

ya, i did see that under collector info

humble trout
#

the python version hasn't been removed because we try to follow deprecation notice rule - if it wasn't announced we woudln't remove.

#

docker will be deleted right after 1.37.0, but 1.37.0 has been delayed and python docker is still around

ashen cipher
#

okay

humble trout
#

docker ps all are in healthy state, none is exited.

if exited is != 0 then you have some stopped containers

ashen cipher
#

no stopped containers, i looked

humble trout
#

those counters come from docker engine, no processing

ashen cipher
#

okay, so see this

#

right now

#

let me do docker ps on all those 25 nodes

humble trout
#

any chance you can adjust configuration?

#

disable docker in python.d.conf

ashen cipher
#

a docker ps on all 25 nodes through saltstack

humble trout
#

with health healthy/unhealthy filters and len() the result

ashen cipher
#

shouldn't this say healthy & unhealthy? 😛

Screenshot_2022-11-17_at_10.17.47_AM.png
#

and not double healthy

humble trout
ashen cipher
#

haha KEK

#

also that

#

so what should i try to do? disable docker in python.d.conf?

humble trout
#

I think python version adds some mess but I am 100% confident the numbers are correct (not aggregated), because go/python uses official libraries to connect to docker engine

#

yes, I am thinking about disabling docker in master branch

ashen cipher
#

so simply just.

[plugins]
  docker = no

in /etc/netdata/python.d.conf ?

humble trout
#

no no

ashen cipher
#

🙈

humble trout
#

python.d.conf has no [pligins]

#

it is yaml

ashen cipher
#

we don't have that that config file

humble trout
#

it is literally just

dockerd: no
ashen cipher
#

ahh, okay

humble trout
#

dockerd

ashen cipher
#

in that location i wrote?

humble trout
#

if you install path is / - yes.

ashen cipher
#

just regular install, no special installation path

humble trout
#

sometimes it is /opt/netdata so it woil;d be /opt/netdata/etc/netdata

#

ok, I believe /opt/netdata is the default for static builds

#

that are default for systems we have no binary packages for

ashen cipher
#

pushing my changes to our repo 🙂

#

okay, so the config is pushed out to my nodes & i have restarted the agent

#

this is also confusing to me

Screenshot_2022-11-17_at_10.30.12_AM.png
humble trout
#

change all dimensions to "exited" and group by node

Screenshot_2022-11-17_at_12.09.11.png
#

to find out which nodes have exited

ashen cipher
#

hmm

#

its running, e.g. just one of them that has 3.0 count

Screenshot_2022-11-17_at_12.05.40_PM.png
humble trout
#

do you do docker ps -a ?

#

-a means all, w/o -a it shows only running from what I remember

ashen cipher
#

nop, i can

#

ahh yes, that ofc shows old ones

#

3 exited, 1 healthy