HassOS `ha resolution info` just | Home Assistant | Page 1

void holly May 6, 2022, 9:01 PM

#

Anything in the logs for DNS or supervisor?

#

anything in the logs for dns or supervisor?

#

also what does ha network info say?

jolly shard May 6, 2022, 9:08 PM

#

https://paste.pfak.org/wx413p86

#

as for the dns container, anything in particular that you're looking for?

void holly May 6, 2022, 9:10 PM

#

ok that looks fine. and so what specifically doesn't work? core_addon_ssh isn't a hostname of anything I know. Do you mean core-ssh and core-ssh.local.hass.io don't work?

#

fwiw the line you linked isn't involved in container name resolution. Supervisor provides a hosts file to the dns container that it uses to handle intercontainer communication. To my knowledge nothing about that was changed

jolly shard May 6, 2022, 9:13 PM

#

# docker container inspect addon_core_ssh | jq '.[] | .NetworkSettings.Networks'
{
  "hassio": {
    "IPAMConfig": null,
    "Links": null,
    "Aliases": [
      "core-ssh",
      "735275fb43a1"
    ],
    "NetworkID": "c7a3fd5d20a8e589823bcb2afc714fa71a8feee06e554d6a81488433330cd309",
    "EndpointID": "7f664412741d7ecea3f3ecca75ff9b6e0f5faab4c3af8205414470c6fe01057a",
    "Gateway": "172.30.32.1",
    "IPAddress": "172.30.33.3",
    "IPPrefixLen": 23,
    "IPv6Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "MacAddress": "02:42:ac:1e:21:03",
    "DriverOpts": null
  }
}

#

Let me see if core-ssh is also unresolvable, I suspect the answer is it is. But you should be able resolve the docker container name (addon_core_ssh, as well as any aliases (core-ssh, $CONTAINER_ID)

void holly May 6, 2022, 9:15 PM

#

This is the resolv conf handed to all containers managed by supervisor:

> cat /etc/resolv.conf               search local.hass.io
nameserver 172.30.32.3

#

And for the dns plugin, here's what its given in its host file:

127.0.0.1 localhost localhost.local.hass.io
172.30.32.2 hassio hassio.local.hass.io supervisor supervisor.local.hass.io
172.30.32.1 homeassistant homeassistant.local.hass.io home-assistant home-assistant.local.hass.io
172.30.32.3 dns dns.local.hass.io
172.30.32.6 observer observer.local.hass.io
172.30.32.1 a0d7b954-ssh a0d7b954-ssh.local.hass.io
172.30.32.1 a0d7b954-adguard a0d7b954-adguard.local.hass.io

jolly shard May 6, 2022, 9:16 PM

#

[core-ssh config]$ cat /etc/resolv.conf
search local.hass.io
nameserver 127.0.0.11
options ndots:0

#

HassoS host:

bash-5.1# cat /etc/resolv.conf
search local.hass.io
nameserver 172.30.32.3

void holly May 6, 2022, 9:19 PM

#

ah yea I see that too, I was looking at a host network addon

#

so I mean all the addons point at hassio_dns and to my knowledge that's never been able to resolve core_addon_ssh

jolly shard May 6, 2022, 9:20 PM

#

sorry im trying to put together a repro

void holly May 6, 2022, 9:20 PM

#

it doesn't deal in docker names, it deals in the hostnames supervisor gives it

jolly shard May 6, 2022, 9:20 PM

#

the system should be able to resolve any docker container name or alias..

#

[core-ssh config]$ ha dns info
fallback: false
host: 172.30.32.3
llmnr: true
locals:
- dns://192.168.3.1
mdns: true
servers:
- dns://127.0.0.11
update_available: true
version: 2022.04.0
version_latest: 2022.04.1

void holly May 6, 2022, 9:21 PM

#

I mean the only way to know the docker names is to gain access to the docker CLI

#

its not published anywhere, the names we support are

jolly shard May 6, 2022, 9:21 PM

#

[core-ssh config]$ dig addon_core_ssh. @127.0.0.11

; <<>> DiG 9.16.20 <<>> addon_core_ssh. @127.0.0.11
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7841
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;addon_core_ssh.                        IN      A

;; ANSWER SECTION:
addon_core_ssh.         600     IN      A       172.30.33.3

;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Fri May 06 14:21:38 PDT 2022
;; MSG SIZE  rcvd: 62

#

so HASSIO is relying on whatever is written to /etc/hosts in each container?

#

im not sure that's the case?

# docker container exec -it homeassistant cat /etc/hosts
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.30.32.6     observer
172.30.32.2     supervisor

void holly May 6, 2022, 9:23 PM

#

no. supervisor writes a file mapped to /config/hosts in the hassio_dns container

#

the dns plugin resolves names within addons/core/supervisor

jolly shard May 6, 2022, 9:23 PM

#

which is coreDNS?

void holly May 6, 2022, 9:24 PM

#

yea

jolly shard May 6, 2022, 9:24 PM

#

maybe I dont understand why it isnt just using dockers built in DNS ..

void holly May 6, 2022, 9:24 PM

#

https://github.com/home-assistant/plugin-dns this repo

jolly shard May 6, 2022, 9:24 PM

#

clearly someone thought its being used that way, or they wouldnt have defined aliases in networks for hassio lol

#

yes ive looked at that repo a bunch and scratched my head a lot

#

what does servers in ha dns info output look like for you?

#

I think I set 127.0.0.11 myself to work around the docker DNS resolution issues ... so im wondering if 2022.4.1 broke servers ?

#

something changed between .0 and .1

void holly May 6, 2022, 9:25 PM

#

it has nothing for me, I just use the dhcp provided one. or well I chagned to a static config so that one

jolly shard May 6, 2022, 9:26 PM

#

127.0.0.11 is sytemd-resolved.

void holly May 6, 2022, 9:26 PM

#

fallback: true
host: 172.30.32.3
llmnr: true
locals:
- dns://192.168.1.5
mdns: true
servers: []
update_available: false
version: 2022.04.1
version_latest: 2022.04.1

jolly shard May 6, 2022, 9:26 PM

#

ahhh so its not respecting servers anymore

void holly May 6, 2022, 9:26 PM

#

oh wait

#

I think I know what happened

jolly shard May 6, 2022, 9:27 PM

#

wait, you took over this plugin. ah ha.

void holly May 6, 2022, 9:27 PM

#

so as part of fixing https://github.com/home-assistant/plugin-dns/issues/70 we did an audit. One of the things realized was that we were handling multicast names incorrectly

jolly shard May 6, 2022, 9:27 PM

#

yes you were

#

(=

void holly May 6, 2022, 9:27 PM

#

yea lol

#

so what we do now is hand off .local and single name queries to systemd-resolved

jolly shard May 6, 2022, 9:28 PM

#

if thats the case I wonder why I cant resolve any container names

void holly May 6, 2022, 9:28 PM

#

but that's it. If systemd-resolved does not come back with an answer we respond with nxdomain

#

because that's also what systemd-resolved does. It will not hand off single name or .local queries to any external resolvers

jolly shard May 6, 2022, 9:28 PM

#

becasue I acn do dig $CONTAINER_NAME @127.0.0.11 and they respond ..

void holly May 6, 2022, 9:28 PM

#

yes currently we don't incorporate docker's dns servers in there

jolly shard May 6, 2022, 9:29 PM

#

I always thought 127.0.0.11 was systemd resolved.. maybe im wrong

void holly May 6, 2022, 9:29 PM

#

I don't believe so. docker normally has its own dns for containers

jolly shard May 6, 2022, 9:29 PM

#

and lsof on HASSIO must be busybox because it doesnt actually show me everything ..

void holly May 6, 2022, 9:30 PM

#

you mean on the host os? yea its busybox

jolly shard May 6, 2022, 9:30 PM

#

Additionally, systemd-resolved provides a local DNS stub listener on the IP addresses 127.0.0.53 and 127.0.0.54 on the local loopback interface

void holly May 6, 2022, 9:31 PM

#

believe so anyway. its pretty limited, especially with squashfs making everything read only

jolly shard May 6, 2022, 9:31 PM

#

so 127.0.0.11 must be docker, and servers must be no longer used ...

void holly May 6, 2022, 9:31 PM

#

servers is used but not for single name or .local queries

#

because those are expected to be llmnr and mdns queries

jolly shard May 6, 2022, 9:32 PM

#

so just to clarify, from >= 2022.4.1 any DNS that is a shortname or local is handed off to systemd-resolved (127.0.0.53)?

void holly May 6, 2022, 9:33 PM

#

yes with one exception. If you go into the hassio_dns container and do cat /config/hosts that's the file supervisor writes out for it

#

that takes precedence

jolly shard May 6, 2022, 9:33 PM

#

ok

void holly May 6, 2022, 9:33 PM

#

sorry. honestly this was the first I heard of someone trying to use the docker container names. it does make sense though. but yea I can see now how that did work and doesn't anymore

#

within supervisor anyway

#

obviously normal in standard docker practice

jolly shard May 6, 2022, 9:34 PM

#

that's the only way to reference a container between reboots, I cant use the IP

void holly May 6, 2022, 9:34 PM

#

well you can just use the hostname listed on the addon

jolly shard May 6, 2022, 9:34 PM

#

docker image pull ..

void holly May 6, 2022, 9:35 PM

#

ugh I can't put in images

jolly shard May 6, 2022, 9:35 PM

#

heh

#

Now im wondering if I can set the systemd primary resolver to docker

void holly May 6, 2022, 9:35 PM

#

Like here: https://ha.local.degatano.com/hassio/addon/core_ssh/info

#

if you go there look on the right side

#

it says "Hostname" and lists a hostname

#

that's there for every add-on and supervisor makes sure that name keeps resolving

jolly shard May 6, 2022, 9:36 PM

#

yep

void holly May 6, 2022, 9:36 PM

#

can also see if if you do ha addons info core_ssh

jolly shard May 6, 2022, 9:36 PM

#

but is there a reason for coredns? this is all natively handled by docker ...

#

i know you've put a lot of work into it.

void holly May 6, 2022, 9:38 PM

#

so I'm not 100% on the history but I've pieced some of it together. There's a few reasons but I believe the biggest one is what I detailed out here: https://community.home-assistant.io/t/local-dns/178108/119?u=centralcommand

#

I wasn't working on supervisor when it happened but I gathered when we really leaned into musl we hit major issues with DNS where people had DNS servers that returned NXDOMAIN responses for AAAA queries on domains that only resolved for A queries

jolly shard May 6, 2022, 9:39 PM

#

I remember that.

void holly May 6, 2022, 9:39 PM

#

a big one that has this is github.com and ghcr.io

#

so that was a really big deal

#

that's why the fallback DNS option exists and why it handles NXDOMAIN

jolly shard May 6, 2022, 9:40 PM

#

so im basically stuck on 2022.4.0 or I have to fork the core DNS addon? any way to support the old behaviour for short names and .local?

#

maybe somewhere in the docs explaining the change in behaviour would help as well

void holly May 6, 2022, 9:41 PM

#

can you switch to using the hostnames published by the addons?

#

I mean tbh we don't actually provide any guarantees the container names will stay the same

#

they have so far but that's not something we really guarantee since its supposed to be internal

jolly shard May 6, 2022, 9:41 PM

#

These are containers I've launched myself, totally unsupported.. but I imagine (And ive already seen open issues) that changing .local behaviour has broken stuff for people

#

and yes I know .local shouldnt be used (=

void holly May 6, 2022, 9:42 PM

#

it did yea. I've responded to a couple posts on it. People actually seemed to get that handing a .local query off to an external resolver was wrong and basically just switched to .home.arpa so far or a different internal tld

#

the spec publishes a number of pure internal TLDs for this purpose

jolly shard May 6, 2022, 9:43 PM

#

yeah i remember the RFC

void holly May 6, 2022, 9:47 PM

#

so I could fix the docker container ones but it won't be immediate and tbh I'm not a huge fan because I honestly don't know if the container names are fully stable. The hostnames are but I'm not sure if there's some edge case that can cause us to pick different ones I've never looked into it. Plus it would be tricky to figure out how to solve it without handing single name queries anywher besides the docker resolvers, we don't want to hand them to external resolvers anymore (one of the callouts in the CVE)

#

out of curiousity, what is wrong with the mariadb addon?

jolly shard May 6, 2022, 9:47 PM

#

okay. so I thought I was being smart. because $ALIAS.$NETWORK should be resolvable, i.e. core-ssh.hassio
but .. in 2022.04.1 it also doesnt work
I thought you said it was just short names?

#

container names are stable.

#

out of curiousity, what is wrong with the mariadb addon?

#

I had some performance issues that were resolved with a newer version of MariaDB and being able to set InnoDB memory.

#

and there is no deepstack addon.

void holly May 6, 2022, 9:49 PM

#

core-ssh.hassio
That... is confusing. I don't know why that one would've stopped, hrm

jolly shard May 6, 2022, 9:49 PM

#

https://github.com/home-assistant/plugin-dns/commit/8e0ea74e46bd85579948b0e3b1e0bbad35bbacc7
Was this the commit?

void holly May 6, 2022, 9:50 PM

#

oh

jolly shard May 6, 2022, 9:50 PM

#

man I have PTSD of k8s.

#

sets computer on fire and throws it out the window

void holly May 6, 2022, 9:50 PM

#

yea that would do it

#

template ANY A local.hass.io hassio {
        rcode NXDOMAIN
    }

#

since we assume if the name wasn't in the hosts file its not in hassio

jolly shard May 6, 2022, 9:51 PM

#

im just rtfm'ing coredns

#

well that answers my question.

void holly May 6, 2022, 9:53 PM

#

so tbh my suggestion would be this. Not exactly fork the repo (since there's a lot of addons in there) but copy just the mariadb addon into a new repo. Then tweak its config to your needs

#

And tbh I think deepstack should be pretty easy? Let me look at it

#

Like addon configuration has an image fields wher eyou just put an image

jolly shard May 6, 2022, 9:54 PM

#

yeah for deepstack I just do:

docker run -d --net hassio --name deepstack --restart unless-stopped -e VISION-DETECTION=True -v localstorage:/datastore -p 80:5000 deepquestai/deepstack

void holly May 6, 2022, 9:54 PM

#

so if you have an existing image you just put it there

jolly shard May 6, 2022, 9:56 PM

#

I think when I implemented this local addons weren't a thing.

void holly May 6, 2022, 9:56 PM

#

ah

#

yea you can just dump it in that folder

jolly shard May 6, 2022, 9:56 PM

#

im just rtfming that page

void holly May 6, 2022, 9:56 PM

#

The only downside with local addons is backups. Since it builds the image locally it includes the entire image in your backup

jolly shard May 6, 2022, 9:56 PM

#

meh. that sounds actually better.

void holly May 6, 2022, 9:57 PM

#

oh perfect then, that's easy

jolly shard May 6, 2022, 9:57 PM

#

i decided to update in the middle of the day, noticed nothing was working, tried to roll back and of course the schema had been updated ...

void holly May 6, 2022, 9:57 PM

#

if that does bother you at some point can just dump what you've already got working into a git repo and tell github actions to build the image for you on pull

#

that's what I do for my addons

#

(I copied frenck, I'm not that cool lol)

jolly shard May 6, 2022, 9:58 PM

#

is just a lowly sysadmin

#

hackhackhack

#

Maybe consider dropping at some point all the custom hosts and using dockers built in DNS resolution, may simplify things for you ..

#

I run a bunch of docker swarm clusters and I have had zero issue with DNS resolution in docker betwen containers.

void holly May 6, 2022, 9:59 PM

#

yea its fair. although it would be quite a headache now with the change in hostnames from stuff like core-ssh to addon_core_ssh

jolly shard May 6, 2022, 9:59 PM

#

someone, at some point, clearly knew about them.. or aliases wouldnt be defined for each container ..

#

no no, it can support multiple ..

void holly May 6, 2022, 9:59 PM

#

oh right duh yea the alieas

jolly shard May 6, 2022, 9:59 PM

#

someone already defined core-ssh for the containers ...

void holly May 6, 2022, 10:00 PM

#

yea I mean where we are now was a process. like I said I don't know the whole history but I'm fairly certain that A vs. AAAA queries thing on musl had a prominent role

jolly shard May 6, 2022, 10:00 PM

#

same for the other ones.

#

AAAA returning NXDOMAIN , and then people have to implement a bunch of hacky workarounds.

#

i think ill stay on 2022.4.0 for now, and then make custom local addons for mariadb and deepstack..

void holly May 6, 2022, 10:02 PM

#

ok. yea sorry for the hassle

jolly shard May 6, 2022, 10:02 PM

#

and/or turn HA into a VM and then create a second VM for mariadb and deepstack.

#

not even sure why I have a separate machine for hassio

void holly May 6, 2022, 10:03 PM

#

honestly that sounds like a solid plan

#

in the end addons are basically just preconfigured docker containers. If you want more control and know what you're doing breaking them out sounds like a better plan

jolly shard May 6, 2022, 10:05 PM

#

void holly ok. yea sorry for the hassle

at least you got back to me, i appreciate it, and while it's annoying that the change was made, I understand why

#

I have no idea what I'm doing 😄

#

oh i remember why i didnt just make a VM, i dont have any free RAM on my kvm box. Ryzen maxes out at 128gb 😦

void holly May 6, 2022, 10:06 PM

#

128gb of ram?! what are you doing on that thing lol

jolly shard May 6, 2022, 10:07 PM

#

my wife has her Linux VM which she does like pytorch on, i have a file server VM, and a windows VM, and a jenkins VM.

void holly May 6, 2022, 10:08 PM

#

interesting. yea i mean just for point of comparison I have HA and 25 addons running on my HA blue right now which has 4gb lol

#

oh wait, I guess a couple of these are grayed out. installing stuff for testing. lets go with 20 addons

jolly shard May 6, 2022, 10:09 PM

#

oh nice

#

I am using an Intel NUC for HASSIO or whatever its called tehse days

#

with 16gb of ram

void holly May 6, 2022, 10:09 PM

#

oh wait I thought you said 128? did you mean HD space?

jolly shard May 6, 2022, 10:09 PM

#

Hostname: kvm - OS: Linux 5.4.0-80-generic/x86_64 - Distro: Ubuntu 18.04.5 LTS - CPU: 24 x AMD Ryzen 9 3900X 12-Core (2123.165 MHz) - Processes: 420 - Uptime: 271d 2h 13m - Users: 1 - Load Average: 1.39 - Memory Usage: 101229.64MB/128760.38MB (78.62%) - FS Usage: 25.26GB/155.49GB (16.25%) - LVM Usage: 18299.80GB/20025.70GB (91.38%)

#

anyway this is super offtopic. can take this to a PM .

void holly May 6, 2022, 10:10 PM

#

yea good point. anyway happy I was able to help

#HassOS `ha resolution info` just