External calls are not routed to the vpn | Dagger | Page 1

desert tiger Sep 6, 2024, 10:24 AM

#

Hi,

I have an openvpn client which will route only a part of my traffic like calls to my eks cluster / private services / ...
This client is only for running traffic from laptop, when we are in the CI we have specific egress IPs.
Before dagger I was running that with a mix of docker compose + makefile and with this setup the routing is working correctly but with dagger when I'm checking the egress IP, I can see this my personal one.

services:
  infrastructure:
    image: myorg/infra:0
    network_mode: host
    volumes:
      # some volumes to do a mapping
    environment:
      # some env vars specifics to some software like creds for terraform providers

Maybe it's due to the network mode

nimble bear Sep 6, 2024, 2:27 PM

#

desert tiger Hi, I have an openvpn client which will route only a part of my traffic like ca...

👋 this should work. Make sure the Dagger engine is started after you setup the connection with the VPN. This is so Docker can setup the network correctly so packets can be routed

desert tiger Sep 6, 2024, 2:27 PM

#

I tried that but I can give another try to be sure

#

👋

nimble bear Sep 6, 2024, 2:28 PM

#

so in summary:

Make sure the Dagger engine is not running (docker ps should not show the Dagger engine)
Connect to the VPN
Call your pipeline

desert tiger Sep 6, 2024, 2:29 PM

#

ok I try that now

#

should I stop it or remove it ?

nimble bear Sep 6, 2024, 2:29 PM

#

desert tiger should I stop it or remove it ?

remove it. docker rm -fv $name

desert tiger Sep 6, 2024, 2:37 PM

#

it's working thank you, I restarted the docker engine but didn't think to remove the container

nimble bear Sep 6, 2024, 2:55 PM

#

desert tiger it's working thank you, I restarted the docker engine but didn't think to remove...

👍

#

yeah.. Docker sets the networking when the container is created

#

so restarting doesn't help

#

it's a Docker thing mostly

desert tiger Mar 4, 2025, 7:19 PM

#

Hi, I repost a message here but I'm not sure if I shoud open a new thread.
I'm a bit loss with network issues, I setup the vpn then remove the dagger container but I still have an issue to contact my eks cluster, I have a timeout so I think my queries are not going through my openvpn network

#

from my terminal it's working, from a container with network host it's working

#

Do you have an idea what can I do, re-reading the post it seems one moment it was working but I'm not able to query my kube cluster from dagger and this one allow only queries from the egress IPs of the vpn

nimble bear Mar 4, 2025, 7:52 PM

#

desert tiger Hi, I repost a message here but I'm not sure if I shoud open a new thread. I'm a...

@desert tiger are you using docker for mac? or linux?

desert tiger Mar 4, 2025, 7:52 PM

#

linux

nimble bear Mar 4, 2025, 7:52 PM

#

can you reach your k8s cluster within a "stand alone" container via docker run for example?

desert tiger Mar 4, 2025, 7:53 PM

#

let me try that

nimble bear Mar 4, 2025, 7:53 PM

#

if you can't do that then Dagger won't also work

#

you need to figure that out first

#

dagger doesn't work with host networking like you mentioned above so you need to be able to access your VPN endpoints within a traditional network namespaced container

desert tiger Mar 4, 2025, 7:54 PM

#

it's working with the "stand alone"container

nimble bear Mar 4, 2025, 7:55 PM

#

desert tiger it's working with the "stand alone"container

ok, then it should work then.

if you run dagger -c "container | from alpine | terminal". Does it work there?

#

alpine or whatever other container you tried before

desert tiger Mar 4, 2025, 7:56 PM

#

I have the terminal then you want I install something ?

nimble bear Mar 4, 2025, 7:57 PM

#

desert tiger it's working with the "stand alone"container

how did you try this?

desert tiger Mar 4, 2025, 7:58 PM

#

I used this docker compose file with the network host commented

services:
  infrastructure:
    image: 009160054486.dkr.ecr.eu-west-1.amazonaws.com/infrabox:5
#    network_mode: host
    volumes:
      - $HOME/.aws/:/root/.aws
      - $HOME/.kube:/root/.kube
      - $HOME/.cache/snowflake:/root/.cache/snowflake
      - ./terraform:/terraform
      - $HOME/.terraform.d/cache/:/opt/terraform-plugin-dir
      - $HOME/.config/auth0:/root/.config/auth0
    environment:
#      TF_LOG: TRACE
      TF_WARN_OUTPUT_ERRORS: 0
      KUBE_CONFIG_PATH: /root/.kube/config
      KUBECONFIG: /root/.kube/config
      TF_PLUGIN_CACHE_DIR: /opt/terraform-plugin-dir
      SOPS_KMS_ARN: "arn:aws:kms:eu-west-1:009160054486:key/41f348d3-21eb-4a7f-91ea-88cad5565b43"
      PYTHONIOENCODING: UTF-8
      LANG: en_US.utf-8
      LC_ALL: en_US.utf-8
    env_file:
      - $HOME/.heuritech/infrabox/global.env

nimble bear Mar 4, 2025, 7:58 PM

#

ok, I see

desert tiger Mar 4, 2025, 7:58 PM

#

docker compose run --rm infrastructure /bin/bash
[+] Creating 1/0
 ✔ Network infra_default  Created                                                                                                                                                                                                                                                                     0.1s 
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf-8)
root@aa8108ccd611:/# kubectl get po
No resources found in airflow-system namespace.

nimble bear Mar 4, 2025, 7:59 PM

#

ok, what you can do is get the ip or hostname of your kube cluster

#

and within the dagger terminal try running curl against it

#

you can install curl with apk add curl

#

that will let you know if you can at least reach the service

desert tiger Mar 4, 2025, 8:00 PM

#

ok I'm trying but it seems it will timeout

nimble bear Mar 4, 2025, 8:01 PM

#

does your kube config has an ip or hostname on your kube server endpoint?

desert tiger Mar 4, 2025, 8:02 PM

#

yes there is aa field server with the hostname

#

when I'm looking my terraform logs I have something like that: Error: Get "https://<cluster id>.<region>.eks.amazonaws.com/api/v1/namespaces/inference-system/serviceaccounts/triton-runner": dial tcp <PUBLIC IP>:443: i/o timeout

#

from dagger

nimble bear Mar 4, 2025, 8:09 PM

#

@desert tiger could you install apk add bind-tools in the terminal container

#

and run dig <cluster id>.<region>.eks.amazonaws.com ?

#

just checking if it's a DNS and/or connectivity thing

#

trying to see if you can get the cluster IP from the DNS resolve within Dagger

desert tiger Mar 4, 2025, 8:11 PM

#

yes it's working

nimble bear Mar 4, 2025, 8:11 PM

#

ok and when you do curl https://$IP it just hangs?

desert tiger Mar 4, 2025, 8:12 PM

#

yes but it's hanging from dagger and my terminal if I curl directly the IP

nimble bear Mar 4, 2025, 8:13 PM

#

desert tiger yes but it's hanging from dagger and my terminal if I curl directly the IP

hmmm that shouldn't happen

#

are you connected to the vpn? lol

desert tiger Mar 4, 2025, 8:13 PM

#

yes

#

openvpn3 sessions-list
-----------------------------------------------------------------------------
        Path: /net/openvpn/v3/sessions/6aeb1f73se158s42d9s814esab75e093c8a0
     Created: 2025-03-04 20:20:54                       PID: 216662
       Owner: dgoldenberg                            Device: tun0
 Config name: /home/dgoldenberg/openvpn_laptop.ovpn  (Config not available)
Connected to: udp:[2001:49f0:d0ea:2::6]:1194
      Status: Connection, Client connected
-----------------------------------------------------------------------------

#

curl -k https://<cluster id>.<region>.eks.amazonaws.com/
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}

nimble bear Mar 4, 2025, 8:16 PM

#

it's quite strange that curl https://$IP fails from your machine also

#

unless dig is giving you a wrong ip

#

can you run curl -v -k https://<cluster id>.<region>.eks.amazonaws.com/ and verify if the IP curl resolves to is the same one as dig?

desert tiger Mar 4, 2025, 8:16 PM

#

yes I'm agress

nimble bear Mar 4, 2025, 8:17 PM

#

which might explain why this fails in Dagger I think

desert tiger Mar 4, 2025, 8:17 PM

#

not the same ...

nimble bear Mar 4, 2025, 8:18 PM

#

ok, try with the curl IP within the dagger terminal now

#

to see if that works

desert tiger Mar 4, 2025, 8:18 PM

#

yes I just try and it's working

nimble bear Mar 4, 2025, 8:18 PM

#

ok, it's a resolution issue

#

seems like openvpn might not be setting your DNS correctly

#

and curl is probably resolving the ip from a different place like your hosts file for example?

desert tiger Mar 4, 2025, 8:20 PM

#

not sure because the dns is the dns provided by aws and my openvpn doesn't have any dns override

nimble bear Mar 4, 2025, 8:20 PM

#

well.. if dig and curl are returning different IPs then it's definitely something in your system that's being resolved differenly

#

does dig in your local machine return the same IP as curl?

desert tiger Mar 4, 2025, 8:21 PM

#

yes

nimble bear Mar 4, 2025, 8:23 PM

#

ok.. interesting..

#

if you access your dagger-engine container, can you check if resolution works ok there?

#

docker exec -ti $dagger_engine

#

you can run nslookup there

desert tiger Mar 4, 2025, 8:24 PM

#

ok

#

wrong IPs

nimble bear Mar 4, 2025, 8:25 PM

#

ok, that doesn't make any sense

desert tiger Mar 4, 2025, 8:26 PM

#

yes I'm a bit loss I'm trying to understand what is the difference

nimble bear Mar 4, 2025, 8:28 PM

#

can you validate one more time that docker run -ti alpine sh works?

#

because you were trying with your compose container before

desert tiger Mar 4, 2025, 8:28 PM

#

yes i'm trying that

#

same issue

nimble bear Mar 4, 2025, 8:31 PM

#

ok, it's a docker issue then

#

you can try restarting the docker engine and trying again with docker run

#

sudo service docker restart

#

docker is not very happy when the underlying host network config changes and the engine is not restarted

desert tiger Mar 4, 2025, 8:32 PM

#

ok

nimble bear Mar 4, 2025, 8:33 PM

#

gotta run sorry

#

I'm wrapping up for today. At least you were able to repro the same issue with plain docker

#

which basically discards Dagger being the issue 🙏

desert tiger Mar 4, 2025, 8:34 PM

#

ok no problem thank you for your help

#

ok so the restart didn't change anything I tried it earlier, so at the end I tried to restart the laptop in order to "reset"my network and now it's working

#External calls are not routed to the vpn