paper remnant
Is it on purpose that dagger cannot access the ec2 instance role metadata? If I run a pod with aws-cli straight in kubernetes this works:
curl -X PUT "http://169.254.169.254/latest/api/token" \
> -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"
But if I do the same after running same via:
dagger core container from --address amazon/awscli
And run that same command it hangs. We try to avoid using aws credentials...
lucid glen
Hi @paper remnant when you are running the awscli container, that is run in a container that does not have access to the 169.* address/network as that is only available when directly on the instance.
Usually you can replace the 169.* address with something like host.docker.internal but I'm pretty sure that will not work while in the dagger core container since its not on the actual instance... That depends on the network mode the container is running in, which I'm assuming is not host and you cannot current specify when calling dagger core container.
Also, can you confirm you can run a container with --network host and access the 169.* address just to confirm?
paper remnant
So here is the weird thing. Within the dagger container I can do a GET and it just gives me a 401 which means it sorta works. But the PUT to get the api token doesn't.
Within k8s which is a pod it does have access though.
So let me restate what works:
What does not work:
Also if I borrow the token from the ec2 instance a head of time I can access the meta-data via the token. So the only thing that doesn't work is the PUT to get the token in the first place within the dagger container
paper remnant
@latent ice Do you know if this is a bug or issue I should write or works as intended?
latent ice
@paper remnant IIRC the AWS metadata service had a limit on the amount of hops that the response could actually do before returning to the client. I recall this was a common issue when running workloads in containerized environments
there's a way you can modify the instance metadata to increase the number of hops. Have you checked this already?
also changing the metadata service auth scheme from IDMSv2 to IDMSv2 should also work
That's my theory about what could be happening here particularly becuase the simple GET request works
lucid glen
I think the real issue is that the 169.* address is APIPA, which the container can only access when exposed on the host network - which I do not believe dagger containers are run with that network access?
Start a pod via kubernetes on a host
If you look at the network for a pod, can you verify it is exposed to the networks host. That is probably why it works there vs in a dagger core container command.
Another option, you can embed the SDK/Dagger Client into a program (Go, Typescript) and use the code running on a pod container to get those and pass to a container as secrets/env vars?
paper remnant
This is not true. I am able to get to 169 and everything works if I have a token.
Just the PUT doesn't work.
latent ice
Let's wait and see if @paper remnant changes the hop attribute in the instance and see if that makes it work
@lucid glen according to what Chad mentioned above, the container can access the endpoint. It's the PUT request that gets stuck
and I recall seeing this before given the info I've shared above
paper remnant
Yes, I'll change that and see what I can find.
lucid glen
Ok, sorry I definitely misread @paper remnant
paper remnant
@latent ice you are amazing. Always impressed with your depth of knowledge and experience. That is a long way to say that increasing the number of hops worked. Thanks!
latent ice
<@336241811179962368> you are amazing. Always impressed with your depth of knowl...
always happy to help Chad! glad you were able to get unblocked! 🙏
lusty night
+1 on this. We had the same issue and it looks like it has been solved by increasing the hop count.
Thanks for the tag @lucid glen
primal echo
