Continuing on from the discussion in #general
#Gitlab CI - Runner issues with resolving docker host
dull rose
(Starting #general message)
@proven idol Continuing here rather than the general channel if that's ok
dull rose
Gitlab file is a very minor alteration on the documentation example:
image: docker:latest
tags:
- webcms-dev
services:
- docker:dind
variables:
DOCKER_HOST: tcp://docker:2376
DOCKER_TLS_VERIFY: '1'
DOCKER_TLS_CERTDIR: '/certs'
DOCKER_CERT_PATH: '/certs/client'
DOCKER_DRIVER: overlay2
DOCKER_VERSION: '20.10.16'
.dagger:
extends: [.docker]
before_script:
- apk add python3 py3-pip docker-cli curl aws-cli
- curl -sSL https://install.python-poetry.org | python3 -
- "$(aws ecr get-login --no-include-email --region eu-west-1)"
build:
extends: [.dagger]
script:
- python -m pip install --upgrade pip
- /root/.local/bin/poetry install --no-root
- /root/.local/bin/poetry run python3 build.py
proven idol
LMK if my example makes sense or if you want me to go through it
dull rose
Does that example fully replicate the issue?
I can already use docker in scripts, just not in the python file
proven idol
no, it's an example on how you run docker withing docker
TL;DR for docker to work in dagger you need to set the DOCKER_HOST variable in your Dagger pipelines to: tcp://$DOCKER_PORT_2375_TCP_ADDR:2375
dull rose
I'll test it now - I can just add that line docker run -e DOCKER_HOST=tcp://$DOCKER_PORT_2375_TCP_ADDR:2375 docker:cli docker ps to my .dagger stage?
proven idol
that won't work
you need to set the DOCKER_HOST in your pipelines with the same value I'm setting in my example
dull rose
Oh, so it's just dealing with the failure to resolve docker
dull rose
And presumably I'll need DOCKER_PORT_2376_TCP_ADDR:2376
proven idol
if you want to use TLS , yes
in my example I'm not using TLS connection to the engine
dull rose
Already done
That's running in GL now, few minutes
Fell into the old CI trap of pushing an update to see if it worked over... just running it locally 🤷♂️
Didn't await a value, trying again
Didn't get the port right, seems to have defaulted to 2375 so I've added collecting DOCKER_PORT_2376_TCP_PORT from the host as well
That's done it @proven idol. Thank you for your patience on this! I'd definitely suggest expanding the documentation for the GL examples, other people will no doubt hit this issue
proven idol
That's done it <@336241811179962368>. Thank you for your patience on this! I'd d...
🎉 happy to help. We'll definitely improve the docs: cc @cinder dust
dull rose
The very last line on https://docs.dagger.io/454108/python-ci - python main.py - if you could add a sample main.py below that example that demonstrates adding those variables that'd do it
Unfortunately it's made my lovely clean python look less minimal and clean but that's the price of dind I guess!
cinder dust
^^ @shadow breach fyi
shadow breach
The very last line on https://docs.dagger.io/454108/python-ci `- python main.py...
This is how I tested the GitLab CI example that we have in the docs today. I'll see if it's still working.
https://gitlab.com/jpadams301/test-ci-python cc @proven idol @cinder dust
dull rose
That'll work I expect. Add a with_exec docker operation to test the problem I encountered today
shadow breach
That'll work I expect. Add a `with_exec` docker operation to test the problem I ...
@dull rose so, to make sure I understand, you want to run something like
client.container().from_("docker:cli").with_exec(["docker", "ps"]) ?
dull rose
Should do the trick. My use was docker build but without providing the certs and actual docker host address/port no docker commands were working
Specific to Gitlab with docker dind, works fine with the client Unix socket on gitlab
Gitlab do warn the user about using both dind and mounting the Unix socket and I'd hazard a guess that using dind is more common across orgs (though my sample size is small), so providing a slightly more detailed example should help others here
shadow breach
Ah, okay. Did you also try using
https://dagger-io.readthedocs.io/en/sdk-python-v0.3.2/api.html#dagger.api.gen.Directory.docker_build
Wondering if you can just read in a directory of stuff with a Dockerfile in it and use that ☝️
dull rose
👀
I'm trying to recall if I've seen that before, hah. Does it work in the same way? I.e I can call publish on the returned container?
shadow breach
Yes, should just work. That way you can keep it all in the Dagger Engine vs using Docker itself. Let me get you an example.
There is also https://dagger-io.readthedocs.io/en/sdk-python-v0.3.2/api.html#dagger.api.gen.Container.build
import anyio
import sys
import dagger
async def main():
async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as client:
build = client.host().directory(".").docker_build()
await build.publish("jeremyatdockerhub/dbexample:latest")
anyio.run(main)
in the root of my project I have a Dockerfile 👇
FROM alpine
I might have seen this already and written it off due to various dependencies for cloud/secrets. I'll look into it tomorrow; I've got time this week to get this off the ground for teams that don't control their Gitlab runners and are stuck with dind.
shadow breach
Sounds good. Would love to know if you have requirements that can't be served by this docker_build 🙏
Here's the working example in GitLab. Had to do a docker login so I could push my image 🙂
Stored my DockerHub token in a GitLab secure variable: $CI_REG_TOKEN
https://gitlab.com/jpadams301/test-ci-python/-/tree/docker_build
https://gitlab.com/jpadams301/test-ci-python/-/blob/docker_build/.gitlab-ci.yml#L16
https://gitlab.com/jpadams301/test-ci-python/-/jobs/3803470735
dull rose
This worked, thanks for making me aware! Useful to learn how to configure docker-in-docker anyway but for simple builds docker_build is obviously useful.
cinder dust
@dull rose although this doesn't directly address your use case and also uses a different language (Go), there's a tutorial on using Gitlab CI + Dagger at https://docs.dagger.io/sdk/go/759201/gitlab-google-cloud, in case it's useful to you or others who come across this thread in search