#how to tag a container image after call build?

1 messages · Page 1 of 1 (latest)

gritty oak
#

im building using

await (dag.container()
               .build(context=self.working_dir, dockerfile=dockerfile, target=target, build_args=build_args))

I want to be able to tag that image as if i were run docker build -t name:tag

lofty torrent
#

@gritty oak are you looking to push the container as an OCI image to a registry? Or are you looking to load it into a Docker engine running your machine?

pale sphinx
gritty oak
#

im migrating my GHA pipeline to dagger. which is usual, 1. build - 2. test - 3. push. So in #1 step i build the container images as show and then on #2 i run a docker compose file which set the test env

#

how can i accomplish this with dagger?

lofty torrent
#

You can do 1, 2, 3 all as part of the same Dagger pipeline 🙂

gritty oak
#

do you have an example on how to do it @lofty torrent ? i know i can build the docker compose file by code using the sdk by i already have the .yaml file the thing is that it trust the images on step #1 to have certain tags lets say api:9asd87as and i cant find a way to do that part

lofty torrent
#

Are you able to share the yaml file here?

gritty oak
#

sure

lofty torrent
#

And also the Dagger code you've already written (if any)

gritty oak
#

i was unable to write any. im kind of struggling to find doc on how to

lofty torrent
#

No worries! That's why we're here 🙂

#

Your language of choice is typescript?

#

Are you able to share the Dockerfile too?

gritty oak
#

python in this case

lofty torrent
#

That's a serious dockerfile!

#

So pre-Dagger, once the image is built, what would running the e2e test look like? Is there a particular command you'd run, or a particular section of the compose file I should look at?

gritty oak
#

the steps usually looks like

  1. docker compose up -d
  2. docker compose exec test composer behat
lofty torrent
#

how do you aim compose at the right image tag, do you pass the tag an env variable to compose?

gritty oak
#

no, is just configure there.
…
api:
image: snoutzone/ci
…
test:
<<: *common
image: snoutzone/dev
…
the build step tag the images as that so composer can use them

lofty torrent
#

Ah I see, you just hardcode docker build -t snoutzone/dev

gritty oak
#

exactly

#

and thats what im trying to do now with dagger so composer can find those images

lofty torrent
#

@wooden prairie I'm experimenting with Docker Compose compat in your docker module for this, is it known to work? 🙂

#

@gritty oak so there are 2 possible ways to do this:

Option 1. Reimplement (parts of) the Docker Compose file, plus the glue commands to integrate build+compose-exec, as native Dagger calls. Almost everything maps one-to-one, so it's tedious but straightforward, and only needs to be done once. Let's say 50 lines of code fully loaded.

Option 2. Only implement the glue commands in Dagger, and use a Docker Compose compat module to just pass your compose project as an argument. Probably requires light editing of the Compose file, which may or may not be ready. This requires a working docker compose compat. I'm investigating.

wooden prairie
#

by the looks of it, this pattern won't work, since it just does a From() when an image name is set, which will try to pull it from a registry. it'll work if you set build: <path> instead though

lofty torrent
#

@wooden prairie I tried dagger call -m docker compose --project . all up But it failed with "no ports to forward"

#

Also, can I run an individual service instead of all of them combined?

#

And, if the compose file defines dependencies between projects, will they work?

wooden prairie
#

starting one services isn't implemented, but it should be easy to, since the dependencies are expressed as a DAG

lofty torrent
#

Unfortunately service dependencies via docker networks aren't a DAG..

#

To my knowledge we haven't found a solution to that yet

wooden prairie
#

it can be cyclic?

#

or i guess you're all just peers, and depends_on is just the dependency order, if you define it

lofty torrent
#

Yes exactly. Network dependency DAG is actually not statically defined anywhere, it's whatever hostnames each service decides to lookup at runtime

#

All services can connect to each other in the network namespace. N-to-N

wooden prairie
#

i wonder if that could be implemented in a module thinkies

#

technically all containers can already reach each other, it's just an addressing issue

lofty torrent
#

An explicit DAG would work fine 99% of the time, but can't be generated from the compose file unless very disciplined about depends-on

lofty torrent
#

if you require that compose file explicitly define the dag with depends_on, it could work

#

at least as an initial crutch until we figure out better magic

wooden prairie
gritty oak
#

interesting topic. but definetly something that can block dagger adoption in some cases. maybe an interpreter that generates dagger code to conver compose file to dagger code could solve it

gritty oak
#

ok but getting back to the original question, there is no way to tag a container/image with dagger right? the dagger flow goes as container -> build -> use it as you want -> publish (this actually support tagging)

#

i got it right?

wooden prairie
lofty torrent
#

you can also run an ephemeral docker engine in the container, then aim regular docker-composr at it. That works quite well actually, @gaunt gate did it for slim integration

#

I will try for you this weekend @gritty oak 😁

gritty oak
#

thanks a lot guys, awesome team

pale sphinx
pale sphinx
pale sphinx
lofty torrent
#

😛

gritty oak
#

so if all services in compose are relaying in depends_on and build a DAG it can be done right? that said which module do you guys advice to try this, i mean load a compose file and try it?

pale sphinx
gritty oak
#

will take a pick into it

lofty torrent
#

Hello! I am currently stuck on a weird dagger error, which prevents me from installing the docker module 😦

#

I figured I would start with the full compat option: embed a docker engine + docker-compose, get everything running. Then gradually refactor to make it more native & efficient

lofty torrent
#

OK, I am unstuck!

lofty torrent
#

@gritty oak @pale sphinx @wooden prairie taking a break for lunch, I have a basic POC that's not fully functional, but gives an idea of what "option 1" would look like:

# 
dagger call -m github.com/shykes/daggerverse/sratch/bitgantter \
 test-env --app index.docker.io/nginx \
 terminal

--> This will setup a test environment with an ephemeral docker engine + docker-compose CLI, inject the given app container into the same engine, and set everything up to run docker-compose exec test ... against it

(in my example, I just point to nginx as a fake app container)

gritty oak
#

amazing @lofty torrent will read that

#

thanks so much

lofty torrent
#

I intentionally don't handle the build, because that's easy to do. Will add to the complete demo after

#

To recap, there are 3 options:

  1. Run docker-compose in Dagger: most compatible, least efficient --> This is the one I'm prototyping
  2. Run a docker-compose compat module: intermediate
  3. Reimplement the compose file in code: least compatible, most efficient
#

In the terminal you can access $APP_IMAGE which is the ID of the injected container to test

#

It's not tagged yet, I'll take care of that after lunch break

lofty torrent
#

FYI I'm pushing an update in the next few minutes 🙂

lofty torrent
#

OK here we go 🙂

# 
dagger call -m github.com/shykes/daggerverse/scratch/bitgantter@v0.2.1 \
 test \
  --app index.docker.io/nginx \
  --command sh,-c,'curl http://app'
#

It takes longer to run than necessary, because of a temporary bug which causes the ephemeral docker engine to be re-started in between each client call. It works because the docker engine state is persisted in a cache volume. But still adds a little overhead

#

I took the approach of not supporting passing arbitrary docker-compose files to the funciton, like I originally did. That way, I can enforce a simple convention in the compose file. At the moment the convention is:

  • There should be a service called app using image app, where the Dagger Function will inject the app container to be tested
  • There should be a service called test using image test, and executing something that never returns (so that docker-compose up doesn't fail

This basically makes the compose file an implementation detail of the test-env function. Later it could be refactored away entirely.

#

I also added a test-source function that takes a source directory as argument, rather than a container. It simply does a standard docker build, and passes the result to test.

Example building a remote repo:

dagger call -m github.com/shykes/daggerverse/scratch/bitgandtter@v0.2.1 \
 test-source \
  --source https://github.com/dagger/hello-dagger \
  --command sh,-c,'curl http://app'

And building a local source checkout:

dagger call -m github.com/shykes/daggerverse/scratch/bitgandtter@v0.2.1 \
 test-source \
  --source . \
  --command sh,-c,'curl http://app'
#

In all the commands above, you can pass a custom test container with --test-container

gritty oak
#

Thanks @lofty torrent saw the work there and its an starting point. But on my scenario i dont think its going to work. As you can see my testing env relies a lot on the compose file and all moving pieces (secrets, volumes, and so on). That said i think i know the path now and i can start building something around it. Will share if i manage to make it work

lofty torrent
gritty oak
#

yes i do thing that starting an ephemeral docker engine can help, at least easy the path. but i was tryionh to go the dag path you know

lofty torrent
#

Yeah, honestly after going through the process of wrangling that docker integration, I kind of feel that it's easier to bite the bullet and convert to native Dagger code.

gritty oak
#

@here if i parse the docker compose file and i intend to create the same ecosystem using dagger api, which its going to be the best way? given that they must complain to networks, volumes, and so on?

pale sphinx
#

I'd say that most of the things are there, so it's mostly a matter of hooking your compose file and see what doesn't work

gritty oak
#

got you. my main stop holder is that im not a go developer so wont be able to tweak things. thanks will read whats in there

pale sphinx
wooden prairie
pale sphinx
lofty torrent
#

@wooden prairie we should merge our compose packages 😛

#

I kind of want to start an unofficial distro

#

just missing a silly name

#

I could bring back "stackbrew"

#

or "dagbrew"?

#

dagopedia

wooden prairie
wooden prairie
lofty torrent
#

of modules

#

ie a collaborative daggerverse repo, separate from my individual one

wooden prairie
#

ah got it. yeah, i guess that's one downside to monorepos, don't think there's per-directory write permissions, at least not on GitHub

#

unless you just do PRs

#

in lieu of that, feel free to just scrap mine for parts, and I can delist/remove mine to consolidate

gritty oak
#

yes that will require PR and review from you guys, the symfony php community does similar with what they call flex packages

gritty oak
#

ok, i know i have ask this question before but here comes again. The main reason for asking again is that i dont completely understand the concepts of: pipeline and https://docs.dagger.io/api/reference/#definition-Service. So the question. If i want to implement something like compose, few services/containers sharing same network, which dag resources are recommended and why?

gritty oak
#

@lofty torrent @wooden prairie any advice?

lofty torrent
gritty oak
#

exactly "Are you trying to reimplement the stack from that docker compose file, in pure code?" but i dont know which primitive to use, i was looking for arch concepts on the doc but cant find any. So if you can point me to which resources should i use. For example as docker compose i would like the containers to be isolated in their own network, volumes, etc

lofty torrent
#

The general idea:

  • For each service configuratin in the compose file, write a Dagger function that returns an equivalent Container.
  • For each dependency in the compose file (depends_on): add the dependency as an argument to the client container. The argument should be of type Service.

In Go:

// The 'db' service from compose file, ported to a function
func DB() *Container {
 return dag.
   Container().
   From("postgres").
   WithExposedPort(...).
   WithBlaBlaBla()
}

// The 'api' service from compose file, ported to a function
func API(source *Directory, db *Service) *Container {
  return dag.
    Container().
    From("php-server-image").
    WithDirectory("/app", source).
    WithBlaBlaBla().
    WithServiceBinding("db", db)
}
#

I think I remember Python is your language of choice, sorry, my Python is quite rusty. But it all transposes 1-1 to Python code.

gritty oak
#

ok but i have few more questions:

  • dont quite get db container build on first function can be pass as the db Service argument of the second one? so container == service? or something more needs to be done in the middle?
  • calling the WithServiceBinding method is like the old link config in docker compose?
  • There is any way to create a dedicated network for all this containers/services; like compose does? as you can see in my compose file some internal dns requires the ip to work
lofty torrent
#

a service is an abstraction for a network endpoint: basically one or more ports that you can connect to from a container (by binding it).

Dagger has 2 underlying service implementations: container services (call Container.asService()). And host services, which are tunnels to endpoints running outside Dagger.

#

this means you can call Api() and pass it a dv service that is either running in another container run by dagger, OR outside dagger, and the Api function won't know the difference

#

better composition

#

yes it's very similar to old docker links (which I designed ironically)

gritty oak
#

cool. i think i got it. let me see where i can get it, having fun with this

gritty oak
#

@lofty torrent maybe you can take a look at this PoC https://github.com/wildbitca/daggerverse/tree/master/docker-compose and give me some feedback. Im just starting with dagger im sure there are better ways to do stuff or even i may be doing some of them the wrong way. That said its build in base of my personal requirements which is been able to bring a full/complex docker compose project and reutilize dag containers previously build from dockerfiles without pushing them to any registry

GitHub
daggerverse/docker-compose at master · wildbitca/daggerverse

Contribute to wildbitca/daggerverse development by creating an account on GitHub.