Container.up breakages

[keen latch]

actually let's thread 🧵

https://github.com/levlaz/snippetbox/blob/main/.dagger/main.go#L126-L131

i get what's happening - we return a Container here.
on the cli, when we do Up, then this does an AsService + Up together using the v0.15.2 api

the expectation i guess is that we would have kept this working - but imo, this is correctly breaking

the issue is, I think all usages of Container.up are broken - there's no way anymore to specify the args right?

what args should we actually be using here?

[golden depot]

this used to work with v0.14.0, so what I was thinking was to have a legacy version and a new version of container.up. The legacy version would call asService with view: v0.14.0, and it will work like it worked before the breaking change. (use the last withExec and enable the entrypoint etc.... ).

[keen latch]

(jumping to #911305510882513037 if you want to sync)

[golden depot]

sure 👍

[keen latch]

x-posting https://github.com/dagger/dagger/issues/9190#issuecomment-2545604302

[golden depot]

while I am making changes, I ran into a scenario that is weird, but seems valid:

The python docker image has defaultArgs: [python3]. so now our check to identify if the user forgot to specify args/entrypoint etc fails as defaultArgs are indeed set. BUT, it will result in hang of service I am assuming when we try to run defaultArgs

I dont think there is a good way to identify this (unless we go the route of starting a background process or goroutine with a specific timeout or something?)

[keen latch]

do you mean if a user has added defaultArgs of python3? docker images don't have the concept of defaultArgs, that's entirely a dagger concept

[golden depot]

                "python3"
            ],```
docker has cmd. I think that is similar to `defaultArgs`.

^^ above from output of docker inspect python

[keen latch]

🤔 you're right

but like

shouldn't we ignore the Cmd in the image for the purpose of services?

also, it looks like there are codepaths where you can use the Cmd of an image, but without using the Entrypoint of an image? isn't that just going to give incorrect results a lot of the time?

cc @cunning elm, really unsure of what the design should be here

imo, UseEntrypoint should determine whether we use the images default Cmd as well, right?

[golden depot]

I think that was a known decision (to use default cmd if available and prepend entrypoint if its available and enabled).

[keen latch]

yeaaaa, i mean it's just going to give incorrect results lots of the time

but it also means it's really hard to implement the error check you want to do

[golden depot]

in docker world:

• it runs entrypoint + default cmd if entrypoint and default cmd is available
• it runs default cmd if entrypoint is not set but default cmd is set.

e.g. if you run docker run -it python, it will run defaultcmd only as no entrypoint is available

but during the discussion for these changes, we decided to try and do the right thing instead of trying to do what docker do today.

[keen latch]

but if i have an image that's got entrypoint "my-wrapper-script-to-setup-the-environemnt" and cmd "do some things", then just running the cmd is going to fail in really non-obvious ways

i don't think the check is possible to implement then if Cmd is set on the image

[golden depot]

yep... that is what i am running into right now

[keen latch]

🤷‍♀️

[golden depot]

maybe @frail flare has opinions/suggestions about this. He helped derive a lot of discussion around this area.

[keen latch]

if i had to choose a behavior, i'd go with:

• if no withDefaultArgs are set, we default to entrypoint + cmd (both from the image - since these are often designed to go together)
• if yes withDefaultArgs are set, we only use the args from there, ignoring both the entrypoint + cmd (unless useEntrypoint is set to true)

[golden depot]

Isnt cmd in docker same as defaultargs in dagger

[keen latch]

no, because cmd is always suffixed to entrypoint in docker

if you set cmd, you still keep entrypoint

but even if you do this, the check you want to implement to detect the hanging in withExec isn't really super easy to implement

i actually think it's impossible to detect this case, since most images are going to have a cmd or defaultargs set on them somehow

it feels like the best you could do, would be to try and detect long-running WithExecs that are immediately before an AsService and provide a warning? but this is a bad user experience

[golden depot]

Principle: embrace "default args" (aka CMD) as the correct way to persist a default command in a container

from https://github.com/dagger/dagger/issues/8140#issuecomment-2450747681

[keen latch]

yeah, i'm not arguing that we should use entrypoint for any "dagger-native" containers, i'm just suggesting we should keep using it when working with containers that have set these from outside-of-dagger

[golden depot]

I think I remember from our discussion that we discussed and agreed that this will break for people who come to dagger with an assumption that it behaves similar to how docker works. and that we want to get rid of entrypoint and it should be sufficient to just return a useful error if "as service fails and we identify that there was a entrypoint configured but not used"

(i am just trying to recall the discussions that led us to make this decision).

[keen latch]

fair, i just think it's quite difficult to give a useful error here - the service might "appear" to work, but not actually functionally do the right thing

[keen latch]

@silver sorrel @delicate harness

[cunning elm]

withEntrypoint has a keepDefaultArgs: Boolean = false arg. Maybe withDefaultArgs could have a keepEntrypoint: Boolean = false too.

But even if we do what you say, the Python image isn’t supposed to be run as a service. It’s up to the user to decide if it’s a withExec or a asService, this needs to be properly documented. The same thing can happen in Docker. Would you put a long running program in a RUN step? What if CMD is ["python", "-V"]? It’s not going to stay running, but does the docker output help the user realize why?

[keen latch]

yeah, i don't really mind about running the "python" image as a service - i more care about running a third party image as a service, like postgres (or some other database) - if the entrypoint is set to postgres, and the args are set in cmd, then we just won't be able to run it (where previously we could)

if you choose to override using withDefaultArgs I think it's fine to say "you're on your own", and not use the entrypoint

[silver sorrel]

yes, up to now we were mixing the "docker build" and "docker run" type behaviors and it mostly "just worked". Now we've made things more explicit, which means services with legacy docker images that rely on entrypoint all break with defaults.

So, if we're going to be explicit and I need to move from
last with_exec(<args>) to as_service(<args>), I should see that in an error message

If we find an entrypoint set in the Container's image and the user hasn't set use_entrypoint (as is the case in most db images today), I would want a warning about it so I know I need to explicitly use it.

I never really used with_default_args before, but maybe I need to learn how now.

Even if we think that using entrypoint is a hack, it's the fact on the ground for important service-related images in the docker ecosystem, so we need an elegant way to work with them until we change the behavior of all container image builders.

[keen latch]

but i think if you just take a container like registry:2 and do AsService, it should "just work" - instead of manually needing UseEntrypoint https://github.com/dagger/dagger/blob/97f4996392efbbb6947a974576d1f901f4f3ef05/.dagger/test.go#L371
it feels like a small compromise to make to ensure our compatibility with service images like users have so far been used to

I should see that in an error message
I think this is where I'm struggling - I agree we need better prompts here, but I really don't know what they should look like, or how we would even detect this error case

[silver sorrel]

Big ecosystem question for me if we're going to try to go without need for entrypoint for service containers that need setup like postgres db is: What would be the suggestion to give to container image builders for Postgres, for example, that would allow them to build a valid OCI image with any tool that would work with Dagger out of the box? Like, do they keep using entrypoint for docker compat, but add something else for Dagger compat?

[frail flare]

Joining late sorry

[silver sorrel]

that was my argument as well, since I liked how things "just worked" without needing to think about it, but we decided to go with consistency with with_exec's behavior around entrypoints, for now. I think the old inconsistency was ok if we think of as_service as being docker run -d/docker compose type stuff and with_exec as being docker build type stuff, so they don't need to treat entrypoints the same. They can be consistent without their own spheres.

[frail flare]

Given where we are now, what options should we consider?

One quick note: those images don't "just work" anyway, because you have to manually expose ports

[silver sorrel]

good point

[frail flare]

I guess I didn't realize the extend of breakage, because most of the time when I run an externally built image as as long-running service. it's nginx, and that one has been working fine (I think)

As a short-term fix, wouldn't it help to have asService print a warning or info message when there's an entrypoint in the image, and it's not being used, something like: ignoring image entrypoint, to change: 'useEntrypoint:true'

Just as a hint if things break, and users go looking to find out why?

[silver sorrel]

yep. I agreed above.

Also think that since for services we require the args to be in as_service instead of in a final with_exec we have two different (but related) types of animals here. Build spec vs Run-as-service spec.
Breaking up the example from docs into two functions https://docs.dagger.io/api/services/#bind-services-in-functions

@function
    def build(self) -> dagger.Container:
        """Build container for web server."""
        return (
            dag.container()
            .from_("python")
            .with_workdir("/srv")
            .with_new_file("index.html", "Hello, world!"))

@function
    def serve(self) -> dagger.Service:
        """Start and return an HTTP service."""
        return (
            self.build()
            .with_exposed_port(8080)
            .as_service(args=["python", "-m", "http.server", "8080"])

So if I wanted my build to work for folks running it with Docker, I'd set with_default_args?

[frail flare]

Can we take the official postgres image as a baseline? What was the shell command to run it correctly as a service, before and after the change?

My assumption:

• Before: container | from postgres | with-exposed-port 5432 | as-service | up

• After container | from postgres | with-exposed-port 5432 | as-service --use-entrypoint | up

--> I'm trying the former in 0.15.1, and getting an error that seems unrelated:

"root" execution of the PostgreSQL server is not permitted.
The server must be started under an unprivileged user ID to prevent
possible system security compromise.  See the documentation for
more information on how to properly start the server.

[silver sorrel]

dagger core container from --address postgres \
with-env-variable --name DATABASE_URL --value "postgres://postgres:postgres@localhost:5432?sslmode=disable" \
with-env-variable --name POSTGRES_PASSWORD --value postgres \
as-service --use-entrypoint up --ports 5432:5432

works today

[frail flare]

Is that the before or after?

[silver sorrel]

figuring out. just a sec

[frail flare]

oh yeah seems to work with 0.15.1 👍. Shell version:

container | from postgres | with-exposed-port 5432 | with-env-variable DATABASE_URL "postgres://postgres:postgres@localhost:5432?sslmode=disable" | with-env-variable POSTGRES_PASSWORD postgres | as-service --use-entrypoint | up

[silver sorrel]

I think in the old world you used to have to do horrible final with_exec

so the as_service is an improvement there, for sure.

[frail flare]

Ah ok. What's the canonical example where the recent change makes things worse/harder then? That's what I was looking for

[silver sorrel]

Let's try registry:2. Guessing it's mostly going to be matter of --use-entrypoint being tacked on ...

container | from registry:2 | with-exposed-port 5000 | as-service --use-entrypoint | up

pseudo pre-0.15: container | from registry:2 | with-exposed-port 5000 | as-service | up

So yes, if I don't use with-exposed-port today I get a somewhat helpful error message (we should likely improve this):

Error: input: container.from.asService.up failed to select host service: select: no ports to forward

if I don't use --use-entrypoint today I don't get a helpful error message:

Error: input: container.from.withExposedPort.asService.up failed to start host service: start upstream: exited: exit code: 2

so indeed, seems next step is to produce a helpful error message on failure.

The insidious thing could be that a service "seems" to work, doesn't fail, but isn't working properly (mentioned above as well by someone, I think). So suggests we should always warn if entrypoint present, but not used.

[silver sorrel]

@errant bridge working through some of this here: #daggernauts message

Guessing you'd agree to helpful error and warning messages @errant bridge ?

===
Okay, this breakdown works well to produce an image that can be run by docker run as well as dagger:

import dagger
from dagger import dag, function, object_type


@object_type
class TestService:
    @function
    def build(self) -> dagger.Container:
        """Build container for web server."""
        return (
            dag.container()
            .from_("python")
            .with_workdir("/srv")
            .with_new_file("index.html", "Hello, world!")
            .with_exposed_port(8080)
            .with_default_args(["python", "-m", "http.server", "8080"])
        )
    
    @function
    def serve(self) -> dagger.Service:
        """Start and return an HTTP service."""
        return (
            self.build()
            .as_service()
        )

dagger call build publish --address jeremyatdockerhub/test-service

docker run -p 8080:8080 docker.io/jeremyatdockerhub/test-service@sha256:7b52ecfc5b0c1f4d87448c9d5d6930e7a7d5236faa3d8a3b200a409846ef8627

dagger call serve up

https://dagger-io.readthedocs.io/en/sdk-python-v0.15.1/client.html#dagger.Container.as_service

Turn the container into a Service.

Be sure to set any exposed ports before this conversion.

Parameters:
args – Command to run instead of the container’s default command (e.g., [“go”, “run”, “main.go”]). If empty, the container’s default command is used.

use_entrypoint – If the container has an entrypoint, prepend it to the args.

For compat with docker seems better to use with_default_args in the build in general and not use the args in as_service unless overriding.

[errant bridge]

with the issue I was hitting today I don't know if there's much dagger could have done to help tbh. The postgres image has both an entrypoint and cmd, so it's not like nothing gets executed if entrypoint goes away. The error just comes from the postgres side because the default entrypoint does some stuff to setup the nss wrapper

[silver sorrel]

yep, we're proposing a warning if entrypoint is set on the image and --use-entrypoint is not used in the as_service

since we can know that ahead of time

_type: Container
defaultArgs:
    - postgres
entrypoint:
    - docker-entrypoint.sh
mounts: []
platform: linux/arm64
user: ""
workdir: ""

maybe there is need for a global --use-entrypoint, could see that leading to "magic" bugs/vulns as well, though that was the prior behavior.

[errant bridge]

I think a warning would be fine. I don't think a global is necessary. A lot of the pain atm is just from upgrading, and that's temporary. As long as there's a warning and our cookbook snippets have the appropriate options set so people copy/pasting have success I think that's ok

[errant bridge]

the docker compat question is a can of worms though... if you choose to set with_default_args and your base had an entrypoint, it's still going to behave differently in dagger vs in docker

[silver sorrel]

yeah, trying a wrapper thing now

[errant bridge]

so best practice would have to be using with_default_args AND without_entrypoint on every container you want to publish and use in a docker environment...

[keen latch]

I think with a warning is a good shout - but I guess my question is, wouldn't we want to always use the entry point? So why force the user to explicitly opt in, if we're always prompting them to do it?

And how would we know to silence such a warning if the user has explicitly chosen to not use the base images entrypoint? Maybe we could suggest either setting UseEntrypoint or using WithoutEntrypoint

Potentially we could think of it this way - we set our dagger default args to be docker entrypoint + docker args. When using withDefaultArgs, we override all of that.

Anyways, sounds like thats not the consensus anyways - so I don't think it's worth rushing a release of v0.15.2 for this this week anyways

If we can make a warning work sometime soon, then for sure, we can release, but I don't think there's a need to rush if we're not going to change how the API works

[errant bridge]

wouldn't we want to always use the entry point?

I would agree, but I don't know if going back to that is an option 😂

[keen latch]

It's not a total reversion 😂 so maybe that helps.
It's just for the case where no args were specified by the user either through withDefaultArgs or through asService.
I think that given the pain of upgrading it, we should at least consider the option of changing it - it's just code, we can really pick any behaviour here, it's really up to us how we make it for our users.
Given it's really early on for v0.15, if we chose to change back for this one specific default case - it's probably okay. Anyone who updated to add explicit useEntrypoint won't break, since that would be the effective default if no args were specific.

In addition to not really breaking any users who'd already gone to v0.15, we'd help anyone who was about to upgrade from lower versions

[errant bridge]

just for the case where no args were specified by the user
I'm in favor of this

[silver sorrel]

@frail flare I agree with this too. In keeping with the goal of making Dagger easier to adopt.

[keen latch]

Just to clarify - I'm suggesting that withDefaultArgs keeps the behavior today of v0.15 of not using the entrypoint.
What I want to change is just the case where no withDefaultArgs is specified, no args in asService is specified - in v0.15 we take the default config container args from the image and ignore the default config entrypoint - I want to use the default config entrypoint from the image config only in this case (which is how this case functioned in v0.14 and before).
The moment the user interacts with the container to set args, the entrypoint disappears and has to be explicitly enabled (since I think it really does just confuse things in dagger world)

100% agreed on the last withExec removal, that was so difficult to understand and had so much associated complexity - the new API is clean and neat, and we should keep it as is for sure

[frail flare]

What makes all this more complicated, is that the current state is itself a stopgap... remember we had a "phase 1" and "phase 2"... We haven't shipped phase 2 yet

[frail flare]

Will it still feel consistent overall? I get the appeal of "out of the box convenience" but want to weigh the cost of "lots of special cases you have to remember, which makes the overall experience worse".

I guess you could argue: withExec doesn't have an equivalent, because there's no situation where it makes sense to execute an empty command. Therefore, there is no consistency problem.

Counter-argument: withExec can be called with an empty args, even if it's useless. Therefore consistency demands that this also uses the entrypoint, if the image is pristine and args have not been interacted with.

BUT in all cases this means you could pass useEntrypoint: false and the entrypoint would actually be used, if the conditions are right (arguments untouched). that would be confusing no?

eg.

# This would actually use the entrypoint.... :-/
container | from postgres | as-service --use-entrypoint=false

[keen latch]

I think the current state I have a couple of issues with:

• using some of the config (the cmd) but not other parts of the config (the entrypoint) can result in: it works entirely, it doesn't work at all and crashes, it doesn't work at all in an invalid state, it appears to work but doesn't etc - imo we should either always consume the config entrypoint+cmd together, or not at all.
• it's just adding another step to consuming pre built docker images - yes, we can fix it in docs, and add some warnings etc - but if we're always guiding users towards a specific outcome, then wouldn't we want to just pick that as the default anyways? The before+afters above are always showing the after as longer and more verbose

[keen latch]

We can make it not use the entrypoint - we can pick the behaviour here - if the user has specified to explicitly disable it, we can just disable it.

But this is also a weird case - I am struggling to understand why a user would want this combination, though I suppose you could always create one.
I guess this would be more naturally expressed as WithoutEntrypoint in the chain, but yeah, if a user does ask for this, we can just turn it off

[frail flare]

But the SDKs don't let us distinguish between "flag not set by user", and "flag set to false", do they?

[keen latch]

Only go 🙂

But yes you're right

[frail flare]

My initial motivation was this:

• Entrypoints are a pseudo-API buried in artifact metadata. An early attempt at making Docker a platform. It's scope creep.
• With Dagger you can create an actual API, using actual code. I want entrypoint logic to be expressed in code, cleanly
• I want good compatibility with the images that exist out there. But not at the price of the Dagger API being stuck with this shitty pseudo-API forever, making your Dagger code less complete and less expressive.

So it's not about making entrypoints unusable right away. That's definitely a non-goal. It's about creating a path towardsw entrypoints gradually going away, so that we're not stuck with them forever.

unfortunately it's hard to one without the other, it turns out

[silver sorrel]

import dagger
from dagger import dag, function, object_type


@object_type
class Wrap:
    @function
    async def wrap(self, image: str) -> dagger.Container:
        """Wrap."""
        ctr = dag.container().from_(image)
        ep = await ctr.entrypoint()
        cmd = await ctr.default_args()
        if len(ep) > 0:
            ctr = ctr.without_entrypoint().with_default_args(ep + cmd)
        return ctr
    
    @function
    async def serve(self) -> dagger.Service:
        """Start and return a postgres db service."""
        w = await self.wrap("postgres")
        return w.with_env_variable("POSTGRES_PASSWORD", "postgres").as_service()

voila, no more entrypoints.

[keen latch]

Agreed, I think we definitely want to get rid of users needing to think about setting entry points themselves entirely.
I think this is actually the main argument for doing this weird little dance for imported images - so users don't need to set useEntrypoint everywhere. If it's just something users do automatically, then we run the risk of it just being an "incantation" that users add without thinking about, which hurts us when we want to remove more of the entrypoint api in the future.

I think we all agree adding the warning would be good? But if we're always pushing users to add this extra thing, and consider the absence of it to be a bug or non-idiomatic dagger, I think it's worth just fixing it at the API level so you can't do it wrong. fewer potential foot-guns = fewer things users get frustrated with while daggerizing

[silver sorrel]

It took me a minute to find the right use_entrypoint incantation in Python
In Go it looks like this: AsService() vs AsService(dagger.ContainerAsServiceOpts{UseEntrypoint: true})
In TypeScript: asService() vs asService({ useEntrypoint: true })

        # I suspect most people just want to say, and will try first:
        return (
            dag.container().from_("postgres")
            .with_env_variable("POSTGRES_PASSWORD", "postgres")
            .with_exposed_port(5432)
            .as_service())

        # versus:
        return (
            dag.container().from_("postgres")
            .with_env_variable("POSTGRES_PASSWORD", "postgres")
            .with_exposed_port(5432)
            .as_service(use_entrypoint=True))

need to look into why my IDE didn't suggest use_entrypoint

but happy once in place

[frail flare]

My hope initially by making useEntrypoint: true an opt-in, was not to make it something you copy-paste everywhere just in case, but rather something that you would prefer to remove from your code - so you get trained to think of entrypoint as something that one shouldn't use by default

"oh this image uses an entrypoint shell script to run" -> this should be synonymous with "this is a poorly designed image"

Just to put this whole user experience glitch in perspective:

• It's not the case that one can pick any image from docker hub, and have it "just work" reliably with just Container().From().AsService(). There are almost always other chenanigans or verifications required anyway.

• Copy-paste still rules the world. Assuming you will copy-paste the correct incantation anyway, it's not a major difference whether the copy-paste has an extra bool argument or not

• In the long term it is a win if the Dagger ecosystem is weaned off shitty entrypoint scripts, while the rest of the docker ecosystem is stuck with it.

• Modules are a multiplier on the above

• Here is my ideal "it just works" command in Q1 2025: github.com/dagger/postgres | server | up; github.com/dagger/docker | registry | up

[keen latch]

I guess I worry about the perspective of entrypoints are bad and we want to get rid of them - people still use them, they're well through the ecosystem (and in OCI as well, and throughout the docker official images). Others might keep disagreeing with us, and keep using their entry points, and it makes getting new images into the dagger ecosystem and building that neat wrapping module just a touch harder - we can't realistically maintain official modules for everything, it's a huge amount of effort - we need the community help here, so making it easy is important.

But I'm running out of steam honestly, so if we're happy to eat the upgrade trouble for all users using services, and happy to explain this in docs somehow, then we can leave it as is.

I don't see that we're sacrificing the core idea by just changing this for images that are being imported from docker "world" into dagger "world" - like you mention, there are bigger consistency problems.

[frail flare]

I'm playing devil's advocate but not definitely about the limited reversal you mentioned. Just thinking through it outloud, there are so many dimensions...

In any case it seems we agree on the first incremental step of adding an info message to help debugging. Doesn't close any doors while we figure out what to do next.

One thing I worry about is not just existing docker images, but also new dagger modules relying on setting the entrypoint as a "shadow API", which adds a new dimension of complication every time a dagger module receives a container from another.

It just makes our Container type slightly less reliably for interop. Kind of like usb-C cables that you can always plug in, but may or may not work - there are other dimensions of interop now than just "can I plug it in"

"oh my function sets an entrypoint". "Ah - but my function calls sh in the containers it receives; but it didn't manage entrypoint. I guess my function needs a new argument to configure whether to use the entrypoint it the containers it receives"

[keen latch]

(will pick this up again in the morning, it's bed time 😴 regardless, it doesn't feel like we want to do anything particularly urgent here, so happy to keep it async)

[frail flare]

"oh my function function sets an entrypoint, did you disable that?" "Ah yes, I always sanitize my inbound containers with .WithoutEntrypoint in my modules because I find it more reliable that way" "well I consider my entrypoint arg an implementation detail, currently it's foo but I can't promise it won't change/

[silver sorrel]

If someone is preparing postgres for their org, sure than can get it all prepped and ready with a multi-stage build, or by using a dagger module.

I think the postgres folks are building general-purpose images that they want to be compatible with k8s, docker, podman, dagger, etc, and thus they run scripts on container startup for db customization based on env vars, etc. In the ENTRYPOINT script they do a bunch of prep and then look for sql scripts and such in a docker-entrypoint-initdb.d directory, etc. A whole entrypoint initialization aparatus which many folks are accustomed to and kubernetes runs by default.

k8s init containers are an interesting/related study: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
https://www.youtube.com/watch?v=Ezz03l2JDmE

Some k8s init containers vs docker entrypoint:
https://chatgpt.com/share/6760cb00-3550-800c-a9df-ef823ede1b03

[golden depot]

Here is my ideal "it just works" command in Q1 2025: github.com/dagger/postgres | server | up; github.com/dagger/docker | registry | up

one more thing that entrypoints do today is e.g. seeding the database on startup. You place some sql files in /init-database and entrypoint seeds the database with those sql data.

[golden depot]

to collect some datapoints, I looked at top 25 docker images and ALL of them have atleast CMD set. and about 18 of them had ENTRYPOINT set.

[dense wraith]

catching up with this thread. I was having a quick sync with Rajat this morning and he told me that we're still spiraling around this issue. It seems to me that going back to the original proposed solution of raising a friendly error when Entrypoint and DefaultArgs are not set so we overcome the hanging issue as well as adding the warning message are the reasonable things to do for 0.15.2. Otherwise, if we're thinking about making bigger API changes and potentially releasinga nother MINOR, that'd be super confusing to users IMO.

In my experience, the whole CMD, ENTRYPOINT thing has always been a mess in the Docker ecosystem and never has been properly addressed. That's one of the reasons why I am content with the current decision as it helps the user to at least stop for a second and thing what they're actually trying to do in their pipelines while still allowing them to fallback to the "traditional" way of using the ecosystem images via the UseEntrypoint argument.

[keen latch]

i'm still a bit frustrated that it's possible to consume the image's defined CMD without it's defined ENTRYPOINT - as mentioned here, it's a genuine footgun

i feel like the final design for this (ignoring what we do in the short term for now) should make sure this isn't possible - either by always bundling entrypoint+cmd together when reading the builtin image config, or by requiring something like a UseCmd or something

but to do a warning, I think should be very doable - the first step (implementation wise) should be to separate out WithDefaultArgs to write to a dedicated field, and not overwrite the Config.Cmd - then we can actually detect at the point of exec when this has happened - now when we fallback to loading the Config.Cmd without UseEntrypoint being set, we can warn. It'll also be easy to then change the behavior later to combine Config.Entrypoint and Config.Cmd is no args were set (as I was proposing further up, if we go that direction)

[dense wraith]

yes, agree. I forgot about addressing the CMD thing which is super confusing as it is today as well. I'll summarize the action items below and wait for consensus then to see if we can move forward

[keen latch]

i've done some of the hopefully uncontroversial items in: https://github.com/dagger/dagger/pull/9231

[dense wraith]

So, according to my understanding, here's the latest summary of this thread as well as a possible path forward towards fixing some of the most important effects of v0.15 service changes:

• There's still not a solid consensus about the changes introduced in v0.15 around the Service entrypoints. Even though the change makes sense from the design perspective, looks like we're still worried about the potential confusion it might cause by breaking Docker's ecosystem. On the other hand, we're aware that the docker ecosystem has its flaws mostly due to the lack of consistency on how images are built and seems like we still haven't agreed on a solution which could both be compatible with the ecosystem will keeping Dagger's design consistent and simple

• We need to make sure that Container.up and Container.asService are consistent. Currently being addressed by https://github.com/dagger/dagger/pull/9231

• To promptly fix the issues introduced in v0.15, given the current design, we want to throw a user error if neither WithDefaultArgs not UseEntrypoint is set when defining the service

• We should separate WithDefaultArgs to its own field so it doesn't collide with the container Config.CMD so we can detect when the user intentionally set it

• We can add a warning when the image has an Entrypoint and the user didn't specify it when creating the service. This is a way to mostly bring down the confusion from users migrating from < v0.15

Thoughts?

[keen latch]

agreed with all points, but less convinced on the idea of doing a warning

[silver sorrel]

It's interesting that you mention Config.CMD as I was going to ask if we could just use the stringcmd instead of default_args. When I hear "default args," I think default arguments to what? The ENTRYPOINT, presumably. It feels confusing to me. Command/CMD feels semantically like, "the command I want to run".

[keen latch]

if we're erroring when no args are provided (the case where we'd maybe automagically just fallback to the previous behavior), is that not enough?

[frail flare]

I think I'm ok with @keen latch 's suggestion, it's not really a reversal of the change, more like a refinement

but definitely DO NOT want to add one more field, that's how entrypoint started, "let's just add another keyword"

what's the rationale for that part @dense wraith ?

[keen latch]

just to clarify - this wouldn't be api facing - it's how we would do the implementation

it's just one possible way to do it, but it preserves the most information, otherwise we probably have to have a bunch of floating booleans

[silver sorrel]

Since CMD has this dual nature as a "default, user-overridable command" and as the "default args for the entrypoint", and we want to emphasize the "default command" case, our api's "default_args" could be semantically confusing.

It's a bit different in k8s as well, of course.
As for existing images, here's some more data on their use of ENTRYPOINT: https://docs.google.com/spreadsheets/d/1dHc4XbdlczaqTc7Fut2p0-Wi_E3Jc6mGoaKeX0lM2vo/edit?gid=0#gid=0

ChatGPT's take for Docker/containerd:

Key Points to Understand:
CMD provides a default command or arguments for the container.
If ENTRYPOINT is set, CMD becomes the default arguments to the ENTRYPOINT process.
Users can override the default behavior of CMD easily when running the container.

Summary Analogy:
Think of CMD as a fallback plan:
It provides default instructions for the container.
If the user specifies a command when running the container, that command replaces the CMD.
If used alongside ENTRYPOINT, it becomes the default arguments passed to the fixed ENTRYPOINT command.

I notice that in k8s, command maps to/overrides ENTRYPOINT and args maps to/overrides CMD in the pod spec.

ChatGPT's take for Kubernetes:

If neither command nor args is specified, Kubernetes uses the ENTRYPOINT and CMD from the image as-is.

ENTRYPOINT + CMD = Default behavior in Docker images.
Use command in Kubernetes to override ENTRYPOINT.
Use args in Kubernetes to override CMD.

[frail flare]

ok you're right that's different. I just want to avoid users having to juggle too many dimensions in their head

[keen latch]

agreed 😆

[silver sorrel]

Thought this made sense.

Community Trends:
Many base images (e.g., nginx, postgres) use ENTRYPOINT to define non-overridable behavior since they are purpose-built.
Application developers often prefer CMD to maximize flexibility for downstream users.
There’s a preference in the community to make ENTRYPOINT more extensible by always supporting arguments passed with CMD.

Conclusion:
Avoid ENTRYPOINT if:
Flexibility is paramount.
Your image doesn't require mandatory initialization logic.
You want to make it easy for users to override the startup command.

Use ENTRYPOINT if:
The container needs to enforce a particular behavior.
Initialization logic is essential before running the main process.

Ultimately, it's not a mistake to use ENTRYPOINT, but if simplicity and usability are your priorities, CMD alone often suffices.

Possible summary note for Dagger 0.15+:
Unlike Docker, containerd, or Kubernetes, Dagger does not use the ENTRYPOINT by default, but the user may specify use-entrypoint when adding a with-exec or as-service. This is important for many existing containers for long-running services like dbs, caches, message queues, proxies, and webservers that currently use ENTRYPOINT for initialization.

[frail flare]

Maybe we should draft the proposed change to that paragraph (let's call it "the justin amendment" 😁) to make sure we're all on the same page?

btw better than a log warning: what if every exec always emitted a span with the exact arguments being executed? that way the user could look for that, and quickly realize if the arguments look wrong

[keen latch]

Happy to pick that up tomorrow morning 🙂

I was thinking consistency stuff earlier as well - I'd potentially like to apply the same amendment to the empty exec arg string for WithExec
Atm, I guess it's kind of undefined right? Since it's previous purpose was really only applicable for setting overridable options for a service, and we dont get those from withExec now

Not really sure what other behaviors make sense - but again, was hoping to remove the discrepancy where we sometimes read the CMD without the ENTRYPOINT (from the base images config)

[silver sorrel]

So the "amended" paragraph is this?

To promptly fix the issues introduced in v0.15, given the current design,
[ in a container with an existing ENTRYPOINT ]
we want to throw a user error if neither WithDefaultArgs (override) nor UseEntrypoint is set when defining the service
[ in the case where they specify WithDefaultArgs without UseEntrypoint, the user has overridden DefaultArgs/CMD and thus we will ignore the docker default of the ENTRYPOINT+CMD pair (though we don't know if that is their "intent" or accidental usage if they assume the pair will execute - see *note)]
[ in the case where they specify UseEntrypoint AND WithDefaultArgs, we use the existing ENTRYPOINT and new overridden DefaultArgs/CMD as the pair]
[ in the case where they specify WithEntrypoint AND UseEntrypoint, they will override or set the ENTRYPOINT and use that together with any DefaultArgs/CMD present as the pair (seems uncommon)]
[ in the case where they specify WithEntrypoint AND UseEntrypoint AND WithDefaultArgs, we get a completely new ENTRYPOINT+CMD pair that will execute and persist in a published image]

*note: always emit a span with the exact arguments being executed, so the user could look for that and see what combo of Entrypoint, DefaultArgs, or AsService args are being executed (useful for WithExec too)

[dense wraith]

SGTM! thx for sharing "Justin's amendment" 🙏

[frail flare]

Right but we should review the proposed user-facing paragraph. "This is how it works" and make sure it's not confusing and terrifying

[errant bridge]

[ in the case where they specify WithEntrypoint AND UseEntrypoint, they will override or set the ENTRYPOINT and use that together with any DefaultArgs/CMD present as the pair (seems uncommon)]
I think we should probably not use default CMD in this case. Feels like it's the exact same issue as using the predefined entrypoint when the user hasn't specified

[silver sorrel]

hmmm...yeah, the rule then simplifies to
You can specify the whole ENTRYPOINT+CMD pair and we'll exec it (if you say UseEntrypoint), else if you only specify one, then we only exec one

[dense wraith]

I think that's what currently happening. I understand how this could be confusing though and I agree it's less error prone to not use the image's CMD but at the same time it's less consistent with the current behavior and overall Docker experience 🤔

[errant bridge]

based on this #general message I think you're right. And that makes sense. The way I think about docker images I'm pulling, it only ever makes sense to use the default CMD with the default entrypoint

[silver sorrel]

from my spreadsheet above. Feels like history...in olden times, folks put all kind of logic in entrypoints, then cmd came along to give users an overridable interface, but many containers with heavy init/setup needs (e.g. dbs) continue to use the entrypoint script.

[silver sorrel]

oh, right
dagger core container from --address postgres

_type: Container
defaultArgs:
    - postgres
entrypoint:
    - docker-entrypoint.sh
mounts: []
platform: linux/arm64
user: ""
workdir: ""

Need one of the invocations below to replicate this one using the stock ENTRYPOINT+CMD pair:
dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres as-service --use-entrypoint up

Sets ENTRYPOINT for container and sets args for service
dagger core container from --address postgres with-entrypoint --args docker-entrypoint.sh with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres as-service --use-entrypoint --args postgres up
or
Sets ENTRYPOINT+CMD for container and by saying "--use-entrypoint", we mean use the ENTRYPOINT+CMD pair, that is, there is no separate use-default-args (use CMD) in the API.
dagger core container from --address postgres with-entrypoint --args docker-entrypoint.sh with-default-args --args postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres as-service --use-entrypoint up

PERHAPS, most Dagger-idiomatic way to run this?
dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres as-service --args docker-entrypoint.sh,postgres up move the needed entrypoint script to the as-service args

use-entrypoint implies use ENTRYPONT+CMD if no with-exec args override the CMD/default-args

$ dagger core container from --address alpine with-entrypoint --args echo with-default-args --args hello,from,default with-exec --args "" --use-entrypoint stdout

hello from default

Override default-args with with-exec args:

$ dagger core container from --address alpine with-entrypoint --args echo with-default-args --args hello,from,default with-exec --args hello,from,withexec --use-entrypoint stdout

hello from withexec

CMD/default-args reign supreme in the default case:

$ dagger core container from --address alpine with-entrypoint --args ls with-default-args --args echo,hello,from,default,sorry,kyle with-exec --args "" stdout

hello from default sorry kyle

cc @errant bridge @dense wraith

[silver sorrel]

Possible summary note for Dagger 0.15+ v2:
Dagger does not use the container ENTRYPOINT by default in WithExec or AsService functions.

Since many popular container images for long-running services like dbs, caches, message queues, proxies, and webservers currently specify a script for initialization in the ENTRYPOINT args array, we advocate moving both the ENTRYPOINT and CMD args to the AsService args (as shown below) to make the behavior most explicit and predicable.

dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres as-service --args docker-entrypoint.sh,postgres up

It's easy to discover the values for the ENTRYPOINT and CMD/defaultArgs by running a command like:
dagger core container from --address postgres

_type: Container
defaultArgs:
    - postgres
entrypoint:
    - docker-entrypoint.sh
mounts: []
platform: linux/arm64
user: ""
workdir: ""

Alternatively, you can use UseEntrypoint as a convenience to use the default ENTRYPOINT specified in the image together with either the CMD/defaultArgs or args you specify in the WithExec or AsService function calls.

==
Note:
While Dagger does not use the container ENTRYPOINT by default in WithExec or AsService functions, it does use the container CMD/defaultArgs by default if no overriding args are provided to WithExec or AsService.

[errant bridge]

we advocate moving both the ENTRYPOINT and CMD args to the AsService or WithExec args
Do we, though? It seems to me that setting WithoutEntrypoint and WithDefaultArgs(args...) would be the better approach, even if it doesn't look as clean. That way when you publish your image it can be run the same as in dagger as-service

[silver sorrel]

yeah, there's the, I just wanna consume this upstream image that I don't control and the, I'm going to publish an image for others to consume

We show the approach I drafted here:
https://docs.dagger.io/features/services/
.as_service(args=["python", "-m", "http.server", "8080"])

kinda

[errant bridge]

Yeah, totally looks cleaner, just means we have 2 different best practices depending on whether you're going to publish the container

[silver sorrel]

 @object_type
class TestService:
    @function
    def build(self) -> dagger.Container:
        """Build container for web server."""
        return (
            dag.container()
            .from_("python")
            .with_workdir("/srv")
            .with_new_file("index.html", "Hello, world!")
            .with_default_args(["python", "-m", "http.server", "8080"])
            .with_exposed_port(8080)
        )

    @function
    def serve(self) -> dagger.Service:
        """Start and return an HTTP service."""
        return (
            self.build()
            .as_service()
        )

this looks clean too, to be fair

Could also do

@object_type
class BuildService:
    @function
    def build(self) -> dagger.Container:
        """Build container for web server."""
        return (
            dag.container()
            .from_("python")
            .with_workdir("/srv")
            .with_new_file("index.html", "Hello, world!")
            .with_default_args(["python", "-m", "http.server", "8080"])
            .with_exposed_port(8080)
        )

@object_type
class UseService:
    @function
    def serve(self) -> dagger.Service:
        """Start and return an HTTP service."""
        return (
            dag.build_service().build()
            .as_service(args: ["python", "-m", "http.server", "8080"])
        )

but yeah, redundant, but explicit

hmmm...
dagger core container from --address postgres without-entrypoint with-default-args --args docker-entrypoint.sh,postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres

Now I've made something that "just works" but has a different interface than the original, so other tools may need to be adjusted.

_type: Container
defaultArgs:
    - docker-entrypoint.sh
    - postgres
entrypoint: []
mounts: []
platform: linux/arm64
user: ""
workdir: ""

vs original interface
dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres

_type: Container
defaultArgs:
    - postgres
entrypoint:
    - docker-entrypoint.sh
mounts: []
platform: linux/arm64
user: ""
workdir: ""

Possible summary note for Dagger 0.15+ v3:
Dagger does not use the container ENTRYPOINT by default inAsService or WithExec functions, but uses CMD/defaultArgs by default. This allows the CMD/defaultArgs to be overridden by args provided to AsService or WithExec.

How to handle entrypoint scripts/commands in ENTRYPOINT:

AsService:
Since many popular container images for long-running services like dbs and webservers currently specify a script for initialization in the ENTRYPOINT args array, we advocate combining the ENTRYPOINT + CMD args and setting that as the new CMD/defaultArgs to make the behavior most explicit and predicable at runtime and on publish. For example:

dagger core container from --address postgres without-entrypoint with-default-args --args docker-entrypoint.sh,postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres as-service up

It's easy to discover the values for the ENTRYPOINT and CMD/defaultArgs by running a command like:
dagger core container from --address postgres

original

_type: Container
defaultArgs:
    - postgres
entrypoint:
    - docker-entrypoint.sh
mounts: []
platform: linux/arm64
user: ""
workdir: ""

then examine the preferred container:
dagger core container from --address postgres without-entrypoint with-default-args --args docker-entrypoint.sh,postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres

_type: Container
defaultArgs:
    - docker-entrypoint.sh
    - postgres
entrypoint: []
mounts: []
platform: linux/arm64
user: ""
workdir: ""

WithExec:
Some container images may similarly have a command like git in the ENTRYPOINT. We advocate for moving commands like that to the args of the WithExec where you execute that command. For example alpine/git has git in the ENTRYPOINT and ``--helpin theCMD/defaultArgs`:

dagger core container from --address alpine/git without-entrypoint without-default-args with-exec --args git,version stdout

As above:
dagger core container from --address alpine/git

_type: Container
defaultArgs:
    - --help
entrypoint:
    - git
mounts: []
platform: linux/arm64
user: ""
workdir: /git

Alternatively, you can use UseEntrypoint as a convenience to use the default ENTRYPOINT specified in the image together with either the CMD/defaultArgs or args you specify in theAsService or WithExec function calls.

[frail flare]

Can we make this an issue please?

[dense wraith]

We started discussing here.

https://github.com/dagger/dagger/issues/9190

[silver sorrel]

added my draft above 👆

[keen latch]

draft looks mostly okay:

• i would suggest that instead of hardcoding the with-default-args, we pull it dynamically from the entrypoint if we suggest that
• does this include the amendment we were talking about? in the case that no args have been set anywhere.

[silver sorrel]

please expand on your points so there is no room for confusion 🙂 since I missed capturing your exact meaning

There's the "this is how it works" and "this is what we think you should do given how it works" (could be several options for different circumstances, use-cases). The mechanics and the implications.

[keen latch]

for the first point - should we not recommend something more like this

instead of specifying that users should manually get the entrypoint, and insert it into withDefaultArgs - since the upstream image could change it's entrypoint later

[silver sorrel]

So not a recommended practice for the user, but new behavior for Dagger?

Or the user implements that pattern in code?

[dense wraith]

re-reading your comment @silver sorrel I'm also confused about why we're explicitly telling the user to unset the entrypoint in the container. What's the reason behind that if AsService won't use it anyways by default. Feels to me like an unnecessary cognitive overhead to do that 😬

does this include the amendment we were talking about? in the case that no args have been set anywhere.

it should. Should we add a note that not setting WithDefaultArgs and not using UseEntrypoint will yield into a user error?. cc @keen latch

[silver sorrel]

my most updated version is in the issue comment, can update.
I was sort of taking the extreme position of, "Yes, sometimes you might need to run some setup commands for a database or webservice (could even be a shell script), but we should not be using or relying on ENTRYPOINT and this problem ends with me! Going forward, the images I create/curate for my team will not use ENTRYPOINT."

I was curious to see if taking that approach would lead to issues/contradictions.

[dense wraith]

I see.. and I agree with the statement. I'm a bit worried it might confuse users (like I got confused) as they could think it's the recommended practice to remove the container Entrypoint

[silver sorrel]

My brain is not toally online yet, but I suspect that there are different contradictions that arise from removing it and leaving it depending on whether you plan to run images later in k8s, docker, or dagger.

[dense wraith]

yes. I have the feeling that we might be mixing two things which are generating a bit of confussion:

1. The AsService change we introduced in v0.15
2. How we think Dagger "improves" the whole ENTRYPOINT+CMD mess.

OFC the two are connected, but trying to reflect everything in a changelog might be confusing for users

[silver sorrel]

I figured the blurb I was working on might go into docs eventually. For sure we'd want something shorter in the changelog

[dense wraith]

@keen latch to recap the v0.15.2 changes so we can close (https://github.com/dagger/dagger/issues/9190), the only thing missing IIUC is to raise an error when both UseEntrypoint and WithDefaultArgs are not set, correct?

[silver sorrel]

Especially k8s where someone might have set command (overrides ENTRYPOINT) or args (overrides CMD/defaultArgs) in the pod spec yaml.

[keen latch]

@dense wraith yeah the point in this comment here

erroring is acceptable yup

[silver sorrel]

you mean raise error when:
((ENTRYPOINT is set) AND ((CMD is set) OR (CMD is NOT set))
AND
((UseEntrypoint NOT set) AND (WithDefaultArgs NOT set))

[dense wraith]

perfect, I'm also somehow lost again about what the final decision is around using the defaulting on the image's CMD or not. Again, reading @silver sorrel 's comment, I see **Dagger does not use the container ENTRYPOINT by default inAsService or WithExec functions, but uses CMD/defaultArgs by default. **

does this mean that we'll use the image's default CMD if that's set? If the entrypoint it also set will that make it ENTRPOINT+CMD?

[keen latch]

I mean, an error is raised if all of the following are true:

• UseEntrypoint is not specified to AsService
• args is not specified to AsService
• WithDefaultArgs is not specified on the container prior to calling AsService
  The container config having CMD shouldn't affect this

ATM, in v0.15.1, the behavior when all of those are true is to fallback to the container CMD - this behavior is inccorect IMO (as mentioned before) - running the container CMD automatically without the ENTRYPOINT is kind of weird behavior.

Erroring is one acceptable way to solve it - and what we should do for v0.15.2. In the longer term, we should maybe consider my amendment, where instead we run ENTRYPOINT+CMD.

[dense wraith]

ok, so AsService will ignore the container's CMD by default then. This needs to be updated in Jeremy's comment if we decide it should work this way

[dense wraith]

SGTM

[keen latch]

yes.

if you want to use the container config's CMD, then you must specify useEntrypoint as well - if you don't, then that's the error case IMO

[dense wraith]

yes, makes sense to me also

@silver sorrel WDYT?

[silver sorrel]

@keen latch all of your examples above assume there is an ENTRYPOINT set on the container image or doesn't matter to you?

[keen latch]

if there is no entrypoint set, we don't need to error

[silver sorrel]

just being explicit since it's easy to get lost here 🙂

[keen latch]

the way to think of the error that works for me: it's a protection from the footgun where you run the container CMD without the container ENTRYPOINT. IMO, we should never allow the user to do this. So if we would, we should error out, so that it doesn't happen.

(to clarify - i mean the container config that we import from an external image)

[dense wraith]

wait, I thought the image having an entrypoint or not was independent of erroring given that the user could still not set the WithDefaultArgs

that will results in the current hanging behavior right?

[keen latch]

oh i see

yeah we also want to protect that case too

okay, so, it doesn't matter if the image has no ENTRYPOINT

better way to think of it then: we need the user somehow explicitly indicate what command to run, or you get an error. falling back to weird implicit behavior is complicated, so we should avoid that.
you can indicate the command by:

• using WithDefaultArgs
• using AsService args
• using AsService useEntrypoint

this should catch most of the issues that users have

and enforce the changes you need when you upgrade

[dense wraith]

@silver sorrel going back to your statement, if we proceed this way, this sentence needs update: "Dagger does not use the container ENTRYPOINT by default in AsService or WithExec functions, but uses CMD/defaultArgs by default"

What we're saying here is that AsSerivice won't use anything by default. You need to set either UseEntrypoint or WithDefaultArgs / args

[silver sorrel]

that's how it is here:
https://github.com/dagger/dagger/issues/9190#issuecomment-2549676262

but needs update. got it

[dense wraith]

yes, needs some minor fixes. The sentence I pasted above is from there which brings some confusion

[silver sorrel]

So after this change, it won't use it anymore

it will error

[dense wraith]

correct

[silver sorrel]

but in the case where there is NO ENTRYPOINT, then AsService will use CMD by default, right?

[dense wraith]

no

[dense wraith]

^ this

we're proposing that for v0.15.2AsService won't use anything by default

even if the image has or doesn't have ENTRYPOINT and/or CMD

the user needs to set UseEntrypoint and/or WithDefaultArgs / args

[dense wraith]

@keen latch is proposing that we can do this later

because we need to decide what we do in the cases where the image has a CMD and an ENTRYPOINT since you can't just run the CMD by default as this could yield inconsistencies

[silver sorrel]

hmmm...okay. Since many/most containers that would be used for services do have the ENTRYPOINT+CMD combo today, that would make sense, but would impact some containers that are meant to be services, but only use CMD like
httpd
tomcat
(I guess Apache stuff 😆 )

https://docs.google.com/spreadsheets/d/1dHc4XbdlczaqTc7Fut2p0-Wi_E3Jc6mGoaKeX0lM2vo/edit?gid=0#gid=0

I don't see an issue when an image has CMD but NO ENTRYPOINT

[dense wraith]

I don't think there's an issue in this case. I'm assuming we're taking the conservative route so we can fix v0.15.1 and introduce this "default" behavior later

[keen latch]

yes exactly! imo, it's very easy to lift an error case, and make an error case a non-error case

it's much harder to backpedal and make previously good cases error cases (though i guess that's sort of what we're doing here - but slightly different because we're trying to do a fast-follow, and hopefully improve the upgrade experience by making it clear how users should upgrade)

this isn't the final pass

[dense wraith]

In fact, this is how it works in v0.15.1 right @keen latch ? With the exception that if the image has an entrypoint, it won't be appended to the CMD. So super confusing 😬

[keen latch]

yeah, that's how it works right now

but, one reason it's frustrating having different behavior is that it forces users to think about the fact that the images they consume have entrypoints

if our goal is to stop people from using entrypoints, then we should treat images that have entrypoints and don't the same

all about removing the stray footguns

@dense wraith who's picking up the code work for this? I haven't made any progress on it today, I'm feeling a bit ill annoyingly, so not really ended up touching code
I can probably take a bit more of a look tomorrow, unless someone else is on it?

[dense wraith]

I can take it today Justin

No worries

@silver sorrel can you help me by updating the comment in the issue?

[silver sorrel]

exactly, even if we're not trying to "stop" them, but gently nudge them to a new standard.

The more I read about this topic, the more I hear about the original intent of ENTRYPOINT to be the the thing that devs DON'T override (except as a last resort) because its the main executable OR "has to run to guarantee the thing (like a db) is set up right"
and
CMD's intent to be overridable by devs.

So @frail flare it's sounded like you think there should not be this split of intent (it was a mistake), but that everything should be either in the CMD/defaultArgs or in the args to AsService, eh?

I'm guessing that if I extend that a bit, you might say, don't leave any commands hidden as defaults on the image in CMD at all but set them all when invoked by AsService

wow, looking at Docker Compose adds a little more confusion potential for folks:

k8s command overrides ENTRYPOINT
k8s args overrides CMD
docker compose command overrides CMD but uses ENTRYPOINT

both tools run ENTRYPOINT by default (typically with CMD, if present)

Asked ChatGPT for a decsion tree @keen latch @dense wraith going to look at

START
  ├── Does the container have ENTRYPOINT?
  │     ├── No
  │     │     ├── Does the user specify `WithDefaultArgs`, `args`, or `UseEntrypoint` in `AsService`?
  │     │     │     ├── Yes → Use specified configuration to execute the container.
  │     │     │     └── No → Raise an error (v0.15.2 behavior).
  │     │
  │     └── Yes
  │           ├── Does the user specify `UseEntrypoint` in `AsService`?
  │           │     ├── Yes → Run ENTRYPOINT + user-specified `args` or container's `CMD` (if no `args` specified).
  │           │     └── No
  │           │           ├── Does the user specify `args` or `WithDefaultArgs`?
  │           │           │     ├── Yes → Run user-specified command with ENTRYPOINT ignored.
  │           │           │     └── No → Raise an error (v0.15.2 behavior).
  │
  └── Decision Notes:
        - Defaulting behavior removed in v0.15.2 to prevent "footguns."
        - Users must explicitly configure the execution method.
        - ENTRYPOINT and CMD must be handled explicitly for clarity.

Key Points:
No Defaults by v0.15.2: If users do not explicitly set UseEntrypoint, WithDefaultArgs, or args, Dagger will raise an error.
ENTRYPOINT + CMD Handling: For containers with both ENTRYPOINT and CMD, the combination will only be executed if UseEntrypoint is explicitly enabled.
Error Prevention: The changes aim to prevent unintentional behavior, especially when using CMD without ENTRYPOINT.
This structure ensures consistent and explicit behavior, making it clear to users how container commands are executed in Dagger.

[dense wraith]

https://github.com/dagger/dagger/pull/9247 now waiting for tests to pass to see check if this PR breaks something else 🙏

[silver sorrel]

I spoke with Solomon more just now.

He's actually on board with having as-service use the unchanged pair of [ENTRYPOINT+CMD] by default (one of them could be empty and that's okay, of course). Fact-check me on that @frail flare 🙏

That would give us out of the box compat with existing images for services like postgres, nginx, etc.

So that would revert the requirement introduced in 0.15 for as-service --use-entrypoint even when there are no as-service --args and no with-default-args.

I think most of our wrangling was trying to find a new solution while KEEPING the decision made in 0.15 to require as-service --use-entrypoint when you wanted to use the container image's built-in ENTRYPOINT+CMD with no changes.

[dense wraith]

@silver sorrel so this will revert the change introduced in v0.15?

does this mean that we have to release a new minor?

because we're breaking the behavior again

[dense wraith]

well..not breaking the behavior, but does this mean that the UseEntrypoint flag is useless now?

given that by default AsService will go back to using ENTRYPOINT+CMD again?

[dense wraith]

this is confusing me. We had long meetings in the past where everybody was involved about not making this the default behavior .We were also aware that this was going to break the Docker ecosystem and we were ok with that. We're now rolling back the decision for the same argument we decided to implement UseEntrypoint in the first place?

[dense wraith]

just moved my PR to draft as it's still no clear how we should proceed here. Let's have a quick chat tomorrow @keen latch @silver sorrel @frail flare and agree on the path forward 🙏

[keen latch]

yes - on board with this as well i think

this is what someone called the "justin amendment" above

essentially @dense wraith - this behavior is mostly the same as in your pr
except, instead of erroring out, we set cmdArgs to default to entrypoint + cmd

however, this doesn't help to resolve the case where a user has set a WithExec that would otherwise hang - which is really the main issue with the amendment I want

detecting that case is really hard though, it's still possible to miss it with the error in 9247:

• take nginx as an image
• we do withDefaultArgs([/bin/sh])
• do a withExec([nc -tlp 4000]) to open just a random tcp socket
• now if we do asService + start, previously this would have launched nc as a service! in 0.12.5, we just hang on waiting for nc to complete successfully (which it won't ever).
• we can't detect this case by just looking at entrypoint + cmd + default args + any combination. we've set defaultArgs - so now, we won't hit the error!

i think this is a case that we absolutely can cover in a migration guide - i don't think it's possible to catch this in every case for the hanging problem - so proposing that we just don't handle that.

[keen latch]

if we remove that from the equation, then just doing this makes sense to me

[keen latch]

i'm feeling very flu-y today - so probably not able to jump into a call - so did a brain dump above

one thing to be aware of - i don't think we're doing v0.15.2 before the holidays. i don't think we want to risk needing to debug potential new issues over christmas, and from tomorrow, quite a few folks are out
so if that's the case, i don't feel like we need to rush at top speed

[silver sorrel]

Agreed that it's worth getting our next move as right as possible, and it's complex. Thanks for the brain dump!

[keen latch]

(as a fun side note, i was talking about this with tianon who maintains the official images for docker, who also mentioned offhand that the entrypoint feature was probably a mistake as well)

[frail flare]

@keen latch can you please say hi to Tianon, and tell him I have something I really want to show him? 🙂

[dense wraith]

@silver sorrel while working in the magicache tests, I've realized that in v0.15.0 we've broken this use-case which I think it's not possible anymore https://github.com/dagger/dagger.io/blob/cf82780c4f7f4718508e154d608bf3e120c61ceb/magicache/cmd/ci/services.go?plain=1#L159-L172

        WithExec([]string{
            "/usr/local/bin/dagger-engine",
            "--config", "/etc/dagger/engine.toml",
            "--debug",
            "--addr", "unix:///var/run/buildkit/buildkitd.sock",
            "--addr", "tcp://0.0.0.0:8080",
            "--network-name", fmt.Sprintf("dagger%d", engineInstance),
            "--network-cidr", fmt.Sprintf("10.88.%d.0/24", engineInstance),
        }, dagger.ContainerWithExecOpts{
            UseEntrypoint:            false,
            RedirectStdout:           "/var/log/engine/out.log",
            RedirectStderr:           "/var/log/engine/err.log",
            InsecureRootCapabilities: true,
        }).
        AsService()

^ I think we don't have a way to express that anymore (redirection + InsecureRootCapabilities) with the updated Services API

[silver sorrel]

See Paul Dragoonis in General. Same issue, I wonder

[dense wraith]

oh, we can do the InsecureCapabilities thing, but we've lost the ability to do redirection

[silver sorrel]

Oh…

[keen latch]

Oh and then using a cache volume to read it?

Hm, yeah missed that - we can add the redirect APIs into asService pretty easily I think

[silver sorrel]

https://docs.google.com/document/d/1ed5W9Pz7-Av8IWPk-7dKcOwKrNZWZdQ9tqPR9TibrHI/edit?tab=t.0

🙏 https://github.com/dagger/dagger/issues/9190#issuecomment-2558254529 🙏

Proposal 👆

Trying to make this part from the 0.15 blog post true:

We’ve enhanced Dagger’s service execution by ensuring that services start with their default command arguments when none are specified. This simplifies workflows by allowing services to operate as intended without requiring manual configuration for default settings. Services now just work out of the box, saving you time and eliminating unnecessary steps.

We’ve improved the developer experience when using Dagger’s AsService API. Previously, if you wanted your service to run with its image’s default command (CMD and ENTRYPOINT), you had to jump through hoops—like adding empty WithExec steps—just to ensure it did the right thing. This was confusing and slowed you down.

AsService now seamlessly respects an image’s built-in defaults whenever you don’t provide explicit command arguments. Under the hood, this means we no longer rely on fallback logic that varied depending on prior WithExec calls. Instead, you get a straightforward, “just works” behavior that matches what you’d expect from running the same image with docker run. The result: less friction, clearer expectations, and a smoother path to getting your services up and running.

[dense wraith]

Yep, I'm currently doing the redirection via shell but we should definitely add it

[silver sorrel]

can't figure out why

dagger shell -s -c 'container | from python | with-exposed-port 8080 | with-default-args python,-m,http.server,8080 | up'

and

dagger shell -s -c 'container | from python | with-exposed-port 8080 | with-default-args python,-m,http.server,8080 | as-service | up'

work, but with --args doesn't

dagger shell -s -c 'container | from python | with-exposed-port 8080 | as-service --args python,-m,http.server,8080 | up'

Error: input: container.from.withExposedPort.asService.up failed to start host service: start upstream: exited: exit code: 2

SInce it's our example in docs:
https://docs.dagger.io/api/services/

 @function
    def http_service(self) -> dagger.Service:
        """Start and return an HTTP service."""
        return (
            dag.container()
            .from_("python")
            .with_workdir("/srv")
            .with_new_file("index.html", "Hello, world!")
            .with_exposed_port(8080)
            .as_service(args=["python", "-m", "http.server", "8080"])
        )

Seems our dumb-init has no PATH
[dumb-init] [python: No such file or directory

https://v3.dagger.cloud/dagger/traces/071e2b179aa4518c06d7fdcf30139187?span=69fbeae0b1e6839e#69fbeae0b1e6839e:L1

[keen latch]

What is [python 🤔

Looks like a typo maybe?

Ohhh this is a bug in shell I think - helder should have merged a fix for this recently

Should be fixed on main

[silver sorrel]

verified, works as expected in

dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres as-service --args docker-entrypoint.sh,postgres up

also verified that this with-exec version sorta looks like it works, but it just hangs. We should error:

dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres with-exec --args docker-entrypoint.sh,postgres up

[dense wraith]

Another reason to move away from using Entrypoints https://x.com/iximiuz/status/1871929315581161529?t=sNQd8X7edXBVm_z7ZdQpxQ&s=19

Ivan Velichko (@iximiuz) on X

How using an entrypoint script can break the container's graceful termination 🔽

A new Linux & Docker challenge: Can you fix the graceful termination for this containerized Go application?

https://t.co/8KYzQUriq2

Twitter · Dec 25, 2024, 2:41 PM

[dense wraith]

@silver sorrel @keen latch https://github.com/dagger/dagger/pull/9247 is ready for 👀 to address the AsService changes

[dense wraith]

@frail flare @silver sorrel while working in the above PR I left this comment https://github.com/dagger/dagger/pull/9247#discussion_r1907264949 which requires one last final decision before merging. Exchanging ideas with @keen latch about this we're not sure how to proceed. The main concern here is that the use-case I expressed in my comment seems to be a miss from the original proposal here https://github.com/dagger/dagger/issues/8140. Particularly around point # 2.

Quoting my comment, the situation is the following:

// this is my app container that the pipeline builds which later I want to publish to my OCI registry. 
// Let's say this image already has an entrypoint defined.
myApiContainer := dag.Container().From("img-with-entrypoint").WithDefaultArgs("blah") 

//  as part of myApiContainer tests somewhere else in the code I do: 

svc := myApiContainer.AsService()

dag.Container().WithServiceBinding("api",svc)......

^ in this case, if we apply the logic to disable the entrypoint when setting WithDefaultArgs the above scenario ends up being super confusing since if you pull the container from the OCI registry it works, but if I use the one you've built it doesn't. 😬

currently in v0.15.1 this is how it works and we'd like to verify if we should leave this as-is or if we should also revert the decision of point #2 in #8140 about WithDefaultArgs invalidating the Entrypoint when using services. cc @silver sorrel since I'm not sure if you thought about this case while drafting your recent services proposal

[dense wraith]

@frail flare @silver sorrel friendly reminder about this. It's time sensitive for the release next week 🙏

[silver sorrel]

yep, I've been working on this. Thanks @dense wraith

[silver sorrel]

We disable ENTRYPOINT when you set args on asService even with useEntrypoint: true which makes no sense to me (see bottom cases).

Ran from PR:
dagger /mnt $ dagger version
dagger v0.15.2-250108145053-5bc1bd0b3baa (registry.dagger.io/engine:8c0cefb0e8d3f16307e7c93c2a11e9095d501fed) linux/arm64

dagger core container from --address postgres

_type: Container
defaultArgs:
    - postgres
entrypoint:
    - docker-entrypoint.sh
mounts: []
platform: linux/arm64
user: ""
workdir: ""

===========
CASE 1
dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres as-service up

This works as expected. Running ENTRYPOINT+CMD

===========
CASE 2
dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres with-entrypoint --args docker-entrypoint.sh as-service up

! failed to start host service: start upstream: service exited before healthcheck
because the CMD/defaultArgs was not used

===========
CASE 3
dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres with-entrypoint --args docker-entrypoint.sh with-default-args --args postgres as-service up

"root" execution of the PostgreSQL server is not permitted.
The server must be started under an unprivileged user ID to prevent
possible system security compromise.  See the documentation for
more information on how to properly start the server.
! exit code: 1
because ENTRYPOINT was not used, but defaultArgs was used 

===========
CASE 4
dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres with-entrypoint --args docker-entrypoint.sh with-default-args --args postgres,-c,log_min_messages=notice as-service --use-entrypoint up

✔ container: Container! 0.0s
$ .from(address: "postgres"): Container! 0.6s CACHED
✔ .withExposedPort(port: 5432): Container! 0.0s
✔ .withEnvVariable(name: "POSTGRES_PASSWORD", value: "postgres"): Container! 0.0s
✔ .withEntrypoint(args: ["docker-entrypoint.sh"]): Container! 0.0s
✔ .withDefaultArgs(args: ["postgres", "-c", "log_min_messages=notice"]): Container! 0.0s
✔ .asService(useEntrypoint: true): Service! 7.3s
∅ .up: Void 7.0s

WORKS! runs both ENTRYPOINT and defaultArgs (using more than just `postgres` to prove that)

===========
CASE 5 >>fixed by change made below<<
dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres with-entrypoint --args docker-entrypoint.sh as-service --args postgres,-c,log_min_messages=notice --use-entrypoint up

✘ .asService(args: ["postgres", "-c", "log_min_messages=notice"], useEntrypoint: true): Service! 0.4s
"root" execution of the PostgreSQL server is not permitted.
The server must be started under an unprivileged user ID to prevent
possible system security compromise.  See the documentation for
more information on how to properly start the server.
! exit code: 1

FAILS! this "should" work by same logic above, I'd think and use ENTRYPOINT + asService's args ??? Indicates that overridden ENTRYPOINT not being used, even though useEntrypoint is TRUE!

============
CASE 6 >>fixed by change made below<<
dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres as-service --args postgres,-c,log_min_messages=notice --use-entrypoint up

same issue as above, doesn't run default image ENTRYPOINT either when asService's args are set

Marcos called out CASE 3, which I acknowledge could confuse some folks, but I'm more concerned about CASES 5 and 6 which do not respect useEntrypoint: true when asService args are used.

[dense wraith]

you're right, case 5 and 6 was a miss. I'll fix that really quick 🙏

we clearly missed a bunch of tests in the initial implementation for v0.15.1

[silver sorrel]

I suspect it's one of those things we we "just flipped one bit" but the relationships are more complex

[dense wraith]

done, just pushed the fix with the test @silver sorrel

https://github.com/dagger/dagger/pull/9247/commits/acd885ac6e06ff14a746a5e9e2a7cc25fc70ed4b

[silver sorrel]

Feels to me like things "should" be:
ENTRYPOINT is run in these three separate cases:

• you use useEntrypoint: true then it will run ENTRYPOINT + CMD/DefaultArgs
• you set withEntrypoint args then it will run ENTRYPOINT + CMD/DefaultArgs <<< note: this is not current behavior. withEntrypoint blots out original CMD/defaultArgs unless you use keep-default-args
• you don't touch withDefaultArgs or asService's args (docker compat https://github.com/dagger/dagger/issues/9305)

Then with CMD/defaultArgs it feels like it should run in this case:

• you set withDefaultArgs args (whether it also runs entrypoint first is depedent on first case above) and don't also set asService args (because these override)

If asService's args are set, they take precedence over CMD/defaultArgs

kinda like

asService(args) > CMD/defaultArgs > (ENTRYPOINT)

where those "greater-than" signs (>) signify precedence and ENTRYPOINT is easily muted by the other two unless set (withEntrypoint) or explicitly used (useEntrypoint: true)

[dense wraith]

gotcha.. I think that's the state of the current PR

[silver sorrel]

testing now. Note that this PR doesn't have the asService().up() == up() feature in it, but I'd expect up() alone to act the same. Seems like up will have args and useEntrypoint etc
https://github.com/dagger/dagger/pull/9231/files#diff-74a9c9e133ee9e1936ab0992a17621f577bf69d08455e7610afef8ce50cc033fR70-R74

[silver sorrel]

In current PR functionality,
If you specify withEntrypoint then it WILL execute the entrypoint even without useEntrypoint: true, BUT it ignores the CMD/defaultArgs (whicih is postgres in the image):

CASE 7

dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres with-entrypoint --args docker-entrypoint.sh as-service up

You can tell this, because if you shove the contents of the original ENTRYPOINT and CMD all into the entrypoint, then it works

dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres with-entrypoint --args docker-entrypoint.sh,postgres as-service up

just like if you shove it all into defaultArgs

dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres with-default-args --args docker-entrypoint.sh,postgres as-service up

or asService's args:

dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres as-service --args docker-entrypoint.sh,postgres up

oh, I see case 3 again,
If I try to "fix" CASE 7 above by adding a withDefaultArgs, then it blows away the withEntrypoint and fails since I'm back at having no non-root user prepared by the entrypoint script

dagger core container from --address postgres with-exposed-
port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres with-entrypoi
nt --args docker-entrypoint.sh with-default-args --args postgres,-c,log_min_messages=notic
e as-service up

Whereas, I would assume that if I specified BOTH withEntrypoint AND withDefaultArgs then I'd get both of them to run as a set.

but to get that behavior, you need to add useEntrypoint: true I suppose...which I wasn't mad at before...but I might be getting there because it's a bit frustrating.

dagger core container from --address postgres with-exposed-
port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres with-entrypoi
nt --args docker-entrypoint.sh with-default-args --args postgres,-c,log_min_messages=notic
e as-service --use-entrypoint up

yep, that's what it needed ✅

But it is consistent with the spec according to --help which says pre-pend to args not to defaultArgs...

 --use-entrypoint                    If the container has an entrypoint,
                                          prepend it to the args.

And I hadn't looked carefully at withEntrypoint:

ARGUMENTS
      --args strings        Entrypoint to use for future executions (e.g., ["go",
                            "run"]). [required]
      --keep-default-args   Don't remove the default arguments when setting the
                            entrypoint.

So, if you use withEntrypoint, you blot out the original CMD/defaultArgs unless you keep them.

If you use withDefaultArgs you blot out the original ENTRYPOINT unless you useEntrypoint: true on the asService or up

That's why, even if you sepcify both of them, with withEntrypoint AND withDefaultArgs you still NEED useEntrypoint: true to use the things you specified. That seems okay in a world where we advise folks to move things to asService/up's args or defaultArgs...though it does spoil the intent of having a fixed ENTRYPOINT that out of the box could take different arguments specified by the developer at runtime...but yeah, I'd say we're in a good spot for our ethos where you CAN have that behavior as long as you explicitly say useEntrypoint: true. 👍

======

Of course it's about intent. Is my intent to RUN this container with this entrypoint and CMD/defaultArgs or to PUBLISH it with those set in the image, or BOTH.

[silver sorrel]

checked Justin's PR by cherry-picking his two commits on top of the PR from Marcos. No conflicts. Works as expected. Consistent with asService().Up

[dense wraith]

merged! thx everyone for all the chicken and/or tofu for the vegans! :tofu: 🐤

also merged https://github.com/dagger/dagger/pull/9231 🙌

[silver sorrel]

https://github.com/dagger/dagger/issues/9363

[dense wraith]

replied

[keen latch]

cheers y'all ❤️