#Explainers for Dagger vs Docker

1 messages · Page 1 of 1 (latest)

waxen oyster
#

Context: our devs keep using Dagger as a scapegoat, my understanding is that "modern" Docker will also spin up BuildKit-in-a-container (BKiaC), and that this is nearly the same setup. Our devs have not seen Docker do this. Do you have any explainers that show

  1. Dagger vs Docker in this respect
  2. Docker before/after the transition to BKiaC

Or are you aware of any docs et al out there? Being asked to come up with some things by the CTO

maiden vortex
waxen oyster
#

there isn't one, half the time it's not Dagger, but today, it's the same DNS in a container not working I asked about last time.

#

Like, I explained and showed them that it is reproducible in Docker only then. They seem to have forgotten that and claim it works in Docker, but are not following the repro steps provided facepalm

#

To help with this, the CTO has asked me to collect some educational materials

#

More about the internals than how to use things

fringe rose
#

@waxen oyster is it my impression or the devs that you're working with have 0 interaction with their build tools at all? It seems to me like from their perspective, dagger, docker, nix, bazel / whatever is doesn't really matter that much and they only see those tools as the implementation details of whatever build system they have?

waxen oyster
waxen oyster
maiden vortex
#

So it seems we're getting the worst of both worlds - your team isn't using Dagger the way we've designed it to be used. But Dagger still gets blamed when the custom software stack has issues.

#

A good summary of why we don't like being wrapped

#

Responsibility without control

waxen oyster
#

I don't think it has to do with the wrapping, it's more a human bias thing to blame the thing they don't understand first

#

The RCA is rarely Dagger

#

like, they have k8s running on the same vm on baremetal, messing with IP tables and the whole shebang, it's unsurprising to me we have issue like this

#

And we did find a way to resolve this specific issue, they have to nuke /var/lib/docker and clean up IP tables

#

and to be fair (?), the complaining is largely driven by one individual, occasionally it rubs off on others

#

but to circle back, the goal is to give them something that shows Dagger and Docker work essentially the same with the BKiaC, eg. that time spent moving files from host into a container, that's not going to go away if we go back to Docker

#

They remember a world where Docker wasn't doing this, but Docker has changed since

maiden vortex
#

Honestly I would fire up Codex 5.3, aim it at the codebase, and ask all the questions

waxen oyster
#

which code base?

maiden vortex
#

if it's a question about dagger: the dagger codebase
if it's a question about docker: the docker codebase (plus abundant open knowledge about its internals)

fringe rose
# waxen oyster and to be fair (?), the complaining is largely driven by one individual, occasio...

I can relate to this.

Have you had the change to get together with this person, explain why you have chosen to use Dagger and how it helps to build and scale their build system in a different way than docker (or any other build tool)?

A big part of blaming the unknown and new things in my experience is also a result of people's insecurities of being left over and feeling that they don't have a voice

waxen oyster
#

Oh man, we went through all of that and have tons of docs about the rationale, including having this person in on the editorial process for that

#

Did a full hour presentation, outlining all the pains they were complaining about, maybe they just need to watch that again 🤣

#

(pre-dagger pains)

#

part of this is that fading of memory, forgetting how much worse it was beforehand

#

My RCA for the repeated issues is that we have two teams using three distinct processes that all manipulate their dev-vms.

fringe rose
# waxen oyster Did a full hour presentation, outlining all the pains they were complaining abou...

lol. Sometimes having a pair-programming session with these individuals by implementing a specific request or showing them some new feature of the tools helps to calm the waters.

To give you a personal pointer to your question specifically aimed for a C-Level audience I'd maybe frame this conversation towards the current AI trend of devs producing more code while dev tools and CI pipelines still being relegated. Regardless of implementation details like BKiaC and all those things, in my opinion the strongest argument of dagger is maintainability, repeatability and portability.

Having a Docker based build system only gives you a small part of those properties. That's the main reason why we've built Dagger and there's a lot of material we've produced over the year to explain this vision

waxen oyster
#

This is not a C-level problem, the CTO gets it. It's an ENG lead, and I think undue extrapolation

#

At this point, it seems like they want to blame Dagger for everything without actually looking or thinking about it

#

Jenkins used to be the target of ire, but we've made that simple and solid

fringe rose
#

In that case maybe giving a "Dagger internals" presentation to show them how Dagger effectively works, how to troubleshoot it and the fact that the execution runtime architecture is similar to Docker's might help? I don't think we have any specific docs on that but there's plenty of material in the server + the #1075928318802657340 forum.

#

FWIW we're very close to remove buildkit from Dagger's core since it has been a constant source of issues which are super hard to address and pinpoint because of the complexity and friction on Docker's codebase. Even the buildkit maintainers that we have in the team agreed that it is the best thing for the project.

There's several PR's in the dagger/dagger repo documenting all this work

waxen oyster
waxen oyster
#

fyi, looking more like host level NetworkManager is the culprit, and one may need to make sure some/all of the docker related ifaces are not managed by it?

#

and/or tailscale

#

hopefully some of this can become an entry in "common networking pitfalls/setup" with Dagger

#

if we can actually RCA it this time

waxen oyster
#

Some tailscaled , all of those lxc...., those are dagger created/mangaged, service binding releated? (scroll right on the first line to see)

📎 tailscaled-logs.txt
#

I believe this is someone sharing some of these logs in an AI session, which captures a likely sequence of events

image_8.png
#

It looks like Dagger builds cause iface churn, tailscaled and NetworkManager bork themselves, most likely resolution is to config tailscaled better

fringe rose
waxen oyster
#

Actually, the config is NetworkManager

waxen oyster
#

The lxc... interfaces are Dagger named though, yea? (eg. lxc03d96a9277f7)

#

(that's my memory, from seeing what I think looks like this in Dagger logs for service bindings)

#

we're also running containerd/nerdctl/k8s-on-metal, so it could be coming from those, but the logs align with Dagger builds (our devs are not doing k8s related things for several weeks now)

fringe rose
fringe rose
waxen oyster