Nixpkgs

[prime osprey]

Hi Mark, 1) it’s amazing that you’re maintaining a distro package of dagger 🙂 Unfortunately our trademark policy doesn’t allow redistributing unofficial builds under the Dagger trademark. In other words, if it wasn’t built by Dagger, it can’t be called Dagger. This policy is informed by past experiences with Linux distributions. We’re open to discussing this policy, or perhaps changing it in the future, but at the moment that is the policy.

This is our first case of applying that part of the trademark policy , that I know of. So I’m not sure how to go about it to be honest. Does nixpkgs support bundling our official binary releases? That would be compliant.

Sorry to be a downer..

[lusty rose]

Well, nixpkgs does not always provide its own build. It's somewhat similar to how Arch linux works: it builds the software from source. It may provide an optional, intermediary cache though of the build results.

In other words: the normal operation is that when you run nix-env -i dagger, Nix will run go build for you. If you have the nixpkgs "cache" (called substituters in Nix terminology) enabled, you may get that version though.

Technically, it's possible to use prebuilt binaries as well, but that may cause problems if Dagger uses any cgo code.

Does this violate your trademark policy? :/

Also, can you point to that part of your trademark policy?

If we have to switch to the prebuilt binaries, we will have to provide a justification and an explanation for people looking at it in the future.

(Also, switching to prebuilt binaries might require marking the package as "unfree" in nixpkgs, but I'm not sure about that. Unfree is basically a safeguard for people who don't want any non open source software on their computers, it's a true/false flag that prevents installing unfree software)

I think we should start a discussion in the Nix community (Discord or nixpkgs repo) about this. I'm happy to open an issue, but it would be great if you could chime in as well @prime osprey

[deft jolt]

This is similar to homebrew, right? Is homebrew also infringing on this policy or does it do something different to stay within the policy?
https://github.com/Homebrew/homebrew-core/blob/53261b4c94e837978ac75c60afd0a8691d85b222/Formula/d/dagger.rb

Both nixpkgs and brew are building at install time and in a very similar manner, curious how they are different.

For reference, the dagger install script through nixpkgs
https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/development/tools/continuous-integration/dagger/default.nix

[lusty rose]

Both nixpkgs and brew are building at install time

That isn't necessarily true. There is a considerable chance you'll get it from cache.

[deft jolt]

Right, my point is not factoring in cache, it'll build on the local machine, right?

[lusty rose]

Yep

[deft jolt]

Also as one last example, it looks like the Arch distribution also builds on the machine at install
https://gitlab.archlinux.org/archlinux/packaging/packages/dagger/-/blob/4e3dd92f5a51c609807a3af2629d8a42ee792bae/PKGBUILD

Is this more a concern of who is maintaining the distribution maybe?

[lusty rose]

But since we can't say for sure where you get it from, and the cache potentially infringes the trademark policy, the safer path is distributing the prebuilt binaries (unfortunately 😄 )

I guess it's about faulty/old software and the lack of control over how distros distribute packages.

[deft jolt]

Hmm, interesting. Will definitely be following the discussion. I don't know anything about trademark so really want to know the differences and nuances to why one distribution might be allowed but others aren't, especially between nixpkgs, homebrew, and Arch, and how a cache factors into that.

[deft jolt]

This cache is really tricky though, because for me, I just use my local computer's cache, I don't use anything like cachix. And that's the default for nix out of the box. You can probably cache other package distros as well.

[lusty rose]

Chances are you use cache.nixos.org, aren't you?

[deft jolt]

Maybe? When I do a full garbage collection and dump generations I'm rebuilding everything, though

[lusty rose]

I guess building from source is fair play, but I'm not sure. If it isn't, chances are Homebrew is also violating the trademark policy 😄

[prime osprey]

I believe homebrew distributes the binary?

if they rebuild, then yes they are also in violation

[deft jolt]

From the sources I linked above, all three, arch, homebrew, and nix, all build from source

[prime osprey]

then all three would be in violation

I guess I’ll need to kickoff some sort of process to address all three… (sigh)

[deft jolt]

But if they called the package something different they wouldn't be in violation anymore?

I don't particularly care if I get my dagger binary from pre-built or by source, but I know there are some people who want everything from source.

[prime osprey]

it’s important that we do enforce it consistently, even with “friendly” redistributors, otherwise it may weaken our ability to enforce it later with hostile ones

[prime osprey]

correct. As long as it doesn’t hurt the trademark or cause confusion with it

[deft jolt]

This sounds similar to vscode and vscodium, maybe?

[prime osprey]

more like firefox/iceweasel

but even better is to simply not redistribute custom builds at all

[deft jolt]

I'm curious what you would consider enough of a change in name. Would daggerx or dagger-source be enough change, or would you rather it not include dagger/be a bit more cleaver

I'm playing devils advocate for the people who want everything from source

I think that question about naming is my last

[prime osprey]

if you can find it easily, it’s not enough

we distribute a source release, of course you can build it yourself from source

you don’t need a distro for that

what is not allowed is redistributing your build to others and calling it Dagger

[deft jolt]

And it's still redistribution if the package manager, at install time, is pulling down the source and running go build on your machine?

(not factoring in caches)

[prime osprey]

yes, if they claim the result will be dagger

by the way that will not give you a working dagger engine

[deft jolt]

I see, so install process must be pulling down the release from an official dagger repository like the dagger github

[prime osprey]

the build & release process is much more complex

distros will screw it up, then users will remember dagger doesn’t work well

[deft jolt]

Ooh, interesting view. Hadn't thought of that

[deft jolt]

Curious about this. I've only used dagger through nixpkgs and homebrew on my local machine and haven't hit any issues with the dagger engine, so not sure what is not supposed to work.

[prime osprey]

the CLI is half the engine; it auto-downloads the other half and runs it as a worker container

my guess is the package only builds the cli?

[deft jolt]

  def install
    ldflags = %W[
      -s -w
      -X github.com/dagger/dagger/engine.Version=v#{version}
    ]
    system "go", "build", *std_go_args(ldflags: ldflags), "./cmd/dagger"

    generate_completions_from_executable(bin/"dagger", "completion")
  end

Specifically from homebrew

[prime osprey]

If I knew for sure that architecture would never change, I would almost consider relaxing the policy for the CLI since it’s so trivial, but that may change in the future.

We’re considering a build of the CLI that embeds the worker container, for airgapped environments

[deft jolt]

I do understand what you mean though, all three of these distros re-implement the build process for dagger

[rocky zinc]

But the Homebrew can distribute a formulae from software owner. That would not violate the trademark, right?

[prime osprey]

Yes, I believe Dagger releases official builds on homebrew already

[deft jolt]

Just employ @lusty rose then nixpkgs will be solved 🙂 /s

[prime osprey]

ha 😁 actually now I wonder what will happen to our brew package if/when we embed the worker container into the CLI..

[lusty rose]

LOL

@prime osprey I opened an issue: https://github.com/NixOS/nixpkgs/issues/260848

[prime osprey]

I hope I didn’t just start a shitstorm…

[lusty rose]

No idea 🤷‍♂️

I’m doing my best here.

[prime osprey]

Thank you I appreciate it @lusty rose

The last thing I want is to create headaches for you when you're investing so much energy into the Dagger community, which I greatly appreciate

[lusty rose]

Well, it’s not like I’m gonna be held accountable or anything.
The worst case scenario is that Dagger gets removed from nixpkgs which would be sad.

[prime osprey]

@lusty rose if we do reach that worst case, is there a way to have an upstream-maintained nixpkg? In other words, could we ship our own nixpkg package, built our way, so that nix users still have an easy way to install dagger?

This is what we do with homebrew, and what Docker does for eg. apt, rpm packages etc

(we might do apt/rpm in the future too)

[lusty rose]

I don’t actually have the power to make that call though. Other than being tagged for reviews, I don’t have much power as a “maintainer”

[lusty rose]

Yes, through Flakes. If we get to that, I can help set that up.

It might make sense to do regardless. Getting updates to nixpkgs is always slower

[prime osprey]

This whole discussion gave me an idea: with Zenith we can encode our official build process as module. That module in turn can be used by anyone who wishes to build Dagger Engine from source correctly. In theory, we could make an exception to the trademark rule where unmodified source code, built with the unmodified official dagger build module, is allowed to be called Dagger 🙂

Of course that would require having a build-time dependency on a bootstrap version of the dagger binary. This is technically sound, but Distros tend to have strong NIH syndrome when it comes to their build process and policy, so they may not be willing to do that.

Would be cool though 🙂

[lusty rose]

That may actually not be that big of a deal. I believe that's how Go is bootstrapped for example.

The build time dependency on Docker is an entirely different case though. I don't even know if it's possible to do with Nix.

Given both Nix and Homebrew run on both Linux and mac and both provide different solutions for running these containerization solutions, I think it would certainly be challenging.

[prime osprey]

It's possible to run dagger without docker installed. You do need a way to run an OCI container since the core of the engine is only available as an OCI image

That's a good point though

[lusty rose]

https://github.com/NixOS/nixpkgs/issues/260848#issuecomment-1763033934

@prime osprey this may be a bigger can of worms than you realized 😄

[deft jolt]

The decision by nixpkgs to remove the package entirely instead of packaging the prebuilt binary is a little sad to see, especially since there seems to be other instances of pre-built binary packages like firefox-bin.

[deft jolt]

I wonder if they would accept a PR for a dagger-bin package

[prime osprey]

that’s ok, we can try again later. It’s easier to fix in this direction.

[lusty rose]

I'm guessing the reason for removal isn't the binary distribution, it's the perceived "legal minefield".

[lusty rose]

FYI the python SDK weren't accepted into Nix either: https://github.com/NixOS/nixpkgs/pull/236916

[prime osprey]

“legal minefield”: when you can’t do whatever you want with other people’s software… 🙄

[lusty rose]

It's not even the software, but the name.

I have to be honest: I find this whole situation incredibly weird. I mean, I've been working on Open Source in my whole life and never once had to think about trademark.

But I totally understand your position.

[prime osprey]

It was the same for me when we launched Docker. But Red Hat and others taught me a brutal lesson in the importance of protecting your trademark.

Their entire business model is built on the premise that they can use the names of all open source software they package, but nobody can use theirs.

[lusty rose]

Started working on this: https://github.com/sagikazarmark/dagger-flake

@prime osprey you said you would be interested in hosting this?

[prime osprey]

Paging our resident nix expert @coral scroll 🙂 And @tulip tulip for infra perspective. wdyt guys?

[lusty rose]

There is no infra need, it’s just a repo

[coral scroll]

This looks fine, we'll just need to integrate it into our release process and likely merge it into our monorepo.

What I came in here paranoid about was adding a from-source flake.nix to dagger/dagger, which IME has been a source of constant toil with Bass because maintaining a sha256 of your go.mod dependencies is a huge pain in the ass, to the point that I've regretted accepting the PR that added it. (This bit: https://github.com/vito/bass/blob/main/default.nix#L8)

But as long as it's just a post-release step that bundles our official binary distribution, SGTM.

[lusty rose]

That's precisely why I generally like keeping the flake in a separate repo, so there is no circular dependency between the tag and the source code.

[prime osprey]

Thank you for the contribution @lusty rose 🙏

[tulip tulip]

Nice one @lusty rose !

Ideally, we would have this in the OSS repository - in the flake directory? - and release it together with everything else, part of https://github.com/dagger/dagger/blob/main/RELEASING.md

Interested to talk more about this option?

[lusty rose]

Ideally in the repo root.

There are two schools of thought:

a) Keep it in the repo root.
b) Keep it in a separate repo.

Happy to talk more about these.

[tulip tulip]

OK! Curious to dig into both. Slight preference of a). DM-ing to find a time.

[lusty rose]

The primary reason to consider option b) is that people often try referencing a specific tag in a repo. For example in this case, people could reference github:dagger/dagger/v0.9.0....but it would actually download an older version, because in order to update a package in the repo, first you need the tag (and in Dagger's case, the published binaries to GitHub).

And this confuses newcomers (but it's also partly specific to Dagger, due to not building Dagger from source). Nix is primarily a rolling release kind of package manager, so you would basically always reference main/master that will always be updated to the latest version at some point.

We can discuss more in private, but if the above is acceptable for you, I see no reason why we couldn't go with option A.

[tulip tulip]

What are your thoughts on github.com:dagger/dagger/nix-flake/v0.9.0 ? We take the same approach for everything else that depends on the CLI & Engine.

The main reason for keeping everything together, in a single repo, is that everyone is looking at the same stream of issues & PRs. We all benefit from the same tooling, infrastructure, release cycle, etc. And while we don't take the same approach with all tooling that is built on Dagger, in this particular case, I would enjoy having you as a fellow contributor in the dagger repository.

Let's talk more tomorrow.

[lusty rose]

To be honest, I would also prefer keeping it in the same repo.

At the same time, I've seen people getting confused about the above several times. Not seasoned Nix users though.

Having to use the prebuilt binaries also complicates the process a bit.

It's just something to keep in mind when making that decision.

If the flake is in a separate repo, there is no need to tag the flake repo. Nix always locks to commit hash anyway.

[prime osprey]

Could our flake build from source using dagger as a build dependency? Would that solve some of these ux problems?

[lusty rose]

Maybe....but I'm not smart enough to put that together 😄

[prime osprey]

do you mean the dagger part, or nix part, or both? Asking because if the nix part is trivial, we could supply a formally supported dagger module to build dagger, invokable from the CLI

(sorry if I’m complicating things, just looking for the best possible experience for Nix & Dagger users)

[lusty rose]

Kinda both. THe part where you call dagger from Nix to build dagger.

I'm the one who probably overcomplicates things a bit. We should probably just start doing it and iterate based on user feedback. 🙂

[prime osprey]

ah I see. I assumed build dependencies on various domain-specific binaries were common , but I’m a nix noob

[lusty rose]

Everyone except a few dozen people on Earth qualifies as Nix noob 😄 I'm a noob as well

[prime osprey]

basically why Dagger exists 😂

[lusty rose]

Nice! 😄

[south vale]

Well, that's a bummer.

[prime osprey]

see #nix , thanks to @lusty rose and @tulip tulip we have a nix flake coming up

[deft jolt]

Looks like it's going into NUR as well? Can't wait for that.

[south vale]

Yeah, I saw that. Appreciated. Though, honestly, it might just be the final push I need to move my laptop away from NixOS and my homelab to something else. I've used Nix for about a year now and it costs me more than it brings to the table.

Well, I'll probably keep using devenv because that's a pretty cool project.

[lusty rose]

Haven't opened the NUR PR yet, but yeah, that's the plan.

Flake is actually up and running: https://github.com/dagger/nix

However, the real test will come with the next release to see if the automation works.

[tulip tulip]

Thank you for helping with this @lusty rose 💪

FWIW, we will need to bump to GoReleaser 1.22.0 for everything to work as intended: https://github.com/goreleaser/goreleaser/pull/4389#issuecomment-1785646378 cc @split fjord