#buildpacks

1 messages · Page 1 of 1 (latest)

hasty glacier
#

continuing in a thread 🙂

#

so, the integration with buildpacks is possible but probably not in the way you think

#

could you share a link to the buildpack sdk please, so I can make sure?

azure sigil
#

Here's a quick example:

    packClient, err := pack.NewClient(pack.WithLogger(NewLogger(io.Discard)))
    if err != nil {
        l.Errorf("Could not create pack client: %s", err.Error())
        os.Exit(1)
    }

    opts := pack.BuildOptions{
        Image:        "registry.mydomain.app/" + accountID + "/" + appName + ":" + revision,
        Builder:      "heroku/buildpacks:20",
        AppPath:      appPath,
        Publish:      true,
        ClearCache:   false,
        TrustBuilder: func(_ string) bool { return true },
    }


    // set context with timeout
    err = packClient.Build(context.Background(), opts)
    if err != nil {
        l.Error(err.Error())
        os.Exit(1)
    }
#

I'm not sure if it would be an addition of Dagger client in there or if pack can somehow directly connect to dagger engine via docker connection

hasty glacier
#

Unfortunately not. These are both SDKs so you need to write code against their respective API to use them. For them to work together you would have to either 1) use a plugin/extension mechanism in one to integrate with the other, or 2) modify one to use the other’s API

azure sigil
#

What do you recommend?

hasty glacier
#

You can definitely implement buildpack support on top of the Dagger Go SDK. Possibly by modifying the pack tool to use Dagger, or just borrowing parts of its code into a new, simpler tool.

It all depends on what your goal is, and what is a reasonable effort to reach it.

#

What are you trying to achieve in the end?

azure sigil
#

I'd like to use Pack client to connect to dagger engine to build images

#

After that, I could use dagger SDK for rest of the operations

hasty glacier
#

I see. How do you run the pack client today? From CI, a shell script, makefile, other?

azure sigil
#

It's just a binary which is invoked whenever someone does a git push on their repo

#

It does nothing other than build from git repo and push out the image but looking at dagger Go sdk, I felt it could do a lot more.

#

Feels intuitive to add dagger client to pack SDK. I'm not sure if that's something pack community will accept but that's okay.

#

Once pack is connected to dagger engine, I'm not sure how it can instruct for builds.

hasty glacier
#

That would be option 2: modify pack to use the Dagger API. It's definitely possible, but it would probably be a lot of work. Not the kind of thing you'd do quickly this week to have a working project the next week.

hasty glacier
azure sigil
#

Sounds worth trying 😄

hasty glacier
#

Another approach would be to configure pack to not orchestrate docker

#

or use another buildpack tool that does not require docker

#

then that tool can just be run inside a container, and orchestrated by Dagger

azure sigil
hasty glacier
#

This is how I would do it personally 👆

hasty glacier
azure sigil
#

Cool

hasty glacier
#

There are other options too, shortcuts for compatibility

#

If you're just looking to get compatibility with what you have, then build new features directly on Dagger, then you could simply run a Docker engine on the side, and run the pack CLI in Dagger, configured to send instructions to that external Docker engine.

#

In that case the integration would be less good with Dagger: the buildpack build itself would appear to Dagger as just the execution of the client. Dagger wouldn't know anything about the rest of the buildpack process. That stuff would not be magically cached for example.

It only makes sense if you just need to get that pack thing out of the way, to focus on a more interesting part.

If integrating pack is the main part of using Dagger for you, this option doesn't make sense.

#

Another option that is not yet available, but could be interesting in the future: in theory we could develop a Docker engine API proxy, backed by Dagger. So you could "fool" a regular docker client into thinking it's talking to a regular docker engine, when in fact it's talking to a facade to Dagger.

This proxy would never have 100% compatibility (an explicit non-goal), but it could support enough potentially to fool simple tools like pack that only use a fraction of the docker API

azure sigil
#

Yeah, integrating pack with dagger is important. Really want to avoid running the docker engine and just rely on dagger-buildkit

hasty glacier
#

yet another option: maybe wrap podman inside a container? I keep hearing that podman is "daemonless". So I guess that means you can run a podman container inside a dagger container? In that scenario you would do dagger -> pack -> podman

azure sigil
#

Oh, docker engine proxy sounds nice.

hasty glacier
#

yes but again: does not exist today 🙂

azure sigil
#

Not really in a rush tbh.

hasty glacier
#

honestly I bet someone with a little time on their hand, and equipped with a reference implementation of the buildpack spec, could reimplement pack on top of dagger in less than a week

#

with 10x less code too probably

#

kind of makes me want to try...

azure sigil
#

I am more than happy to contribute to make this happen

#

I dont know either side well enough to start though, haha

hasty glacier
#

Ha ha let's try to figure that out 🙂

#

So, here's what we have so far:

GitHub
GitHub - buildpacks/pack: CLI for building apps using Cloud Native ...

CLI for building apps using Cloud Native Buildpacks - GitHub - buildpacks/pack: CLI for building apps using Cloud Native Buildpacks

GitHub
GitHub - tonistiigi/buildkit-pack: buildkit frontend for buildpacks

buildkit frontend for buildpacks. Contribute to tonistiigi/buildkit-pack development by creating an account on GitHub.

GitHub
GitHub - buildpacks/lifecycle: Reference implementation of the Clou...

Reference implementation of the Cloud Native Buildpacks lifecycle - GitHub - buildpacks/lifecycle: Reference implementation of the Cloud Native Buildpacks lifecycle

azure sigil
#

Feels like the docker engine proxy could be beneficial to other such projects too. Whenever someone looks at Dagger and wants to substitute it with Docker engine, things would work out of the box(hopefully all of them)

hasty glacier
#

It definitely would not be all of the things though.. The Docker engine API has a lot of compatibility bagage

#

But you could probably get the 10% of features used by 90% of people

#

we already support docker build out of the box 🙂

#

as well as the core of docker run (not fancy stuff like volumes, network etc)

azure sigil
#

That's cool

hasty glacier
#

my guess is several months before a docker engine proxy makes any sense to create or use

#

and even then it will be kind of a stopgap. Something you use when you don't have anything more native available

azure sigil
#

Yeah, that's true..

hasty glacier
#

That buildpack stuff is a beast. Looks quite bloated and complex

#

Probably 80% of the work is just understanding all the complexity. Then the remaining 20% is using the parts of the code we actually need once identified, and writing the Dagger part

azure sigil
#

I don't connect how buildpacks/lifecycle and dagger can integrate...

hasty glacier
hasty glacier
#

it's initially very simple code that got wrapped in complicated words

#

Each of these scripts can run in a container - something which Dagger can do easily

#

Looking at the spec and the pack codebase.. It has turned into quite a beast of complexity

#

It's probably most realistic to modify pack directly to talk to Dagger, rather than start from scratch

azure sigil
#

As in have pack execute those scripts in Dagger, right?

hasty glacier
#

Yes, replace all calls to the docker client to equivalent calls to the dagger client

#

(probably easier than a proxy)

#

how much easier, not sure

#

I'm looking at the source code now, my curiosity is piqued 🙂

azure sigil
#

Haha, me too.

hasty glacier
#

No idea

azure sigil
#

I'll have a closer look over the weekend

hasty glacier
#

Huh, yes that seems like it could be if not 100% of the work, then a solid chunk of it at least

#

cc @tepid furnace @wild hawk @thin atlas this could be an interesting "stress test" of API features 🙂 In addition to CUE SDK 0.3, and Bass

azure sigil
#

Thank you for having this chat and creating something amazing 🙂

hasty glacier
#

And thanks for being patient with us! Happy to chat about this any time, and of course work on the code too!

hasty glacier
hasty glacier
#

Hum, it appears that this 👆is for implementing a buildpack. But we want to implement a buildpack runner (like pack). So not as useful as I initially thought.

azure sigil
#

They were considering the buildkit frontend route. It's possible Buildkit frontend might offer a lot more but dagger client can be a drop-in replacement for Docker client

#

Just noticed in the end that there's a dagger issue linking this one. 😄
Kinda glad that there are other people in both communities that want this

gentle path
#

This takes me back 😄 I was looking into this 1yr ago. It was the first thing I asked when I came here (#🔭universe message) and it was my first conversation with @hasty glacier too 😁. It was also my first GitHub issue on the repo.

#

The problem was exporting the result of the build into the DAG so it can be reused in other steps (as with a Dockerfile, i.e., Container().Build(...)). I gave up on using a buildkit frontend because of performance reasons and not supporting all features from CNBP.

You can see in https://github.com/dagger/dagger/issues/1053#issuecomment-972770824 that my conclusion was to use the lifecycle directly:

Use lifecycle directly. Run inside a builder image (which includes everything you need), without having to bind the docker socket (unless you want to build to the local daemon). Also, since it's the lifecycle that does the exporting, we can explore how the output can be integrated with dagger's DAG.

Dagger would become the platform then.

In the next comment there's further explanation:

Another way of describing this is: the lifecycle is comparable to kaniko or other unprivileged image building tools. The pack CLI is glue code that makes it easy to use lifecycle with the Docker daemon. We could expand the functionality of the pack CLI so that it acts as glue code for other container runtimes, but that glue code is only necessary when containers are not natively accessible already. [...]

So Pack is the glue, but we can use Dagger as the glue instead.

GitHub
Support Buildpacks.io · Issue #1053 · dagger/dagger

It would be great if Dagger could support building images with Cloud Native Buildpacks. More specifically, I'd like to integrate with Pack, which is both a Command Line Interface and a Go l...

azure sigil
#

So are you suggesting we make our own builder and run it with Dagger and ditch pack? If so, problem is that we'll have to add a lot of features to the builder image then. Currently, Pakteto and heroku builders have a lot of functionality for multiple languages built over the years which will be hard to port over.

gentle path
#

Lifecycle: The lifecycle orchestrates buildpack execution, then assembles the resulting artifacts into a final app image.
Platform: A platform uses a lifecycle, buildpacks (packaged in a builder), and application source code to produce an OCI image.

#

Pack is a Platform. But Dagger can be a platform as well. You already have all the components.

#

Final Gitlab CI pipeline:

image: paketobuildpacks/builder

stages:
  - build
 
# We somehow need to access GitLab Container Registry with the Paketo lifecycle
# So we simply create ~/.docker/config.json as stated in https://stackoverflow.com/a/41710291/4964553
before_script:
  - mkdir ~/.docker
  - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_JOB_TOKEN\"}}}" >> ~/.docker/config.json

build-image:
  stage: build
  script:
    - /cnb/lifecycle/creator -app=. $CI_REGISTRY_IMAGE:latest
#

The lifecycle is a binary that's available on the builder image.

#

Pack calls it.

azure sigil
#

Thanks for sharing the blog post, that was informative! 🙂

#

I understand now

gentle path
#

Sure! I didn't attempt it but I'm interested in how this would work with Dagger 🙂

azure sigil
#

Solomon mentioned earlier in the thread today that Dagger supports docker runbut doesn't support volumes

#

I think we can get away without mounting volumes though but looks like we'll really need exec. I am not sure if dagger supports that

#

Taking a look at the SDK now...

gentle path
#

We can mount volumes but they're restricted to workdir which may not be an issue for you. Also, these volumes aren't bidirectional, they're mounted read-only so if you want to write to the host you need another call. Writing something to a mounted volume won't write it back to host automatically.

azure sigil
#

There's an Exec method but I'm not sure it'll work on running containers. In the example, it looks like it's running the Dockerfile cmomands

gentle path
#

If you mean the Exec() in Container, it allows running any command inside said container.

azure sigil
#

Yeah, you're right. I'll try to port that gitlab example with dagger-go-sdk tomorrow

gentle path
azure sigil
#

Is there a way to debug the container we're working on?

Screenshot_2022-10-26_at_8.02.06_PM.png
#

I couldn't wait, haha.

gentle path
azure sigil
gentle path
#

You need workarounds for now.

azure sigil
#

Oh, that's amazing. I did feel like debugging was going to be a huge pain point, it always was with Dockerfile. Glad to see it's already worked on

gentle path
#

Yeah @thin atlas did an awesome demo where he attaches the terminal to a container in the middle of the pipeline and you can run around that container executing commands, like a breakpoint. Very powerful.

azure sigil
#

Yeah, I did expect it to fail as I didn't initalise the docker.json file but I don't think this is due to that. It looks something from qemu. Is there a way we can do equivalent of docker build -platform=linux/amd64 or something?

gentle path
azure sigil
#

I'm using the embedded engine and I think that's picking up the arch of my host machine, wonder if I can start my own

gentle path
wild hawk
# azure sigil I'm using the embedded engine and I think that's picking up the arch of my host ...

One thing, dagger.json is only needed when using extensions, which aren't released yet, so unless you want to explore the dev frontiers you can ignore that for now 🙂

The current behavior is that your platform should default to using the platform where buildkit is running. So yeah, by default on macos m1 machines you'll end up running it inside the docker desktop linux VM, which is also ARM now I believe.

I think the reason you are picking up qemu is that the image you're using is x86 only: https://hub.docker.com/r/heroku/buildpacks/tags

So Buildkit sees that, pulls the x86 image and then tries to run it anyways using qemu. Unfortunately, it looks like docker is saying this is just a qemu issue, they aren't going to do anything about it, and instead you should try to use ARM images...: https://github.com/docker/for-mac/issues/5123

#

I don't love that we just have to repeat that unfortunate "solution" but it is indeed hard to fix without fixes from qemu or the buildpack image maintainers publishing arm images. I suppose one possibility would be to rebuild the buildpack image yourself as part of your pipeline instead of just pulling it, but after a brief glance that may be easier said than done: https://github.com/heroku/builder

azure sigil
#

I could actually host a buildkit daemon/dagger engine on a linux/amd64 machine

#

If you can point me to a doc/tell me at a high level how to host one, that would be great! 🙂

wild hawk
# azure sigil If you can point me to a doc/tell me at a high level how to host one, that would...

Oh sure, though major forewarning that there's a good chance these details will change and this isn't really documented, tested or officially supported right now (it will be in the future though).

I suspect the easiest (though it's still not easy) way would be to setup a remote docker engine on the amd64 machine and then tell your local docker CLI to use that (via DOCKER_HOST, as documented here: https://docs.docker.com/engine/security/protect-access/). If you have the env var exported when you invoke dagger, dagger should use that remote docker host instead of the local one.

If that doesn't work, there's also undocumented support for connecting to buildkit running in a kubernetes pod or podman via a BUILDKIT_HOST env var. Can give pointers for that too.

azure sigil
#

Yeah, no worries if it's a thing in works. I know it's early days.

#

I was actually thinking if I can host the the dagger engine itself on a different machine and then connect my program via DAGGER_HOST but I'll need to know how to setup buildkit first and then start dagger engine on top of it.

#

How can I start Dagger engine on top of it once buildkit is deployed?

hasty glacier
#

@azure sigil there's a short term answer and a longer term answer

#

short term: not supported right now, but we could quickly easily add a way for you to run a buildkit daemon yourself and configure the API router to connect to it. BUT that will not be supported long term, as the engine really bundles its own buildkit engine as a sub-component. We don't want to deal with the matrix from hell of supporting any variation of dagger/buildkit versions.

#

longer term: dagger engine is its own unit, with its own configuration and deployment options. It includes buildkit but you don't need to individually deploy that (nor can you). All the deployment features of buildkit are available for dagger engine

azure sigil
#

Oh, that makes sense. I am just looking at how the code for how dagger engine starts up

hasty glacier
#

the engine is really 2 distinct components right now:

  1. An API router that runs on the client machine
  2. A runner (buildkit engine + helpers) that runs in a container on a worker machine
#

You are welcome to hack away in the engine package (cc @thin atlas @wild hawk for more details on the internals), but just know that we don't guarantee that code's stability, we can and will move things around in that go package

azure sigil
#

Yeah, no issues with that...

#

Because when the engine.Start method initialises the buildkit client, it relies on the BUILDKIT_HOSTvariable.

wild hawk
# azure sigil Because when the `engine.Start` method initialises the buildkit client, it relie...

Yeah the problem right now is that BUILDKIT_HOST assumes that buildkit is already running whereever you point it to. Dagger won't spin it up for you if it's not there. That's why I suspected that setting DOCKER_HOST might actually be the easiest route. By default, dagger spins up a buildkit for you in a docker container, but it does that just by shelling out to docker, so if you have DOCKER_HOST set then you can get our automatically setup buildkitd, but on whatever remote host you want.

azure sigil
#

Got it to work without having to run docker locally 😄
I did start up a buildkit daemon on a remote machine

Screenshot_2022-10-28_at_4.00.18_AM.png
azure sigil
#

Is there any plan to have dagger engine exposed via http? Happy to contribute to make it happen

azure sigil
#

Back to the buildpack integration, I'm not able to figure this one out. For some reason, writes to /src fails.
Logs:

#15 13.42 go: extracting github.com/gofrs/uuid v3.3.0+incompatible
#15 13.43 error writing go.mod: open /src/go.mod966711297.tmp: permission denied
#15 13.43 ERROR: failed to build: exit status 1
------
input:1: container.from.withMountedDirectory.withWorkdir.exec.directory process "/_shim /cnb/lifecycle/creator -app=. ttl.sh/random-new-image:latest" did not complete successfully: exit code: 145
azure sigil
#

I'll try copying the contents over rather than mounting

#

Oh, got it to work by using /tmp!! 😄

#

Thanks @gentle path @hasty glacier @wild hawk
You've all been a great help! 🙂

hasty glacier
#

@azure sigil FYI I have a working POC of a docker2dagger bridge 🙂

azure sigil
#

Nice.

#

I wonder if I can still avoid using docker though

#

I remember the dagger-buildkit version matrix concern you had but I wonder if Dagger engine is only going to support one buildkit version at a time, then maybe we can have enforce it by not letting a dagger engine start if buildkit version doesn't match for the particular dagger release.

hasty glacier
#

it may be more than version though

#

there are specific combinations of configuration flags that we may need to hardwire or manage

azure sigil
#

Oh, ok.
I really liked it when I could just connect a locally running stateless dagger engine to a remote buildkit where things are cached. It seemed like a much simpler version with host writes and everything just working out of the box.

#

I don't know how much of it was intended but I was wondering if I could have dagger engines connected to a single buildkit

hasty glacier
#

Yes you will still be able to do this, we will release the dagger engine as a standalone docker image that you can run as a stateless, shared service. Just like builkitd today - but with a higher-level API

#

until that’s ready we’ll still allow swapping out buildkitd, but with the understanding that it will be deprecated eventually in favor of the above

hasty glacier
#

by the way, even after that transition you will still be able to build your own engine, including the buildkit component, and deploy that, since it’s all open source and still powered by buildkit under the hood

azure sigil
hasty glacier
#

I believe there’s already another env variable that you can use. But things are moving fast so I’ll wait to see what the review says on that PR 🙂

#

(thanks for taking the time to contribute this)

azure sigil
#

Oh, cool.
I'll fill in the description meanwhile for the setup I wanted . If it's already supported, that'd be amazing.