#Use remote BUILDKIT with Python SDK

1 messages · Page 1 of 1 (latest)

glacial yacht
#

Hello,

I'm trying to run a remote buildkit in kubernetes. Currently I have a pod running with buildkit exposed on the port 1234 and a Service exposing it. Then I run my python in a docker-compose.yaml setting the env var BUILDKIT_HOST.

I'm following this docs: https://docs.dagger.io/1223/custom-buildkit/#using-a-custom-remote-buildkit-running-in-docker


services:
  test:
    build: .
    volumes:
      - /var/run:/var/run
      - .:/home/app
    working_dir: /home/app
    environment:
      - BUILDKIT_HOST=tcp://myendpoint.com:1234

Then I have a Dockerfile with python, where I run the python script.


RUN pip install dagger-io

RUN pip install python-dotenv

RUN apt-get update && \

    apt-get install -y \
    ca-certificates \
    curl \
    gnupg \
    lsb-release && \
    mkdir -p /etc/apt/keyrings && \
    curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \
    echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian" \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null && \
  $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null && \
  apt-get update && \
   apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

Then I run in my local container the python.

root@150254f210b1:/home/app# python dagger_test_and_push.py 

Is this setup enought to run dagger in the kubernetes buildkit? I don't think so beacuse I can't see any log in the kubernetes pod.

Thanks in advance.

Using a custom buildkit daemon

#

Use remote BUILDKIT with Python SDK

smoky thunder
#

hey @glacial yacht ! those docs don't apply to the current Python and SDK projects. The current features supported are the ones listed under docs.dagger.io.

#

Having said that, setting the BUILDKIT_HOST has been disabled since we're re-architecting the way remote engines work. We'll share some more details about this soon

cedar hawk
smoky thunder
#

as I mentioned before we're re-working the client / remote architecture, the thing about using BUILDKIT_HOST is that it enabled a lot of undesired behaviors.

glacial yacht
#

Thank you @smoky thunder for the answer, I will be on the ball. We were thinking on remote buildkit to take advantage of the cache for faster builds connecting our github pipelines to an external buildkit.

smoky thunder
cedar hawk
smoky thunder
#

@cedar hawk thx for the feedback. If you could post this on #maintainers for visibility that'd be much appreciated.

opaque tide
#

/cc @bitter raft @frozen gazelle

opaque tide
#

Just to better understand the issue — were you relying specifically on a shared buildkit container or just a shared something (e.g. doesn’t matter if it’s buildkit as long as it’s a shared service holding the cache)?

frozen gazelle
#

yeah that’s a miscommunication on our part, I agree we should leave BUILDKIT_HOST as an escape hatch until remote engine is ready, BUT with the understanding that it’s not officially supported and may not be exempt of problems (which is why we’re moving away from it in the first place)

opaque tide
#

Yeah I was going to suggest something along those lines — we can provide a workaround, as long as you’re ok it being a workaround that’ll change at some point once we have a proper solution

opaque tide
#

(or something — but it’s not vanilla buildkit anymore)

#

That should work with all SDKs @glacial yacht, including Python (re: your original question)

More details soon

frozen gazelle
opaque tide
cedar hawk
cedar hawk
cinder relic
#

Is there an issue or somewhere to follow when the remote/shared runner option will be available again in the SDKs? The sidecar method for like k8s or equivalent is what I'm interested in doing.

opaque tide
frozen gazelle
true coral
#

Hey guys - trying to follow along here, was a little surprised this AM to find remote buildkit host gone 😦 This prevents us from running in jenkins

I've been reading through docs and experimenting for a little while this am, while I understand that this is all highly unstable and moving quick... is it safe to say that at this point I cannot run dagger without a docker daemon? (I'm just trying to execute GO SDK against a remote host)

bitter raft
# true coral Hey guys - trying to follow along here, was a little surprised this AM to find r...

Yeah the latest Go SDK does not support BUILDKIT_HOST and also doesn't work w/ moby/buildkit, it needs our custom engine image (which helps us provide more features out of the box). Apologies about the instability here, the interface that existed before was kind of haphazard and inherited from previous version of dagger. I didn't realize how many people were relying on it because it was undocumented in the newer docs, but in retrospect should have made this transition more painless 😕

In the upcoming release you'll be able to use:

In the meantime before that release, if you want a quick temporary escape hatch, w/ the 0.4.2 go sdk it's possible to use an DAGGER_RUNNER_HOST env var (we temporarily switched to that before settling on the final interface) in combination w/ ghcr.io/dagger/engine:v0.3.7

true coral
#

Hmmm. Yeah I'm running 0.4.2 SDK, 0.3.7 engine, exposed on TCP 1234 , set DAGGER_RUNNER_HOST=tcp://3.4.5.6:1234 and I get this panic.

#

If this is a big rabbit hole I guess I can wait, I'm just at the final step putting a recent dagger pipeline into prod - was hoping to get this across the finish line.e

bitter raft
# true coral Hmmm. Yeah I'm running 0.4.2 SDK, 0.3.7 engine, exposed on TCP 1234 , set `DAGGE...

That error message in particular is quite odd, it seems like it would imply you have . somewhere in PATH, but a user who encountered this previously fixed it a different way: #1048087694758137906 message

The escape hatch in the meantime is that you can build the binary it's trying to obtain from here: https://github.com/dagger/dagger/tree/sdk/go/v0.4.2/cmd/engine-session
And then set an additional env var DAGGER_HOST=bin:///path/to/that/binary

This again is something that is all massively simplified in the next release, shouldn't be a problem there, so this extra complication is only needed temporarily

frozen gazelle
#

Sorry about that @true coral . We can commit to give you two things quickly:

  1. A short term solution that may break later but is reliable now

  2. A clear plan on what the final design will be, so you can prepare & give feedback

true coral
#

If you guys know a quick fix that'd be great, I'm happy to update my stuff later, I know I'm kind of running on the dev branch here 😅

frozen gazelle
#

That's a question for @bitter raft, I don't have enough information on the current state of the various versions to be useful

#

I will focus on part 2

bitter raft
# true coral If you guys know a quick fix that'd be great, I'm happy to update my stuff later...

I think the quickest possible fix would be what I mentioned above: you can build the binary it's trying to obtain from here: https://github.com/dagger/dagger/tree/sdk/go/v0.4.2/cmd/engine-session
And then set an additional env var DAGGER_HOST=bin:///path/to/that/binary

The other possibility could be to go back to the 0.4.0 version of the Go SDK, which still used BUILDKIT_HOST as before. If that works for you that's probably even easier

true coral
#

I'll give em both a shot. Thanks for the help!

true coral
#

Rollback worked fine. Just had to fix up my stdout calls again. I'll pin to 0.4.0 for now, and follow along to see how things settle for the final architecture. Thanks again!