#The_Automator pairing

1 messages Β· Page 1 of 1 (latest)

sage kelp
#

Hi, yep. Here it is πŸ˜‡

naive path
#

Hey, how are you?
I lost everything in the last two weeks from my mind. πŸ˜‰
So I had a discussion with our Salesforce Team which explained their pipeline to me.

So the scenario is that:

  1. Build the docker image based on a docker file
FROM salesforce/salesforcedx:latest-rc-full

WORKDIR /salesforce/salesforcedx

ENV DEBIAN_FRONTEND noninteractive
ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8
ENV LANGUAGE en_US.UTF-8

COPY package.json ./
COPY python3requirements.txt ./

RUN apt-get update && \
    apt-get install python3-pip -y && \
    apt-get clean && \
    pip3 install --no-cache-dir -r python3requirements.txt

RUN npm install && \
    npm cache clean --force
  1. Push the image to the gitlab container registry.

I wanna do that with cloak in combination with TypeScript. But to be honest I don't know how to start.
I already build cloak on my local machine. But what is next?

#

As I understand that right I have to create a cloak.yml file at the root of the repository, right?

sage kelp
# naive path As I understand that right I have to create a cloak.yml file at the root of the ...

Exactly.
What I would do is run the yarn example with cloak dev: cloak dev -p examples/yarn/cloak.yaml --local-dir src=".". It is relying on the Typescript SDK.

I would then play with the playground. Everything is a graphQL query. You just wrap these queries in TypeScript. Try to see how to access the dockerbuild function from the playground. (prior making extensions out of it, but we'll save that for later)

The yarn example will be a good reference πŸ˜‡

The next step is to play with the core primitives on the GraphQL playground:

  1. Make a query to load the dockerfile and all necessary content
  2. Be able to build it (using the dockerbuild primitive)

Here is an example: https://github.com/dagger/cloak/blob/fc5744a46c87571e218e6511e927b562bf4c8709/examples/queries/docker_build.graphql

  1. Be able to push it (Not sure it works with remote registries atm though). You will be able to push it to a local registry for sure (need @quasi ingot confirmation for that)

here is the reference for the push: #maintainers message

By the way, to run the local registry and make it work with Dagger, all the context is here: https://discord.com/channels/707636530424053791/1019222022007554068

#
  • One you have 1 and 2, check how the yarn example is done, and how you can wrap those graphQL queries in TS
#

If anything seems unclear, please let me know ahaha πŸ˜‡

naive path
#

What is this command doing? cloak dev ...

#

Ah ok it starts a webserver with the playground.

sage kelp
#

Rectification -> you should look at the netlify example

naive path
#

Ok I will start with the next step. 'Make a query to load the dockerfile'.

Actual I have this:

{
  core {
    image(ref: "alpine") {
      dockerbuild(dockerfile: "/examples/salesforce/Dockerfile") {
        id
      }
    }
  }
}
#

Actual there is something wrong. "message": "failed to read dockerfile: open /tmp/buildkit-mount1630103192/examples/salesforce/Dockerfile: no such file or directory",

naive path
#

@quasi ingot do you know what is wrong?

sage kelp
#

You need to load from equivalent of the client api

#

Would you be ok to have a 1:1 tomorrow, to unlock you ?

naive path
#

Yeah is ok for me.

#

Hmm seems that I understood it wrong.

#

πŸ˜“πŸ˜“πŸ˜“

sage kelp
#

Tell me your hour πŸ˜‡

naive path
#

08:00 - 11:00 or from 13:00 - 14:00

sage kelp
#

10->11 ?

naive path
#

Perfect

naive path
#

Zoom-Meeting beitreten
https://us04web.zoom.us/j/7241362068?pwd=WmRUSUJQWkJLSGlRTlVmMlZseFhEZz09

Meeting-ID: 724 136 2068
Kenncode: b9ZWWn

naive path
#

@sage kelp are you here?

sage kelp
sage kelp
sage kelp
sage kelp
#

Thanks @humble fulcrum for the help πŸ˜‡ πŸ™

naive path
#

@sage kelp here are some impressions from my vacation.

sage kelp
naive path
#

@humble fulcrum I have another question.
We designed the build.mjs file

import { gql, Engine } from "@dagger.io/dagger";
new Engine().run(async (client) => {
  const fileContents = await client
    .request(
      gql`
        {
            host {
                workdir {
                    read {
                        dockerbuild(dockerfile: "Dockerfile") {
                            id
                        }
                    }
                }
            }
        }
      `
    )
    .then((result) => result.host.workdir.read.dockerbuild.id);
  console.log("Output: " + fileContents);
});

Now when I try to make my push.mjs I realized that I need the dockerbuild.id in my push.mjs file. Is it possible to save the id in a variable and use it in the push.mjs again? Or do I have to do both in one file?

#

@sage kelp what is expected here in the ref field? The id of the build image? How can I configure the url and token of the storage where I wanna publish my image?
pushImage(ref: "")

humble fulcrum
#

I think you should use it in the same file

naive path
#

Sounds logical.

humble fulcrum
#

Not sure about splitting in multiple file

sage kelp
naive path
#

To push an Image I have to make a docker login to the gitlab registry: docker login registry.gitlab.com

Is it possible to realize that?

#

Or is that a case for an extension?

sage kelp
#

I don't think you can atm, as it's a core primitive that needs to include the credentials. I'll ask for confirmation

naive path
sage kelp
#

It's more that the pushImage core API doesn't permit to add credentials inside, and if it's not implemented, I don't think we can bypass it

naive path
#

From my point of view it could be that:

core {
    addSecret(plaintext: "")
    image(ref: "") {
      pushImage(ref: "")
    }
  }

I will give a try.

#
"message": "failed to load cache key: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed",
sage kelp
#

Yeah, I'm 99% sure the secret would be an input to pushImage

naive path
#

I make this:

{
  host {
    workdir {
      read {
        dockerbuild(dockerfile: "Dockerfile") {
          id
        }
      }
    }
  }
  core {
    addSecret(plaintext: "mySuperSecretToken")
    image(ref: "id") {
      pushImage(
        ref: "registry.gitlab.com/p7s1/sales/salesforce/p7s1cicdtemplate/id:cloak"
      )
    }
  }
}
#

Seems that I have to make a login before...? My docker login on the local machine is working.

sage kelp
#

However, this requires some design thinking (how to efficiently load secrets across all actions), that's why we haven't implemented it yet

sage kelp
naive path
#

On my local machine yes.

#

But not with cloak. Is it possible to do that?

sage kelp
naive path
#

Yeah that that is the machine where I run cloak.
Do I have to use that

 exec(input: {args: "", secretEnv: {name: "", id: ""}})
#

?

sage kelp
#

Hmm, ok. I think I found where the secrets are passed in the typescript sdk

#

I'm going to try tweaking it

naive path
#

That happens after a mac update...

#

132 hours and 29 minutes... In 2022... Awesome Apple...

naive path
#

Any ideas how to do that when I use a dockerfile?

quasi ingot
#

(reminder to self for after community meeting, provide example on how to configure creds and how to push image to specific ref)

quasi ingot
quasi ingot
quasi ingot
naive path
# quasi ingot <@960764294851608606> what Guillaume suggested here should be all you need. `clo...

Ok now it is working.
The problem was, that my access token had as scope only api. Which means in gitlab Grants complete read/write access to the API, including all groups and projects, the container registry, and the package registry.. But it doesn't worked.
After recreating my access token with more scopes it works.

My GQL-Statement looks like that now:

import { gql, Engine } from "@dagger.io/dagger";
new Engine().run(async (client) => {
  const fileContents = await client
    .request(
      gql`
        {
          host {
            workdir {
              read {
                dockerbuild(dockerfile: "Dockerfile") {
                  id
                  pushImage(
                    ref: "registry.gitlab.com/salesforce/salesforcedx:dagger"
                  )
                }
              }
            }
          }
        }
      `
    )
    .then((result) => result.host.workdir.read.dockerbuild.id);
  console.log("Output: " + fileContents);
});
naive path
#

@sage kelp and @quasi ingot my code is working now and still I'm confused.
When I publish the image with dagger I have a container in the container registry with 20.03 MiB . At the same time when I execute a docker build and docker push I have a new entry in the registry with 905.84 MiB. Despite the fact that the image is based on the same docker file.

How is that possible?

#

I mean the base image

salesforce/salesforcedx:latest-rc-full
Digest:sha256:73aba3b4f210df3141becc2a6bb14ba97a9e62729690c5c68525ad9d90c91fea

already has 2.08 GB. I don't get it.

sage kelp
#

Hi @naive path
Did you make sure that the context on which you run that docker build command and the the workdir inside your graphQL query is the same ?

#

The dockerfile command is literally calling the docker frontend, so that sounds weird

naive path
naive path
#

@sage kelp how can I see the context which cloak is using? My command docker context show prints out default. Is cloak using the same?

#

From my point of view there is a problem with the query itself.
Could be that the workdiris really wrong.

sage kelp
naive path
sage kelp
#

Currently making a repro to progile

#

Profile

naive path
#

@sage kelp that is my dockerfile.

FROM salesforce/salesforcedx:latest-rc-full

WORKDIR /builds/salesforce/salesforcedx

ENV DEBIAN_FRONTEND noninteractive
ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8
ENV LANGUAGE en_US.UTF-8

COPY package.json ./
COPY python3requirements.txt ./

RUN apt-get update && \
    apt-get install python3-pip -y && \
    apt-get clean && \
    pip3 install --no-cache-dir -r python3requirements.txt

RUN npm install && \
    npm cache clean --force

And here is my new query:

{
  host {
    dir(id: "src") {
      read {
        dockerbuild(dockerfile: "Dockerfile") {
          id
          pushImage(
              ref: "registry.gitlab.com/..."
          )
        }
      }
    }
  }
}

But for me seems that the dockerbuild is not working.

sage kelp
#

ok, setup is finally good. Now, trying out your queries

naive path
#

@sage kelp do you need my other files too?

sage kelp
#

The size is ok for me

naive path
#

Seems good.

#

Ok what did you right, what I did wrong?

sage kelp
#
import { gql, Engine } from "@dagger.io/dagger";
new Engine().run(async (client) => {
    const fileContents = await client
        .request(
            gql`
        {
        host {
            workdir {
            read {
                dockerbuild(dockerfile: "script/Dockerfile") {
                id
                pushImage(
                    ref: "YOUR_REGISTRY"
                )
                }
            }
            }
        }
        }
      `
        )
        .then((result) => result.host.workdir.read.dockerbuild.id);
    console.log("Output: " + fileContents);
});
#

I tweaked a little the Dockerfile because I didn't have all the files

#
FROM salesforce/salesforcedx:latest-rc-full

WORKDIR /builds/salesforce/salesforcedx

ENV DEBIAN_FRONTEND noninteractive
ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8
ENV LANGUAGE en_US.UTF-8

RUN apt-get update && \
    apt-get install python3-pip -y && \
    apt-get clean
#

But it shouldn't impact the size

naive path
#

I believe I see that...

sage kelp
#

I believe that this dockerbuild(dockerfile: "script/Dockerfile") {

naive path
#

You put the dockerfile on the same level as the mjs file.

#

I also did it now and now the docker build takes much longer as before. And it uses the salesforce image.

sage kelp
#

Did you put the path to the dockerfile before, related to the cloak.yaml file ?

naive path
#

Nope.

sage kelp
#

That shall be the reason then. Thanks for your question, we learned something today ahah πŸ˜‡

naive path
#

Yeah of course.

#

So the workdir is the directory where the mjs file lives.

sage kelp
naive path
#

Nope.

sage kelp
#

that's why I put script/Dockerfile in my dockerbuild command

naive path
#

That was original

#

So it can build the image.

sage kelp
naive path
#

And now I get the following error:
#9 23.62 npm ERR! An unknown git error occurred
#9 23.62 npm ERR! command git --no-replace-objects ls-remote ssh://git@github.com/dagger/cloak.git
#9 23.62 npm ERR! Warning: Permanently added 'github.com,140.82.121.3' (ECDSA) to the list of known hosts.
#9 23.62 npm ERR! git@github.com: Permission denied (publickey).
#9 23.62 npm ERR! fatal: Could not read from remote repository.

naive path
# sage kelp Ooh, then there might be an issue yes

To prove that I change the query to that:

{
          host {
            workdir {
              read {
                dockerbuild(dockerfile: "../../Dockerfile") {
                  id
                  pushImage(
                    ref: "registry.gitlab.com/p7s1/sales/salesforce/salesforcedx:dagger"
                  )
                }
              }
            }
          }
        }
#

And delete the dockerfile from the scripts folder.

sage kelp
#
1. In order to pull cloak dependencies, cloak and yarn will need the ability to pull a private git repo

   - Setting up an ssh-agent with credentials that can pull the `dagger/cloak` will cover all cases and is recommended for now.
     - Github has [documentation on setting this up for various platforms](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent).
     - Be sure that the `SSH_AUTH_SOCK` variable is set in your current terminal (running `eval "$(ssh-agent -s)"` will typically take care of that)
     - Without this, you may get error messages containing `no ssh handler for id default`

1. Add a dependency on the cloak nodejs sdk:

   - `yarn add --dev git+ssh://git@github.com:dagger/cloak.git#main`
   - `yarn install`
#

You seem to have an issue your with ssh key here

naive path
#

With the changed path the image build is also running. So the workdir points to the scripts -> build -> build.mjs

#

Do you have a link to the documentation? Then I can fix this ssh issue

sage kelp
#

Taking a plane tomorrow, need to go buy some missing stuff. Will be back in a few hours. I might reply slower

naive path
#

@sage kelp I don't get the issue with the ssh thing.
Because I have in my package.json already this:

"devDependencies": {
    "@dagger.io/dagger": "git+ssh://git@github.com:dagger/cloak.git#main"
  },

And before a yarn build can run I had to install this dependency with yarn install .... that worked.
I believe the problem is that the package.json is copied to the image COPY package.json ./ and from there it can`t access the github repo via ssh...

#

Sounds that valid to you?

sage kelp
naive path
#

Ok how can we fix that in a good way... I have to think about... Or do you have any suggestions?

quasi ingot
#

(catching up on the most recent discussion, will let you know what's possible after)

quasi ingot
#

hey @naive path sorry about the delay... we had a meeting and suddenly decided to get a bunch of emergent work done really quickly, I'm going to take a look again asap later today!

naive path
quasi ingot
# naive path Hey Erik, sounds good. The most blocking issue is only the ssh part. When the Im...

So the change we just made should actually help fix the SSH auth part because our cloak repo moved to be a branch on dagger/dagger, so no more pulling of private repos needed!

There are a few tiny adjustments needed:

  1. Switch your dev dependency to: git+https://github.com/dagger/dagger.git#cloak. Then you'll have to run yarn install
  2. If you have any dependencies in cloak.yaml from the old cloak repo, change them to use the new branch in dagger/dagger:
  - git:
      remote: https://github.com/dagger/dagger.git
      ref: cloak
      path: ...

But then you should be good to go, everything should just work w/out any ssh creds

naive path
#

I will try that later

naive path
#

@quasi ingot boom it works!!!! Thank you!

naive path
#

Next stop gitlab! πŸŽ‰

naive path
#

@quasi ingot & @sage kelp how does the access to the container registry work in a gitlab ci? As I know you can normally do that: docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY how would I do that in a dind?

naive path
#

@humble fulcrum you already published an application to vercel? Do you have any idea how I can push an image from gitlab-ci in a dind to the gitlab container registry?

sage kelp
quasi ingot
# naive path <@949034677610643507> & <@274903880343748619> how does the access to the contai...

So is the setup that you are running cloak inside a docker container where the docker socket has been mounted in? And you want to use those creds to push an image to gitlab using cloak? If so, I would actually have expected just running that docker login ... command before you execute cloak should do the trick, but let me know if the scenario is different or if that's not working

naive path
#

@quasi ingot you are right. That is the scenario and it works. I ask myself if it would be better to have a call like this:

pushImage(
    ref: "registry.gitlab.com/p7s1/sales/salesforce/salesforcedx:dagger", user: $MyUser, token: $MyToken
)

I would doing that way without an explicit docker login... in the pipeline. What do you think?

quasi ingot
naive path