#The_Automator pairing
1 messages Β· Page 1 of 1 (latest)
Hey, how are you?
I lost everything in the last two weeks from my mind. π
So I had a discussion with our Salesforce Team which explained their pipeline to me.
So the scenario is that:
- Build the docker image based on a docker file
FROM salesforce/salesforcedx:latest-rc-full
WORKDIR /salesforce/salesforcedx
ENV DEBIAN_FRONTEND noninteractive
ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8
ENV LANGUAGE en_US.UTF-8
COPY package.json ./
COPY python3requirements.txt ./
RUN apt-get update && \
apt-get install python3-pip -y && \
apt-get clean && \
pip3 install --no-cache-dir -r python3requirements.txt
RUN npm install && \
npm cache clean --force
- Push the image to the gitlab container registry.
I wanna do that with cloak in combination with TypeScript. But to be honest I don't know how to start.
I already build cloak on my local machine. But what is next?
As I understand that right I have to create a cloak.yml file at the root of the repository, right?
Exactly.
What I would do is run the yarn example with cloak dev: cloak dev -p examples/yarn/cloak.yaml --local-dir src=".". It is relying on the Typescript SDK.
I would then play with the playground. Everything is a graphQL query. You just wrap these queries in TypeScript. Try to see how to access the dockerbuild function from the playground. (prior making extensions out of it, but we'll save that for later)
The yarn example will be a good reference π
The next step is to play with the core primitives on the GraphQL playground:
- Make a query to load the dockerfile and all necessary content
- Be able to build it (using the
dockerbuildprimitive)
Here is an example: https://github.com/dagger/cloak/blob/fc5744a46c87571e218e6511e927b562bf4c8709/examples/queries/docker_build.graphql
- Be able to push it (Not sure it works with remote registries atm though). You will be able to push it to a local registry for sure (need @quasi ingot confirmation for that)
here is the reference for the push: #maintainers message
By the way, to run the local registry and make it work with Dagger, all the context is here: https://discord.com/channels/707636530424053791/1019222022007554068
- One you have 1 and 2, check how the yarn example is done, and how you can wrap those graphQL queries in TS
If anything seems unclear, please let me know ahaha π
What is this command doing? cloak dev ...
Ah ok it starts a webserver with the playground.
Rectification -> you should look at the netlify example
Ok I will start with the next step. 'Make a query to load the dockerfile'.
Actual I have this:
{
core {
image(ref: "alpine") {
dockerbuild(dockerfile: "/examples/salesforce/Dockerfile") {
id
}
}
}
}
Actual there is something wrong. "message": "failed to read dockerfile: open /tmp/buildkit-mount1630103192/examples/salesforce/Dockerfile: no such file or directory",
@quasi ingot do you know what is wrong?
Hi Patrick,
You cannot do it that way. Dockerbuild loads from the rootfs of the alpine image
You need to load from equivalent of the client api
Would you be ok to have a 1:1 tomorrow, to unlock you ?
Tell me your hour π
08:00 - 11:00 or from 13:00 - 14:00
10->11 ?
Perfect
Zoom-Meeting beitreten
https://us04web.zoom.us/j/7241362068?pwd=WmRUSUJQWkJLSGlRTlVmMlZseFhEZz09
Meeting-ID: 724 136 2068
Kenncode: b9ZWWn
Zoom is the leader in modern enterprise video communications, with an easy, reliable cloud platform for video and audio conferencing, chat, and webinars across mobile, desktop, and room systems. Zoom Rooms is the original software-based conference room solution used around the world in board, conference, huddle, and training rooms, as well as ex...
@sage kelp are you here?
https://github.com/dagger/cloak/blob/main/docs/guides/oy1q7-writing_extensions_nodejs.md <== everything seems to be here
Zoom is the leader in modern enterprise video communications, with an easy, reliable cloud platform for video and audio conferencing, chat, and webinars across mobile, desktop, and room systems. Zoom Rooms is the original software-based conference room solution used around the world in board, conference, huddle, and training rooms, as well as ex...
Thanks @humble fulcrum for the help π π
The famous robot you talked about π
@humble fulcrum I have another question.
We designed the build.mjs file
import { gql, Engine } from "@dagger.io/dagger";
new Engine().run(async (client) => {
const fileContents = await client
.request(
gql`
{
host {
workdir {
read {
dockerbuild(dockerfile: "Dockerfile") {
id
}
}
}
}
}
`
)
.then((result) => result.host.workdir.read.dockerbuild.id);
console.log("Output: " + fileContents);
});
Now when I try to make my push.mjs I realized that I need the dockerbuild.id in my push.mjs file. Is it possible to save the id in a variable and use it in the push.mjs again? Or do I have to do both in one file?
@sage kelp what is expected here in the ref field? The id of the build image? How can I configure the url and token of the storage where I wanna publish my image?
pushImage(ref: "")
I think you should use it in the same file
Sounds logical.
Not sure about splitting in multiple file
Totally
To push an Image I have to make a docker login to the gitlab registry: docker login registry.gitlab.com
Is it possible to realize that?
Or is that a case for an extension?
Let me check
I don't think you can atm, as it's a core primitive that needs to include the credentials. I'll ask for confirmation
I found this:
core {
addSecret(plaintext: "")
}
But I am not sure if that could help.
It's more that the pushImage core API doesn't permit to add credentials inside, and if it's not implemented, I don't think we can bypass it
From my point of view it could be that:
core {
addSecret(plaintext: "")
image(ref: "") {
pushImage(ref: "")
}
}
I will give a try.
"message": "failed to load cache key: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed",
Yeah, I'm 99% sure the secret would be an input to pushImage
I make this:
{
host {
workdir {
read {
dockerbuild(dockerfile: "Dockerfile") {
id
}
}
}
}
core {
addSecret(plaintext: "mySuperSecretToken")
image(ref: "id") {
pushImage(
ref: "registry.gitlab.com/p7s1/sales/salesforce/p7s1cicdtemplate/id:cloak"
)
}
}
}
Seems that I have to make a login before...? My docker login on the local machine is working.
You can locally yes, but as the containers run inside buildkit it has to be passed to the buildkit engine
However, this requires some design thinking (how to efficiently load secrets across all actions), that's why we haven't implemented it yet
@naive path A question though, you did docker login registry.gitlab.com ?
Is it the machine on which you run cloak ? I overheard that the login were passed, that's what I'm currently looking for in our codebase
Yeah that that is the machine where I run cloak.
Do I have to use that
exec(input: {args: "", secretEnv: {name: "", id: ""}})
?
Hmm, ok. I think I found where the secrets are passed in the typescript sdk
I'm going to try tweaking it
That happens after a mac update...
132 hours and 29 minutes... In 2022... Awesome Apple...
Ok first problem.
I have to respect the naming convention:
https://docs.gitlab.com/ee/user/packages/container_registry/#image-naming-convention
<registry URL>/<namespace>/<project>/<image>
Any ideas how to do that when I use a dockerfile?
(reminder to self for after community meeting, provide example on how to configure creds and how to push image to specific ref)
In terms of the naming, all you have to do is specify that name as the ref when you use pushImage. Nothing in the dockerfile needs to be connected to that name. You can push any dockerfile as any name you want.
@naive path what Guillaume suggested here should be all you need. cloak is setup to automatically use any credentials you have configured using docker login <url>. So if you have run that command on the host where you are using cloak, your credentials should be configured, nothing else needed
It's setup here: https://github.com/dagger/cloak/blob/ee6dd357870fc94c3119bbbf305c59abccddf71b/engine/engine.go#L120
Very easy to miss since it's a lot of functionality to get with just a single line of code!
Ok now it is working.
The problem was, that my access token had as scope only api. Which means in gitlab Grants complete read/write access to the API, including all groups and projects, the container registry, and the package registry.. But it doesn't worked.
After recreating my access token with more scopes it works.
My GQL-Statement looks like that now:
import { gql, Engine } from "@dagger.io/dagger";
new Engine().run(async (client) => {
const fileContents = await client
.request(
gql`
{
host {
workdir {
read {
dockerbuild(dockerfile: "Dockerfile") {
id
pushImage(
ref: "registry.gitlab.com/salesforce/salesforcedx:dagger"
)
}
}
}
}
}
`
)
.then((result) => result.host.workdir.read.dockerbuild.id);
console.log("Output: " + fileContents);
});
@sage kelp and @quasi ingot my code is working now and still I'm confused.
When I publish the image with dagger I have a container in the container registry with 20.03 MiB . At the same time when I execute a docker build and docker push I have a new entry in the registry with 905.84 MiB. Despite the fact that the image is based on the same docker file.
How is that possible?
I mean the base image
salesforce/salesforcedx:latest-rc-full
Digest:sha256:73aba3b4f210df3141becc2a6bb14ba97a9e62729690c5c68525ad9d90c91fea
already has 2.08 GB. I don't get it.
Hi @naive path
Did you make sure that the context on which you run that docker build command and the the workdir inside your graphQL query is the same ?
The dockerfile command is literally calling the docker frontend, so that sounds weird
Ok I will not ping Erik.
How can I validate if the context and the workdir is the same? I mean I am working in the same directory with both commands.
@sage kelp how can I see the context which cloak is using? My command docker context show prints out default. Is cloak using the same?
From my point of view there is a problem with the query itself.
Could be that the workdiris really wrong.
It's possible yeah. Maybe the dir alternative might work
I see that. What is expected as id? The path?
@sage kelp that is my dockerfile.
FROM salesforce/salesforcedx:latest-rc-full
WORKDIR /builds/salesforce/salesforcedx
ENV DEBIAN_FRONTEND noninteractive
ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8
ENV LANGUAGE en_US.UTF-8
COPY package.json ./
COPY python3requirements.txt ./
RUN apt-get update && \
apt-get install python3-pip -y && \
apt-get clean && \
pip3 install --no-cache-dir -r python3requirements.txt
RUN npm install && \
npm cache clean --force
And here is my new query:
{
host {
dir(id: "src") {
read {
dockerbuild(dockerfile: "Dockerfile") {
id
pushImage(
ref: "registry.gitlab.com/..."
)
}
}
}
}
}
But for me seems that the dockerbuild is not working.
ok, setup is finally good. Now, trying out your queries
@sage kelp do you need my other files too?
Currently pulling the uploaded image, when done I check the size
The size is ok for me
import { gql, Engine } from "@dagger.io/dagger";
new Engine().run(async (client) => {
const fileContents = await client
.request(
gql`
{
host {
workdir {
read {
dockerbuild(dockerfile: "script/Dockerfile") {
id
pushImage(
ref: "YOUR_REGISTRY"
)
}
}
}
}
}
`
)
.then((result) => result.host.workdir.read.dockerbuild.id);
console.log("Output: " + fileContents);
});
I tweaked a little the Dockerfile because I didn't have all the files
FROM salesforce/salesforcedx:latest-rc-full
WORKDIR /builds/salesforce/salesforcedx
ENV DEBIAN_FRONTEND noninteractive
ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8
ENV LANGUAGE en_US.UTF-8
RUN apt-get update && \
apt-get install python3-pip -y && \
apt-get clean
But it shouldn't impact the size
I believe I see that...
I believe that this dockerbuild(dockerfile: "script/Dockerfile") {
You put the dockerfile on the same level as the mjs file.
I also did it now and now the docker build takes much longer as before. And it uses the salesforce image.
Did you put the path to the dockerfile before, related to the cloak.yaml file ?
Nope.
That shall be the reason then. Thanks for your question, we learned something today ahah π
I don't think so, it's where the cloak.yaml file lives
Nope.
that's why I put script/Dockerfile in my dockerbuild command
Ooh, then there might be an issue yes
And now I get the following error:
#9 23.62 npm ERR! An unknown git error occurred
#9 23.62 npm ERR! command git --no-replace-objects ls-remote ssh://git@github.com/dagger/cloak.git
#9 23.62 npm ERR! Warning: Permanently added 'github.com,140.82.121.3' (ECDSA) to the list of known hosts.
#9 23.62 npm ERR! git@github.com: Permission denied (publickey).
#9 23.62 npm ERR! fatal: Could not read from remote repository.
To prove that I change the query to that:
{
host {
workdir {
read {
dockerbuild(dockerfile: "../../Dockerfile") {
id
pushImage(
ref: "registry.gitlab.com/p7s1/sales/salesforce/salesforcedx:dagger"
)
}
}
}
}
}
And delete the dockerfile from the scripts folder.
1. In order to pull cloak dependencies, cloak and yarn will need the ability to pull a private git repo
- Setting up an ssh-agent with credentials that can pull the `dagger/cloak` will cover all cases and is recommended for now.
- Github has [documentation on setting this up for various platforms](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent).
- Be sure that the `SSH_AUTH_SOCK` variable is set in your current terminal (running `eval "$(ssh-agent -s)"` will typically take care of that)
- Without this, you may get error messages containing `no ssh handler for id default`
1. Add a dependency on the cloak nodejs sdk:
- `yarn add --dev git+ssh://git@github.com:dagger/cloak.git#main`
- `yarn install`
You seem to have an issue your with ssh key here
With the changed path the image build is also running. So the workdir points to the scripts -> build -> build.mjs
Do you have a link to the documentation? Then I can fix this ssh issue
Taking a plane tomorrow, need to go buy some missing stuff. Will be back in a few hours. I might reply slower
@sage kelp I don't get the issue with the ssh thing.
Because I have in my package.json already this:
"devDependencies": {
"@dagger.io/dagger": "git+ssh://git@github.com:dagger/cloak.git#main"
},
And before a yarn build can run I had to install this dependency with yarn install .... that worked.
I believe the problem is that the package.json is copied to the image COPY package.json ./ and from there it can`t access the github repo via ssh...
Sounds that valid to you?
You're right, that totally may be the root cause
Ok how can we fix that in a good way... I have to think about... Or do you have any suggestions?
(catching up on the most recent discussion, will let you know what's possible after)
hey @naive path sorry about the delay... we had a meeting and suddenly decided to get a bunch of emergent work done really quickly, I'm going to take a look again asap later today!
Hey Erik, sounds good. The most blocking issue is only the ssh part. When the Image is built based on the dockerfile.
So the change we just made should actually help fix the SSH auth part because our cloak repo moved to be a branch on dagger/dagger, so no more pulling of private repos needed!
There are a few tiny adjustments needed:
- Switch your dev dependency to:
git+https://github.com/dagger/dagger.git#cloak. Then you'll have to runyarn install - If you have any dependencies in
cloak.yamlfrom the old cloak repo, change them to use the new branch in dagger/dagger:
- git:
remote: https://github.com/dagger/dagger.git
ref: cloak
path: ...
But then you should be good to go, everything should just work w/out any ssh creds
I will try that later
@quasi ingot boom it works!!!! Thank you!
Next stop gitlab! π
@quasi ingot & @sage kelp how does the access to the container registry work in a gitlab ci? As I know you can normally do that: docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY how would I do that in a dind?
@humble fulcrum you already published an application to vercel? Do you have any idea how I can push an image from gitlab-ci in a dind to the gitlab container registry?
Hi @naive path, we're on a team retreat this week. Sorry for the response delays π
So is the setup that you are running cloak inside a docker container where the docker socket has been mounted in? And you want to use those creds to push an image to gitlab using cloak? If so, I would actually have expected just running that docker login ... command before you execute cloak should do the trick, but let me know if the scenario is different or if that's not working
@quasi ingot you are right. That is the scenario and it works. I ask myself if it would be better to have a call like this:
pushImage(
ref: "registry.gitlab.com/p7s1/sales/salesforce/salesforcedx:dagger", user: $MyUser, token: $MyToken
)
I would doing that way without an explicit docker login... in the pipeline. What do you think?
Yep thatβs a great point, Iβll make an issue for supporting that
@quasi ingot I already did. https://github.com/dagger/dagger/issues/3138