Dagger execution taking to long | Dagger | Page 1

formal cloak Oct 17, 2023, 6:23 PM

#

I have created a dagger project to build image from dockerfile and upload to gitlab container registry

import { connect, Client, Directory, Secret } from '@dagger.io/dagger'
import Path from 'node:path'

const gitlabRepositoryUrl = '...'
const gitlabRepositoryUsername = '...'
const gitlabRepositoryPassword = '...'

connect(async (client) => {
  const jmixPath: string = Path.join(process.cwd(), 'app')

  const jmixContextDir: Directory = client.host().directory(jmixPath)

  const gitlabRepositoryPasswordSecret: Secret = client.setSecret(
    'gitlabRepositoryPassword',
    gitlabRepositoryPassword
  )

  const imageRef: string = await jmixContextDir
    .dockerBuild()
    .withRegistryAuth(gitlabRepositoryUrl, gitlabRepositoryUsername, gitlabRepositoryPasswordSecret)
    .publish(`${gitlabRepositoryUrl}/repo/app:3.0.5`)

  console.log(`Published image to: ${imageRef}`)
}, { LogOutput: process.stderr })

And the docker file is

FROM gradle:8.4.0-jdk17 as Builder
WORKDIR /app
COPY . .
RUN ./gradlew -Pvaadin.productionMode=true bootJar

FROM openjdk:17.0.2-jdk-slim
WORKDIR /app
COPY --from=Builder /app/build/libs/knowledge-0.0.1-SNAPSHOT.jar /app/app.jar
CMD java -jar /app/app.jar

When I build the image locally by running ./gradlew -Pvaadin.productionMode=true bootJar it takes 1 minute or less (obviusly, by caches saved on my machine) but when I run from a bash script it takes 5-7 minutes (which is still and acceptable time)

KNOWLEDGE_DOCKER_NAME="registry.gitlab.com/repo/app"

docker build --tag $KNOWLEDGE_DOCKER_NAME:3.0.4 --file dockerfile .
docker push $KNOWLEDGE_DOCKER_NAME:3.0.4

But when I execute from dagger dagger run node --loader ts-node/esm ./src/index.ts it is taking 1h45m and still not finishing 😦

I guess dagger is not using my full hardware to build the image (i don't know), any one knows how can I reduce that execution time?

wind raptor Oct 17, 2023, 6:41 PM

#

Do you have a sense of which step is taking the longest time? That should help narrow down what might be wrong.

formal cloak Oct 21, 2023, 10:45 PM

#

Yes I know which line is the problem
Is this line

RUN ./gradlew -Pvaadin.productionMode=true bootJar

when I run locally ./gradlew -Pvaadin.productionMode=true bootJar takes a minute, when I build with docker docker build --tag registry.gitlab.com/repo/app:3.0.4 --file dockerfile . it takes 5-7 minutes, but with dagger I leave for 10 hours and still not ends :c

wind raptor Oct 23, 2023, 3:42 PM

#

Thanks for sharing the additional info, is there any more information that you could share on what this step is acutally doing. I'd love to be able to reproduce this somehow locally, is the repo you are working on public or private?

Could you also try running dagger run with --debug --progress=plain so we can see if there is any additional logging that shows why we are getting stuck?

formal cloak Oct 25, 2023, 4:12 PM

#

Hello I have published the code on this repository
https://github.com/danny270793/Jmix-Dagger

GitHub

GitHub - danny270793/Jmix-Dagger

Contribute to danny270793/Jmix-Dagger development by creating an account on GitHub.

old salmon Oct 26, 2023, 3:48 PM

#

@formal cloak do you figure this is the same issue?
https://github.com/dagger/dagger/issues/5791
cc @broken quail

GitHub

🐞 secrets scrubbing implementation causes excessive log output late...

What is the issue? Context We have a dagger pipeline which, at some point, mounts secrets to a container which is then runs integration tests by with_exec'ing a task in gradle. We are just as b...

broken quail Oct 26, 2023, 4:02 PM

#

I'm still working on a repro case tho!

formal cloak Oct 26, 2023, 8:15 PM

#

Have you run the code on my repo? now it takes normal time 5-7 minutes from dagger, I think that was something network releated becasue my internet was too slow those days

deep trout Oct 27, 2023, 10:22 AM

#

old salmon <@903784691939819580> do you figure this is the same issue? https://github.com/d...

this definitely isn't the same issue: https://github.com/dagger/dagger/issues/5791 doesn't actually cause dagger to behave slower - it's not a performance issue. the issue is that logs don't appear as soon as they could possibly do, they're delayed as we try and scrape secrets from them.

the issue in this thread seems more like a caching issue: from a clean setup, the local gradle command actually takes 4m26s - which is much longer than in the README.

to get the docker build to be faster on subsequent runs (since you're using the dockerBuild helper https://github.com/danny270793/Jmix-Dagger/blob/master/ci/dagger/src/index.ts), you'll want to use a cache-mount to ensure that these cached resources are used next time: https://docs.docker.com/build/cache/#use-the-dedicated-run-cache. you can also convert your dagger pipeline to directly create this - see WithMountedCache: https://docs.dagger.io/quickstart/635927/caching/

#Dagger execution taking to long