#Has dagger been tested much on ARM macs

1 messages · Page 1 of 1 (latest)

dry zephyr
deft sorrel
#

Yes, I'd say it has been reasonably tested since multiple devs in the dagger project are using M1/2 macs

dry zephyr
#

That's what I assumed. sorry, the way I phrased my question made it sound like I presumed it wasn't. Not my intention! 🙈

deft sorrel
#

no worries! no hurt feelings 🙂

dry zephyr
#

so there shouldn't be anything we need to include or look out for to ensure that they can also run this pipeline locally? I'm thinking of the --platform arg in the docker CLI, for example.

deft sorrel
#

Dagger automatically detects and tries to use the platform based on your local setup. For M1's, that'd be aarch64/armvX. You just need to make sure that the images that you're using have support for that architecture

#

it'd be awesome if you could share a simple repro example we can run in a Mac

dry zephyr
#

It's a big pipeline but you should be able to clone this and just run go run ./cmd --verbose package

https://github.com/grafana/grafana-build

This is the image in question I think: https://hub.docker.com/layers/library/node/18.12.0/images/sha256-16258bac1617ed4456e7f540c4468478fd7341ffeed478ffba1920013efc1348?context=explore

which has arm/v7 and arm64/v8

GitHub
GitHub - grafana/grafana-build: GitHub actions and packages for bui...

GitHub actions and packages for building Grafana. Contribute to grafana/grafana-build development by creating an account on GitHub.

dry zephyr
#

I wanted to give a smaller example but everyone who was helping me on their ARM computers have now logged off for the day 😦

deft sorrel
#

np, that works

#

summoning someone with a mac here. cc @trim sapphire @viral burrow @rough coyote @nimble quest 🙏

rough coyote
#

Testing now!

deft sorrel
rough coyote
#

So it ran fine on my machine (M1 pro), but it did fail for a different reason:

#3 ERROR: process "docker-entrypoint.sh yarn run build" did not complete successfully: exit code: 129
2023/03/08 13:15:47 input:1: container.from.withDirectory.withWorkdir.withExec.withExec.directory process "docker-entrypoint.sh yarn run build" did not complete successfully: exit code: 129

...

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
#

so it was able to run the node image at least

dry zephyr
#

same, and for my coworker he saw a bunch of buildkit panics and the program never exited

deft sorrel
dry zephyr
#

I think the nodejs heap limit is more likely. I've seen that more than once with the Grafana frontend

#

in fact I think we manually increase it in our Dockerfile

#

so maybe my coworker's docker version is out of date by a lot or something?

deft sorrel
#

I'd say it might be fixable by increasing the docker for mac VM ram size

rough coyote
# 
> docker version
Client:
 Cloud integration: v1.0.29
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:01:18 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.14.1 (91661)
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 17:59:41 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.9
  GitCommit:        1c90a442489720eec95342e1789ee8a5e1b9536f
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
rough coyote
#

I have it at 8gb right now btw

deft sorrel
dry zephyr
#

interesting. At least this shows me that there might be something weird with his setup! Are there known requirements that maybe he's missing? I asked him for his docker version output but I might not get that from him until tomorrow.

rough coyote
#

Failed again without even using 50% of the VM's memory

deft sorrel
#

seems like something that needs to be tuned indeed

rough coyote
#

Yeah, is it possible that node has a different default heap size based on arch?

dry zephyr
deft sorrel
#

maybe for whatever reason the arm node runtime uses more memory and it's hitting that max old space limit? All the google results talk about bumping that flag

viral burrow
#

Trying on my Mac M1 also

dry zephyr
# 
time="2023-03-08T17:13:14Z" level=error msg="/moby.buildkit.v1.frontend.LLBBridge/StatFile returned error: rpc error: code = Unknown desc = process \"docker-entrypoint.sh yarn run build\" did not complete successfully: exit code: 129"
process "docker-entrypoint.sh yarn run build" did not complete successfully: exit code: 129
1 0.0.0+unknown /usr/local/bin/dagger-engine --debug --config /etc/dagger/engine.toml --oci-worker-binary /usr/local/bin/dagger-shim
github.com/moby/buildkit/executor/runcexecutor.exitError
/go/pkg/mod/github.com/moby/buildkit@v0.11.0-rc3.0.20230222163458-34a576c411ea/executor/runcexecutor/executor.go:360
github.com/moby/buildkit/executor/runcexecutor.(*runcExecutor).Run
/go/pkg/mod/github.com/moby/buildkit@v0.11.0-rc3.0.20230222163458-34a576c411ea/executor/runcexecutor/executor.go:334
github.com/moby/buildkit/solver/llbsolver/ops.(*ExecOp).Exec
/go/pkg/mod/github.com/moby/buildkit@v0.11.0-rc3.0.20230222163458-34a576c411ea/solver/llbsolver/ops/exec.go:360
github.com/moby/buildkit/solver.(*sharedOp).Exec.func2
/go/pkg/mod/github.com/moby/buildkit@v0.11.0-rc3.0.20230222163458-34a576c411ea/solver/jobs.go:901
github.com/moby/buildkit/util/flightcontrol.(*call).run
/go/pkg/mod/github.com/moby/buildkit@v0.11.0-rc3.0.20230222163458-34a576c411ea/util/flightcontrol/flightcontrol.go:121
sync.(*Once).doSlow
/usr/local/go/src/sync/once.go:74
sync.(*Once).Do
/usr/local/go/src/sync/once.go:65
runtime.goexit
/usr/local/go/src/runtime/asm_arm64.s:1172

🤔

viral burrow
#

similar to @rough coyote 's

Stderr:
:0x17d82950]    55701 ms: Mark-sweep 1965.7 (2094.5) -> 1965.7 (2089.7) MB, 586.6 / 0.0 ms  (average mu = 0.094, current mu = 0.026) allocation failure; scavenge might not succeed
[52:0x17d82950]    56322 ms: Mark-sweep 1973.2 (2095.2) -> 1973.0 (2091.7) MB, 601.0 / 0.0 ms  (average mu = 0.063, current mu = 0.033) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb5eb74 node::Abort() [webpack]

Wonder if this related. Going to look at code:
https://github.com/nodejs/help/issues/2388#issuecomment-570881808

GitHub
Allocation failure scavenge might not succeed · Issue #2388 · nodej...

Node.js Version: v10.16.0 OS: macOS 10.15.2 (19C57) Scope (install, code, runtime, meta, other?): Runtime/Code Module (and version) (if relevant): Unsure I'm getting a crash on Node.js. It ...

rough coyote
dry zephyr
#

It makes sense then that he ran into the same bug then!

The log format he sent me looks a lot different than mine. Is it possible that the container is failing for the same reason but I'm looking at the buildkit logs or something?

#

guess I'll have to ask tomorrow.

dry zephyr
rough coyote
#

here's a snippet of my engine logs for the same time:

2023-03-08 13:23:21 time="2023-03-08T18:23:21Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = process \"docker-entrypoint.sh yarn run build\" did not complete successfully: exit code: 129"
2023-03-08 13:23:21 process "docker-entrypoint.sh yarn run build" did not complete successfully: exit code: 129
2023-03-08 13:23:21 1 0.0.0+unknown /usr/local/bin/dagger-engine --debug --config /etc/dagger/engine.toml --oci-worker-binary /usr/local/bin/dagger-shim
2023-03-08 13:23:21 github.com/moby/buildkit/executor/runcexecutor.exitError
2023-03-08 13:23:21     /go/pkg/mod/github.com/moby/buildkit@v0.11.0-rc3.0.20230222163458-34a576c411ea/executor/runcexecutor/executor.go:360
2023-03-08 13:23:21 github.com/moby/buildkit/executor/runcexecutor.(*runcExecutor).Run
2023-03-08 13:23:21     /go/pkg/mod/github.com/moby/buildkit@v0.11.0-rc3.0.20230222163458-34a576c411ea/executor/runcexecutor/executor.go:334
2023-03-08 13:23:21 github.com/moby/buildkit/solver/llbsolver/ops.(*ExecOp).Exec
2023-03-08 13:23:21     /go/pkg/mod/github.com/moby/buildkit@v0.11.0-rc3.0.20230222163458-34a576c411ea/solver/llbsolver/ops/exec.go:360
2023-03-08 13:23:21 github.com/moby/buildkit/solver.(*sharedOp).Exec.func2
2023-03-08 13:23:21     /go/pkg/mod/github.com/moby/buildkit@v0.11.0-rc3.0.20230222163458-34a576c411ea/solver/jobs.go:901
2023-03-08 13:23:21 github.com/moby/buildkit/util/flightcontrol.(*call).run
2023-03-08 13:23:21     /go/pkg/mod/github.com/moby/buildkit@v0.11.0-rc3.0.20230222163458-34a576c411ea/util/flightcontrol/flightcontrol.go:121
2023-03-08 13:23:21 sync.(*Once).doSlow
2023-03-08 13:23:21     /usr/local/go/src/sync/once.go:74
2023-03-08 13:23:21 sync.(*Once).Do
2023-03-08 13:23:21     /usr/local/go/src/sync/once.go:65
2023-03-08 13:23:21 runtime.goexit
2023-03-08 13:23:21     /usr/local/go/src/runtime/asm_arm64.s:1172
dry zephyr
#

hahaha no way

rough coyote
#

looks like the same output 😂 so I guess your teammate was looking at those logs instead of the build stdout?

dry zephyr
#

i wonder why he sent me that instead of the program's stdout

#

🙃 well there it is. I'll ask him about that tomorrow then. thanks for running it on your macbooks for me! sorry about the red herring hahaha

rough coyote
#

no problem! curious to hear why they saw those logs and also the solution for the heap problem 🙂

dry zephyr
#

me too! i'll let you know

deft sorrel
#

@rough coyote have you tried bumping the Dockerfile limit to see if that fixes it?

#

since Kevin also doesn't have a mac 😄

dry zephyr
#

I think I can add that real quick.

rough coyote
#

testing

viral burrow
#

@dry zephyr how could I test this with a prior version of grafana/grafana repo?

dry zephyr
#

you can provide a git ref via the --version arg I believe. I haven't actually tested that yet though.

Example, go run ./cmd --verbose --version=v9.4.4 package

If you have a cloned Grafana repo you can also provide the path to it as the last argument, go run ./cmd --verbose package ./grafana

#

is the main branch broken? lol

#

because we've been so happy lately about our 95% build success rate lately! up from like 60% two weeks ago

rough coyote
#

passed 🎉

#3 342.2 Entrypoints:
#3 342.2   app (7.58 MiB)
#3 342.2       runtime.80b03277d6646478b33a.js
#3 342.2       1670.060c741b60f5333e0569.js
#3 342.2       9767.38f7498988ef027ab6ce.js
#3 342.2       1020.d68b113f55c9b2367c1b.js
#3 342.2       6418.cbe9393b622ff6e08caa.js
#3 342.2       639.923973d791450a8d491f.js
#3 342.2       app.fe274256335372a86ce1.js
#3 342.2
#3 342.2 webpack 5.75.0 compiled with 2 warnings in 67997 ms
➜  grafana-build git:(main)
> echo $?
0
deft sorrel
#

🚀

dry zephyr
#

🎉

deft sorrel
#

@dry zephyr I saw in the internet that some people add that flag to the package.json instead of GHA or Dockerfiles. That way developers running npm build in their machines won't trip with the same issue

viral burrow
dry zephyr
dry zephyr
viral burrow
#

seems to be running
go run ./cmd --verbose --version=4a1c18abf64046a28a214dca18c86dc2d9aeacc2 package

lots of messages that suggest why Cache was turned on perhaps 🤣

w3c-xmlserializer@npm:2.0.0 can't be found in the cache and will be fetched from the remote registry
#3 216.5 ➤ YN0013: │ w3c-hr-time@npm:1.0.2 can't be found in the cache and will be fetched from the remote registry
...

but yeah, same Heap exhaustion issue

<--- Last few GCs --->

[52:0x1ff6d950]    52762 ms: Mark-sweep 1953.4 (2090.9) -> 1953.4 (2085.1) MB, 279.2 / 0.0 ms  (average mu = 0.185, current mu = 0.188) allocation failure; scavenge might not succeed
[52:0x1ff6d950]    53422 ms: Mark-sweep 1984.8 (2116.3) -> 1984.8 (2110.0) MB, 640.3 / 0.0 ms  (average mu = 0.093, current mu = 0.030) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb5eb74 node::Abort() [webpack]
 2: 0xa81dc0 void node::FPrintF<>(_IO_FILE*, char const*) [webpack]
dry zephyr
#

nice!

#

the intention with this repo is it should be able to build/package any version of Grafana instead of us having it all defined alongside the code, which has been really tedious.

getting it working with github actions was super easy, was very excited to see it running with very little effort.

dry zephyr
deft sorrel
dry zephyr