#Building multi-arch docker containers with cross-compilation

1 messages · Page 1 of 1 (latest)

left silo
#

Hello, at my company we have been using dagger for quite some time. In my project, one part is to build docker containers, and recently there appeared a need for multi-arch containers.

I did successfully manager to use https://docs.dagger.io/cookbook/#build-multi-arch-image with natural emulation, however in one case the emulation is not good enough. So I tried https://docs.dagger.io/cookbook/#build-multi-arch-image-with-cross-compliation but that did not work for me.

I did install the containerd module, but when I try to use it, Python can't find it.

AttributeError: module 'dagger' has no attribute 'containerd'. Did you mean: 'Container'?

I think it is because I am not using dagger to execute the script as module, but I call the script directly (aka python .dagger-ci/daggerci/main.py -d <DOCKERFILE>). I do not understand how dagger modules work, I have never touched that stuff.

The project is open-source and here is the offending code: https://github.com/9elements/firmware-action/blob/feat/cross-compile/.dagger-ci/daggerci/lib/orchestrator.py#L329

Can someone tell me please what am I doing wrong?

pearl hull
#

Hi @left silo 👋 To use a module like containerd, you basically have to use modules for the whole stack. There's no mixing modules and non-modules today. I can try to help solve this without modules, or we can figure out if it's worth it to migrate your project to use modules all the way.

Here's some info on how modules work https://docs.dagger.io/features/reusable-modules

Here's the github issue to support mixing modules with non-modules https://github.com/dagger/dagger/issues/5993

left silo
#

Oh I see, thank you, that is good to know.

#

My question then would be - is using containerd the only way to get cross-compilation to work?

pearl hull
#

No, it will be easy to use the core API to accomplish this. As an aside, all modules are functions written with the core API, we just have to reimplement it in your code.

It looks like the containerd module just parses a platform string (like "linux/arm64") into a platform type. So it's actually the same as the cross-complication cookbook example that uses emulation. When you said that did not work for you, was it a performance issue or an issue with the code?

left silo
#

From my experiments is seems that the emulation does not emulate all of the necessary instructions required by the application. Specifically, I am trying to compile compilers and tool-chains and they fail with odd errors (like segmentation faults and so on).

I did run the exact same process on both x86 and arm64 machines and I can confirm that the building process works as long as I am building for a native architecture. But when I try to build container (for example on x86) for the other architecture (for example arm64) then it fails.

#

If containerd does the emulation anyway, then I guess it is pointless to try use it 😦

runic sequoia
#

@left silo yeah.. unfortunately the only way to do cross compilation is either by native toolchain support or emulation. IIUC the tools you're trying to build don't support native cross-compile and since emulation doesn't work, apparently the only way would be running the pipeline in a machine with the target architecture

#

Out of curiosity, what's the project that you were getting weird errors to do emulated cross compilation?

left silo
#

I though it was possible to replace emulation with virtualization (simulation).

#

The projects for which I tried to build the tool-chains are edk2 and coreboot.

runic sequoia
left silo
#

Yes, that is what I meant. It would solve the CPU architecture issue, but would be super slow.

runic sequoia
left silo
# runic sequoia Not sure I follow. VMs don't virtualize the underlying CPU architecture generall...

Not sure how it is done under the hood, but VM can run different architecture than the host machine. Not sure if it is some kind of native emulation.

Anyway, this was dead end anyway. I though that containerd dagger modules was the solution, but I was mistaken.

I suppose at this point the only way to actually build multi-arch containers in my usecase is to use different machines with different CPU architectures to build their respective builds and then to merge these together into single manifest.

runic sequoia
# left silo Not sure how it is done under the hood, but VM can run different architecture th...

Not sure how it is done under the hood, but VM can run different architecture than the host machine. Not sure if it is some kind of native emulation.

Limitations on Cross-Architecture VMs: Running a VM on a different CPU architecture than the host (e.g., an ARM VM on an x86 host) is generally not feasible with standard virtualization technologies. This is because VMs rely on the underlying hardware's instruction set and architecture. As such, VMs cannot directly utilize resources from an incompatible architecture without significant overhead or emulation, which can lead to performance degradation.

https://www.perplexity.ai/search/can-vms-run-with-a-different-c-4_aeaGHUQ.SUS1c75cgi8Q

Where I'm going with this is that if you manage to make it work with VM's it should also work with Dagger

#

I suppose at this point the only way to actually build multi-arch containers in my usecase is to use different machines with different CPU architectures to build their respective builds and then to merge these together into single manifest.

Yes, that seems to be way in this case.

Having said that, I'd still check if there's a public issue about edk2 and coreboot not working with qemu as it'd be awesome if you could cross-compile in the same machine 🙏

left silo
#

I would assume full-blown VM has more options than container regarding CPU architecture.
And yeah, I was aware that it would be slow as heck if I chose to not emulate but simulate (not sure if that is the correct word)

left silo
#

... at least I think ...

#

I am getting lost in this.

So I have x86 machine. And I want to build arm container with dagger. So I am using a native compiler (arm) on emulated arm machine to compile the toolchain.

runic sequoia
left silo
#

I suppose if I instead used x86 compiler to cross-compile the toolchain for arm on x86 machine and then just put it into arm container ... that might work ???

runic sequoia
#

so you can cross-compile to arm from the x86 compiler?

left silo
#

I don't know 😄

runic sequoia
#

some compilers allow it

#

like the Go compiler, for example .Allows you to compile to arm from x86

#

so it's defienitely worth checking that out

#

that's actually how we do Go cross-compilation generally. We compile the binarny native CPU architecture, and then we just put the resulting binary in the proper container architecture image

left silo
#

Yeah yeah, but this is kinda complicated 😄
I will ask my colleague who is expert on coreboot.

#

I have never seen someone cross-compiling the coreboot toolchain which itself allows for cross-compilation 😄

#

Should be doable

left silo
#

So the cross-compilation is rather complicated because of missing dependencies (namely zlib-aarch64 which is in AUR for ArchLinux, and gcc with ada support - don't know where to get that one).

So to save myself from this headache I decided to go at this from another angle. I am building the toolchains separately and respective x86 and arm64 runners and no longer as part of the Dockerfile. The get downloaded and copied into the container.

pearl hull
#

I've also run into this issue. Some distros are pretty lacking on the arm64 package repos. In those cases I've had to either download the binaries from another place or build from source

left silo
#

Yeah, I also thought about that, but who knows how many packages I would have to make and maintain from there on (because this build setup is not one-off)