#Integration with HPC: converting Docker image to Singularity/Apptainer

1 messages · Page 1 of 1 (latest)

oblique pasture
#

Hi, I am currently developing a Dagger CI pipeline to build, test, and push Docker images that will be used on HPC.
It is very common for HPC centers to provide Singularity/Apptainer instead of Docker, due to better security guarantees and improved integration with the SW stack provided by the HPC.

At the moment, every time I build a new Docker image, I pull it on the HPC and convert it to a Singularity image. Example: singularity pull ubuntu.sif docker://ubuntu:latest
The command above pulls (quick) and converts (slow) the Docker image to a SIF (Singulartity Image File).

The worst part is when this operation is done within a job allocation, as some compute time will be "wasted" to do the conversion. To streamline the process, I would like to perform the Docker to SIF conversion in my CI pipeline managed by Dagger... Has any of you ever done that?

Naively I would look for a Docker container with Singularity installed and run the conversion inside it, having the whole process managed by Dagger... However, | was wondering whether this would be the best way to go. Also, that's just an intuition which I never tested, and it may not even work.

Also, I don't want to open the Pandora's box here 😂 , but have you ever considered some degree of support for Singularity/Apptainer in Dagger?

hybrid ether
#

for the hpc-docs link, you can do docker-in-dagger to replicate this within a dagger pipeline

oblique pasture
#

Thank you @hybrid ether ! I'll give it a try. When you say "docker-in-dagger" you mean translating the docker run and the mounts to Dagger functions?

hybrid ether
#

something like that, like there is docker-in-docker, there is docker-in-dagger (dind image) and even dagger-in-dagger

you can also mount the docker socket and use the docker cli in a container to interact with the docker daemon dagger is running on (which is probably what you want)

#

seems like you want to run the docker2singularity image in this setup, because it looks to interact with the docker daemon directly

#

but yea, I would try to convert their docker run example into dagger code

oblique pasture
#

I see, thanks for the clarification

oblique pasture
#

I was able to write this, which is still somehow incomplete though

    @function
    async def singularity(
        self, container: str, output_dir: dagger.Directory, socket: dagger.Socket
    ) -> str:
        return await (
            dag.container()
            .from_("quay.io/singularity/docker2singularity")
            .with_mounted_directory(path="/output", source=output_dir)
            .with_unix_socket(path="/var/run/docker.sock", source=socket)
            .with_exec(
                [
                    "bash",
                    "-c",
                    f"/docker2singularity.sh --name /output/container.sif {container}",
                ],
                insecure_root_capabilities=True,
            )
            .stdout()
        )

Called as

dagger call singularity --container "python:3.12" --output-dir "$PWD" --socket /var/run/docker.sock

When called following this example the conversion is successful, whereas the dagger CLI results in a file of ~6MB which doesn't look like the expected SIF. Any idea?

Also, do you know if I could instantiate the dagger Socker inline with something like dagger.Socket("/var/run/docker.sock")?

dusk ether
#

@oblique pasture does docker2singularity support loading the docker image from an OCI archive file, or a registry, instead of from a docker engine? That would allow removing the docker engine from the loop - definitely the most heavyweight component in all this

#

@oblique pasture here's my little POC, let me know if it helps. It's using the experimental dagger shell feature, but it's easy to transcribe to code or dagger call - all standard Dagger API under the hood

oblique pasture
#

Hey @dusk ether , I can export to Singularity without using the Docker runtime. The built-in conversion script of quay.io/singularity/docker2singularity uses Docker engine, but I can simplify the conversion as singularity pull my_container.sif docker://image:tag.
However I wasn't able to export the generated file to the host... What am I doing wrong?

@function
    async def singularity(
        self, src_container: str, dst_container: str, output_dir: str
    ) -> str:
        """Convert Docker to Singulartiy/Apptainer"""
        return await (
            dag.container()
            .from_("quay.io/singularity/docker2singularity")
            .with_exec(
                [
                    "bash",
                    "-c",
                    f"singularity pull {dst_container} docker://{src_container}",
                ]
            )
            .file(dst_container)
            .export(path=(Path(output_dir) / dst_container).resolve().as_posix())
        )

This is called with

dagger call singularity --src-container "busybox" --dst-container sif.sif --output-dir "$PWD"
#

@dusk ether I think I didn't receive your POC 😅

dusk ether
dusk ether
#

@oblique pasture here's my little POC, using the experimental shell feature for fun:

#!/usr/bin/env dagger shell -q

.container |
from singularityware/singularity |
with-file img.tar $(.container | from alpine | as-tarball) |
with-exec singularity,build,img.sif,oci-archive://img.tar |
file img.sif
oblique pasture
#

Thank you! I will give it a try

brazen pawn
dusk ether
#

this will only print metadata about the file, because of a bug in the shell (can't export a file because it conflicts with the bash export, we're working on it)

#

@oblique pasture I converted it to a regular module real quick. Try:

dagger call -m github.com/shykes/x/singularity import --source=index.docker.io/nginx -o ./nginx.sif