#do you have tips to create a small python app image ?

1 messages · Page 1 of 1 (latest)

mild quail
#

Hi, I'm building a product and I'm giving a try to dagger (it's AWESOME !)
I'm using poetry as dependencies manager, the issue is it bloat the size image, so I'm doing multistage build: a container which install poetry (from pip) and export a classic requirements.txt, then in a fresh container I use pip as normally

#

do you have any more tips ?

#

also does distroless images are a good idea for production ?

glossy pawn
#

Hi! I have a few suggestions. What does your production image need to run? Web framework, cli run...?

mild quail
#

@glossy pawn Hi, didn't saw your message sorry 😅

#

a all in on webserver + admin cli

#

also i'm targeting ["linux/amd64", "linux/arm64", "linux/arm/v7"] as platform, but some python libs requires building from source 💀

mild quail
#

finally I'm using poetry in the container (to speed up deps installation)

# 

async def build_yauks(
    client: dagger.Client,
    platform: str,
) -> dagger.Container:
    """Build the containerized version of the app."""

    backend_path = get_project_path() / "backend"
    frontend_dist = await build_frontend_dist(client)
    system_deps = [
        "git",
        "curl",
        "gcc",
        "musl-dev",
        "jpeg-dev",
        "zlib-dev",
        "libffi-dev",
        "cairo-dev",
        "pango-dev",
        "gdk-pixbuf-dev",
        "cargo",
    ]

    commands = [
        ["apk", "add"] + system_deps,
        ["pip", "install", "poetry==1.4.2"],
        ["poetry", "config", "virtualenvs.create", "false", "--local"],
        ["poetry", "install", "--only", "main"],
        ["pip", "uninstall", "-y", "poetry"],
        ["apk", "del"] + system_deps,
        ["rm", "-rf", "/var/cache/apk/*"],
        ["rm", "-rf", "/root/.cache/"],
        ["rm", "-rf", "/usr/local/src/*"],
    ]

    commands_as_str = " && ".join([" ".join(cmd) for cmd in commands])

    ct = (
        client.container(platform=dagger.Platform(platform))
        .from_("python:3.11.3-alpine3.17")
        .with_label("maintainer", "Yauks")
        .with_label("platform", platform)
        .with_workdir("/app")
        .with_env_variable("PYTHONUNBUFFERED", "1")
        .with_env_variable("PYTHONDONTWRITEBYTECODE", "1")
        .with_env_variable("TZ", "UTC")
        .with_env_variable("CARGO_NET_GIT_FETCH_WITH_CLI", "true")
        .with_directory(".", client.host().directory(str(backend_path), include=["pyproject.toml", "poetry.toml"]))
        .with_exec(["sh", "-c", commands_as_str])
        .with_directory(
            ".", client.host().directory(str(backend_path), exclude=[".venv", ".vscode", ".ruff_cache", "__pycache__"])
        )
        .with_directory("frontend-dist", frontend_dist)
    )

    return ct
glossy pawn
#

I want to make an example for this use case, in the meantime here's some quick tips:

  • If you turn that into a multi-stage build (https://docs.dagger.io/544174/multistage-build), you don't need to worry about cleaning up;
  • Set a VIRTUAL_ENV=/opt/venv env var to a known place and add it to the PATH=${VIRTUAL_ENV}/bin:${PATH}, this is all you need to activate and use a virtual environment, Poetry will respect it as well (https://pythonspeed.com/articles/activate-virtualenv-dockerfile/, #1083752578032095263 message)
  • Add a cache mount to pip's cache so you don't install the whole thing on any change to pyproject and the lock file
  • Install poetry globally (root) and the project packages with --user, this way poetry won't need to be removed from /opt/venv later
  • Copy /opt/venv from the build container into the run container
  • Only add your app's files to the run container, since build only needs pyproject and the lock file
  • If you have a .dockerignore that you want to reuse, you can probably read it, split the lines and send that to exclude=
mild quail
#

Thanks for the tips thanks

#

Another thing, a friend complained about it need "too much dependancies where I just want to do docker build"

#

The only way to solve this and keep dagger is to create an image with dagger inside

#

Whichh sound bad ?

glossy pawn
#

You mean running the python dagger script in a container?

mild quail
#

yes, that sound a bad idea

#

but it's the only way to allow using docker run, I think ?

mild quail
#

hmmm

#

Error relocating /usr/local/bin/python: Py_BytesMain: symbol not found

#

copying venv is probably not safe as copying a binarie

glossy pawn
glossy pawn
glossy pawn
# mild quail copying venv is probably not safe as copying a binarie

I've done it quite a lot, so it depends on what you're doing. Here's an old Dockerfile of mine:

# syntax=docker/dockerfile:1
FROM registry.gitlab.com/xxx/fastapi:3.8-build as python-build

COPY pyproject.toml poetry.lock ./
RUN --mount=type=cache,target=/root/.cache poetry-install

FROM registry.gitlab.com/xxx/fastapi:3.8

COPY --from=python-build /opt/venv /opt/venv
COPY --chown=app:app app /app/app
COPY --chown=app:app static /app/static
COPY --chown=app:app templates /app/templates
...
mild quail
#

I don't need it, but a friend complained about "creating my own cli/custom script in order to build oci images is reinventing the wheel"

glossy pawn
mild quail
#

also, for now there is no way to do local dev with hmr right ?

#

I mean, in the container I would like to run a long running process (uvicorn --reload as example)

#

but from what I understood volumes binds are not yet(?) a feature

#

so I must have to export the image (either on fs host or registry) and load/fetch it and use docker directly ?

glossy pawn
#

Yep, dagger is more focused on CI like tasks. There's workarounds you can do but I'm not sure it'll be worth it. I know others have wanted it but I don't know if anyone's done it successfully. I remember doing web dev with my team using docker compose, but the difference in macos vs linux with user permissions in the bind mounts pushed us to use a tool that integrated with compose to synchronize a named volume with the host disk (basically a side car container with a daemon doing the sync). I can't remember the name of the tool.

glossy pawn
mild quail
#

I have a nuxt3 app for the frontend with HMR, I just would like to have hmr working when I edit somes files, same for the backend (fastapi behind)

#

splitting the build (of dev images, npm install as example) and the (dev, npm run dev) run is probably unavoidable

glossy pawn
#

You can watch the files and trigger a pipeline run that rebuilds the image, deploys to your local docker engine and runs the image for you. So there's a lot of automation you can do in dagger.

mild quail
#

I already tried with watchfiles (from samuelcolvin), for the backend it would work not so bad

#

but doing a full rebuild for the frontend is not optimal, it will rebuild the whole frontend app builde instead of patching only what it need

#

and I also need to access to the http dev servers from the host

#

so I must use a trick like some ssh port forwarding or network hack

glossy pawn
mild quail
#

for frontend nope

#

I mean, vite will have ~3s of coldstart as it need to look if any file have change