#using poetry to build and publish package

1 messages · Page 1 of 1 (latest)

remote cloud
#

I am using poetry within the pipeline to build and upload the python package I created. I tried it locally and it works fine. Within the pipeline, I grab the PyPi token via an env and that is also (apparently) working. Afterwards, I use poetry config to configure this token. No problems here either. But than, when I use poetry publish --build it fails, telling me problems with the authentication.

Anyone an idea what that could be?

heavy inlet
#

cc @hushed frigate since he's our python guru 🧙

remote cloud
#

Here the idea how I pictured it:

"""Pipeline for building and publishing python package"""
import os
import sys

import anyio
import dagger
from dotenv import load_dotenv


async def build_and_publish_python_package():
    """"""
    async with dagger.Connection(dagger.Config(log_output=sys.stdout)) as client:
        load_dotenv()
        secret = client.set_secret("PYPI_TOKEN", os.environ.get("PYPI_TOKEN"))

        runner = (
            client.container()
            .from_("python:3.10-slim-buster")
            .with_directory(
                path=".",
                directory=client.host().directory((".")),
                exclude=[".venv", ".vscode"],
            )
            .with_secret_variable("PYPI_TOKEN", secret)
            .with_exec(["pip", "install", "poetry"])
            .with_exec(["poetry", "install"])
            .with_exec(["poetry", "config", "pypi-token.pypi", "$PYPI_TOKEN"])
        )

        build_package = await runner.with_exec(
            ["poetry", "publish", "--build"]
        ).stdout()

        print(build_package)


anyio.run(build_and_publish_python_package)
hushed frigate
#

Hi @remote cloud. with_exec doesn't expand env vars by default, you need to wrap the command in a shell for that (sh -c "....") but I actually prefer using env vars to configure poetry instead of writing to disk like with poetry config:

Analyzing your config, allow me the following tips:

  • Set the POETRY_PYPI_TOKEN_PYPI env var which is directly picked up by Poetry instead, without the need for poetry config;
  • Move the dir exclude from Container.with_directory to Host.directory, otherwise you're still uploading them to dagger;
  • Put your files in a subdir of /—which is the default workdir for that docker image—to avoid any conflicts;
  • Reorder your steps so that the thing you change more often comes later, like in a Dockerfile;

All of this together:

        runner = (
            client.container()
            .from_("python:3.10-slim-buster")
            .with_exec(["pip", "install", "poetry"])
            .with_secret_variable(
                "POETRY_PYPI_TOKEN_PYPI",
                client.set_secret("pypi_token", os.getenv("PYPI_TOKEN")),
            )
            .with_directory(
                "/src",
                client.host().directory(".", exclude=[".venv", ".vscode"]),
            )
            .with_workdir("/src")
            .with_exec(["poetry", "install"])
        )

Note: that's from top of mind, didn't test it but the approach is sound.

remote cloud
#

Thank you a lot! This makes so much sense on so many levels 😄 - I just started with dagger and I like the general idea a lot. Everything you just recommended is something I wanted to take a look into step by step :). Even better with your example!.

Btw.: I exclude the .venv folder, would you think that it makes sense to use a cache for that? It could improve the speed, wouldn't it?

remote cloud
#

Worked like a charm 🙂

hushed frigate
#

Yeah, you can add a cache volume mount for pip cache and poetry cache here. It'll help not having to re-download the packages from pypi if poetry install executes again (which is easy if you change anything in your source code). It'll still create the venv and install all the packages, but they'll be collected from cache instead of re-downloaded. You can also use a cache mount for the venv itself like you said, but you need to set the initial dir first and set it as base for the mount (source argument: https://dagger-io.readthedocs.io/en/sdk-python-v0.8.4/client.html#dagger.Container.with_mounted_cache).

For now, to keep it simple, just adding the cache mounts is an improvement:

        runner = (
            client.container()
            .from_("python:3.10-slim-buster")
+           .With_mounted_cache("/root/.cache/pip", client.cache_volume("pip_cache")).
            .with_exec(["pip", "install", "poetry"])
+           .With_mounted_cache("/root/.cache/poetry", client.cache_volume("poetry_cache")).
            .with_secret_variable(
                "POETRY_PYPI_TOKEN_PYPI",
                client.set_secret("pypi_token", os.getenv("PYPI_TOKEN")),
            )
            .with_directory(
                "/src",
                client.host().directory(".", exclude=[".venv", ".vscode"]),
            )
            .with_workdir("/src")
            .with_exec(["poetry", "install"])
        )
remote cloud
#

Great idea 🙂 - thanks for the hint. I have so many more questsions. But let's start simple first 😉

remote cloud
#

Again, worked like a charm - and the speed is drastically increased 🙂