What is the best way to cache the pip cache? | Dagger | Page 1

upper fiber · 2022-11-12T08:04:20.180Z

I tried to do `.with_mounted_directory("/root/.cache/pip", cache_id)` but that doesnt seem to work. Is the mount read only? If so can I make it read-write? Thanks

Hi @upper fiber 👋 You want to use Cache Mounts. Here's an example:

"""
Run tests for a single Python version.
"""

import time
import sys
import anyio
import dagger


async def test(repo_url: str):
    config = dagger.Config(log_output=sys.stderr)
    async with dagger.Connection(config) as client:
        repo = client.git(repo_url)

        # get reference to the project's directory
        src_id = await repo.branch("master").tree().id()
        cache_id = await client.cache_volume("pip_cache").id()
        t = time.time()
        python = (
            client.container()
            .from_("python:3.10-slim-buster")
            # mount cloned repository into image
            .with_mounted_directory("/src", src_id)
            # set current working directory for next commands
            .with_workdir("/src")
            # use epoch time to always bust layer cache
            .with_env_variable("CACHEBUSTER", t)
            # use pip cache for installs
            .with_mounted_cache("/root/.cache", cache_id)
            # install one new package to show rest are cached
            .exec(["pip", "install", "kombu"])
            # install test dependencies
            .exec(["pip", "install", "-e", ".[test]"])
            # run tests
            .exec(["pytest", "tests"])
            # uninstall new package to remove from cache
            .exec(["pip", "uninstall", "--yes", "kombu"])
        )

        # execute
        await python.exit_code()

        print("Tests succeeded!")


if __name__ == "__main__":
    # Using the popular FastAPI library as an example
    anyio.run(test, "https://github.com/tiangolo/fastapi")

Note the cache_id used by with_mounted_cache.

If you comment out the CACHEBUSTER line, you'll get layer caching after the first run, but the cache buster simulates a change in the underlying code (mounted in a line a bit above it), which will likely happen a lot in real life. The install of thekombu package represents changes in the list of packages to install, which will happen, but we want to see the majority of packages cached since they won't change as often.

#What is the best way to cache the pip cache?