Freeze requirements and output to file

[willow flume]

I can make a working example, but hopefully this gets the idea across, as I was ironing out

• build base image for mkdocs
• if requirements file is found install from that
• if not install from string array then persist a new requirements file.

How am I supposed to get the output from the ctr.Stdout(ctx) and persist. I've tried freeze with inline shell redirect and even capturing the string of the output but neither seem to persist out the requirements file.... pointer on what I'm doing wrong?

// Container is the mkdocs container with thea dditional python packages to install
func InstallPipPackages(ctx context.Context, ctr *dagger.Container, packages []string, dir *Directory) *dagger.Container {
    // support using requirements.txt to install, as can use for caching optimization.
    // however, if you delete it or it doesn't exist, it will install the required packages and then provide a newly generated requirements file for you.
    if _, err := os.Stat(qualifiedInternalRequirementsFile); os.IsNotExist(err) {
        fmt.Println("requirements not found, so installing packages from code first exists")
        for _, p := range packages {
            ctr = ctr.WithExec([]string{"python", "-m", "pip", "install", p}, dagger.ContainerWithExecOpts{
                SkipEntrypoint: true,
            })
        }

        ctr = ctr.WithExec([]string{"python", "-m", "pip", "freeze", ">", qualifiedInternalRequirementsFile}, dagger.ContainerWithExecOpts{
            SkipEntrypoint: true,
        })
    } else {
        fmt.Println("requirements file exists")
        ctr = ctr.WithExec([]string{"python", "-m", "pip", "install", "-r", qualifiedInternalRequirementsFile}, dagger.ContainerWithExecOpts{
            SkipEntrypoint: true,
        })
    }
    // for _, overlay := range overlays {
    //     ctr = ctr.WithDirectory("/", overlay.Rootfs())
    // }
    return ctr
}

second part

func (m *Devex) MKDocsBaseImageBuild(ctx context.Context, dir *Directory) (string, error) {

    packages := []string{
        "mkdocs-glightbox",
               // etc
    }
    ctr := dag.Container().From(qualifiedImageName).
        WithWorkdir(daggerInternalMkdocsWorkingDirectory).
        WithMountedDirectory("/docs", dir)
    ctr = ctr.WithEnvVariable("TINI_SUBREAPER", "true").
        WithEnvVariable("PIP_ROOT_USER_ACTION", "ignore")

    ctr = InstallPipPackages(ctx, ctr, packages, dir)
    output, err := ctr.Stdout(ctx)
    if err != nil {
        return "", err
    }
    if err := os.WriteFile(localRequirementsFile, []byte(output), permissionUserReadWriteExecute); err != nil {
        return "", fmt.Errorf("error writing requirements file: %w", err)
    }
    return output, nil
}

I

[scarlet lotus]

@willow flume you seem to be passing a shell command as arguments to withExec. That > redirect character seems out of place

If you want to use the shell redirect syntax >, you need to execute an actual shell, eg. withExec(["sh", "-c", "python -m pip freeze > bla.txt"])

Alternatively, you can keep withExec and pass the optional argument redirectStdout

See RedirectStdout in ContainerWithExecOpts: https://pkg.go.dev/dagger.io/dagger#ContainerWithExecOpts

This should do it:

ctr = ctr.WithExec(
 []string{"python", "-m", "pip", "freeze"},
 dagger.ContainerWithExecOpts{
  SkipEntrypoint: true,
  RedirectStdout: qualifiedInternalRequirementsFile,
 },
)

[willow flume]

i spent too long on this. I really appreciate the quick response. Helped a lot with quick MVP step by step.

[willow flume]

now that snippet and docs say that this is based on the path in the container, not local for redirect right? Asking because not suing results show up yet and trying to iron out why. Seems like it should be now and I provide the qualified path of /docs/requirements.txt to the command.

[scarlet lotus]

path is inside the container. either absolute, or relative to workdir at the time of exec

[willow flume]

ok, so the redirect seems to be mapped to the same directory but when I run nothing ever seems to persist.

I see it output that requirements should freeze but no file gets created. I've tried /docs/requirements.txt and requirements.txt and such to see if I can get it to show up from the mounted directory in case i had path wrong but nothing changes.

sorry lag...

here's what i have so far... maybe it's an issue with build?

just flipped back to the output on the function of (output string, err error) { and return return ctr.Stdout(ctx) where I'm still stuck

[scarlet lotus]

@willow flume one possibility is that you're writing the file to a mounted directory. Then the mount disappears, taking the file with it

[willow flume]

so when I tell it /docs/requirements.txt it might be reading this as a directory? I use Wd mounted to "/docs" in the container so I thought that would be considered a file in that path.

i guess i thought this was the equivalent of mounting a synced directory with -v ${PWD}:/docs in docker command

[scarlet lotus]

I didn't fully understand that. But what I mean is that in this scenario:

1. You mount directory /docs from somewhere else (using withMountedDirectory, withMountedCache or withMountedTemp)
2. You write a file inside /docs, for example /docs/requirements.tx, whether with redirectStdout, or other means
3. You later try to read the file, /docs/requirements.txt

It's possible that at this last step, the file is not there, because /docs has been unmounted

[willow flume]

ok, i'm looking into it then and also replaced dag.Directory() which was a test case i forget to remove with "dir"

[scarlet lotus]

Are you able to share the entire code? Or at least relevant parts?

[willow flume]

hmmm. So I build up all the exec commands in order. Seems correct and the output shows up but never seems to go to a file.

https://gist.github.com/sheldonhull/0b6a278a3cb418eafbd0cdcb05f3eef5

i also was trying to return it as a container type so I could have this invoke as required from the service stuff, but step by step 🙂

[scarlet lotus]

Do you have a Trace URL you could share by any chance? Would be even easier to debug

(ie. via Dagger Cloud)

[willow flume]

i haven't tried dagger cloud. this is in azure devops so do you have link to docs on how to do this if I've not connected any of this and am running completely locally?

i want to, but not something i can do with my company. would have to be a personal experiment 🙂

[willow flume]

Ok, so even if I have requirements.txt in the repo, it doesn't recognize it so maybe I'm doing something wrong with MountedDirectory expecting requirements.txt and repo contents to be there?

[willow flume]

I have logic that says above:

• if requirements.txt isn't found build exec commands for pip install based on string slice of packages.
• then persist to requirements.txt

I can't figure out 2 things....

• why requirements.txt which I manually created still doesn't get found by logic in here.... it's qualified to the /docs/requirements.txt which is the local working directory too and isn't found.
• how I can get the pip install exec commands to not rerun everytime when nothing has changed, essentially like docker layers would normally do with the RUN command.

make sense or should I record a clip to better explain?

[scarlet lotus]

how I can get the pip install exec commands to not rerun everytime when nothing has changed, essentially like docker layers would normally do with the RUN command.

This part should happen automatically. Container.withExec is exactly the same as a RUN dockerfile command

[willow flume]

any last tip on why mounted directory doesn't seem to make requirements.txt show up? If not no biggee i'll try and export a trace if i can't end up figuring it out

[willow flume]

ok, I'm certain it has to do with confusion from me on how I get my local directory synced similar to docker mounting a volume.

I've tried

• Directory(foo).Export
• WithMountedDirectory
• Directory

and even though in the container the mkdocs build is working, when I use dagger instead of my equivalent mage command it doesn't give me any output to my artifacts directory.

// Build generates all the artifacts to publish for the mkdocs site.
func (Mkdocs) Build() error {
    wd, err := os.Getwd()
    if err != nil {
        return err
    }
    return sh.RunV(
        "docker",
        "run",
        "--rm",
        "--env",
        "TF_BUILD=true",
        "-v",
        fmt.Sprintf("%s:/docs", wd),
        "dev.local/mkdoc:latest",
        "build",
        "--clean",
        "--verbose",
    )
}

That works... this or the other variations doesn't seem to persist anything

func (m *Devex) MkdocsBuild(ctx context.Context, dir *Directory) *Directory {
    
    return m.MkdocsBaseContainer(ctx, dir).
        WithWorkdir("/docs").
        WithMountedDirectory("/docs", dir). // 👈 .artifacts/_site should be dropped in the dir of my repo i'm running this from
        WithExec([]string{"mkdocs", "build", "--clean", "--verbose", "--site-dir", qualifiedInternalArtifactsSiteDir}, dagger.ContainerWithExecOpts{
            SkipEntrypoint: true,
        }).Directory(qualifiedInternalArtifactsSiteDir)
}

[scarlet lotus]

I think it's because what you write on the mount is not persisted after each exec. but thought Container.directory() did allow reading new files from the mount after they're created by withExec? @mossy latch am I saying something stupid? 🙏

[mossy latch]

Mounts are "sticky"; any changes you make to them during one withExec with be present in subsequent operations chained after

Basically, they act like mounts do on your filesystem. Different than mounts in Dockerfiles though which are ephemeral

[scarlet lotus]

That's what I thought - so, we have a mystery on our hands right? We're trying to understand why `Container.WithMountedDirectory("/docs").WithExec().Directory("/docs") doesn't work -> the file created by the exec doesn't seem to be there

[willow flume]

If it's better I'll convert this over to my public mkdocs repo for my blog. Nothing is sensitive really so I should be able to reproduce and demonstrate the issue by trying to build it there. Then you can tell me how bad my dagger code is 😆

[willow flume]

Ok, I've got it available for cloning now

git clone https://github.com/sheldonhull/digital-garden
dagger call mkdocs-service --dir ${PWD} up

That works... though I'm still not certain if I can specify serve on 127.0.0.1 and not 0.0.0.0 via dagger per #1235333292799033527

Next

mkdir .artifacts/
dagger call mkdocs-build --dir ${PWD} -v

I'd expect it to show up there... nothing though?

[scarlet lotus]

What kind of directory can I pass? Any python project?

[scarlet lotus]

This worked for me:

• dagger call -m github.com/sheldonhull/digital-garden mkdocs-service --dir=https://github.com/sheldonhull/digital-garden up
• browse https://localhost:8000

This works for me:

dagger call -m github.com/sheldonhull/digital-garden mkdocs-build --dir=https://github.com/sheldonhull/digital-garden

But I don't know what is supposed to be in that build directory, that isn't there

Is it a requirements.txt file?

$ dagger call -m github.com/sheldonhull/digital-garden mkdocs-build --dir=https://github.com/sheldonhull/digital-garden file --path=requirements.txt contents
✔ Garden.mkdocsBuild(
    dir: ✔ GitRef.tree: Directory! 0.0s
  ): Directory! 1.3s
┃ package listing: [mkdocs-glightbox mkdocs-rss-plugin mkdocs-autolinks-plugin mkdocs-git-revision-date-localized-plugin mkdocs-exclude
┃ kdocs-git-authors-plugin mkdocs-swagger-ui-tag mkdocs-glightbox markdown-callouts mkdocs-awesome-pages-plugin]
  ✔ Container.from(address: "squidfunk/mkdocs-material:latest"): Container! 0.7s
    ✔ HTTP GET 0.3s
    ✔ remotes.docker.resolver.HTTPRequest 0.3s
      ✔ HTTP HEAD 0.3s
  ✔ Container.directory(path: "/docs/.artifacts/site"): Directory! 0.3s
    ✔ cache request: git://github.com/sheldonhull/digital-garden#main 0.3s
✘ Directory.file(path: "requirements.txt"): File! 0.0s
! lstat .artifacts/site/requirements.txt: no such file or directory

Error: response from query: input: container.from.withMountedDirectory.withWorkdir.withEnvVariable.withEnvVariable.withExec.withWorkdir.withMountedDirectory.withExec.directory.file resolve: lstat .artifacts/site/requirements.txt: no such file or directory

[willow flume]

for that build output a reporoot/.artifacts/_site content should come out. I gave up on requirements.txt since apparently Exec() is supposed to be caching (not doing it like I expect but that's a separate issue I'll open up after the more important stuff is working)

i'd expect it to do what mage docker:build mkdocs:build does...

[willow flume]

⬆️ bump. I have detail above to reproduce and just need to figure out why i'm having problems persisting data out of my mounted directory. See screenshot above. That's what runs fine with the mage command I gave you above. Run that and then see the difference. I want to replace the mage command with dagger command instead.

[upper kettle]

Hey @willow flume - its not enough to return directory, you also need to export it

this

dagger call mkdocs-build --dir ${PWD} -v

should be this

dagger call mmkdocs-build --dir ${PWD} export --path .artifacts

[scarlet lotus]

But I think even listing the contents of the directory, shows that .artifacts isn't there

[upper kettle]

Hm, I see it on my end

[scarlet lotus]

Oh! mmm

This is what I run:

dagger call -m github.com/sheldonhull/digital-garden \
 mkdocs-build --dir=https://github.com/sheldonhull/digital-garden \
 directory --path=.artifacts \
 entries

[upper kettle]

I was going off the original command that @willow flume mentioned

mkdir .artifacts/
dagger call mkdocs-build --dir ${PWD} -v

Where .artifacts is created first manually, but after running export I see a bunch of stuff in the folder

[scarlet lotus]

Ah I see. I wasn't sure what was supposed to be in that dir argument, so thought I was playing it safe by passing the contents of the digital-garden repo itself

[upper kettle]

@willow flume let me know if you wanna pair on your module for a bit - I am around for the rest of the day, its easier to explain some of the more complicated bits live. 🙂

[willow flume]

ok i'm looking at your update and will take you up on that if it doesn't solve the issue.
I wasn't understanding I have to use an export command to get content out. Thought mounted directory == same as a mounted directory in docker, so any artifacts should show up.

This is really helpful and just when I'm starting on either doing back in mage or dagger, so will keep ya posted

🙏

dagger call mkdocs-build --dir ${PWD} export --path .artifacts did exactly what I needed. What does am i missing on this part.
Is this only for a Directory return type or for all?

I thought if I did WithMountedDirectory that the if I create a directory in my repo then it should be visible on my host, but I guess it's copying the directory in not mounting the same way I assumed.

[upper kettle]

Yeah this is a common source of confusion that we are still working out ways to make more clear. There are a few things happening here.

1. Mounting volumes does not work the same in Dagger as it does in Docker Run or Docker Compose. There is this open issue that aims to make this a reality https://github.com/dagger/dagger/issues/6990

2. export is a built in function that works with directories and files (you can see that yourself if you run dagger call mkdocs-build --dir ${PWD} --help

3. All commands that you execute are not happening on your host but instead inside the dagger engine in an intermediate container (check out this diagram https://docs.dagger.io/manuals/developer/overview/482011/architecture) - so even if you did the export programatically it still would not work as you expected since the files would end up in the intermediate container instead of on your host. This issue aims to give you the option to make your functions more "host aware": https://github.com/dagger/dagger/issues/5993

[willow flume]

will work more on this next week then. Thank you for double checking this.

[willow flume]

So if I want to serve the file contents and do live reload how do I mount the markdown files so they are in sync? That part I've a little confused on.

[upper kettle]

This isn’t possible yet, that’s the key issue. You can serve but not live reload.

[willow flume]

ok, so to be clear there's no mounting in terms of 2 way file sync, i need to consider anything with dagger a "black box" that copies the contents internal and then works completely in the sandbox.
And then can i not use export command and instead have it export as part of the function?

[upper kettle]

Yes the first part is spot on.

And then can i not use export command and instead have it export as part of the function?

You can, but you have to keep in mind that when you use it via the CLI you specify output to some local pathway.

When you call export programmatically you have to remember that the "host" is actually the dagger engine, so the file wont be on your laptop like you may expect.

There are use cases for doing this programatically, for example if you're trying to pass files around between multiple modules.

But if your goal is to get some file to your local machine, then calling via the CLI using export is the way to go.

[willow flume]

ok. That's interesting. I find this a challenge to consider now since that means tools like "mkdocs serve" for example i still have to run via mage to do live reload. Local services building and deploying would also be the same way.

The ability to watch and copy files back in to rebuild for tools like NPM is important.

1️⃣ Is there any general work around ya'll are doing to allow live reload oriented approach?
2️⃣ Could I be pointed to an example of what it would look like to trim the dagger call/export options I showed if the goal was to produce artifact directory anyway, as in using an export() method but showing me how to invoke this?

[upper kettle]

The ability to watch and copy files back in to rebuild for tools like NPM is important.

Yeah, agree with you 100% this is going to open up a ton of use cases. I made this issue a while ago please add your feedback and support there 🙂 https://github.com/dagger/dagger/issues/6990

1 -> no, there is no way to do this right now
2 -> Im not sure what you mean here, you cannot use the export method and bring it back to your host, see this previous point

But if your goal is to get some file to your local machine, then calling via the CLI using export is the way to go.

The way you did it here: dagger call mkdocs-build --dir ${PWD} export --path .artifacts is the way to do it.