writing to context dir

[quartz crane]

ooh, sounds doable by exporting only the diff/upper layer. which would also eliminate complicated Directory.diff calls we usually do (and I always screw up the order of)

actually, is it possible to delete files right now? even with -o PATH?

maybe this connects to ideas around syncing filesystems, which we talked about before but going in the other direction (syncing local changes to a long-running service for web dev)

[glass yew]

Yeah it all seems connected, I'm struggling to picture a sane DX

but from a technical standpoint, no doubt it's possible

My use case is generating Github Actions configs

I've been trying this fun pattern where I make .github itself a dagger module 🙂

Then I run dagger call -m .github generate -o .github

While playing with that, I ran into the brittleness of the -o PATH pattern, can't just call that function from anywhere, gotta be careful of my workdir so I can adjust the path accordingly, etc.

[glass yew]

Yes there is export --wipe, I learned this recently from Erik accidentally wiping his .git 😅

• Additive-only is definitely awkward. For example when I rename a pipeline in my config, the old generated workflow file is left behind at its old filename.
• On the other hand --wipe feels a little too nuclear 😛

[quartz crane]

--wipe will delete everything though right? i think you'd want more specifically to delete files that were deleted, so you don't also lose your filtered-out stuff. I wonder if that's possible with the filesync code - pretty sure it can do deletes, maybe by looking for tombstones? (or whatever the snapshotter uses)

[glass yew]

Yeah agreed. I almost added an input source dir so that I could start doing diffs myself in my codegen function, but didn't feel worth the effort

So instead I decided to start this thread!

https://tenor.com/view/im-doing-my-part-serious-stare-gif-11979677

[quartz crane]

https://github.com/tonistiigi/fsutil/blob/91a3fc46842c58b62dd4630b688662842364da49/diskwriter.go#L105 <- that looks promising? just a piece of the puzzle though.

[glass yew]

what if Directory.export() took an optional source: Directory argument to enable all that fancy diffing?

but then there's a risk of a "shadow return value" which is the mirror of the "shadow argument" you raised a red flag about when we were designing context dir

--> instead of func Codegen() *dagger.Directory you get func Codegen(source *dagger.Directory) { doCodegenStuff().Export(source) }

Then we lose the nice "no implicit side effect" thing

[quartz crane]

Yeah exactly - typed out a whole message and it mainly gets back to whether we want to allow functions to write to the host all on their own. There are arguments for composability (it's nice to return a Directory than have a side effect), but you can just split out a separate function for the export, for CLI/API convenience.

Personally, I think "implicit side effects" are fine (and usually kind of the whole point) as long as it's sandboxed - i.e. they can't write to arbitrary paths on the filesystem, and just like with h2c/c2h, the "host" is the local host, so a deep module dependency can't arbitrarily write to the host. (Unless we make our API able to do that somehow, in a controlled/safe manner.)

previously: #1174585230649217055 message

[glass yew]

by the way, see this concern I have #maintainers message

I worry that we accidentally made context dir too safe

[glass yew]

My CI in Go:

// Generate our CI config
// Export to .github at the repository root
// Example: 'dagger call -m .github -o .github'
func (m *Github) Generate() *dagger.Directory {
    return dag.
        Gha().
        WithPipeline(
            "Demo pipeline 1",
            "git --url=https://github.com/$GITHUB_REPOSITORY branch --name=$GITHUB_REF tree glob --pattern=*",
            dagger.GhaWithPipelineOpts{
                Module: "github.com/shykes/core",
            }).
        WithPipeline(
            "Demo pipeline 2",
            "directory with-directory --path=. --directory=. glob --pattern=*",
            dagger.GhaWithPipelineOpts{
                SparseCheckout: []string{"misc", "scripts"},
                Module:         "github.com/shykes/core",
            },
        ).
        WithPipeline(
            "Demo pipeline 3",
            "directory with-directory --path=. --directory=. glob --pattern=*",
            dagger.GhaWithPipelineOpts{
                Module:   "github.com/shykes/core",
                Dispatch: true,
            }).
        // Trigger 'Demo pipeline 1' on:
        //  - push to main branch
        //  - push to any tag
        //  - pull requests
        OnPush(
            "Demo pipeline 1",
            dagger.GhaOnPushOpts{
                Branches: []string{"main"},
                Tags:     []string{"*"},
            }).
        OnPullRequest("Demo pipeline 1").
        // Trigger 'Demo pipeline 2' on all pull requests
        OnPullRequest("Demo pipeline 2").
        Config().
        Directory(".github")
}