#First very visible workflow, wrapping up changie in dagger and need a few design pointers

1 messages · Page 1 of 1 (latest)

uncut sedge
#

I am wrapping up changie, using Go SDK.

Goal will be to run the basic commands but also include some git workflow so it will

  • checkout the sha of the pushed commit on main
  • run the batch and merge commands.
  • commit these new files and updated changelog
  • tag it
  • push commit these+ tags

1️⃣ Since this involves the git workflow piece is there a better way than checking out the repo inside of dagger and flipping to the commit and running and pushing all internally? The more I do inside dagger the more auth/environment variables become challenging so making sure.

2️⃣ I think my biggest confusion besides this is chaining commands because of handling output becoming confusing. So if I have a "latest" command and it outputs just the current version and run that via cli great, but if I want it to be input for another command if chained together in a combination of commands, what's the way to return output so it can be used, still stdout? A code link would help me if you have an example in a module so I can emulate this.

3️⃣ is there anything I need to do to make this not interact with any dagger cloud stuff for privacy at this point, just the https://consoledonottrack.com/ ? Dagger cloud is great but cannot use at this stage as would be an involved legal vetting process.

This will be the first workflow shared across team in a meaningful way so hoping it's successful and shows value to everyone!
I'd like to eventually connect to azure OpenAI and then have it automatically cleanup grammar and wording with a flag so I think lots of promise here!

uncut sedge
#

@umbral rain doing some more work on this today and maybe you can tell me...

  • should i be posting in github discussions instead for better engagement (no complaints, but since this is a visible project internally, knowing the best place to post for maximum visibility/engagement would be good).
umbral rain
# uncut sedge I am wrapping up changie, using Go SDK. Goal will be to run the basic commands...

👋 sorry @uncut sedge here is the best place to start these conversations since there's someone from the Dagger team always watching. The team has been in focus mode the past weeks so that's why your message very likely went unnoticed.

Regarding your observations here are my thoughts:

  1. Doing this in Dagger should be the best way. For authentication, either passing the repo as a function argument or mounting the SSH_AUTH_SOCK should work. Having said that, @velvet raft 's PR here: https://github.com/dagger/dagger/pull/10697 will allow to inject default secrets in case you need to pull credentials from somewhere else without to manually pass them each time.

  2. Instead of returning (string, error) as you're returning now, you should return an object like Changie or any struct defined in your module. It's pretty well explained here (https://docs.dagger.io/api/return-values/#chaining).

  3. Dagger cloud won't be used at all if you haven't run dagger login from the CLI.

#

let us know if you have any other questions. Tagging @trim tendon @urban bramble and @red bolt here as well 🙏

uncut sedge
#

thank you!
Followup quick questions....

  • Chaining sounds great, but is that supposed to mean I can return the object directly like "Latest" could be chained, but also invoked directly, or is it chained == for reusability as building blocks, but not meant to be invoked directly by user?
  • is there a local viewer in the same web ui for traces? If not, I'll look perhaps in future of submitting for allowing internally at company as more adopt.
  • any examples of connecting AI function to azure openai? don't dig, just asking cause I'm under a time crunch and can't go back and dig around.

I haven't gotten any time to explore the AI stuff, so it's a low priority except for me as an interest, since things are moving so much on MCP client/server front. Main thing is my design so I can properly show demo.

I'll post a few snippets and maybe you can tell me if anything is a "smell" or not how you would do it

#

Example of bump + release with git repo... no obligation to review, but feel free to provide any thoughts if you get downtime

#

edit... nvm, will post as a gist, to make easier to quickly see and then feel free to provide any guidance if you see anything way off... 🙂 cheers

#

This will be rolled out in devex team for 10-15 repos with basic improvements to changelog and versioning improvements, so good first visible simple usage at company. 🙂

trim tendon
# uncut sedge thank you! Followup quick questions.... - Chaining sounds great, but is that s...

here are a few good pages that might be related if you have not seen them (re-chaining, constructors, defaults)

https://docs.dagger.io/api/return-values?sdk=python#chaining
https://docs.dagger.io/api/constructor

Constructors are great for things that are shared among functions (like the source code directory)

Also, the term chaining can be a bit ambiguous. The above page refers to Function Chaining (e.g. don't create the base/build container each time - the function that needs it should call the build/base function that returns a container)

#

(sorry selected Python as I thought you were using that for some reason)

uncut sedge
#

no prob, i see the tab. That helps point me to the abstraction I've wondered about. I've done a "runner" for returning a container, but wasn't clear on why the extra layer, and now I have a reference for that. Appreciate it

umbral rain
# uncut sedge thank you! Followup quick questions.... - Chaining sounds great, but is that s...

is there a local viewer in the same web ui for traces? If not, I'll look perhaps in future of submitting for allowing internally at company as more adopt.

no there's not besides the TUI. You can connect any OSS OTEL collector and use their viz tool if you can't use Cloud.

any examples of connecting AI function to azure openai? don't dig, just asking cause I'm under a time crunch and can't go back and dig around.

cc @red bolt @somber pond ?

red bolt
#

I've used dag.llm with azure cognitive services with several models. Configuration is the same as any openai endpoint - i set the openai base url and api key and it works ✨

umbral rain
# uncut sedge https://gist.github.com/sheldonhull/a640c1fd9934db88aef7d17e524da4d9 there's wh...

some very high level observations:

https://gist.github.com/sheldonhull/a640c1fd9934db88aef7d17e524da4d9#file-changie-go-L78 > the +defaultPath pragma mighto be handy here: (https://docs.dagger.io/api/default-paths/)

https://gist.github.com/sheldonhull/a640c1fd9934db88aef7d17e524da4d9#file-changie-go-L85 > It's generally best if the constructor returns it's own type and you store state a fields inside of it. We use this pattern a lot in the dagger/dagger repo. i.e: https://github.com/dagger/dagger/blob/main/.dagger/main.go#L15-L90

There seems to be some repetition in the function arguments (https://gist.github.com/sheldonhull/a640c1fd9934db88aef7d17e524da4d9#file-changie-go-L112). This can be solved by the above hints of using constructors and types.

There's a very neat undocumented "beta" feature called "blueprint modules" (https://github.com/dagger/dagger/pull/10712) which improves the UX when calling remote modules that require source arguments and such.

#

there's probably more things to double check. Not having a lot of time in my hands now but LMK if you have any other specific questions 🙏

uncut sedge
#

nice! looking at feedback now. thank you!

#

when you get a chance can you give me a quick 101 on why I'd look at blueprint modules? Should I wait till it matures, or gain benefits immediately on this simple new work?

velvet raft
#

ie. you dont need to wait for it to mature

uncut sedge
#

https://github.com/dagger/dagger/issues/9988

This got me laughcry after all my new constructor stuff. Glad to see it was already caught and well documented. I need to add "search open issues" to my copilot instructions and maybe it will help debugging :-p

GitHub

What is the issue? Go codegen panics when a function is called "New" Dagger version dagger v0.17.1 (docker-image://registry.dagger.io/engine:v0.17.1) darwin/arm64 Steps to reproduce Creat...

uncut sedge
#

When I implemented the constructor approach with the struct, I am not sure why but now I get unknown flag: --source. I've tried invoking command with and without it.

// Changie encapsulates configuration for changie operations
type Changie struct {
    Source  *dagger.Directory
    RepoURL string
    DryRun  bool
    Project string
}

// NewChangie creates a new Changie instance with the provided configuration
func NewChangie(
    ctx context.Context,
    // Source contains the directory of the root of the repo which should have changie config
    // +optional
    source *dagger.Directory,
    // +optional
    repoURL string,
    // +optional
    dryRun bool,
    // +optional
    project string,
) (*Changie, error) {
    // if err := validateSourceOrRepo(source, repoURL); err != nil {
    //     return nil, err
    // }

    return &Changie{
        Source:  source,
        RepoURL: repoURL,
        DryRun:  dryRun,
        Project: project,
    }, nil
}

The commented out piece I tried to validate combination to help guide user


// validateSourceOrRepo ensures that either source or repoURL is provided, but not both or neither.
func validateSourceOrRepo(source *dagger.Directory, repoURL string) error {
    if source == nil && repoURL == "" {
        return fmt.Errorf("either source or repoURL must be provided")
    }
    if source != nil && repoURL != "" {
        return fmt.Errorf("cannot specify both source and repoURL")
    }
    return nil
}

I removed it temp to try and simplify debugging. Any idea what I'm doing wrong now :-p dagger call -m "mymodulerepopath/daggerverse/changie" new --kind "feature" --body "Integrate quantum entanglement for FTL communication" export --source . --path .

Does this have to do with the constructor I changed to NewChangie but the function is "New()" still and should change that?

umbral rain
#

so NewChangie is not really the constructor in this case

uncut sedge
#

Ok but that GitHub issue shows an issue with it? I failed to compile until I removed constructor being "New". Should I revert to a different engine/cli?

umbral rain
#

I think the issue in your cae is that you had a function called New

#

that's why it was complaining

#

constructors functions don't have receivers. It should be a top level package New function

uncut sedge
#

ok, so back to this... tomorrow's the demo! woot woot.

#

ok... so fixed func New() for the return of (*Changie, error), so that matches the expectations.

Changed this

func NewChangieRunner(
    ctx context.Context,
    client *dagger.Client,
    repoURL string,
    source *dagger.Directory,
) (*ChangieRunner, error) {...

Finally got rid of New as a function name, and did

// NewEntry creates a new changelog entry using the Changie container
func (c *Changie) NewEntry(
    ctx context.Context,
    kind string,
    body string,
) (*dagger.Directory, error) {
    runner, err := c.getRunner(ctx) ...

code generate fails to rebuild Error: generate code: template: module.go.tmpl:86:3: executing "_dagger.gen.go/module.go.tmpl" at <ModuleMainSrc>: error calling ModuleMainSrc: failed to parse constructor: ignore pragma "[", must be a valid JSON array: '': source data must be an array or slice, got string

Seems similar to my first issues, can't get my code generation to fix, so dagger develop -m changie fails. Any tip on what I do to get back past this?

#

I think the hardest thing for some of this because I'm jumping in is this stuff normally would be simply/just works cause you are abstracting a lot of complexity away, but when something goes bump due to all the code generation and my ramp up, I'm not quite certain in the time I have how to fix. Not blaming, just hate having to ask for help on this :-p

#

ARGGGGGH

umbral rain
# uncut sedge

well.. at least the error was quite accudate there 😛

#

but yes.. we're constantly improving the error messages and the UX around codegen 🙏

uncut sedge
#

Right now last I checked there's no equivalent to invoking and have it wait for user input like a gum tool right? I see https://github.com/dagger/dagger/issues/10756 and assuming nothing right now?

The only way for me to make it more "interactive" would be to give user a dagger shell tutorial to run certain commands that way, right?

Trying to minimize friction and i'm ok right now just giving dagger call commands but making sure!

GitHub

Quickly extracting this rough proposal from a Discord discussion since it's come up a couple of times now. We have a lot of the primitives already to build a pretty nifty prompting API, buildin...

uncut sedge
#

what would you do?

command is adding a file to git repo and exporting... unknown file name

1️⃣ use go-git and grab the file name then export that
2️⃣ capture before container state and after container state, then export the diffed file

Curious as copilot added via 2️⃣ but seemed odd to me. Wondering if you would just use go-git to diff file changes, or if there's a helper I'm missing to do it that's better

#

more I need to explore but time is limited today so I see things like source.AsGit, etc. Figured there might be a way so when I add the file via an embedded command I could easily get the addition without needing to script anything

red bolt
uncut sedge
#

Oh very nice. ok checking now.
I'm updating my context7 and mcp prompts as we go to so it better knows where to look.

If you all have interest in that for improving enhanced instructions for folks like me as a consumer those can be really helpful.

#

You got labeled as "elegant" today 🏆

#
  • So if I'm in a new directory
  • i run dagger call -m ${fullyqualifiedpathtomoduleWhichIsSomewhereElse} new-entry --kind 'foo' -body 'bar' export --path . ... and i have +defaultPath=".", shouldn't it be mounting the working directory as my context here? Was wanting to to always work in context of where I am,
  • I think it is mounting the module directory path instead of where I am...
red bolt
uncut sedge
#

My goal is a remote module... ie you run dagger call -m (fullpath) new-entry ... and this runs where you are currently. I'm just using a file path to a different directory so I can test outside of my current repo.
I don't think blueprints is what I'm looking for in that scenario, as I'm just looking to have a reusable workflow as a "global tool"

red bolt
#

Yeah thats exactly the idea behind blueprints! The other solution today is to dagger call -m (fullpath) new-entry --source . ... (pass the current directory as an argument), but without blueprints there's no way to default an argument to the users current directory

velvet raft
#

With -m you also miss out on encoding extra intelligence in your dagger dependency graph

#

and that intelligence will find itself in a custom wrapper instead

uncut sedge
#

what wait?

I'm confused.

So here's what I'm trying to wrap up today/tomorrow

    dagger call -m "$DAGGERVERSE_DIR/changie" \
      --source . \
        release \
      --git-push-url https://dev.azure.com/MYORG/devex/daggerverse.git/changie@v0.0.1 \
      --auth-token env://AZURE_DEVOPS_EXT_PAT \
      $SILENT_FLAG

Essentially I've tried to wrap up the changie image to allow CI pipeline to clone the azure repo, checkout the sha/main, run the changie batch auto and changie merge commit those files to repo, tag it, then push these changes back upstream.... all without any CI checkout, mounting, or anything.

In addition, I've done dagger call -m https://dev.azure.com/MYORG/devex/daggerverse.git/changie@v0.0.1 latest so it could do the same thing to calculate the repo version, but toggle to running against local mounted directory with optional dagger call -m https://dev.azure.com/MYORG/devex/daggerverse.git/changie@v0.0.1 latest --source . if I wanted to run against current repo.

I did this for one command that's expected to run in local source context not checkout the repo... aka

This copies over all the changie init contents, but the customized template and types.

My hope was to demonstrate the concept of a "global tool" that I can run local, remote in CI, centralize and code with Go, but have zero dependencies on the system other than dagger/docker.

#

Right now my use case is very simple, doing the checkout/tagging/push.
To be honest it's feeling incredibly overcomplicated right now using dagger to do this, but I'm pushing through to demonstrate the benefit to the consumer.
The CI pipeline will execute but there's no dagger in the various repos that the dagger command would run against.

Blueprint, I'm not understanding how it would help me here, as I thought my usecase described is always been supported.

#

If I was invoking with mage, it's straightfoward, but I'm trying to use modules only because the goal in the first place is a centralized "daggerverse" of modules I could hand to someone else (as I'm on devex, I have lots of glue stuff to share).... and they just run. I will have zero dagger adoption outside my team in the future, so a "here run this" and it automates/improves something is what I have to focus on.

#

and also not looking to templatize any base set of linters/tasks that maybe blueprints is for, right now a single global tool for just changie/versioning is all i want, and then I was going to try adding a few flags for "fix spelling/grammar/wording with azure openai", and "check that changie entries and the changes to source, and fail if a breaking contract change between public contract if found and version change isn't a major"... etc.

velvet raft
#

@uncut sedge sorry for the perceived complexity.

How will your end users consume this? How many different workspaces (repos or portion of monorepos) are we talking about roughly?

uncut sedge
#

This rollout was to demo to my Developer Experience team a new versioning that's more controlled than auto version bumping. We are migrating to changie to control important libraries consumed by others.

As a result, I'm going to create an azure pipeline that just invokes dagger -m changierepo --repo-url {repoURL} call release ... sorta like how I'm doing mkdocs publishing now too (win!)

The downstream consumer of a developer would say, "hey I want this type of versioning too!", and I'd say:

  • paste this: dagger -m changie call init --source . export --path . which maps the files out in command and they get the config ready to go (including a first initial entry to initialize v0.0.1)
  • paste the azure pipeline template that calls changie template (yaml)
  • register the pipeline
  • they run pipeline and dagger runs all the release actions, no checkout or other things other than passing tokens would be involved (haven't checked about azure keyvault support)

My thought was could I hand over a terraform test, lint, format run to someone as is? Yes with trunk check right now, but what if I could get this away from mage, nuke, make, other tools and have a global common tool that is for all my module developers and just run dagger -m {repoURLForTerraformDaggerverseModule} call prepare-pr and it ideally would just run in the current directory, no .dagger folder, no dependencies. More a "platform CLI".

#

Am I misunderstanding something on dagger where I'm using it "wrong"? I know certain constraits are problems for this, like no changie new providing a interactive TUI, verbosity of the full command, but should be 95% of what most folks need.

The perceived complexity is because doing some of this is "easy" in native shell/pwsh but doing with containers that maintain snaphots is more work.
That said I'm optimistic that shifting the complexity to me building is still better than shifting to the end user, just still struggling with the lifecycle/model cause it's more magic than I normally experience happening with Go 🙂

red bolt
#

as an aside (really dont want to derail the overall discussion), that kind of directory filtering can be done directly with dagger's api, it's just not an obvious api and we should probably simplify it. The filtering is hidden in the Directory.WithDirectory options
return dag.Directory().WithDirectory("/", dirA.Diff(dirB), dagger.DirectoryWithDirectoryOpts{ Include: [fileExtension] })

#

My thought was could I hand over a terraform test, lint, format run to someone as is?

Yes, you can definitely do this today. Blueprints aim to make this easier, but in general you can do it. I'm not sure in the previous messages which commands you have working as intended vs what you would like to work isn't. Or possibly I confused the issue by bringing up blueprints in the first place!

uncut sedge
#

the commands i just posted above are what I'm trying to achieve. A way as a member of the devex team to send someone a command to run and output/manipulate the files/onboard etc.
It's not a deal breaking to have to provide the source directory, but my initial goal was to also have that be handled in my runner like this

// NewChangieRunner creates a ChangieRunner.
// If repoURL is empty, mounts the provided source directory (default: current dir). If set, clones the repo to /src.
func NewChangieRunner(
    ctx context.Context,
    client *dagger.Client,
    repoURL string,
    source *dagger.Directory,
) (*ChangieRunner, error) {
    var ctr *dagger.Container

    if repoURL != "" {
        repo := client.Git(repoURL).Head().Tree()
        ctr = client.Container().
            From(changieImage).
            WithDirectory("/src", repo).
            WithWorkdir("/src")
        return &ChangieRunner{Container: ctr}, nil
    }
    // since we aren't passing a repo to issue commands, lets run this in the current working directory, so user doesn't have to map this arg everytime.
    ctr = client.Container().
        From(changieImage).
        WithMountedDirectory("/src", source).
        WithWorkdir("/src").
        WithUser(fmt.Sprintf("%d:%d", os.Getuid(), os.Getgid()))

    return &ChangieRunner{Container: ctr}, nil
}

So essentially i was try to say when URL provided use this approach, else assume the current working directory is the source directory .... and the current working directory as in the user running the command, NOT the module source directory which was is local right now but meant to be a remote module once I deploy in CI/teams.

#

a bit more on my end goals

az acr login --name devextools --subscription {GUID}
  docker pull devextools.azurecr.io/RANDOMCODETEMPLATEREPO:latest
  docker run -it --rm -v $(pwd):/repo devextools.azurecr.io/RANDOMCODETEMPLATEREPO:latest

This output all the content.
But if I could do this in future with dagger call -m dev.azure.com/myorg/repo/daggerverse.git/copyrepotemplate@v0.0.1 --source . export --path . then I'd start to move to this approach.

Only my personal repos have some dagger, so just like a npm install --global myhelpertool or go install myhelpertool@latest or dotnet install --global myhelpertool, my hope was to centralize the logic and use it in any place I needed.

This changie stuff was the easy low impact way for me to try this out and show the team. It's verbose, hard to lose the TUI, but all that for the continued gain with this was my aim

#

No one is gonna replace their azure pipeline dotnet build stuff, too familar, but if I do it pieces and provide it as a "commoditized" thing they just can consume, it'd be a win 🙂

#

and less yaml in my life too ;-p
Love writing Go, but yaml drowning isn't as much fun

velvet raft
#

Yeah this feels like a good use of blueprints.

@uncut sedge in that use case you would tell your teams:

  1. One-time install: run dagger init --blueprint=$DAGGERVERSE_DIR/starterkit

  2. Then from their workspace they can run any function eg. dagger call setup-changie -o . or whatever

  3. As you add more functions to the blueprint, your users can run dagger update and boom they're available

red bolt
#

Yeah exactly, so you could simplify it for the users so they dont have to remember dagger call -m dev.azure.com/myorg/repo/daggerverse.git/copyrepotemplate@v0.0.1 --source . export --path . or wrap it with a script! But the functionality of that command should work today so if it doesn't we should take a closer look

uncut sedge
#

That is nice and clean. So renovate running could run dagger update and bump versions of the global tools used by each repo. The only thing is the json data.

nice!

But while this is fantastic for a shared cli usage to give to someone if i want to provide a version pinned set of tools like build tools and all...
I guess I'm seeing that as orthogonal to my current work, which is meant 98% for just CI, and shouldn't require a dagger json in the repo at all, though I could I suppose.

Assuming I just use current remote modules it's just more wordy and the CI + running from laptop is more verbose with

dagger call -m "$DAGGERVERSE_DIR/changie" release --repo-url https://dev.azure.com/MYORG/devex/_git/dagger-changie-test-sandbox --auth-token env://AZURE_DEVOPS_EXT_PAT

right? meant to be self contained remote module at that point. I just have to deal with a bit more learning curve on dagger to properly auth/clone and push to the main, using diff to find what files got modified and push/tag.

#

and it's more complex that just the CI would be, for me as the module writer, since I don't offload all the git checkout and such natively to the pipeline and have to auth/clone etc in my code. Not a deal breaker, just meaning because I'm not mounting the existing repo, it's a snapshot, i can't easily just mount the repo and start tagging and such like I would in a inline yaml script. I have to remember it's sandboxed away

#

hopefully that makes some sense on why i use the "more complex" phrase

red bolt
#

Yeah exactly, it could be even easier than having to decide between repo-url and source because the dagger CLI can turn a git url into a dagger.Directory
so you could

dagger call -m ... release --source https://dev.azure.com/MYORG/devex/_git/dagger-changie-test-sandbox
OR
dagger call -m ... release --source .

uncut sedge
#

Trying my best here just in pieces. Mage is "dead", and i know it well so translating stuff that has environment assumptions and such sometimes is tricky, but getting there. Proper prompting in agent mode with copilot/context7 has helped accelerate this, just working through it as I can.

Definitely going to remove my shell script tests when done and do some "end to end" Go tests based on the docs

#

Hmmm i'll try and see how that works, that could be cool. I had issues last time on the AZURE_DEVOPS_EXT_PAT being recognized and git-credential-manager etc getting all confused. Let me see if I can figure out from the docs and adapt what you said. that would be nice!

red bolt
#

yeah the git credentials stuff can be tricky, hopefully the docs can help out there but if not we're always here 😄

uncut sedge
#

that's what i got to work in another project for folks before

#

thank you for the responses. Gonna go back and yell at my copilot agent to refactor and wait for it to mess things up again 😂

#

oh one other quick one to make sure...

No way to currently drop into a TUI for something like changie new right?
The only way I can think of is having a terminal command that is for dagger shell and drops into alpine container to just run changie new directly, but not certain if the created file would be able to automatically get dropped out to the mounted directory.

Just a confirmation my understanding on limitations for TUI/prompts is still currently a thang 😁

uncut sedge
#

so can i make sure i get this right

  • "-m privategitrepo" should work as it uses git credential manager
  • "--source privategitrepo" should also work, using the same git credential manager?

The docs/discussion that I found confused me a little. I know I used this successfully in past for someone that has https based auth dagger call -m dev.azure.com/MYORG/devex/_git/renovate-config/dagger@main run \ --token=env:AZURE_DEVOPS_EXT_PAT $BUST_CACHE \ export --path repos.json

If both should work with native git manager just want to make sure so I have to figure out my settings locally then, since I started with "invalid pkt-len found" and assuming my quick refactor is missing something

#

looking to see what my git config options are here

#
file:/Users/MYUSER/.gitconfig     credential.azrepos:org/MYORG.username=MYORG
file:/Users/MYUSER/.gitconfig     http.https://dev.azure.com.extraheader=Authorization: Basic  TOKEN
file:/Users/MYUSER/.gitconfig     credential.azrepos:org/https://dev.azure.com/MYORG.username=MYUSER@MYORG.com
file:/Users/MYUSER/.gitconfig     credential.helper=
file:/Users/MYUSER/.gitconfig     credential.helper=/usr/local/share/gcm-core/git-credential-manager
file:/Users/MYUSER/.gitconfig     includeif.gitdir:/Users/MYUSER/git/ssh.dev.azure.com.path=/Users/MYUSER/.gitconfig.azdos

looking now to see if can reset the token. Just odd no error like "401" but instead that error on pkt

#

ok... i reset my git credentials, removed all the configs I had with headers, and then confirmed git ls-remote triggered my oauth browser signin for identity and keychain.... now i know repo command can work, and trying again...

#

ok, did dagger init in a new directory with only this


// RepoTree generates a tree view of the repository structure up to a specified depth
func (c *Testauth) RepoTree(ctx context.Context, src *dagger.Directory, depth int) (string, error) {
    return dag.Container().
        From("alpine:latest").
        WithMountedDirectory("/mnt", src).
        WithWorkdir("/mnt").
        WithExec([]string{"apk", "add", "tree"}).
        WithExec([]string{"tree", "-L", string(depth)}).
        Stdout(ctx)
}

dagger call repo-tree --src https://dev.azure.com/MYORG/devex/_git/dagger-changie-test-sandbox.git --depth 3 -vvv

got same issue

parsing command line arguments 0.4s ERROR
! failed to get value for argument "src": invalid pkt-len found
╰─✘ git(url: "https://dev.azure.com/MYORG/devex/_git/dagger-changie-test-sandbox.git", keepGitDir: true): GitRepository! 0.4s ERROR
  ! invalid pkt-len found
#

so if you have a tip on what specific piece might have an issue with git credential manager let me know cause the mounting seems to not work right now @red bolt ... I know i did it in the past though with the source module code, so is it just the inability to use --source that's the problem?

#

I tried an older command I setup and got the same issue on a different repo dagger call -m dev.azure.com/MYORG/devex/_git/renovate-config/dagger@main run \ --token=env:AZURE_DEVOPS_EXT_PAT \ export --path .cache/repos.json

I even reset with git-credential-manager configure and configured azure repos is bound. I can run git ls-remote against the url and get results. No change in dagger action.

#

ok, so the only way that worked was doing token injection in url 😦 so https://${MYTOKEN}@remaining url which isn't really ideal, but if it's the only use then maybe I can do that. Trying now with my other code

#

ok, got it to work in the repo-tree test.

The negative is that the token is exposed as part of the url, not an ideal situation as relies on the CI obfuscation.

uncut sedge
#

so other than the cred injection thing, no i am able to run changie latest against a remote repo and get the version just like changie latest does. Figuring out the mapping to make sure it outputs to the right path is next, cause it dropped them to the wrong location when using file return instead of directory

#
▶ connect 0.2s
▶ load module: ${HOME}$ssh.dev.azure.com/v3/MYORG/myorg/daggerverse/changie 0.9s
● parsing command line arguments 1.2s

● changie(
  ┆ source: Directory.withDirectory(
  ┆ ┆ path: "/"
  ┆ ┆ directory: GitRef.tree: Directory!
  ┆ ): Directory!
  ): Changie! 0.2s
▶ .checkUnreleasedChanges: String! 0.4s

Found 9 unreleased changes.
dagger call -m "$DAGGERVERSE_DIR/changie" \
  --source $REMOTE_REPO_URL \
  check-unreleased-changes 

nice!

umbral rain
# uncut sedge oh one other quick one to make sure... No way to currently drop into a TUI for...

yes, you can do this today. The Terminal function has an optional cmd argument which by default is /bin/sh so you can replace it to whatever you need. The main caveat is that if you're expecting to pull files out of the Terminal command, you need to use CacheVolumes for that given Terminal is currently designed as a troubleshooting tool and it won't persist any changes in its filesystem

umbral rain
uncut sedge
#

that's fine. For now I can just guide away from it. I'm more concerned on still having to inject the token in the url, but that's fine for another discussion ... just continuing to log my process in case any insight or useful to y'all to see what painpoints might be there for someone

umbral rain
#

by using the git@$url format?

#

can you do git clone git@dev.azure.com.... in some sort of way?

uncut sedge
#

i use ssh, but i'm not going to even try for that with both CI + sharing with others. most people here just use https auth, and CI expects that too

umbral rain
#

so if you run git clone https://dev.azure.com... it clone the repo automatically without having to provide any inputs?

uncut sedge
#

yeah see above, i showed the steps I did just a bit earlier to test out using without ssh

#

yep just did with https url

umbral rain
#

REMOTE_REPO_URL doesn't contain the hardcoded token doesn't it?

uncut sedge
#

oh sorry LOL

#

let me try again

#

i'm 18 hours on dagger straight in 2 days and brain melt 😆

#

yep worked as well with normal url after prompting in browser to oauth

#

then reran the dagger command with this and got the pkt-len error again

umbral rain
#

@uncut sedge one last thing

#

mind trying this and check if you get a valid username/password in the output?

printf "%s\n%s\n%s\n" host=dev.azure.com protocol=https | git credential fill

umbral rain
#

I know https:// works with Dagger because we've tested it before

uncut sedge
#

yes, I know. I've worked with this and while I use ssh, I have set header approach before, and use git-credential-manager which I had written above after testing, but didn't see it work on my url which is the clone url you get from the repo on azure repos

umbral rain
# uncut sedge

mind running git config --global credential.useHttpPath true

#

and see if that fixes it?

uncut sedge
#

sure, i had removed some settings, will try again... this is before

#

trying now

#

same error

umbral rain
uncut sedge
umbral rain
#

printf "%s\n%s\n%s\n" host=dev.azure.com protocol=https path=$path | git credential fill

#

^ mind trying that

#

and replacing $path with what comes after dev.azure.com in the URL?

#

path= > should start with the forward slash /

uncut sedge
#

i checked my chezmoi config and see this is the standard config i normally did

It does keep telling me the fatal: Cannot determine the organization name for this 'dev.azure.com' remote URL. Ensure the credential.useHttpPath configuration value is set, or set the organization name as the user in the remote URL '{org}@dev.azure.com'. so might look at that

#

ok i can try your snippet

umbral rain
#

does git config -l show the useHttpPath?

#

git configs can be loaded and overriden from multiple places

uncut sedge
#

right, i was checking config against

git config --global --list --show-origin | grep -E "(credential|user|core|dev.azure.com)"

That's the output i was sharing earlier

umbral rain
#

since that will force it to load the global config

uncut sedge
#

do you want the path to fill to just be dev.azure.com/myorg/ or the full url?

umbral rain
umbral rain
uncut sedge
#

ok trying again, then i have to pick up again tomorrow

#

is the path the full url or just the org

umbral rain
umbral rain
#

LMK when you're around tomorrow and we can keep looking. Note: we need to add better logs in the engine for this

uncut sedge
#

timed out

#

i have done extraHeader successfully with some tools before, so can try that quickly if you want me to set a global header to my token for the domain and see if it works

#

the first time i ran that it didn't work

umbral rain
uncut sedge
#

then finally showed up a popup, but email with @ showed encoding issue

#

then tried again and it worked right away with no error

umbral rain
#

anyways.. let's continue tomorrow

#

might be an edge case about how we're handling https auth with azure devops in this case

#

but we're close to get it to work 🙏

uncut sedge
#

much appreciated. catch ya tomorrow. the biggest issue of git auth with token at least is a big one. I cut out a ton of needlessly complex code, so it's a good step!

umbral rain
uncut sedge
#

Gonna be jumping back on this in an hour. I'm up for huddle/call too if easier.

#

I was gonna get a coworker to test so here a followup question

is that how to reference module but use the feat/changie branch and the sub folder for the monorepo?

Wasn't working and figured I got something mixed up. It's a remote module

#

It's a little different from a go install so making sure I'm getting path format right from docs. Wasn't working afraid the slash in branch name was an edge case and breaking it.

#

Tags and branches I typically have slashes in when dealing with a monorepo

umbral rain
uncut sedge
#

k, on it.

#
AZURE_DEVOPS_EXT_PAT=$(az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 --tenant 'mytenant' --query "accessToken" --output tsv 2>/dev/null || true) > /dev/null 2>&1
DAGGERVERSE_REPO="https//dev.azure.com/MYORG/devex/daggerverse/changie@main"
TEST_REPO_PATH="https://${AZURE_DEVOPS_EXT_PAT}@dev.azure.com/MYORG/devex/_git/dagger-changie-test-sandbox"

dagger call -m https://dev.azure.com/MYORG/devex/_git/daggerverse/changie@feat/changie \
  --source $REMOTE_REPO_URL \
  latest --silent

Error: failed to get configured module: failed to resolve git src: failed to resolve git src: select: invalid pkt-len found

#

😢 so close

umbral rain
#

trying to validate if the auth is failing for the module repo or the $REMOTE_REPO_URL

uncut sedge
#
git clone https://dev.azure.com/MYORG/devex/_git/daggerverse .cache/test

works so i am authenticated fyi

#

you meant dagger functions -m https://${AZURE_DEVOPS_EXT_PAT}@dev.azure.com/MYORG/devex/_git/daggerverse/changie@feat/changie so that worked

#

getting close then!

#

darn azure repos

umbral rain
uncut sedge
#

let me add the missing _git and see if that helps

umbral rain
#

since I have a few to huddle about this

umbral rain
#

@trim blaze

uncut sedge
#

thank you 👋

#

You know I've basically broken every tool out there and dagger everytime I use it... and I blame Azure Repos 😂

#

I hope so much we move to github, so much less friction for all these great tools!

umbral rain
uncut sedge
#
[credential]
    helper =
    helper = /usr/local/bin/git-credential-manager
    azreposCredentialType = oauth
    useHttpPath = true
[includeIf "gitdir:/Users/MYUSER/git/ssh.dev.azure.com"]
    path = /Users/MYNAME/.gitconfig.azdos # this just sets my default User Name
# For HTTPS on Darwin, You'll want GCM to SSO
[credential "https://dev.azure.com"]
    provider= azure-repos
    helper = /usr/local/bin/git-credential-manager
    useHttpPath = true
    azreposCredentialType = oauth # https://github.com/git-ecosystem/git-credential-manager/blob/main/docs/configuration.md#credentialazreposcredentialtype
    interactive= auto
    credentialStore= keychain # cache option
    # prefer keychain on darwin for better security
    # credentialStore= cache
    # cacheOptions= --timeout=86400
trim blaze
umbral rain
trim blaze
umbral rain
trim blaze
uncut sedge
#

trying to run dagger -m mymod --source remoterepourl release push ...
What' the way to invoke a chained set of commands via call non-interactively? Do I have to make it a single call and wrap that up inside or is this multiple command thing supported

#

is there someonthing @trim blaze you needed rom me that I missed responding too?

#

and someone let me know if i'm supposed to be able to run dagger call -m mymod release push notify or is that not possible without dagger shell? Out of time to dig more and couldn't figure it out yet

#

@umbral rain last bit to bug you on since wrapping up some stuff and this impacts how I setup my azure pipeline. Can I chain some commands like that above, or am I going to be better served by just having "release" do the push based on a flag or something?

#

thank you! have a great weekend everyone

trim blaze
#

dagger -c ' first command ; second command '

uncut sedge
#

TBD the chained command syntax with mounted directory and remote repo was a little tricky and couldn't get it quiet there.

#
dagger -m $DAGGERVERSE_MODULE_REPO_PATH shell - \
  -c "source $TEST_REPO_PATH | latest && bump && latest"

That variable isn't found, so some clarification needed. Thanks though for helping out on this since I know it's not your focus!

#

talk to you all next week

umbral rain
#

In the path?

trim blaze
uncut sedge
#

one plus from my final wrap up today is I could create a single pipeline and just pick one of many repos from a list, and then run it. This is different than me having to rollout a pipeline with templating in every single target repo, benefiting from the whole approach. 💪

umbral rain
#

let us know if there's anything you need our input on 🙏

uncut sedge
#

Question... I have 2 markdown config files. I want to keep these in my module but not have to put them in the remote repo ideally. Can I reference them from my module instead of mounted source path as a resource, so they get mountedinto my command without having to be permanently in the target repos that are mounted via --source?

#

did my prompt get this right?

uncut sedge
#

so with /changelog github prompt command, it contructs the dagger call and other commands making it very nice. This took away painpoints of wrapper scripts :-p and lets me make a command for any platform dynamically. Just the quality of life improvement to help devs love it.

uncut sedge
#

Ok, I had to use that, but then remove them when done, as the files needed to exist in the repo for markdown linting for example, but I didn't want to commit to all repos. So just a container WithoutFile solved the problem for me. NIFTY.

At the beginning it's hard to see the benefit over just inline shell command, but when you start chaining this it's pretty epic to demo

#

Ok, now any tip on the thing above in chat, just a little, where i was asking if it was possible to easily chain some functions?

1️⃣ Like in mage build release notify how can I for CI do something like dagger call -m mymod --source myremoterepo build release notify ... i don't yet grok the shell so not certain how to leverage that properly.
2️⃣ Is there any native feature to easily run the exact dagger command with a "foreach" built in like... dagger call -m mymod --source myremoterepo1, remoterepo2, remoterepo3 onboard and it iterate through each? Since the mounting is a remote repo, I'm assuming I'd have to just wrap this in shell with a foreach and not have dagger do this without refactoring away from a simple directory mounting right?

uncut sedge
#

finally the llm output working ROCKING

#

i love that it loops built in, true agent like work, no having to handle and manipulate output for a file etc. Beautifully done

somber pond
somber pond
#

Example with required constructor arg (no default)

#
type Loopy struct {
    Data string
}

func New(data string) *Loopy {
    return &Loopy{Data: data}
}

// Returns a container that echoes whatever string argument is provided plus whatever is in Data
func (m *Loopy) ContainerEcho(stringArg string) *dagger.Container {
    return dag.Container().From("alpine:latest").WithExec([]string{"echo", stringArg + " " + m.Data})
}
dagger -c 'for animal in "dog" "cat" "fish"; do . $animal | container-echo hello | stdout; done'
#

For optional constructor arg (perhaps like your --source), note only use param name when overriding a default

func New(
    // +default="horse"
    data string,
) *Loopy {
    return &Loopy{Data: data}
}
dagger -c 'for animal in "dog" "cat" "fish"; do . --data $animal | container-echo hello | stdout; done'
uncut sedge
#

@somber pond is this close?

original


  dagger call -m "$(System.AccessToken)@${DAGGER_CHANGIE_MODULE}" \
    --source . -v ${DYNAMIC_SILENT_FLAG} \
    init ${PROJECT_FLAG} \
    export --path . || {
      echo "[##warning]❌ init exited with issue"
  }

  dagger call -m "$(System.AccessToken)@${DAGGER_CHANGIE_MODULE}" \
    --source . -v ${DYNAMIC_SILENT_FLAG} \
    --allow-llm "all" \
    migrate-changelog ${PROJECT_FLAG} \
    export --path . || {
      echo "[##warning]❌ migrate-changelog exited with issue"
  }

  echo "🧹 running any further cleanup actions"
  dagger call -m "$(System.AccessToken)@${DAGGER_CHANGIE_MODULE}" \
    --source . -v ${DYNAMIC_SILENT_FLAG} \
    --allow-llm "all" \
    ensure-nuget-pipeline-triggers \
    export --path . || {
      echo "[##warning]❌ ensure-nuget-pipeline-triggers exited with issue"
  }

  dagger call -m "$(System.AccessToken)@${DAGGER_CHANGIE_MODULE}" \
    --source . -v ${DYNAMIC_SILENT_FLAG} \
    clean-up \
    export --path . || {
    echo "[##warning]❌ clean-up exited with issue"
  }

  dagger call -m "$(System.AccessToken)@${DAGGER_CHANGIE_MODULE}" \
    --source . -v ${DYNAMIC_SILENT_FLAG} \
    validate-onboarding || {
      echo "[##warning]❌ failure to validate onboarding, will still attempt to push what progress we made."
  }

Next

  dagger -c '
  . --source .  | init ${PROJECT_FLAG} | export --path . |
  . --source . -v ${DYNAMIC_SILENT_FLAG} --allow-llm "all" | migrate-changelog ${PROJECT_FLAG} | export --path . |
  . --source . -v ${DYNAMIC_SILENT_FLAG} --allow-llm "all" | ensure-nuget-pipeline-triggers | export --path . |
  . --source . -v ${DYNAMIC_SILENT_FLAG} | clean-up | export --path . |
  . --source . -v ${DYNAMIC_SILENT_FLAG} | validate-onboarding
' -v ${DYNAMIC_SILENT_FLAG}

This doesn't explain the environment variable mapping and such, so just converted to see if this is what you mean. Is it with-source and verbosity is outside this etc.

somber pond
#

Guessing those are all separate statements, so I'd end the lines with ; chars

uncut sedge
#

so i'm actually close? I was assuming this would be wrong but good to know I'm close!

somber pond
# uncut sedge so i'm actually close? I was assuming this would be wrong but good to know I'm c...

With shell, there aren't arg flags (e.g. --source) for required args, just positionals, that's the other thing

Though I'd like to be able to use either flags or positionals: https://github.com/dagger/dagger/pull/10819

GitHub

fixes #9701
I believe the behavior here is that you can use a --argname flag for any arg including required args where previously Dagger Shell would strictly require a positional arg WITHOUT a flag...

#

But yep, you can do several statements in a row like

dagger -M -c 'version; version; version' # (-M for 'no module')

v0.18.14v0.18.14v0.18.14
uncut sedge
#

so positional --source is how to mount or with-source, a link to the docs is fine, just not certain what part I'm doing wrong. You are saying


dagger -c '
  . --source . | init ${PROJECT_FLAG} | export --path . ;
  . --source . --allow-llm "all" | migrate-changelog ${PROJECT_FLAG} | export --path . ;
  . --source . --allow-llm "all" | ensure-nuget-pipeline-triggers | export --path . ;
  . --source . | clean-up | export --path . ;
  . --source . | validate-onboarding
'

?

uncut sedge
#

yeah so i'm not getting export . to work

tried to simplify and see and this didn't work

dagger -m "${DAGGERVERSE_DIR}/changie" -c '
   init ${PROJECT_FLAG} | export  . ;
' --allow-llm "all"
#

hmm did this and got some output so making progress

dagger -m "${DAGGERVERSE_DIR}/changie" -c "
   init ${PROJECT_FLAG} | export  . ;
" --allow-llm "all"

#

so how do you pass the arg in for "--projects" that normally call would be passed?

#

so give me a hint on how to properly use this in CI so it passes the directory and then i'll hopefully grok this.

dagger -m "${DAGGERVERSE_DIR}/changie" -c "
   with-directory --path . | init ${PROJECT_FLAG} | export  . ;
   with-directory --path . | migrate-changelog ${PROJECT_FLAG} | export  . ;
   with-directory --path . | ensure-nuget-pipeline-triggers | export  . ;
   with-directory --path . | clean-up | export  . ;
   with-directory --path . | validate-onboarding;
" --allow-llm "all"

Didn't work as expected

dagger -m "${DAGGERVERSE_DIR}/changie" -c "
   init ${PROJECT_FLAG} | export  .;
     migrate-changelog ${PROJECT_FLAG} | export  . ;
" --allow-llm "all"

doesn't seem to detect the changes

somber pond
uncut sedge
#

@somber pond I'm up for chatting if you are. I find the call to multiple actions confusing right now so any insight to wrap my head it would be great. Raincheck is fine anytime this week.
@trim blaze when you get a chance mind letting me know when you push some changes so I can try and see if token injection is no longer required for git credentials? No rush, just a followup

trim blaze
uncut sedge
#

thanks for the help @somber pond , i have a good idea on how to organize it now 🙇

uncut sedge
#

ran against dagger and it's nearly there.. combination of git tags + changie version info to see what's pending.

uncut sedge
#

Another question on this .... is a --source that is a git repo a full clone? Am I supposed to have access to all the tag and metadata?

somber pond
# uncut sedge Another question on this .... is a --source that is a git repo a full clone? Am ...

⭐️ EDIT: Short answer is no. YES! example: git <ref> | head | tree --depth=0

dagger -M -c 'container | from alpine/git | with-directory /b $(git https://github.com/dagger/dagger | head | tree --depth=0) | with-workdir /b | terminal'

You can look at

dagger -M -c 'container | from alpine/git | with-workdir /a | with-exec git,clone,https://github.com/dagger/dagger,. | terminal'

vs

dagger -M -c 'container | from alpine/git | with-directory /b $(git https://github.com/dagger/dagger | head | tree) | with-workdir /b | terminal'

and try to run git tag in each.

Though you can use the api to get tags via

dagger -M -c 'git https://github.com/dagger/dagger | tags'
uncut sedge
#

since i'm working with azure repos it complicates things. Good to know i explored all options. made it a little tricky 🙂

somber pond
#

@coarse echo @fossil cedar FYI in case git support in the engine will change in the near future .

somber pond
# uncut sedge since i'm working with azure repos it complicates things. Good to know i explore...

Yep, I did some more analysis and our git API clone is shallow by default (though the depth is a param in the API) and thus lacks the tags info.

You can retrieve that with git fetch --tags

So you can do:

dagger -M -c 'container | from alpine/git | with-directory /b $(git https://github.com/dagger/dagger | head | tree) | with-workdir /b | with-exec -- git,fetch,--tags | terminal'

and then run git tag and you'll have them 🙂

Tells me we should have a option for --fetch-tags

fossil cedar
#

🤔

#

replied in the issue

somber pond
#

@uncut sedge looks like tree --depth=0 will give you a full clone. Somehow I missed that 😅

dagger -M -c 'container | from alpine/git | with-directory /b $(git https://github.com/dagger/dagger | head | tree --depth=0) | with-workdir /b | terminal'

Thanks @fossil cedar
I didn't infer that from:

type GitRefTreeOpts struct {
    // Set to true to discard .git directory.
    DiscardGitDir bool
    // The depth of the tree to fetch.
    //
    // Default: 1
    Depth int
}

and didn't think to try the classic 0 == ALL

#

Tree itself also doesn't indicate that our default is a shallow (depth=1) clone, so maybe we can improve that. Looking at another SDK's docs.

trim blaze
uncut sedge
#

@trim blaze I've seen some changes I need to explore about Latest Tag etc. In the meantime did you ever look into the git credentials manager so I can stop embedding token in the repo and reuse hosts credential setup? Will be cleaner and stop showing up tokens (even if short lived) in logs 🙂

uncut sedge
#

@trim blaze bump, just checking if the git credential manager thing has any work done yet. i would like to remove the need to inject tokens in the repo url

trim blaze
uncut sedge
#

@trim blaze if you have a github issue/draft PR please let me know so I can stop bumping this one and just follow-up separately 🙏 As more see this usage in the company I want to remove the concern on having to run run token injection for usage and hopefully make it a clean call for them. no rush, just a follow-up so i can keep up with appropriate issue

trim blaze
uncut sedge
#

hit me up anytime. If I can save you work by running against my actual environment/repo I'll be glad to. cheers