#breaking change caused by .gitignore

1 messages · Page 1 of 1 (latest)

junior grove
#

Creating a thread to follow it easily ^

#

The latest release broke my module constructor, which is loading a *dagger.File with the default name .args. That is in the .gitignore to prevent adding it accidentaly to git. This module is not loading anymore: failed to get value for argument "args". How to overcome of this? https://github.com/dagger/dagger/pull/10883

open kernel
#

I'm using dotfiles on CI to pass arguments, secrets and such to a complex module. One of those arguments is the .git directory.

junior grove
#

Using dagger shell you can disable .gitignore application but you would need to update your call

open kernel
#

Shell is not an option AFIK as I need to pass files and such from the host.

junior grove
#

Let' say you have the following signature:

func (e *Example) Example(
  ctx context.Context

  //+defaultPath=./args
  args *dagger.File,
  ...
) (...)

On your terminal with shell you can do

dagger -c 'example --args $(host | directory --no-git-auto-ignore . | file ./args)'
open kernel
#

In that case it's easier to use a different filename, this seems quite messy with more than one argument. The whole point was to use default names to shorten the call.

junior grove
open kernel
#

That sounds nice!

junior grove
#

Yeah you are in the little edge case, I'll open an issue and we'll figure out what we can do

open kernel
#

Thanks, please link the issue here, I'll follow!

#

I feel that calling a function from outside and passing a file intentionally should override the .gitignore settings.

junior grove
#

Yeah that's also a possible fix, I didn't thought about it

junior grove
open kernel
#

👍

graceful surge
#

do we need to apply .gitignore at all for File arguments?

junior grove
#

Yeah that's a right point

#

I'll work on a fix next week

median tree
#

something interesting here is most of the people this has broken for were writing modules that weren't callable using -m github.com/...

#

there are local files that aren't checked in

#

imo - I feel like contextual args should only be able to load from files that were checked into git? and any other files/dirs need to be manually passed in (though those are also using gitignore today, so something smarter is needed)

#

but obviously that's a breaking change, I feel like we either:

  • revert this, and make it opt-in, which defeats the performance boost we were going for
  • allow including gitignored files (and just acknowledge that some modules cant be called from remote git, or used as dependencies, not great imo for the future of cloud)
  • decide we're making this breaking change, and just struggle through (adding module compat so older modules don't break etc, cc @junior grove)
#

i also think it's way too clunky to disable gitignore loading for files/dirs on the host. the option is really long.

median tree
junior grove
#

/cc @graceful surge I think your input is needed here 🙂

junior grove
#

It's a little verbose but it's totally possible to disable git ignore when using the shell.
So I would suggest as a workaround to use dagger shell until we find the right way to disable that behaviour on a module

open kernel
#

I'm using the files in gitignore to pass credentials to the dagger module, they should not be in git. Of course, it's a possible hack and add a placeholder in git and override the files when Gitlab or Github calls dagger. In the local environment this will be very messy.

median tree
#

passing credentials in as files is not a safe practice generally - those files are uploaded into dagger, and stored unencrypted, etc

#

ideally you should be using the dagger.Secret type which is built for this

open kernel
#

Yep, credentials are passed as dagger.Secret, but regular argument are just in a file.

graceful surge
#

breaking change caused by .gitignore

open kernel
versed obsidian
#

this got me today as I tried to pass explicit paths that were in gitignore for post processing and encryption (.artifacts , .cache) where in local run mode i persist those to non git commit folders, but with remote workflow i chain dagger directories together.

thanks for posting this as I'll look through to see how to resolve tomorrow from here

junior grove
fair tangle
#

How do you guys handle this with generated code from other tools?

#

I do not run these tools in dagger, but use the generated code that is in gitignore.

median tree
versed obsidian
#

So the solution is dagger shell?
I don't know how this will would help, as I'm not clear on the usage of dagger shell with flags, as I'm using with dagger call syntax. Last I recall I couldn't use flags.

If you have an LLM prompt/specific page that shows how to adjust from using flags to shell format, I'll play with it, but last call I did with a team member I don't think it did what I needed.

Any advice? I need to knock this out today so any quick unblocking tip would be greatly appreciated.

example one

dagger call template-render-with-version --az-creds=file://$HOME/.azure --source=./platform-standard-role --values-file file://vendir/values/TACOAPP/TACOAPP-publicapi-dev.yaml --role=publicapi --namespace=TACOAPP --argo-rollouts-enabled=true --environment=dev export --path=.artifacts/out/TACOAPP/dev/publicapi/

example 2, generating a secrets manifest then reviewing, then next step takes that and runs sops to encrypt.
For CI, i'll just pass dagger.Directory in a chained workflow that does both like "fetch-and-encrypt", but I want to maintain flexibility of testing steps with local outputs.

dagger call azure-keyvault-to-values-file --az-creds=file://$HOME/.azure --vaults=\"listofvaults,vault2,vault3\" --namespace=TACOAPP --role=publicapi export --path=.artifacts/out/TACOAPP/dev/publicapi/"
dagger call sops-encrypt --pgp-directory .cache/pgp --secrets-dir=.artifacts/secrets-azure-generate --mode=ci --vault-urls=\"https://VAULTNAME.vault.azure.net/keys/sops-key/\" export --path=.artifacts/secrets-generate-ci/
fair tangle
#

Yeah the shell is a bit weird to use if you are used to the normal cli.

#

I got it working like this.

dagger -c 'vue-api $(host | directory . --no-git-auto-ignore | file ./go/gen/apidocs.swagger.json) | export ./web/src/grpc-api'

#

You should probaby chain a lot inside 1 dagger -c call.

versed obsidian
#

if you expect args like the files and creds directory you are using those subexpressions, and achieving it this way without any flags?

fair tangle
#

Yeah it looks like it only accepts positional arguments

graceful surge
#

@fair tangle the shell uses positional arguments for required arguments, and flags for optional arguments

fair tangle
#

Ah tnx 🙂

versed obsidian
#

Ok, I'm getting further now, but I'm going to say i REALLY don't like having to do this. I emphasize cause I have no use right now for dagger shell for CI stuff, it's a big regression to me versus clear flags, going to positional args. I know there was some work being done to support flags in the past, but this just feels very janky

dagger -c 'template-render-with-version \
  $(host | directory $HOME/.azure --no-git-auto-ignore) \ # host Azure credentials directory
  $(host | directory ./platformapp --no-git-auto-ignore) \ # chart source (platformapp)
  $(host | directory vendir/values/myapp --no-git-auto-ignore | file ./myapp-publicapi-dev.yaml) \ # role values file
  publicapi \ # role name
  myapp \ # target namespace
  true \ # enable argo rollouts (true/false)
  dev | export ./.artifacts/out/myapp/dev/publicapi'

I had copilot add line ending comments to clarify what's what.

Is there a better way to be handling this. Is this a me problem? 😄

#

well i guess i can't comment it , so i'll remove that as it errored.

graceful surge
versed obsidian
#

scroll up a tiny bit and i have the problem i ran into there

#

like i mentioned in github issue, I like the safety net of the changes, but without generating files in repo that are git tracked now I don't know of any way around this other than the chained dagger shell, which doesn't make sense to me for pluggin into these ci systems for pipelines non-interactively.

graceful surge
#

You can probably do this:

#!/usr/bin/env dagger

template-render-with-version ~/.azure ./platformapp vendir/values/myapp/myapp-publicapi-dev.yaml publicapi myapp true dev | export ./.artifacts/out/myapp/dev/publicapi
#

Ah I see you're specifically trying to disable gitignore

#

so your problem will go away once we revert that unfortunate new default behavior, right?

versed obsidian
#

well... i don't mind it except I'm always using .cache, .artifact directories as gitignored for all my artifacts. I like that .git was ignored by default in the new changes, that's nice, but not having a way to pass in an artifact path for the next step in processing was tough

#

without a dagger shell wrapper can i invoke this with the paths as well ?

graceful surge
#

no, at the moment the "object address" for a directory or file is either 1) local path or 2) git URL. It doesn't support encoding configuration for gitignore behavior.

#

if we found a clean way to encode it, we could add it... not sure how though

#

I guess file://platformapp?gitignore=false ?

#

(but that's not compliant with the spec for that URI scheme)

versed obsidian
#

While I like gitignore improvements for cleaner work...

Unless I'm not every exporting anything out to debug or test with, where would you normally expect output? I've always used .artifacts or equivalent for build/test artifacts to simplify things, but if it's blocked now that means I could never pass incrementally modified directories or content unless it was in the source tracked files.

Am I missing something or on the same page?

graceful surge
#

Didn't understand that sorry

versed obsidian
#

it's friday, my brain is melting 🫠

psuedocode / flow

  • dagger call helm render output export .artifacts/app/renderedmanifest
  • dagger call --source .artifacts/app/renderedmanifest add-secrets-to-manifests export --path .artifacts/app/rendered-with-secrets
  • dagger call --source .artifacts/app/rendered-with-secrets encrypt-with-sops export --path .artifacts/app/ready-for-packaging-and-encrypted

In CI I would create a "package-go-brr" that would chain those and pass the dagger.Directory or file through in a chain, no need to even touch host. However, a huge part of the appeal is that I can iterate through this and export steps to a directory, test, confirm it's good, and final steps don't have to do the dumping to the drive.

I would never want to export files like this in any path that git would track. Therefore I standardized years ago on .cache/ .artifacts as my two working directories.
Before the changes, I could explicitly pass in a path from there and do what I needed, but now even when passed in it gets prefiltered out.

Hopefully that's a more understandable gist and you can let me know if I'm thinking wrong about something here.

graceful surge
#

Yeah it doesn't make sense IMO that when you explicitly pass a directory as input to a function, it gets auto-filtered by gitignore. That's presuming too much of what the intent of that argument is.

#

So, I consider it a bug that you can't do this

#

on your workflow itself: I find it weird that you have to explicitly pass --source. Normally you should be able to set a reasonable default to those

#

Also small detail but, instead of dagger call ... export --path foo/bar/baz you can do dagger call ... -o foo/bar/baz

versed obsidian
#

Probably my PowerShell days but I avoid short code flags when qualified paths are there for readability. If I can do --out then I'd use that over export as more intuitive

Thanks for the help today!

versed obsidian
graceful surge
#

(this is about the gitignore breaking change revert)