#Go get cmd to pull dagger module into different project

1 messages · Page 1 of 1 (latest)

agile imp
#

Hello Team

I have dagger module which is working fine using dagger cmd
dagger -m git@abc.com:namespace/daggerverse.git/module1@v0.0.2

this lives in monorepo where multiple dagger module is there, they live within their own directories containing go.mod

Requirement:
I need to get/install above module in other project(not a dagger module but a dagger project using sdk, kind of legacy orchestrator). I am using go get cmd to get that module but getting this error

<<Cmd>>
go get abc.com/namespace/daggerverse/module1@v0.0.2

<<Error>>
go: module abc.com/namespace/daggerverse/module1@v0.0.2 found, but does not contain package abc.com/namespace/daggerverse/module1

Could you pleae guide

edgy nimbus
agile imp
#

I need to install this module in non module dagger project, how to achieve that?

#

that project is simple go project utilizing dagger sdk only

#

so basically i'm looking for go get cmd

#

may be worth asking, is it possible to use and import dagger module in non module dagger project?

edgy nimbus
agile imp
#

thanks for the information

is there a workaround that you think I can do for now, to use/exeucte module from my project, i have been checking sdk api dag.LoadGitModuleSourceFromID()
but not sure if its right direction 🙂

edgy nimbus
#

Our CLI does this. There's an API to load a module, but then you need raw GraphQL queries to call functions in that module. It's not going to be available as dag.MyModule().MyFunc().

agile imp
#

Could you please guide me here with an example, I would appreciate the help

agile imp
#

@edgy nimbus any example would help me unblock for now 🙏

tawdry quest
agile imp
#

Lovely, thats what I was looking for, I was almost there myself with scratching codebase 😆

#

can it work with already running dagger sdk session?
also i see api doc says this and i need to load multiple module in same session

// Serve a module's API in the current session.
//
// Note: this can only be called once per session. In the future, it could return a stream or service to remove the side effect.
func (r *Module) Serve(ctx context.Context) error {

I have a requirement to move part of old dagger jobs into dagger module one by one

edgy nimbus
#

Thanks @tawdry quest!

#

this can only be called once per session

That's interesting. dagger shell does it multiple times in the same session. It's either no longer an issue or I need to double check if things are actually running as expected. 🤔

tawdry quest
#

I guess what's it's saying there is that you can only call Serve for that specific module once in every session? does that make sense @edgy nimbus ?

agile imp
#

I have one more thing to ask, how to prepare a graphql with file, one of module parameter is dagger file type
file *dagger.File
i was getting error
decode "File" ID: failed to decode base64: illegal base64 data at input by

tawdry quest
#

that should work

agile imp
#

Arg("filepath", dagger.FileID("path/to/file/in/host")
i tried something like this, is that what you mean?
btw i am still getting error

#

i think file.ID works for native dagger file

tawdry quest
#

it needs to be a native dagger file

agile imp
#

but i want to prepare file from host

tawdry quest
#

if you need to pass a file from the host, you need to first get a refference for that file with dagger.Host().File("path/to/file") and then get the ID from there

agile imp
#

it worked 🙏
thank you very much both of you @tawdry quest @edgy nimbus

tawdry quest
agile imp
#

One small query -
Is it possible to access dagger module within mono repo from non module dagger sdk in my above setup, not via git link?

tawdry quest
agile imp
#

yes

#

something that can be packaged during compile time, we have a binary that has dagger related code

agile imp
#

@tawdry quest let me know if you have any solution for this 🙂

tawdry quest
agile imp
#

not sure if that loading would be compile time, we have binary that we built, can this will be included during build phase too?

tawdry quest
#

so that path needs to be present whenever the pipeline is executed

agile imp
#

Btw, Is there a way to secretly pass git credentials while accessing module, similar to secret in containers?
Fo reg: https://username:read-token@gitlab.com/namespace/project.git

tawdry quest
agile imp
#

Please correct me, this should also work if I provide module url with auth in the url itself?
https://username:token@gitlab.com/namespace/daggerverse.git/module@v0.0.5

damn i did not know that this support is automatically in 0.14.0, my bad

we cannot upgrade at this moment, but I beilieve above way should work with 0.13.0 as token is set directly in url, right?

tawdry quest
#

the proper way is to use the PAT supported introduced in v0.14

agile imp
#

thanks for heads up

actually, i have tried using PAT with new dagger sdk and engine, still I'm getting same error
I have already set git credentials in host machine as root
will dagger engine needs to be restrated again? because module still failing for download, the same is working in host machine if I do with cli with module cmd against dagger container

tawdry quest
#

cc @stable salmon 🙏

stable salmon
#

Could you please check that echo -e "protocol=https\nhost=gitlab.com\n" | git credential fill does indeed trigger an authentication prompt for you ?

And, could you please also share the .gitconfig ?

agile imp
#

basically i have to do these in dagger engine host machine in order to run that manually

git config --global credential.helper store
echo -e "protocol=https\nhost=gitlab.onprem.com\nusername=username\npassword=abxcd\n" | git credential approve

it's working for me only when dagger call execute within same host, still trying to figure out why remote call is not working

#

quick question:
I am currently on sdk/engine 0.13.0, so in order to get this feature, do i have to upgrade sdk/engine both or one of sdk or engine should be upgrade? tyring to understand if change is coupled with engine

stable salmon
#

An SDK not updated would run an engine below v0.14, thus not having the feature

agile imp
#

so we need both in order to work properly?

stable salmon
agile imp
#

so sdk, cli and engine must be at 0.14.0

stable salmon
#

Yep

#

Keep me in touch 🙏 And all the Git setup that you showed ⬆️ has to be on the host from which the CLI will run the commands

agile imp
#

wait, our setup is like this

  1. dagger engines runs on remote servers
  2. we have gitlab runner runs on k8s, there we have cli and cli connects to 1.)

now all these settings that is require will be done in 1.) host right? and not in host where cli present?

stable salmon
#

I haven't finished the docs

#

But it's a starter

agile imp
#

but all these settings working for me because i added changes in dagger engine hosts

#

and not in cli host

stable salmon
#

I don't think you need that no ; that's not how the code is designed to work

#

The engine host is useful for the git rewrite rules if you have some or some other configs, but the PAT retrieval does come from the CLI

agile imp
#

but modules download happens in engine side right?

stable salmon
agile imp
#

hmm, may be i have been doing a lot of testing to and fro on cli host and engine host, so i could not notice it

stable salmon
# agile imp hmm, may be i have been doing a lot of testing to and fro on cli host and engine...

The idea was: if git can clone the ref locally, we should work too: as git does have the git credential standard, we should piggyback on that.

Now, the user in this issue had the exact same pattern as you do: https://github.com/dagger/dagger/issues/8989#issuecomment-2484406869. Feel free to peak ideas, but it's a strong start: in this gitlab pipeline, I retrieve the username and secret that were exposed as secrets / env var in the pipeline

#

Keep me updated, happy to sync on the issue 🙏

agile imp
#

yeah sure, thanks for help 🙂

#

we use custom gitlab runner running on k8s, i was thinking if there's merit to directly provide credentials globally instead of doing it for each gitlab job?

stable salmon
agile imp
#

also we have vault access in k8s runner

stable salmon
agile imp
#

gitlab runner are only for gitlab jobs which has cli and we have high resource baremetal boxes which has dagger engines
both has access to vault

stable salmon
agile imp
#

so i think i can configure globle auth in gitlab runner directly, not sure if I hit any blocker there

stable salmon
#

yeah makes sense, I would do that too 👍

agile imp
#

cool i will let you know how it goes 🙂

stable salmon
agile imp
#

btw we are already doing this in before_script, but it is not working

doesn't implementation considering this?

agile imp
#

i tried your example, it's working fine from gitlab job, but when I execute these setting in runner directly(exec in machine), the same did not work

GitHub
Issues · dagger/dagger

An engine to run your pipelines in containers. Contribute to dagger/dagger development by creating an account on GitHub.

agile imp
#

Also during testing I always have to test with fresh module commit version, otherwise there's always a cache delusion got me happy, but its not 😦

stable salmon
stable salmon