#nvm.. fake news.. seems to be uploading

1 messages · Page 1 of 1 (latest)

silent jay
#

I think it should be changed to something like:

includeSet[src.SourceSubpathWithDefault(ctx)] = struct{}{}
merry hollow
#

yep, that's what I was thinking about

#

I can actually confirm that dagger functions does not include it

silent jay
#

idk if you have time to try that quickly, but otherwise just an issue works and we can get to it before the next release

merry hollow
silent jay
merry hollow
merry hollow
#

defintely way faster Erik! the repo I was initializing on has a 300mb docs directory with a bunch of files. That was penalizing the init time a few tens of seconds to upload the whole thing 🙏

#

I'll add a test and send a PR tomorrow 🙌

merry hollow
#

I also have another theory about the codegen penalizing the init time quite drastically since I'd assume it goes through all existing *.go files (in a go SDK project) to scan for candidate functions to be generated?

#

silently pinging @silent jay

#

in my example, in a repo with ~900 go files, a cold codegen takes ~ 50s

      ✔ exec /usr/local/bin/codegen --output /src --module-context-path /src/dagger --module-name tracetest
 --introspection-json-path /schema.json 43.8s
silent jay
merry hollow
silent jay
#

hm, yeah, it's at least faster on subsequent runs once the local dir was loaded right?

merry hollow
silent jay
#

I think we need to do more general performance investigation into the filesync, it feels like something weird is happening if it takes 43s

#

like it's actually doing way more work than it should be

merry hollow
#

is there a way to know that? the engine debug logs don't discriminate that

silent jay
#

oh I see, yeah .resolveFromCalleris the part where filesync happens

#

And it was fast because it was entirely cached

#

so it was just codegen that took forever

#

if you change a single file in the repo does codegen take 50s again?

merry hollow
merry hollow
silent jay
#

oh, maybe I'm misreading the output, yeah actually "cache request" is saying "request to the cache" I think. So I guess the filesync actually was just only 300ish ms, which is good

merry hollow
merry hollow
silent jay
#

One expensive thing codegen needs to do is download all the go mod deps

#

So wondering if that could plausibly explain it. But really the only way to do this correctly would be to put a bunch of a spans in the codegen code so we can drill down to what is taking so long

merry hollow
#

the dep download starts after a few tens of seconds

silent jay
#

Ah I see that now, cool

#

hm

#

Yeah just looking around nothing sticks out as obvious, so I think the best approach would be to put spans all over that codebase

#

That'll make it obvious what's taking time

merry hollow
#

👍 np, I can follow-upt with an issue

#

could it be that the codegen process is actually scanning through all the *.go files uploaded in the context?

#

instead of only looking into the module source files only?

#

well.. the codegen is part has a bunch of pieces.. I guess the best thing to do is to add some spans there to instrument what its' doing 🤞

silent jay
merry hollow
#

definitely doesn't happen when initializing dagger in an empty folder

#

it only takes a few secs

merry hollow