#Does the cloak netlify Go extension

1 messages · Page 1 of 1 (latest)

blissful wyvern
#

it was working for me last week! do you have a snippet of what you're working with?

exotic needle
#

yes, I'm gonna update my branch

#

My use case is to generate the static website with hugo in a core.Exec, then get the updated FS with the site part so I can upload it in GCS

#

So I was trying to look at how netlify was done, but I can't understand where the static site is sent into the deploy.
It is with the contents param, but this param is not used at all by the code.

blissful wyvern
#

which I guess is what you're doing in the WriteWorkDir. Is there a reason to not use that fsid directly?

exotic needle
#

The WriteWorkDir was to try it first, write it back to the host, now I just want to send to GCS and get nothing on the host. I kept it for testing.

blissful wyvern
#

I see what you're saying about the netlify extension now

exotic needle
#

so, is there some hidden magic?

blissful wyvern
#

there must be, my brain broke trying to work through that lol

exotic needle
#

like mount contents FS in /mnt/contents? Is it based by param name? Is it for only 1 FS input, and it's always called contents?

blissful wyvern
exotic needle
#

Yep, I guess that's how we get the magic chaining!

#

so 1 FSID in the params is mounted in /mnt/contents

#

Problem is, I don't know how to mount any FSID like that otherwise

#

I guess I need to dig in the engine

blissful wyvern
#

Really interesting. I'm looking forward to hearing how/why that works 🙂 If you loook at the yarn extension, it explicitly mounts the arg as part of the ExecInput which is probably the way to go

#

it does look like the resolver has some logic to mount filesystems by itself

#

at /mnt/{arg name}

old cliff
#

@exotic needle @blissful wyvern Yeah any filesystem that you accept as an input arg will also be mounted into the extension's container during execution. This is needed so we can support extensions that directly interact with the files, such as the netlify go example where we want to directly import the netlify go library and use it for deployments, which is much more convenient than trying to do the same with separate execops.

It will also be needed for cases where the extension wants to read large files from an input filesystem. Right now you can also do that with the API (filesystem(id:...){file(path:"foo")}), but that is based on buildkit's ReadFile API which is extremely inefficient in that it reads the entire file into memory and then transfers it all at once over grpc. Until we have better upstream support, mounting the filesystem into the extension is the only viable option for some use cases.

That being said, while the fundamental feature is important, the current implementation is obviously really silly in that you have to look for the FS at a path under /mnt. There's an issue here for cleaning this up: https://github.com/dagger/cloak/issues/33

Here's where all the "magic" happens in the codebase if curious: https://github.com/dagger/cloak/blob/3c20ca3eddbeeb993cfa307c545631f7171430c6/project/project.go#L269-L296

blissful wyvern
#

Awesome thanks for the clarification 🙂

old cliff
#

As an addendum, one really cool thing that should be possible after this is cleaned up is the ability to not only mount an input filesystem in, but to also change it directly in the extension and then return that as an output. So you could have an extension that accepts some FS, modifies it directly in place (i.e. adds, deletes, modifies files) and then returns it.

blissful wyvern
#

Nice! So would you consider it "best practice" (if that exists yet) to operate on the /mnt within extensions, even if the above constraints aren't a factor?

old cliff
# blissful wyvern Nice! So would you consider it "best practice" (if that exists yet) to operate o...

That's the right question to be asking, and honestly we're not sure yet. It's important that both approaches exist because some use cases strictly require one or the other, but in cases where it's possible to either A) directly interact with the mount in the extension or B) to "exec-out" (i.e. provide the input mount to an exec call and modify it there) it becomes much more subtle whether one or the other is better.

If I was forced to write a best-practice guide today, I'd probably start by showing B) since it forces the user to break their steps up into separate execs, which will often result in better caching. However, I would include A) soon after too as an alternative for when it's more convenient and/or necessary to directly modify the mount (provided we clean it up from its current hacky implementation of course).

#

Basically, A) is probably not the default path, but it's not a particularly "advanced" path either, if that makes any sense

blissful wyvern
exotic needle
#

Thanks for the clarification @old cliff

#

if you're not in an extension, but in a engine.Start(), what would be the best approach to extract files from an FSID/FS?
While I'm making my POC, doing it in one main() is easier to reason about than splitting in different extension func, even though, it will make sense to have an upload to gcs extension separated from a hugo static site generation

old cliff
# exotic needle if you're not in an extension, but in a `engine.Start()`, what would be the best...

Yeah in a script you have two options:

  1. Use the file(path:...) api to read the file contents as a string. Should only be used for relatively small and non-binary files
  2. Export the files locally (i.e. {host { dir(id:"xyz") { write(path: "foo/", contents: "some fs id")}}}. Can also replace dir with workdir if you are okay putting the files in the project dir. That will handle large and binary files well, but less convenient sometimes since it forces the files to exist locally.
exotic needle
old cliff
# exotic needle In fact, I would like to avoid local. In that case: https://github.com/dagger/cl...

We don't currently have an api for listing the contents of a dir on a given FS. Adding that would be the actual solution. Workarounds possible right now:

  1. Run ls using an exec on the dir, select stdout and parse it from there (yuck, but maybe not that bad)
  2. Run that code in an extension instead of in a script

Depends on the time frame, but I'd say that if you have time it might not be that hard to add support for listDir. Right now the file API on filesystem is ultimately using bk's ReadFile here: https://github.com/sipsma/cloak/blob/3c20ca3eddbeeb993cfa307c545631f7171430c6/core/filesystem/filesystem.go#L95-L95

There is also a ReadDir method on that same object in bk though, so you wouldn't have to implement all that much yourself.

#

We'll want that read dir functionality for the upcoming new core api too, and for the most part I think you'll be able to re-use what you write here, so wouldn't be throwaway code in all liklihood