Concerns about dagger.gql

[rough crater]

Thread here

[wet yacht]

• Hard to generate. I've never tried so don't know how hard it is. But I know it's possible since we're doing it now for XDX 🙂
• Hard to add extra metadata. That's definitely true. Not an obvious fit for that
• Static analysis. Already been discussed to death, nothing to add here
• Hard to do codegen in CLI: I think that's a non-issue, the CLI can embed the engine anyway. So anything the API can do, we can consider that the CLI can do too

[ivory river]

dagger.gql brings us in the opposite direction where users have to write GraphQL anyway for the manifest
I don't like this either, but I still consider this a step up from forcing use of a yaml DSL on top of our graphql api.

I also find it interesting that since we are creating language abstractions on top of graphql queries, there may be a door where we someday support specifying the dagger.gql queries with actual languages too. That's a quarter-baked idea, extremely hypothetical, but it is an example of the potential benefit of dogfooding what we're building rather than just bolting yaml on top.

• GraphQL is hard(er) to programmatically generate. Burden for tooling (e.g. Adding a text editor in playground for dagger.gql -> easy. Adding a UI on top -> hard)
  Doesn't the playground need a graphql query editor anyways though? I'd rather re-use that than have to support a yaml editor too. But I may be misunderstanding the concern.

• Unclear to me how to add extra stuff in there, if need be. Hypothetical example: if we wanted to add something about resources (e.g. cpu/memory limit, etc) where would it go? Not immediately obvious how to design that query
  Wouldn't we have the same questions with yaml?

If we wanted the cli to understand the manifest, it would require to parse and understand gql.
This is possible and not that bad IMO. It's not as nice as yaml unmarshal but the vektah gql library has been pretty reasonable to use I think.

I also think we want codegen in the API anyways so I'm not sure this would be a concern for very long

[wet yacht]

What's in your "pro" column for dagger.gql @rough crater ? Or is it empty?

[next sinew]

I also find it interesting that since we are creating language abstractions on top of graphql queries, there may be a door where we someday support specifying the dagger.gql queries with actual languages too. That's a quarter-baked idea, extremely hypothetical, but it is an example of the potential benefit of dogfooding what we're building rather than just bolting yaml on top.

I've always thought that we would be able to use lang code to generate the necessary gql code instead of having to write dagger.gql (much like we're doing with the schema, i.e., use the gql file if exists, otherwise query the entrypoint). You don't seem too sure, do you predict complications doing that?

[wet yacht]

It complicates cross-language collaboration and component reuse

[ivory river]

I have more thoughts here but it’s far too huge a tangent to go down now 🙂. Either way the point is more just that it’s beneficial to reuse our fundamental features than try to create extra things bolted on top or side channels. It just keeps more possibilities open in the long run I think

[wet yacht]

Yeah I agree. I just wanted to point out why it's not a no-brainer to use native code in this case (unlike for regular dagger client code)

So, I guess that makes it a "brainer"

I quoted Andrea's notes here: https://github.com/dagger/dagger/issues/3016#issuecomment-1268970478

@ivory river do you mind replying again there when you have a minute? So we don't lose context. Important DX decision so worth the communication effort to get it right.

[rough crater]

re

[rough crater]

Used to be "one language to rule them all". XDX+codegen have me rethinking that.

Basically there's these two conflicting threads going on in my mind: 1) do our absolute best to abstract away graphql (and we're creating some of the best tooling out there, at least in go), while at the same 2) switch our manifest files to graphql

[ivory river]

I totally agree with this concern, it's what I was hung up on for a long time. The reason I nonetheless want to make this change is not because I think .graphql is the best possible option for our manifest, it's that I think it's strictly better than yaml and I have not thought of anything better yet.

[wet yacht]

yeah that's my feeling also, "some native code + some graphql" feels better than "some native code + some graphql + also that yaml file"

I also like not reinventing a DSL for fetching directories when our API is right there, feels more elegant

My main issue with dagger.gql is that it's awkward to fit "normal" configuration metadata

[ivory river]

Yeah, it's more like you are telling any users of your project how to obtain the components of your project, rather than writing the configuration of those components directly. It's a slightly different mindset. But on the other hand:

1. When we use yaml, we're doing the same thing ultimately, just with some indirection
2. I think the dagger.graphql format we were talking about looks really close to a "normal" configuration file, especially if you are just doing really simple things.

All this being said, if there are alternative suggestions that aren't yaml and aren't dagger.graphql, then I'm very interested to hear more. I just haven't come up with anything better whenever I've thought about it

[wet yacht]

Concerns about dagger.gql

toml

cue 🙂

[rough crater]

Doesn't the playground need a graphql query editor anyways though? I'd rather re-use that than have to support a yaml editor too. But I may be misunderstanding the concern.

Ehhh ...

So, as an exercise, replace "graphql" with "sql". Now, imagine to configure postgres instead of writing postgresql.conf you have to write a postgres.sql file. And if you want to automatically generate a config file, just sprintf() SQL queries.

Deep down to me it just feels wrong 🙂 Might just be me though

The only (and I'm guessing here) reason graphql has the concept of $variables is so that you can pass them out of band as json instead of trying to manually "sprintf" gql data (same reason SQL has "where user_id=?", userID ... having to manually generate the query language from code is not a best practice. again IMHO)

That's based on new data to me:

Before: manifest files will only ever be written by users

Now, with the past few playground convos: There might be a UI on top (e.g. "Add to project"), so manifests might be machine-generated

[rough crater]

gasp -- json?

a la package.json

[tender cypress]

noob question, can move out of band if it's too unrelated: is it possible to configure extensions within the query/API itself?:

{
  withExtension(image: "vito/dagger-concourse:1.0.0") {
    # this scope extends the parent (Query)
    withExtension(image: "daggerio/banana:1.2.3") {
      # this scope extends the parent (Query + Concourse)
    }
  }
}

or is there more to dagger.gql/cloak.yaml to that?

imo it's always nice to have everything portable within the API, carrying around config files is kind of a pain vs just having everything in code.

[wet yacht]

You make a good point about the "add to project" thing. It seems to me before we discuss the format, we should talk about the content a little bit. I'm not 100% sure what goes in that file in this "add to project" scenario

[wet yacht]

Problem with that is you need to load extensions before introspecting the endpoint

Otherwise you don't get full API docs, query auto-completion, etc

[tender cypress]

gotcha, so it's leveraged by more tooling for a better DX (possibly codegen too?)

[wet yacht]

Source: #maintainers message

🙂

Yes including codegen

[rough crater]

yeah. also "how do we build you"

which is a chicken and egg problem

[rough crater]

eighth-baked idea: it's json, you can manally edit if you want, but there's some tooling on top (e.g. dagger add universe/yarn)

[wet yacht]

can we talk about the content?

Does it need to change in any way with a "add to project" button in cloud?

I just want to double check before we go too far in the format / language discussion

[rough crater]

I'd guess cloud needs to figure out one way or another which extensions it wants. Then ship that to the engine alongside the query

[ivory river]

I think that's different though; dagger.graphql isn't configuring knobs of our server. The closest thing we have to that is buildkit's config file, which is actually TOML 🙂

The way I thought about dagger.graphql in the past is that it's a graphql operations file. It defines the operations clients should call in order to use a given project. I think with that mindset, it actually wouldn't make sense to submit the full dagger.graphql to the server from the browser; you would just submit the individual queries used to setup the extensions.

But then that circles back to what we discussed before; the configuration of extensions wouldn't be a header anymore, it would remain an API. Which I actually think is nice in many ways, except you had concerns about how it would work with the browser, which etc.etc.etc. I'm getting lost in all the details now

[wet yacht]

Yeah but, for example: we discussed in the past the possibility that extensions in universe would be "special" and possibly have an API call of their own?

so in dagger.gql:

query Extensions {
  netlify: universe { netlify(version: "foo") }
}

[ivory river]

As as addendum to the above, can you remind me why it's important for the browser use case that loading extensions not be an API and instead something shipped along-side the query? Was it the "distributed worker" use case? If so I have more follow up questions, but want to double check I'm oriented correctly

[rough crater]

I don't think loading all universe extensions and having them become part of the schema would last past a few dozen. And it doesn't provide any indexing for lookup purposes, the engine would need to load all records at once at boot before being able to serve the API endpoint.

An analogy would be to use this approach in from instead of providing a string, e.g.:

query {
  container {
    from {
      dockerhub {
        alpine {
          exec(...)
        }
      }
    }
  }
}

Also, programmatic generation would be a little tougher, especially for catching errors. (e.g. universe("netlifyZZZ") --> error: package "netlifyZZZ" doesn't exist ; universe { netlifyZZZ } --> graphql syntax error)

[ivory river]

I don't think loading all universe extensions and having them become part of the schema would last past a few dozen. And it doesn't provide any indexing for lookup purposes, the engine would need to load all records at once at boot before being able to serve the API endpoint.
It's hard to say since it's getting into implementation details, but if universe is stored in a git repo somewhere, then I'd imagine the implementation of the universe resolver (and subresolvers) would just be using llb.Git to pull the records and the subresolvers would just be reading from a file pulled there to get the extension and load it? I guess I'm not sure A) if there's anything really wrong with that and B) how we will solve these same problems otherwise.

[rough crater]

Yes, correct. That is if you go with universe(name: "netlify") or something -- as you said the universe resolver would use llb.Git (or something -- to pull one file) and load it on the fly

If you go with universe { netlify } however, then all of universe needs to become part of the schema, not on the fly but at "startup" time

[ivory river]

Yeah for the latter the universe extension would have to do all of that when it's loaded (actually may fit in with the code-first schema model well coincidentally), which adds to the boot time of the engine (but by how much I don't know).

I don't like whipsawing the conversation back and forth between different threads but I do think it's important we are all on the same page about my question above in terms of whether we are intending to support the "distributed engine" use case where a browser may be connected to one engine but then another and everything should work seamlessly. I agree that if we want to support that use case then the whole "load extensions via api" model doesn't really work anymore (the extension config has to be a header or something).

I just think that there's probably a ton of other implications to supporting that model which might greatly complicate it, to the point where we should consider whether we actually want to achieve it. The alternative would be something more like a "session" model where you assume you have a continuous session associated with a single engine and any state like "loaded extensions" is scoped to your session. In that case "load extension via API" is reasonable and probably simpler. That seems more plausible to me atm, but this whole area is a huge unknown so low confidence

[rough crater]

1. stateless

"intending to support the "distributed engine" use case" --> it's not a hypothetical, I don't think GA should ship without at least 3 instances. Not (just) for scale, but for redundancy. I definitely don't want to carry a pager around for a single server instance somewhere 🙂

Stateless is just infinitely better to manage than stateful.

re: session: We should probably have them. But sessions should be treated as best effort, as an optimization layer (e.g. caching). If we rely on sessions to carry state, then it means updating one instance (e.g. bumping cloak, rebooting, failure, or whatever) will break the client session. Which means now it's up to the user or client to try and un-break things.

You mentioned running a playground will give us operation experience with cloak, and this is exactly an example of that. If a user deploy a few instances of cloak on k8s using a Service+LoadBalancer, it WILL round robin (pets vs cattle, etc etc).

2. multi-tenancy

there's different API users. If session is carried on the instance, how do we handle that? Will a user will see in the schema whatever others have loaded on that particular instance (which might be different from browser to browser, session and all). What happens if they load different versions of the same extension? conflicting extensions?

making it stateless solves all of that

[wet yacht]

catching up after a call..

[wet yacht]

universe itself could be an extension 🙂

[ivory river]

Stateless is just infinitely better to manage than stateful.
I fully agree with the sentiment here, but in the general use case (not just playground) there's a fair bit of "session-stateful" baggage we carry over from buildkit's concepts of sessions and some of our own ideas:

1. Local dirs (including workdir) are associated with single buildkit sessions and their loading/caching behavior is reliant on that session
2. Services (aka gateway containers) are associated with a session
3. (probably more depending on where things like secrets end up, but you get the idea)

I realize those two are not things we want to support for the playground, but does that mean that the "playground-flavored" engine is the only stateless engine? And the "real" engine is actually stateful when it comes to sessions?

You mentioned running a playground will give us operation experience with cloak, and this is exactly an example of that. If a user deploy a few instances of cloak on k8s using a Service+LoadBalancer, it WILL round robin (pets vs cattle, etc etc).
I agree but part of supporting that use case will involve managing state associated with a session, unless we don't support the above features.

there's different API users. If session is carried on the instance, how do we handle that? Will a user will see in the schema whatever others have loaded on that particular instance (which might be different from browser to browser, session and all). What happens if they load different versions of the same extension? conflicting extensions?
The idea in my head is that you:

1. Connect to a cloak engine, establish a session (use of websockets would make the most sense here, but I suppose a "cookie" approach that contains a session ID would technically work too)
2. "Configure" your session (load extensions you want to use)
3. Do the stuff you want to do
4. Close the session, at which point the engine releases resources associated with your session

The above approach would support us having multiple engines in our playground backend (we don't just need one, we can have 3 or more), it would support multitenancy, it would support each user getting their own "view" of the schema even when connected to the same engine (no issues with different clients loading conflicting extensions). It would also keep our playground engine in line with the problems our "generic" one is going to have to deal with too.

Right? 🙂 I'm not really that experienced with this sort of thing, so I'm happy to be proven naive. I again completely share the sentiment of "stateless is better", I just don't think we can avoid it in the general case, and then that makes me think we may as well just accept it and design the rest of our stuff around acceptance of that idea.

^ And connecting back that to the OP, part of accepting that we need to support session-state is that the loaded extensions are part of that state, which means that configuring the loaded extensions via API probably makes more sense, which in turn makes the idea of dagger.graphql more sound (you wouldn't need to generate it and submit it in HTTP headers, which I 100% agree would be extremely odd)

[wet yacht]

@ivory river just to be sure, you're talking about a scenario where extensions are loaded by making calls to the core API? Or are you talking about a dagger.gql which happens to contain graphql queries? They both technically rely on graphql queries, but in a very different way...

btw the concept of "session" is confusing to me, as it's described I don't think we want or need it

[ivory river]

I'm saying that I think it makes sense to load extensions via API in a world where we have to support session-specific state anyways. And if we are configuring which extensions are loaded via an API, dagger.graphql makes perfect sense since it just contains the operations you need to perform load the extension in the project, essentially

[wet yacht]

I think there is a natural concept of a "project":

• A project has a file specifying what extensions to load
• Everytime you call the engine, you must pass a project file. The engine is otherwise stateless (no session needed)
• In a multi-tenant cloud you need to attach a project ID to every request. Possibly on a http header although the exact mechanism seems like an implementation detail

[ivory river]

Then we need to fundamentally rethink some basic things like local directories and services 🙂 Which may very well be the case! I again am not opposed to the sentiment of stateless being preferable, it just comes with an enormous number of implications that I don't know how to deal with fully

[wet yacht]

In the CLI you could run dagger --project MY_PROJECT_ID_ON_CLOUD and the client would dynamically query the correct config file from cloud in order to load the correct set of extensions

(perhaps that CLI feature could be instrumented by the playground as well)

[wet yacht]

I'm thinking 🙂

This seems related to our clustering architecture, right? The fact that we have only one engine, but a typical Dagger installation will have more than one engine running, and playing different roles

client --(graphql api)--> local engine ---(grpc api)--> one or more worker engines

when we talk about state we should probably be specific about which part of this diagram we're talking about

(sorry about vague terminology)

[ivory river]

Yeah it's also tied into all of that, which makes this even harder to think about. I suppose what I'm referring to right now is tied to the state in memory in the local engine in your diagram above. As we iterate on daggerd things may shift a bit, but in practice at this moment the local engine has in memory:

1. the buildkit session (which is associated with A) loaded local dirs and B) gateway containers, which are how services will be implemented)
2. the schema for the loaded extensions

[wet yacht]

for point 1. this is as a buildkit client correct?

[ivory river]

Yes correct (daggerd client soon, but same thing), we are just piggy backing off of buildkitd's session management for scoping of those resources to individual clients

[wet yacht]

"buildkitd session management" in its simplest form, you get one session per client connection right? And optionally you can share session ID across connections?

Just making sure I understand technical reality

And, just to make sure: sessions are ephemeral? Once the last client holding that session ID disconnects, that session is discarded by buildkitd? Or is it more complicated than that

[ivory river]

Yeah one session per client. There is also a few fields for re-using a session ID from different client, but Tonis told me they don't actually work, implementation was never finished.

[ivory river]

No that's exactly it

session related state is held in memory by buildkit

and then released once the connection from the client is closed

[wet yacht]

So when you use the word "session" earlier in the thread, are you describing a buildkitd session in this sense, or something else?

[ivory river]

I'm saying that we already have inherited the buildkit session state due to our use of local directories, gateway containers, etc. So maybe we should embrace that concept and re-use it for our own dagger features too (especially makes sense in light of us now wrapping buildkitd), including for things like "loaded extensions".

So to answer your question, I'm describing extending the buildkit session into a new thing (call it dagger session)

A dagger session would encompass:

1. Local dirs and gateway containers (basically, everything in the buildkit session state)
2. Which extensions are loaded for your requests, as configured by api calls made earlier in the session
3. More??

[wet yacht]

Yeah that sounds like a project to me 🙂

My starting point is to just think of Cloud Playground as a regular dagger workflow, but as a service

So you have a host, a dagger engine which can access certain things on that host, and a bunch of workers that can run containers and the engine can farm out jobs to

And of course you have a project file telling the engine what extensions to load

When I imagine a multi-tenant version of that, I see:

• A custom dagger client, probably developed with the Javascript SDK, that is spawned every time a playground user clicks "run"
• The client conjures up a cloak.yaml (or equivalent) in a playground-specific way. Then it spawns an engine the usual way (fork-exec) and passes the cloak.yaml to that engine.
• Workdir is set to "scratch" to make implementation easier (as discussed today)
• Engine then uses a custom provisioner to get access to one or more buildkit workers (no docker for mac here)
• Finally client sends the user's query to the engine, the usual way (via JS SDK)

In a basic implementation, the playground web app itself could import our JS SDK, and fork-exec right there on the web server. Assuming we have the provisioner part in place, that web server may not need to run containers at all - it would just fork-exec one local engine per request, which can probably be optimized but is reasonable for a v1

It seems more sustainable to invest the work in a second provisioner (could be just remote ssh to a separate box), rather than engineer a custom wrapper to remote-execute the local engine

@urban patio 👆

[wet yacht]

So this paints a slightly different picture than what you're seeing @rough crater (I think), because the graphql server part of the engine doesn't need to be made multi-tenant: it's easier to just spawn a new engine each time using the standard SDK. This way no need to send any headers, or any sort of non-standard internal API. The part that is multi-tenant is the buildkit worker pool, but that's already supported by the buildkit API and no custom headers needed there either. We just need to support regular buildkit sessions.

(gotta run and pick up the kids)

browser ---http--> playground web server using dagger SDK ---forkexec + graphql--> local engine with custom provisioner ---bk/grpc---> remote worker

actually I guess a simpler first version is to have everything on the same box (web server, local engine, worker) so no custom provisioner needed . You can still load balance but each box needs the full combo

[rough crater]

This way no need to send any headers, or any sort of non-standard internal API.

How does that work?

[wet yacht]

as described above 🙂

[rough crater]

:?

[wet yacht]

which part doesn’t make sense to you?

(biking for now)

[rough crater]

• The client conjures up a cloak.yaml (or equivalent) in a playground-specific way. Then it spawns an engine the usual way (fork-exec) and passes the cloak.yaml to that engine.

it's a SPA, the code runs in the browser

[wet yacht]

aaah!

yeah that part I’m ignoring

thought you meant headers in some internal api bw web server and multi tenant engine

I’m guessing the SPA would send eg. project ID by header

OR project id is looked up from playground id server side

either way not terribly important to how we implement multi tebancyu

[wet yacht]

in this sentence I mean dagger client (ie js or go code running on the server and embedding the dagger sdk)

[rough crater]

ok, that's a new component

[ivory river]

Sorry my brain is fully dissociated now, we need to figure out all of the above but I also was planning on starting cloak.yaml->dagger.gql tomorrow and we need a decision. It obviously ties into way too many other things that we won't fully figure out for a while.

Looking back through the thread, I remembered this suggestion from @rough crater to just make the project config a json file and then have the interface for interacting with it be via the dagger cli. I think that would not be too bad:
A) the fact that it's interacted with through cli makes it better than the current state of hand writing yaml
B) it's easy to generate json a browser too (no questions of generating full graphql operations files)
C) if we say that the cli is the only official way of interacting with it (contents are internal implementation details) we have paths to migrating to different formats in the future in backwards compatible ways (probably)
D) we can still support all the flexibility of using graphql queries in theory; we'd be embedding them as json strings but if we are saying that the contents of the file are internal only it's okay if they are ugly; we can make the cli interface for interacting with the file the polished part (in time, not right away)

I guess the main idea is that all human interaction should go through CLI or browser UI; contents of file are machine-readable only. We can make it json for now or really whatever we want.

Are there any objections to this?

[wet yacht]

is it? I’m guessing playground has a server already

if not then yes new component (again from just me describing my assumptions, could be all wrong)

[wet yacht]

👀🧠

[wet yacht]

now I understand what you’re saying @rough crater . Will digest and try again

[ivory river]

I'll assume this and start working on it tomorrow until someone raises any objections. All the stuff about session-state vs statelessness is going to be very important in the cloud design obviously but also all of the next steps after Guillume's PR to embed buildkit is merged. But I think the CLI-only config interface makes it completely orthogonal, so can save the other conversations for other threads

[wet yacht]

Works for me

• I’d like to get a few schema changes in rather than transpose cloak.yaml as is. I can rebase my proposed change from the gql version to your json version & we can discuss there (mostly terminology stuff)

• People will still want to read those files and sometimes edit them by hand. We should design accordingly. So making it 100%?CLI-only would be a bit too far. But in practice thay won’t be a problem IMO. having it be json will be enough of a deterrent, documented or not

TLDR ship it

[ivory river]

I’d like to get a few schema changes in rather than transpose cloak.yaml as is
Agreed, there were some other changes being made w/ dagger.gql like only supporting one extension, not needing scripts, etc. I will also make all those changes here too.

People will still want to read those files and sometimes edit them by hand.
Also agreed, can't stop them from doing that. I think the difference is more that we won't optimize for beauty or human-writability of these files; that's secondary now to the CLI (and web browser) interface. But we also don't need to make them machine code or completely unparsable by human eyes. There's a balance point for sure.

Okay hopefully @rough crater doesn't object to his "eighth-baked" idea from a few hours ago and we should be good 🙂

[wet yacht]

I like that

https://tenor.com/view/fortune-teller-crystall-ball-psychic-clairvoyant-gif-5100696

[rough crater]

Yeah, sounds good to me if you think it does the job.

What I like is it's much simpler / smaller project which means:

1. We'll have more time to work together on xdx+codegen? 🙂 Which I think is much more impactful
2. The format is "boring" in a good way. If we ever decide to do something else in the future, it doesn't get any easier than this to have auto-migration

[ivory river]

2. The format is "boring" in a good way. If we ever decide to do something else in the future, it doesn't get any easier than this to have auto-migration
   💯 It's incredibly boring and decoupled from everything else, which is a very good thing when everything is changing every hour

What I like is it's much simpler / smaller project which means:
I actually think it's probably more work 😂 But not so much more that it matters (like maybe an extra half a day). So I'm just going to agree so we can move on with our lives

[rough crater]

Oh you think it's more? I was guesstimating less

[ivory river]

My guesstimate is more (dagger.gql is just "submit the query you read"), but the error bars on estimates are so large that it's not statistically significant to make a comparison in reality. I'm not worried about it

[rough crater]

I was thinking swapping yaml.Unmarshal with json.Unmarshal, porting over the changes from dagger.gql, a bikeshedding session and done

[rough crater]

AFAIK sending dagger.gql wasn't enough -- it doesn't do anything. You'd have to parse back the response and do a depth search of things that look like IDs in order to load them

so it's the same system as a yaml file I guess -- the engine grabs it, selects one query within, does some parsing of the response etc

the client sending verbatim wouldn't have done the trick

I think?

[ivory river]

It's the CLI interface part (crud operations essentially) that is new, but it may turn out to be completely trivial, idk

[rough crater]

oh yeah that 100%

[ivory river]

Either way, it's the right call so we'll figure out how to make it happen before preview, will see how it goes tomorrow

[rough crater]

actually, the CLI work doesn't necessarily have to be done by you. Switching to the json format yes, since it's deep down in the engine. But then we could find a volunteer to add the commands to manipulate it, I think? Unless it's more complex than simple "CRUD" on a json file

(note: a cool part of having commands is we can better integrate them, e.g. dagger add universe/yarn or whatever could kick-off codegen automatically, so it's ready to be used immediately after)

[ivory river]

True, honestly it might end up being extremely simple the more I think about it (just read, update in memory, write back to disk), but if gotchas come up I'll defer it and ask around for help

[serene knoll]

dagger add universe/yarn
love it

[wet yacht]

maybe we should skip the CLI part for now?

also should we talk about relationship to a lock file?

is it the lock file too?

[ivory river]

I think it's important and may even save time in that we don't have to write docs on how to manipulate it manually.

[ivory river]

Was suspecting it may be that, yeah

Not implementing locking tomorrow, but it would make sense to shove that in there too once we add support for it

[wet yacht]

fair, my concern is that the CLI ux is tightly coupled to the cloud ux for managing this file. I worry about designing one without the other

[ivory river]

Yeah totally, I think the ux is the trickiest part. I am just going to try the quickest possible implementation of it, but if it that is hard for whatever reason or has too many one-way doors we can decide to skip it for now. It's just worth a quick shot imo

[urban patio]

TL;DR? IIUC

We'll bring a similar package.json | package.json.lock concept to replace cloak.yaml and @wet yacht will focus / add comments on polishing the current schema?

[wet yacht]

might be just one file but yeah basically

extra benefit @sterile cypress @serene knoll get their json file for free for universe

[serene knoll]

I think the universe json might still have a different shape, I’m not sure. Maybe not though

[wet yacht]

possible

tbh the deno approach is looking more and more attractive

[serene knoll]

Im all for that. I know we were trying to keep it simple with a static site so adding a database would be more scope. Probably justifiable if we know we want to go down that road anyway

[wet yacht]

I just realized they probably can redirect downloads to github which is a nice hack

[rough crater]

Potentially, yeah. Worth looking into both and check for patterns

package.json IIRC is the same for dependencies and also npm publishing. Even if you create a hello world react app, in its package.json there's fields like license, author, etc

I don't think we should force a round peg in a square hole, but if they naturally converge that'd be cool

[wet yacht]

I was wondering about that. You can add custom fields too

It's an intriguing idea. There are risks too though. to be discussed 🙂

[wet yacht]

(sorry I thought you were talking about actually using the actual package.json format, but I think I misunderstood)

[rough crater]

@atomic bobcat More context on cloak.yaml->dagger.yaml in this thread.

@ivory river is starting working today on cloak.yaml -> dagger.json

https://github.com/dagger/dagger/issues/3016

[ivory river]

@atomic bobcat leaving cloak.yaml as cloak.yaml probably will be easier for any existing users, I see what @wet yacht is saying about annoying them by changing it 2 times in rapid succession. If it's not too painful it's worth making that update quickly before merging

[ivory river]

@wet yacht @rough crater Please take a look at the dagger.json PR when you have a chance: https://github.com/dagger/dagger/pull/3281

There are TODOs left (listed in description), but it's clean enough now and passing tests, so want to get feedback before finalizing any more.

The only gotcha I ran into in terms of the ux is the issue I described under sdk field. Let me know if it makes sense. It's one of those things we can safely ignore in the short term but don't want surprises down the road.

[ivory river]

^^ Did some rebasing now so it's not based on other PRs, may ease review of actual code, though right now feedback on the general direction+ux is what's most needed

[rough crater]

Will do! We were meeting in person today, that’s why you didn’t hear back from neither of us

[ivory river]

No worries! Appreciate it whenever you have time

[ivory river]

@wet yacht @rough crater re-pinging for a review on this, time-sensitive because I don't want to leave it hanging when I leave for vacation Saturday morning: https://github.com/dagger/dagger/pull/3281

[wet yacht]

Thanks for the ping, on my list for today