#Object print + module discovery

1 messages · Page 1 of 1 (latest)

tender seal
#

🧵

finite shore
#

I don't think so, because the previous behavior was:

  • Core type: XXX evaluated
  • User object: Error: requires sub-selection
  • Main object: --help

Only the last one provided help, but I think current behavior is better.

jaunty talon
#

Before, when it hit that error state would it show you usage?

#

Yeah I think that's the difference - now you tend to just see _type: MyType without further help

tender seal
finite shore
#

Yep, only when dagger call without any more args.

jaunty talon
#

Some sort of 'choose your own adventure' UX would be nice

tender seal
#

Also it wasn't perfect before, for example if the module constructor has mandatory arguments, you don't get to see anything until you set it

finite shore
#

Yeah, I don't think defaulting to --help is a good idea. A better default is to just list the functions but that's what dagger functions do. This way it's more consistent. And you could have fields in the main object that are useful, like version, etc..

tender seal
#

What I propose is that finish this object-print feature (I don't consider it finished yet, there are 1-2 missing parts) then give it a chance. And then decide what we want to add or change

#

The add-ons I have in mind specifically:

  1. Walk all fields recursively
  2. Objects are always printed regardless of how many fields are printable
#

(I'm using "field" and "function" interchangeably here)

finite shore
#

In 2) isn't it more useful to at least have _type?

tender seal
#

Ah right. Yes agreed

finite shore
#

That's how it works now then.

tender seal
#

Right, just making it explicit in combination with the other point (walk recursively). Basically always print the objects whether they are the top-level or not

finite shore
#

Ah, I see.

tender seal
#

I'm hoping that way it solves 1) function discovery, and 2) elaborate test reports all at once

finite shore
#

We'll need a depth limit for that, I think.

tender seal
#

A depth limit flag is a great idea. You think we'd need a hardcoded depth limit as well? I was hoping not, but maybe not realistic

finite shore
#

We could default to 2 perhaps. But I need to think about this through. Imagine a function that changes the current object and returns itself. How do you exclude the many WithXXX functions in core? And how do you do the same for user modules?

#

Obviously need to handle infinite cycles as well.

#

I don't think this is viable for function discovery since some of the functions need to naturally be ruled out. Including them anyway pollutes the output too much. It's better to just use dagger functions.

tender seal
#

All the WithXXX will be excluded anyway because they always require an argument

#

There are a few exceptions though, for example Container.WithoutEntrypoint

#

So I guess we would have to catch cycles (identical IDs)

#

But there's no specific depth limit that would catch this one

tender seal
tender seal
finite shore
tender seal
#

We could catch functions that return the same type

#

That one would be easy

finite shore
finite shore
tender seal
#

It's not a firm goal for me to use object print for discovery. I'm just curious to see if it might help as a side effect

#

Personally I need recursive walking for my "test reports" use case. I want to dagger call test or dagger call lint and see my full set of checks across several layers of objects, arrays etc

finite shore
#

Yeah, the recursive feature, on its own, is new. Haven't tried it yet and ruling out self chains may help.

#

The "printer" should already be able to render it (i.e., accross lists and objects), just need to figure out the query building part.

#

Don't expect it to be hard, just need some time to try it 🙂

tender seal
#

I'm simulating it by hand on the dagger repo to get a sense

#

Simulation (only a small subset, didn't have the energy to do it all by hand):

$ dagger call -m github.com/dagger/dagger/dev
_type: DaggerDev
_description: Develop the Dagger CLI
check: ✅
cli:
  _type: DaggerDevCli
  _description: Develop the Dagger CLI
  file:
    _type: File
    _description: Build the CLI binary
  publish:
    _type: Void
    _description: Publish the CLI using GoReleaser
  test-publish:
    _type: Void
    _description: Verify that the CLI builds without actually publishing anything
engine:
  _type: DaggerDevEngine
  _description: Develop the Dagger Engine
  container:
    _type: Container
    _description: Build the engine container
  generate:
    _description: Generate any engine-related files
    _type: Directory
    entries:
      <ENTER TOP-LEVEL ENTRIES OF GENERATED SOURCE DIR>
  lint: ✅
  lint-generate: ✅
  publish:
    _description: Publish all engine images to a registry
    _type: String
  scan: |
    <INSERT 100-line long json-encoded scan report>
  service:
    _type: Service
    _description: Create a test engine service
  test-publish: ✅
  with-arg:
    _type: DaggerDevEngine
  with-base:
    _type: DaggerDevEngine
  with-config:
    _type: DaggerDevEngine
  with-race:
    _type: DaggerDevEngine
  with-trace:
    _type: DaggerDevEngine
sdk:
  _type: DaggerDevSdk
  _description: Develop Dagger SDKs
  elixir:
    _type: DaggerDevSdkElixir
    _description: Develop the Dagger Elixir SDK (experimental)
  go:
    _type: DaggerDevSdkGo
    _description: Develop the Dagger Go SDK
  java:
    _type: DaggerDevSdkJava
    _description: Develop the Dagger Java SDK (experimental)
  php:
    _type: DaggerDevSdkPhp
    _description: Develop the Dagger PHP SDK (experimental)
  python:
    _type: DaggerDevSdkPython
    _description: Develop the Dagger Python SDK
  typescript:
    _type: DaggerDevSdkTypescript
    _description: Develop the Dagger Typescript SDK
version:
  _type: DaggerDevVersionInfo
  commit: ""
  dev: 4d09cc3f6020231677e894fd5026aff850fc087f
  string: dev-4d09cc3f6020231677e894fd5026aff850fc087f
  tag: ""
jaunty talon
#

Once it starts printing dozens of lines of YAML I'm gonna start getting K8s flashbacks

tender seal
#

Joke aside, yeah I agree that is a risk. That snippet is not even arguing for anything, just (made up) raw data to observe

tender seal
#

I put those little ✅ emojis as a reminder of that

#

I guess technically that's still yaml but anyway

#

My first observations:

  • You're right it looks "kubey"
  • It's verbose (this is probably 50% of the whole thing)
  • It's very greedy. Will compute a lot of thing out of the box, like a giant make all. Not necessarily a bad thing, but definitely opinionated
  • Weird mix of schema and value
finite shore
#

yaml is difficult to edit but easy to read. In Python, the ecosystem has adopted toml format en masse though. Easy to manipulate, easy to read.

jaunty talon
#

TOML is better suited for shallow ini style configs though in my experience. But yeah valid point about YAML being an implementation detail.

tender seal
#

My dilemma right now:

  • For dagger call test to print the whole test report, I need recursive printing
  • But with recursive printing dagger call at the root of the module will greedily run tons of pipelines, including probably all tests, all linting checks, and scans. It will produce a very complete and useful output (kind of like make all but even cooler) but will take forever to run
#

Then again, maybe greedy execution is a good use of dagger call -m github.com/dagger/dagger/dev ?

#

Or at the very least, not aggressively bad, since nobody relies on it anyway. dagger call is kind of like a scalpel where you call what you need. Maybe it's an edge case to "call" the whole module, or a large subsection of it?

#

@dry lichen you're going to love this discussion 🙂

finite shore
tender seal
finite shore
jaunty talon
dry lichen
# tender seal Or at the very least, not aggressively bad, since nobody relies on it anyway. `d...

I don't like any implicit calling of functions unless there was opt-in via the author saying "call this function by default" (via annotations) or the caller explicitly including the function in their dagger call ....

The behavior of aggressively calling functions by default would create a big restriction on how I structure and write modules+functions. I'd need to be constantly cautious about not accidentally writing a function with a signature that's gonna get auto-called by the CLI and trigger execution I don't want to happen. e.g. I can't have functions for calling different overlapping subsets of tests since they would all start running in parallel unnecessarily.

But if Functions are opt-in via an annotation, then that sounds perfect and like it would be super useful. And doing this all recursively (print fields + annotated functions recursively) makes it even better 👍

tender seal
# dry lichen I don't like any implicit calling of functions unless there was opt-in via the a...

I think it greatly depends on how undesirable that auto-execution actually is. Remember this would happen when you dagger call the whole module, something that nobody does today.

If we were to ship this feature, users would call the top-level module with the explicit intent of "walking" it. In that context I don't think it would be undesirable for your overlapping tests to all be executed - or rather that would be up to the end user to decide.

This is all assuming we find a clean way to skip functions with side effects.

dry lichen
# tender seal I think it greatly depends on how undesirable that auto-execution actually is. R...

For the test use case, if I have a bunch of functions that call different overlapping subsets of tests, I would find it much much more useful if I, as the module author, could just choose one of those functions to be called by default if the user just types dagger call.

And we can enable that pretty easily with annotations on functions that say "call this by default". And requiring that annotation to automatically call a Function also avoid all of the problems with side-effectful functions. So it seems like something that solves a bunch of problems at once.

#

In general I don't love the idea just adding more and more annotations for every conceivable setting we could ever want to support, we should be cautious in adding them, but this seems like a case where it would make perfect sense

tender seal
#

I dislike the idea of an extra annotation so much, that I'm reconsidering my objection to printing pre-computed fields 🙂

#

The root of my objectin was really: the backbone of our API is GraphQL; and GraphQL doesn't differentiate between a field (static) and a function (dynamic). Fields are functions, functions are fields. So it made sense to stick to that model for our own API, since any additional distinction would have to be custom metadata. Which creates a precedent for deviating from GraphQL as source of truth. But if we really want to introduce a concept of field, and are serious about doing it propertly: defining them rigorously; documenting them well; justifying why they're useful, etc. Then perhaps it's worth deviating. And at the same time, there is a precedent for extra annotations already: default path, ignore.. By the way I have no idea how that is reflected in the graphql schema. How do you introspect that? If I had clarity on that, perhaps I would feel even less bad about promoting fields as a first-class concept