#object/scalar laziness and error handling

1 messages · Page 1 of 1 (latest)

wintry minnow
#

(making a thread btw, if ya don't mind)

elder yarrow
# 
secret := dag.Vault().
  WithToken(tkn).
  Read("mysecret")

// if the secret is nil, it could be because the user does not have permission, or the path is invalid
#

originally I would have handled that like so...

# 
secret,err := dag.Vault().
  WithToken(txn).
  Read("mysecret")

if err != nil {
 return err
}
#

Now I have to do something like this

#

i

# 
secret := dag.Vault().
  WithToken(tkn).
  Read("mysecret")

if secret == nil {
  return fmt.Errorf("unable to get secret")
}
#

The error has lost its context

wintry minnow
#

Got it - what I'm saying is the if secret == nil branch will never run; there will always be a *Secret, because it's lazy

#

Instead you'll get an error from wherever that secret value gets un-lazied, later on

elder yarrow
#

ok this is actually worse then as I can not stop execution

wintry minnow
#

Yeah, if you want to actually force it you'll need to make a request against it. Since it's Secret your only real option is secret.Plaintext(ctx)

elder yarrow
#

I appreciate the changes you have made here but in my humble opinion this API is worse than original.

#

You have moved things away from the principle of least surprise.

#

I now have to understand the context of Dagger and not just the context of my language which I think is going to introduce a whole world of pain when debugging

wintry minnow
#

Fair feedback - the lazy vs. non-lazy aspect has been subject of much debate, and this has been the best set of rules we could find so far. The rules themselves haven't changed since our first GraphQL-based release, but the big difference is now it's being applied to user-implemented APIs

elder yarrow
#

I think your problem is that in Go and other languages I do not expect lazy execution. But also 9/10 times in code like this I lazy execution is a hinderance to me as I always need to unwrap and check.

wintry minnow
#

Sure, but I think you'd find if you ripped it out you'd not enjoy writing Dagger code very much; there's a lot of chaining, and a big part of the ethos is building up one big DAG and letting Dagger figure out how to evaluate it efficiently and deduplicate work

#

But it doesn't quite work as well in cases like this where there are side effects (or external state)

elder yarrow
#

I will be honest, the lazy execution and not lazy execution is what will trip up many of your first time users. If you have lazy execution you need to be explicit and return a promise or something similar.

#

As a go programmer if I have to go out of what is the bounds of what I would see as idiomatic in my language that will not ender me to the framework. Constantly unwrapping async is really not ideal.

wintry minnow
#

We do that as appropriate in each SDK - Go doesn't have promises, so the closest indicator we could find without introducing non-Go-like patterns or boilerplate was the presence of ctx and error

#

If you find yourself constantly unwrapping async things, something's definitely wrong there that may be acutely affecting a certain usage scenario; that hasn't been my experience

elder yarrow
#

If you look at the following example there is pretty much nothing beneficial to run as async.

wintry minnow
#

(Also sorry, not trying to be defensive, just trying to fill in gaps in context since there's a lot of history to this topic)

elder yarrow
#

No i get it, I am just giving you my opinion as a user and long term go programmer. I suspect you will hear the same from others.

#

But these little edges are the differce to me tolelating a framework or language and loving it

wintry minnow
#

Totally - I appreciate it! What I hope to find at the end of conversations like this is whether there's a big area to address that we just haven't had the capacity to model in the right way. I know there's a big gap around external state and side effects right now. We're pretty great at building, decent at testing, not so great at deployment/state yet.

#

Which seems pretty Go-like to me at least. You shouldn't have to think about sync/async, you should be able to just follow the type signatures

#

When you call foo.Bar() and get *Baz in dagger, that's effectively just a client-side promise, no request has actually been made yet

elder yarrow
#

No I get the tuple when grabbing the plain text, you problem is the point of execution.

#

If I get a secret and pass that to another function I do not realize the error until I come to unwrap it

wintry minnow
#

Understood

elder yarrow
#

It is also a bit weird for a module author, when you define a signature on a method. You expect the signature.

#

This is what I mean about the priniciple of least surprise. I understand that dagger does things async so that it can optimise into a graph, but, this is not always a benefit to the end user.

#

In this instance my code would fail anyway it is least surprising for me that it fails at the point of execution.

grand path
#

Hi friends, sorry I'm only catching up to this now.

#

Does this problem only apply to functions that return a core type + an error? Or does it also apply to functions that return a custom type + an error?

wintry minnow
#

It applies to any object return type type (i.e. non-scalar). Scalars have the opposite issue; they'll always be n, err := dag.Fooer().Foo(ctx) even if the other side is Foo() int. So it's orthogonal to custom vs. core types, and more about the asymmetry between client and server due to lazy/unlazy semantics.

#

(At least that's my read on it 😁)

grand path
#

Got it. Is the laziness entirely on the client side of the graphql boundary? As in, it doesn't actually shoot a graphql query, it's still building one, so there is no transport error to worry about?

#

Sometimes I get lost in the multiple layers of laziness

wintry minnow
#

Yeah, foo.Bar().Baz() is just building a query internally, so it can't really fail. (Technically you can pass a nil instead of an object but we decided to just make that a panic, since it's really not worth dealing with an , error at every single turn)

grand path
#

Follow-up question. Once the query actually is made, does the function's actual error get passed all the way back to the client?

wintry minnow
#

I'd be a little surprised, we don't have any special plumbing for that yet so it might just be a cryptic exit status 1 type error with something printed to Stderr

#

Either way it'd probably just end up being a string, unless we get super fancy with error codes etc

grand path
#

eg. I call Container.WithMountedSecret(my_lazy_secret).WithExec().Sync() and the secret function fails, does Sync() give me the actual secret-related error?

wintry minnow
grand path
#

Got it thanks

#

This at least makes it possible to experiment with alternative DX, because it's all on the client side. So in theory @elder yarrow could ship an alternative Go SDK that is less lazy, or does laziness in a less confusing way; and anyone using that alternative SDK would instantly get the benefits for 100% of the modules they need to call - without anyone having to rewrite their modules.

#

One possible candidate is the "query builder" pattern, where you explicitly separate the query building and query execution phase. Then query building would intuitively have no errors. And query execution would always return an error.

Of course you lose some benefits of the current DX along the way. But as we already established, those are subjective tradeoffs.

I've been wanting to experiment with the query builder pattern myself for a while. I don't mind the laziness rules, but the current DX is not good for querying lots of fields. I often want to just drop to raw graphql. And query builder pattern would definitely be an improvement from that.

wintry minnow
grand path
#

I see that kind of metadata as doubling down on one DX vs. others

#

ie. the caller's preference for how to call modules, now leaks into the callee's module

wintry minnow
#

eh, it could be used in different ways by different clients, I'd see it more like parallel advancement 😛

#

For modules it would just be automatically done for your module's fields, so module authors wouldn't need to make any conscious decisions for this that they aren't already making when they decide whether something is a field (state) or a function (behavior). Unless I'm missing something

grand path
#

Yeah but not every function is expensive to call. In fact most functions that return a scalar are not. Now we're annotating functions with something like field and it's a slippery slope, what is a field exactly, etc

wintry minnow
#

I'm not talking about annotating functions 🤔 we already have a concept of fields vs. functions, we'd just extend that metadata to the API by having the fields be marked with a @field GraphQL directive. (Also don't think there's a strong correlation between whether a scalar return value is expensive, you can also have cheap object fields and expensive scalars, i.e. container.stderr())

grand path
#

But what a function is cheap? I can't easily move a static field to be a dynamic function or vice-versa, because it will affect how it is annotated to the client

#

Currently we distinguish between fields and functions in a way visible to the client, but we shouldn't. They are all graphql fields in the end - and we should call them all functions (our own term for graphql fields). Whether the underlying code is a static Go field or a Go method should be an implementation detail IMO

wintry minnow
#

If we didn't have a distinction it would become impossible for a module to have any state, no?

grand path
#

Sorry I added in a way visible to the client

#

Because you're right that the SDK and engine must know about the difference, for state marshalling/unmarshalling to work.

#

But that shouldn't leak to the client

#

In fact sometimes I have fields in my objects that I wish didn't show up as functions to the client

#

(ie I want the SDK to expose them to the engine for state marshalling, but I don't want clients to see them as functions they can call)

wintry minnow
grand path
wintry minnow
#

I can buy into that ideology, but I dunno if people will fully buy in to a pure-query-builder approach; the current one is verbose in that particular scenario, but is otherwise pretty nice. Maybe we can do some kind of hybrid, like make it easier to do multiple-field-selects with the current SDK?

#

(I would still like to see what full-query-builder-pattern client would look like, but implying we either maintain both or eventually phase out the current one is a lot of baggage to carry with it, so just looking for a cheaper/easier answer)

grand path
#

Yeah I'm not pushing to move aggressively on builder-pattern, particularly. We're busy enough as it is 🙂 I just like that we can cross that bridge at any time - and so can anyone in the community. If we saw an ecosystem of alternative SDKs flourish, that would be a great sign for the Dagger ecosystem

wintry minnow
grand path
#

thank you!

#

not in the docs

#

🙂