@field in typescript SDK

[stone bison]

Thanks, reading now

is that explanation also available in the "option B"?

What is a field? What's the difference between a field and function?

Is "field" a typescript concept?

Why do I need decorators at all? What happens if I just add a method to my object? Why isn't that enough to make it a function?

[grim dirge]

A field is a variable that is part of the Dagger state, a function is a callable field that can execute logic

[grim dirge]

It's the same in Python, and in Go with public variable

[grim dirge]

Because exposing something to the dagger api is explicit in typescript and python

[stone bison]

OK, they all seem redundant to me, but maybe it's because I'm not used to typescript

I look at a function, and there's a decorator saying "this is a function"

[grim dirge]

Dagger will only expose the fields of a modude that are explicitely exposed using the @field() decorators, the rest will stay private in the context of your class.

[stone bison]

I look at a class and there's a decorator saying "this is an object"

[grim dirge]

Saying this is a func to expose to the Dagger API

[grim dirge]

This is an object to expose to the dagger api

[stone bison]

Yeah I understand.

[grim dirge]

everything is explicit basically

[stone bison]

What about "field"?

There is no concept of field in the Dagger API

[grim dirge]

Same

[grim dirge]

There's a concept of state

fields are the state

Public variable in Go are part of the state, in ts and python it's identified with the @field decorator

[stone bison]

OK. Where's the documentation explaining this?

[grim dirge]

so you can call the field using dagger call

[grim dirge]

This should be part of a global documentation

/cc @ashen sage

I only explain the convention for Typescript, because I assume we'll have a global doc explaining that

[stone bison]

I think we have a problem of passing on major DX problems onto documentation

[grim dirge]

Yep we mainly focused on feature and implementation, the doc isn't ready yet

We're still looking for the right structure

@grand niche You might be interested in this thread too

[stone bison]

This concept of "field" is broken

Docs won't fix it

[grim dirge]

wdym?

it's the same as in Go, why would it be broken?

[stone bison]

Go does not have the word "field" used in a Dagger-specific way

[grim dirge]

Nope because Go is implicit

You expose what is Public during the serialization

[stone bison]

Public to who?

The SDK or the end user?

[grim dirge]

The end user

[stone bison]

Wrong

I want the SDK to serialize my Go struct fields, but I don't necessarily want those fields to be visible to the end user as functions

[grim dirge]

If I have a public variable in a Go struct, I can access it from the dagger API using dagger call no?

[stone bison]

(we don't expose a concept of fields to the end user)

[grim dirge]

So why the end user can access them with dagger call?

[stone bison]

by default yes but I can hide it with // +private. In any case if it is exposed, it is as a function. There is no end-user-facing concept of field

In any case, you answered my original question, so thanks

[grim dirge]

In Typescript and Python, it's the reverse, everything is // +private except if you tell explicitely dagger to expose it

Field is just the way to declare it, we could have name it @property or @state or anything

[stone bison]

Yes but by saying @field as something different than @function, you're implying that there are two different things you can expose to the end user: a field, or a function. That is incorrect.

If it's exposed to the end user, it's a function

[grim dirge]

That would have been strange to decorat a variable @func

[stone bison]

I'm just explaining the problem to you

I don't know what the solution is

[grim dirge]

The other way around is to remove these decorators and not expose things that has the keyword private, which exist in both language

And about the class, theses that have the keyword export

But then, we cannot support alias or we would need a decorator @alias

However, I think being explicit with @field and @func is a pretty nice way to understand dagger: a module has a state composed of fields and a set of functions that are exposed to the Dagger API and callable by the end user

[stone bison]

That's a wrong understanding of dagger though

[grim dirge]

What's the right one?

[stone bison]

What I explained before

[grim dirge]

But it's not actually true, Dagger use GraphQL to expends schema and a GraphQL has a concept of fields and functions

[stone bison]

Your property can be persisted, or it can be exposed as a function, or both

[stone bison]

wrong

[grim dirge]

{
   myFunc(someArg: "aaa") {
     field
   }
}

In a GQL query, the field isn't a function, it's a field

[stone bison]

GraphQL has a concept of field. It has no concept of functions.

What GraphQL calls fields, we call functions.

Confusing I know 🙂

[grim dirge]

But, you can call function in GQL? I don't understand

[stone bison]

You can replace field with field(), they are equivalent in GQL

every graphql field is like a function. As a convenience, if you pass no arguments, you can skip the ()

[grim dirge]

Ah ok I didn't know that

[stone bison]

It's definitely confusing because of the overlapping use of the same words in different parts of the stack

[grim dirge]

Well then this ^^ is a way to be in phase with what you said

So it's the same as in Go, everything is a field (so a function) except if you don't want to by adding a keyword

That would require a LOT of refactor tho...

[stone bison]

We definitely don't have time for a refactor before the release...

All I can do is clarify, and complain 😛

[grand niche]

There's another level of confusion here because solomon isn't aware there's actually the concept of field in our API.

[grim dirge]

So we make a difference between fields and functions

[grand niche]

https://docs.dagger.io/api/reference/#definition-FieldTypeDef

https://docs.dagger.io/api/reference/#TypeDef-withField

[stone bison]

I am aware, and disapprove 😛 as discussed elsewhere

fields should be purely a concern between SDK and engine

And in practice, today it is

But I worry that will not remain the case, because of the confusion

Yeah the description fo FieldTypeDef in that API doc is definitely wrong. It talks about static vs. dynamic, which is not the point of having fields. The only point is state serialization

[grand niche]

Yeah, I hear you. Wasn't as clear to me until recently. But makes it weird when mapping to a language. That internal concept of field was because we wanted to turn struct fields into functions without having to create getters. We can say that a class's properties are conveniently turned into functions. With that argument maybe it's not too weird to use a @func decorator in a class property.

[stone bison]

Right, saying things like "conveniently exposed as functions" would definitely help

[grim dirge]

I find it pretty weird since our internal API has a concept of field

[stone bison]

Even without changing the current DX, if we said @field: expose this property as a persistent field of the Dagger object, to be persisted by the engine in between calls. By default, the property will also be exposed as a function, This can be disabled with ???

[grand niche]

In Python it's a bit different. The @object_type decorator is a wrapper on a @dataclass. Similarly, the "field" in name: str = field() is also a wrapper on top of dataclass.field. So there's a language specific explanation on why it's called "field", without having to say it's a Dagger field.

[stone bison]

only for the purpose of SDK-to-engine communication (for persisting object state)

[grim dirge]

In our case, it's disabled if you don't put @field notation

[grand niche]

You can say that if you don't put @field it just doesn't create a getter function in the API for you, but if there's concern about exposing the internal concept of state, we can just not talk about it.

[stone bison]

It's fine to talk about it, since it does affect how your module works. It would be even better if we find a way to clarify the distinction between 1) functions that the client can call, and 2) state that the engine persists for you

[grand niche]

But 2) is basically what module fields are. So I'm at a loss on how to reconcile here.

Fields do two things: 1) persist state between SDK-to-API calls; 2) create a corresponding getter function to access that state.

[grim dirge]

We could also do something release like it is right now and add a big refactor for v0.11 where you don't need decorators anymore -> fields and functions become implicit like in Go?

[stone bison]

Maybe just saying that is enough, then

Simply saying that a field will get (by default) a corresponding getter function, helps understand that field and function is not the same. It opens the possibility that there could be a field without a getter function, and then the user couldn't access it

I don't think that part was clear to @grim dirge before this conversation

[grand niche]

No, all fields create a corresponding getter function. If not, it's not a dagger field.

[stone bison]

Then what does // +private do in the Go SDK?

[grand niche]

+private simply doesn't expose the struct field as an "API field".

[stone bison]

lol

[grand niche]

With -> https://docs.dagger.io/api/reference/#TypeDef-withField

[stone bison]

so there's "state fields" and "API fields" now?

[grand niche]

There's a language's object state, in memory, and there's API state in the server.

[haughty monolith]

No, // +private just doesn't expose the getter function, that's it. It's still just state that gets passed around

[stone bison]

And then there's what I can see with dagger functions

[stone bison]

OK I'm not crazy then! I was starting to wonder 😛

I just used // +private today, and it did seem like it was doing what I expected, which is 1) continue to persist the field's state. while 2) removing the convenience getter function for it

[haughty monolith]

I think the confusion is that today it's implement by just never telling the engine about //+private fields, as opposed to telling the engine about and saying it's a private field. But it creates the same end effect, total implementation detail (and we may need to modify this in the future, so the engine knows about it but just doesn't create a getter)

[grand niche]

How does that work with typedef system? Do you still call withField on that?

[stone bison]

But if the engine doesn't know about it, how does it get persisted?

[haughty monolith]

It's because the results of graphql functions resolvers are just generic "result maps" (typically serialized as json objects). If you put some object keys in that json that aren't officially in the schema, they still get passed around transparently. This is used all the time in the standard js gql server, it works in the go one we used to use, dagql, etc.

[grand niche]

Oh this wasn't fresh on my mind at all 😄

But Python supports the same.

[haughty monolith]

I don't think the gql spec explicitly says "thou shalt support this" but its in practice a convention that seems to be universal

[grim dirge]

Same for Typescript

We even save in the state variables that have the keyword private

[grand niche]

So @field == objectTypeDef.withField --> creates a getter function

Still have serialization without it.

[stone bison]

maybe if we start talking about "object state" instead of "fields" it would help clarify the terminology over time?

If we had time to refactor before release (which we don't) I would probably use @state instead of @field; and by default not expose a getter function.

(or maybe a bad idea since we have '"field" firmly embedded in the API already)

As long as we agree to talk about "functions" as the only way for a client to query a module, I'm happy

[grand niche]

To not expose a getter function in Python or Typescript, just don't use the decorator. Meaning it's a docs change. Only Go is defaulting to it.

[stone bison]

But what if I want to persist that property in my Dagger state, but not expose the getter function? How do I do that?

[grand niche]

In Python and Typescript? Still works. The decorator is just to create the function.

[grim dirge]

Yep, everything is saved in the state

[grand niche]

For example, in Python:

@object_type
class HelloWorld:
  msg: str = "hi"

  @function
  def say(self) -> str:
    return self.msg

Here, msg is still persisted as state because Python will include the "msg" property when it serializes the object.

[stone bison]

Just to make sure I understand:

In the Typescript SDK, @field has no effect on whether a property is persisted in the state or not? All properties of an object are always persisted?

So @field only exposes that property as a getter function.

Correct?

[grim dirge]

Yes

[grand niche]

You need to additonally do this to create the getter function:

-  msg: str = "hi"
+  msg: str = field(default="hi")

[stone bison]

OK I understand

[grim dirge]

As I said ^^ @field is just a matter of exposing it to the end user or not

[stone bison]

Then I propose that @field be deprecated by @function. Not a release blocker of course.

[grim dirge]

This is why I talk about it in field exposition section

[grim dirge]

Same for Python?

[stone bison]

I don't know enough about the Python SDK to have an opinion. But you can probably determine what I would answer if I did.

[grand niche]

I can't use the same @function decorator for what field does. Need a different name. As I said before, it wraps Python's dataclass.field so to access necessary features from there, users will need to use dataclass.field rather than dagger.field in many cases. Python users know what a dataclass field is and how to work with it.

But I'm ok changing the name, just can't use the same function name.

[grim dirge]

What about func?

(Maybe a dumb idea)

or state or property

[grand niche]

Can't have a func and a function. Can't change both to the same otherwise we're on square one.

[grand niche]

It needs to represent what it actually does in comparison to when you don't use it. Basically creates the getter function (i.e., calls obj.withField).

[grim dirge]

@expose?

[grand niche]

How about this? With the Strawberry library, you can define GraphQL resolvers either from a property:

@strawberry.type
class Query:
    last_user: User = strawberry.field(resolver=get_last_user)

Or a method:

@strawberry.type
class Query:
    @strawberry.field
    def last_user(self) -> User:
        return User(name="Marco")

I can use the same approach. Needs quite a bit of refactoring but I can work with that.

It would allow keeping the same name:

@dagger.object_type
class Hello:
  msg: str = dagger.function(default="hi")

  @dagger.function
  def greet(self) -> str:
    return self.msg

But if you don't want to create the getter function, you may need to use "field" anyway:

class Hello:
-  msg: str = dagger.function(default="hi")
+  msg: str = dataclass.field(default="hi")

[stone bison]

dataclass.field would be needed to serialize the state I guess? So, different from typescript in that way.

[grand niche]

Yes. In the example above it's not necessary, but if your default value is mutable (like []) then you need a factory/callback, in which case you need dataclass.field. Just an example as there's other settings for it. More realistic:

class Hello:
-  msgs: list[str] = dagger.function(default=list)
+  msgs: list[str] = dataclass.field(default_factory=list)

Is it clear to say that dagger.field is like a dataclass.field but which also creates a getter function for it in the API?

[stone bison]

I would say it depends how familiar and standard dataclass.field is to python devs. If it's as familiar as go's convention for json marshalling of structs, then perhaps dagger.field is not needed

If dagger.function could be made to work with both fields (to add getter function) and methods, then that serms to be the best option, best of both worlds

I found this thread very helpful and instructive, thank you all for your patience 😁

[grand niche]

I would say it depends how familiar and standard dataclass.field is to python devs.
It's quite familiar, yes 🙂 It's from the standard library.

If dagger.function could be made to work with both fields (to add getter function) and methods, then that serms to be the best option, best of both worlds
The only problem with that is I don't have time to make that big a change for v0.10.0.

[stone bison]

oh yeah, to be clear I don't think we should make any major change to the SDKs for the release

[grand niche]

But we can do it later and deprecate dagger.field.

[stone bison]

It does help to discuss future design changes now, because it can inform what we say in the docs now. We can put the right "lens" on the current technical reality.

[grand niche]

Yeah, I'm glad we reached some clarity/consensus.

"It's as if dataclass.field and dagger.function got together for a code merge and ended up committing dagger.field to their repository of life."