Go SDK & docs

[viscid solstice]

@dusky laurel @high gate @spark granite @dark kelp @next jungle @wanton dove

[dusky laurel]

The documentation structure I'd proposed some time ago includes a node for SDKs under Reference/SDKs/XX SDK. However, I'm not sure how this content would be generated. My efforts so far were focused only on auto-generating reference docs for graphql.

[dark kelp]

per-language sections makes a ton of sense to me, especially since the SDKs might look pretty different from one another depending on patterns in the host language. it's really tempting to just defer to the generated godoc and call it a day, but I think we should have more example content too. one way to do that while still deferring to godoc would be to add handwritten examples to a package.go in the generated core SDK docs, like this: https://pkg.go.dev/net/http#example-FileServer

i'm leaning pretty hard on godoc here because I think folks are going to end up there anyway, and it'd be nice for it to all be in one place, but I don't know if we'd be able to have an experience consistent with this for other languages. so, just a discussion point.

that also raises a point of how we expect extension authors to maintain example content; it's not really practical for them to write docs for each language 🤔

[spark granite]

@next jungle's PR includes the generation of doc comments on the Go SDK from the graphql schema comments. E.g. https://github.com/dagger/dagger/blob/bf78389144c73b0cfbbc957cf505176ff8b1787d/sdk/go/dagger/api/api.gen.go#L21

So I'd be curious what the generated go docs look like for that package. Maybe they are already good enough to get started?

That, in general, should be a pattern we try to reuse in many ways I think. I didn't add it in codefirst schemas for go yet, but it would be pretty trivial to support turning comments on your go methods+structs into graphql schema comments (believe that's all built into the go ast parser).

When we add support for other languages, should try to use the same patterns again.

[spark granite]

That's a good question, and reminds me of something we talked about in the past where maybe we support two-way conversion in the query builder. Right now we basically have support for converting Go->Graphql, but if we also had the ability to convert Graphql->Go, then all our supported languages could convert queries between one another. You could convert Go->Graphql->Python for example. Graphql becomes the hub, languages are the spokes.

There's a lot to figure out there, but based on my (admittedly limited) experience so far with the codefirst stuff, this isn't quite as insane as it sounds. The tooling around all this sort of ast stuff is pretty decent (thanks in large part to the fact that LSP+linters need it so much I think).

If we had that, then examples could be provided as "query builder snippets" in your language. We could then convert that to documentation in other languages as needed.

That's a pipe dream right now, but is in the realm of plausibility. I'll open an issue

[viscid solstice]

@lapis edge 👆

Threads are weird

[lapis edge]

ah ok, now it shows up. Thanks!

[viscid solstice]

Trying out an idea:

Since we are looking at significant changes to the planned docs contents and structure, on a tight schedule (go sdk preview launch next week), what if we reduced the scope of the docs for that first launch, buying ourselves more time to work on the rest?

Suggestion: since next week is Go SDK preview, why not include only Go SDK docs?

(discuss) 🙂

That could actually be shipped as a standalone docusaurus versioned docs site - assuming SDK has its own release and versioning cadence, which IMO is inevitable

This would allow us to dodge a lot of questions about how everything will fit together in Dagger 0.3. By not shipping a complete docs structure for dagger 0.3 right away, we won’t invite those questions

Even the method of installation of the dagger binary might end up being SDK specific after all (if it’s even required?)

[spark granite]

We could just ship the go docs generated from the client here: https://github.com/dagger/dagger/blob/bf78389144c73b0cfbbc957cf505176ff8b1787d/sdk/go/dagger/api/api.gen.go#L21

I don't know if they would be on our site our linked from our site (haven't done anything with go docs in the past really)

[viscid solstice]

Go api reference would be a great start , but we’d need to add more content probably for it to be useful standalone. Introduction, concepts, examples, architecture explaining how dagger fits into your Go code etc

[dusky laurel]

I have a rough draft of the get started guide in https://github.com/dagger/dagger/pull/3171/files - the final two steps are incomplete as some of the functionality needed was not available when I wrote it.

We could adjust this as follows:

• remove the graphql examples, change the SDK example to use Go
• modify the two complete examples (local and remote deploy) to use Go
• publish this together with the auto-generated API reference

However one issue is that I don't know Go, so one of our team would need to rewrite the examples using the Go SDK and work with me to explain the code.

[viscid solstice]

I hate to throw more meetings at a problem, but @spark granite @dark kelp would you mind joining our live discussion of this topic with @dusky laurel tomorrow ? I think it could help speed things up

[dark kelp]

yeah no worries!

[spark granite]

Yep, happy to

[viscid solstice]

🙏 @fleet cipher @keen spindle would be awesome to have you too, but if one of you could stay & MC community call to the end , that would be awesome (they overlap by 30mn)

It’s a big meeting but we’re getting better at those 😉

My working hypothesis is:

• We define a “minimal awesome docs” for our Go SDK
• Has to be useful standalone (not just generated API docs, that’s necessary but not sufficient). Basically a miniature product
• Launch blog post links to that
• We make a rough task list. Dispatch work across the team. It’s a an all hands on deck situation
• In parallel we continue planning how this will fit into the overall future dagger docs (but we have a little more time to complete that)

All the above is up for discussion and can change. It’s a starting point.

Ideally all work adjacent to Go SDK docs (demos, talks, videos) should contribute to the Go SDK docs somehow. Recycle effort as much as possible

@dusky laurel remains the editorial bottleneck and all content should flow through him for consistency

This group should over-communicate early and often to make sure we get all the content we need and it all flows through the editorial bottleneck efficiently and in time for launch

[keen spindle]

@fleet cipher is going to work on a Go example for a good, old-fashioned build, test, release use case that go devs/devops could relate to and doesn't use extensions. Since cloak can do multi-arch well, we'll highlight that too.

Makes most sense to me to have @fleet cipher join docs meeting and I'll hang out til end of the community call and join the docs group afterwards.

[viscid solstice]

That all sounds perfect, thanks

[fleet cipher]

Started putting the demo together, the new dx looks really great!

Quick question on actually importing the api. Should I be using dagger client-gen in my project, or importing go.dagger.io/dagger/sdk/go/dagger/api?

[spark granite]

Importing

(for now anyways)

[fleet cipher]

Ok, that's what I expected! I'm running into this and I don't quite understand why

go get -u go.dagger.io/dagger/sdk/go/dagger/api@7cbadb5
go: module go.dagger.io/dagger@7cbadb5 found (v0.2.37-0.20221012201757-7cbadb56a08a), but does not contain package go.dagger.io/dagger/sdk/go/dagger/api

[spark granite]

does go get -u go.dagger.io/dagger by itself work?

[fleet cipher]

it does! but go run still tries to look for the api submodule as a separate import for some reason, so there's a disconnect

go run ci/main.go
ci/main.go:8:2: no required module provides package go.dagger.io/dagger/sdk/go/dagger/api; to add it:
    go get go.dagger.io/dagger/sdk/go/dagger/api

cat go.mod | grep dagger.io
    go.dagger.io/dagger v0.2.37-0.20221012201757-7cbadb56a08a

[spark granite]

Oh I think 7cbadb56a08a is from the main branch

try 405b55e9157cb5d880bdfe6e280d9ac4cd0eb25a

[fleet cipher]

did I actually get burned by an abbreviated shasum lol

oh, nope, just copied the wrong one

[keen spindle]

🍿

(while I'm going through a long overdue golang tutorial)
(wow CUE actually helped me a lot with getting the feel for golang stuff. Thanks, CUE)

[viscid solstice]

https://tenor.com/view/thank-you-thanks-thank-q-cute-kid-gif-17045545

[fleet cipher]

it's alive 🎉
new dx: https://github.com/kpenfound/dagger-demos/blob/main/go/hello-container/magefiles/main.go
old dx: https://github.com/kpenfound/dagger-demos/blob/main/go/hello/magefiles/main.go

now to add a more full story around publishing and running

[viscid solstice]

https://tenor.com/view/hug-the-sun-andrea-gibbs-bonnie-davies-radical-rad-gif-21647783

@fleet cipher I pinged the author of Mage so we can show him 🙂 Thank you @stuck orchid for making that connection!

[fleet cipher]

Oh awesome! It feels really natural for dagger tbh

[viscid solstice]

btw it occurs to me we should make sure the Go SDK docs have at least one decent example that is not a magefile, to avoid confusion especially for those that don’t yet know mage

[keen spindle]

yep @fleet cipher and I had the same thought. Mage not universally known

[viscid solstice]

we could accidentally give the impression that they are all the same product. Had this problem in the early days of docker where we relied so heavily on Vagrant for Mac & Windows users that they were super confused

(that started the many iterations that eventually led to docker for mac)

[fleet cipher]

Yeah exactly, same page!

[viscid solstice]

Fun fact: I wondered if Tailscale deviated from their KB article model (similar to ours) for their API reference. It turns out that they don't even bother shipping their own API reference doc: just link to Github. https://tailscale.com/kb/1101/api/

Doesn't mean we should do the same necessarily, their API is peripheral to the core UX. In our case the API is the core UX

Their CLI reference is a regular KB article, ID 1080: https://tailscale.com/kb/1080/cli/#command-reference

They have basically found a way to not deal with versioning of docs at all, by farming out anything that requires versioning to a special case.

• API reference: link to separate system (in their case github, but in our case could be docusaurus)
• Downloads: link to standalone page tailscale.com/download

They don't have SDKs, so who knows how they would handle that. That's a whole new set of problems

Then of course there's the f&^^&**ing voodoo magic of the Stripe API docs... where you can seamlessly switch between languages at any time https://stripe.com/docs/api?lang=ruby

[spark granite]

That's what we should do

[viscid solstice]

Years later that page still makes my brain 🤯

Their docs as a whole are pretty sprawling and I get lost between stripe.com/docs and stripe.dev (they link to each other). But that's just the nature of a giant platform with layers of evolution... Good problems

[viscid solstice]

Yeah it's appealing for sure. But a major tooling endeavor... We would have to consider it as a first-class product engineering effort. That would mean less engineering time available for other things. wdyt? worth it?

Stripe is known for having taken this approach on their docs (critical product feature as opposed to afterthought) and it seems to have worked out well for them 🙂

But it's the kind of thing that doesn't work without full engineering buy-in. Including being OK with the tradeoffs

[spark granite]

Oh yeah "That's what we should do" was short for "That's what we should do in the fullness of time once we have appropriately scoped and prioritized the effort".

It's an absolute killer feature in terms of docs though. It also may have overlapping effort with other nice-to-have ideas like the ability to translate query builder snippets from one lang to another (to enable cross-language examples).

My immediate impression is that we don't have the resources to say "we must do this now". We could probably get away with simpler approaches (different doc pages for each lang) for now.

Which is sad but is what it is.

[keen spindle]

wow, the way they capture ctrl/cmd-F for search too

[viscid solstice]

We do have the resources (in my opinion). However the question is: are we willing to pay the price. It may be too high.

(the price being: lots of other things don't get built, possibly including things we consider absolutely critical)

(also it would not be ready next week, obviously)

but we could conceivably aim for that end result (if we like the architecture) and ship a tiny first slice

[keen spindle]

yeah, I think we can get a "lite" version if we just take approach we have already with our CI tabs here: https://docs.dagger.io/1201/ci-environment

or better, the Install page: https://docs.dagger.io/install/

[viscid solstice]

@spark granite setting aside difficulty, resource constraints etc. Is there any reason we should not structure our SDK docs in that way? Some fundamental mismatch between their product and ours? They are pretty different

@keen spindle you're right that's a good comparison point. However doing it for the API reference doc would be a whole extra dimension. For example does our use of GraphQL instead of REST change anything here? Maybe the 1-1 mapping between HTTP, JS, Go etc. is somehow different?

As opposed to the baseline I suggested above, where each SDK is its own standalone thing. Probably lots of duplication, less wow effect, but also less constraints in having everything structured the same, paragraph for paragraph, no matter what

[keen spindle]

yeah, the unification can come later

[viscid solstice]

Yes but - 1 year later at least

Once we're set in one direction or the other, it's going to be a straight line for a while. Short of blowing it up and rebuilding from scratch IMO

[spark granite]

I really just think we should be able to have snippets in our docs (example queries) and then there should be a button for each of our supported langs (plus raw graphql). Those snippets should be all over our docs, and they should also be interactive too. (I'm speaking idealistically here of course).

I don't think the other aspects of how their docs are something to emulate (as you said quite different). I just love that I can translate from one lang to another by clicking a button. It's both useful inherently and it also makes the general message that dagger supports multiple very clear to everyone

Actually pulumi does the same sort of thing: https://www.pulumi.com/registry/packages/aws/api-docs/s3/bucket/

[viscid solstice]

OK so we could start with snippets as a middle ground, even if the overall structure is not completely tied. Same suggestion you made @keen spindle right?

Basically like that CI example, but for all our snippets

[keen spindle]

Yep. @lone kayak and @rare zenith know how to build React components to model things like that that are super embeddable in our docs/website.

[viscid solstice]

I bet there's a lot of custom tooling behind the scenes to organize the content, keep it up to date etc.

I wonder if there are shortcuts we could take. Tools that are already built etc

[keen spindle]

Took Kyle's magefile and turned into a straight go runnable script:

func main() {
    ctx := context.Background()
    Run(ctx)
    Test(ctx)
}

[viscid solstice]

I wonder what else we could build. Love the custom CLI example too

Maybe a simple webhook server?

Or custom github action? (hint at databricks idea)

Does GHA have a Go SDK?

basically interesting places we could embed the go pipelines

Perhaps bring back that apko merge/diff poc Erik made?

@lapis edge that reminds me we need a hackathon in the checklist 🙂

[next jungle]

Any feedback? Weirdness? Wishlist?

btw for this:

src := api.DirectoryID(ctx.Workdir) // hacky cast cannot use ctx.Workdir (variable of type core.DirectoryID) as type api.DirectoryID

You should be able instead to do core.Host().Workdir().ID() or something like that (ctx.Workdir is a remnant from the old API)

Another (hopefully last) big SDK change is dagger.Connect() (instead of using engine.Start with the callback etc): https://github.com/dagger/dagger/pull/3349

[next jungle]

Neat! Should be trivial to add a switch around args so that it becomes go run code.go test|build … 90% like mage

[next jungle]

Yep totally.

Missed the convo but saw @viscid solstice mention tooling.

Spent some time in the past with docusaurus:

1. there’s snippets support with syntax highlighting

2. there’s tab support to have multiple snippets (Go/Python/etc)

3. tabs are neat: change to “Go”, all page tabs switch to Go

4. Snippets don’t need to be inlined in .md — they can be sourced from an external file

5. which means the external files can be linted/compiled/tested (e.g. maintenance)

6. can be wrapped around our own react component (if we want a “play” button”

As for Pulumi — they actually codegen their examples as well.

This is something we could consider in the future

I think for now, as long as we have some basic tooling for maintenance that blinks red when we break a snippet, we should be good with manually writing/updating them

[fleet cipher]

Ah thanks, I didn't know the ctx.Workdir wasn't sticking around, but that makes sense. I've been following the conversation around dagger.Connect() so I'll get that updated as soon as it's ready!

One thing I ran into, which @dark kelp was able to help me out with, was that tonistiigi/fsutil needed to be a specific version, or else engine.Start would spit out

# github.com/moby/buildkit/session/filesync
../../../../../go/1.19.1/pkg/mod/github.com/moby/buildkit@v0.10.4/session/filesync/filesync.go:112:20: cannot use dir.Map (variable of type func(string, *types.Stat) bool) as type fsutil.MapFunc in struct literal

Using github.com/tonistiigi/fsutil@b19f7f9cb274 resolves that error.

As far as weirdness, one thing that felt weird from a Go dx is that most functions don't have an error returned. I assume that means we panic if something goes wrong, but it feels weird to not handle that from my code.

For wishlist, I'm sure I'll have something when I dig deeper today!

[dark kelp]

Stripe uses https://markdoc.dev/ for all their docs, but I'm sure it's only one piece of the puzzle

they released it to the public ~5 months ago

[stuck orchid]

I've catching up on the activity so this might be irrelevant. I use mkdocs a lot, and with magetools (prepackaged mage tasks I've reused in many places), I setup https://github.com/sheldonhull/magetools/blob/main/docgen/docs.go to run mage docs:generate github and it resulted in something like this: https://github.com/sheldonhull/magetools/blob/main/docs/godocs/pkg/req.md

That tool gomarkdoc is useful for generating markdown driven docs.
I then used the Go TestExample framework for self-documenting tests as much as possible so the docs are driven from the packages.

Ya'll have more complex needs and i think use a different static site generator (docusaurus i think?) but maybe that will give you some ideas.

[viscid solstice]

@dark kelp you mentioned the Stripe docs model is hard to get out of your head. I agree it's super compelling. Do you feel that the current plan leaves the door open to adopt that model? Just wanted to double-check if you have concerns on that part

(same question for anyone else who thinks we should adopt the stripe docs pattern eventually)

[dark kelp]

to me the appeal is demonstrating that every language is a first-class citizen on the Dagger platform, and I figured we could cheat by having the language switcher be "Go / NodeJS / Python / Ruby" icons but with all 3 other ones greyed out until they're done. 😆 it would at least show that they're in mind from day 1, and not look like "oh turns out people want JS, let's pivot and hope we don't overextend ourselves"

but I think starting with something clearly called "Go SDK for Dagger" at least hints that more SDKs are coming, and this is just the Go one, so I'm totally cool with focusing on that

I think the language switcher style is a different type of content, more of a general platform guide showing various common examples (build, test, publish) and we just need to fill in the equivalent for each language and keep the prose itself generic. if we want to get really fancy we could have snippets embedded in the prose like api.Container() that switches between the languages too, but that's probably harder to make generic

the nifty thing about multi-platform is historically container tech has been pretty firmly within the bounds of the Go bubble (with some rust lately), and Dagger completely changes that, so I wonder if devs in other languages have been yearning for something like this to arrive. but I don't have much evidence to back that up.

[stuck orchid]

Pulumi generates their docs with per language toggle, so it's a solid experience overall.

Is there any example of what it might look like to say be writing my pipeline with the Go SDK, but then call a prebuilt dagger action that's in typescript and calling it like a "black box"? Something that shows the mix and match, while not having the actual logic in the same repository?

Low priority question, haven't had time to dig into find out yet.

[viscid solstice]

That’s what we call Extensions. They allow sharing and reusing pipelines across languages, at the Graphql API layer. It’s still under development but you can experiment with it already

[lapis edge]

Go SDK Docs plan for launch -https://github.com/dagger/dagger/issues/3360

[mortal marsh]

I guess this is where we should generate the examples for different language.

To contribute to the discussion (I have not checked your links @stuck orchid ), I agree that Go get the testable example model right. And it's runnable from the Go playground.
Maybe we can tune it so it could link to our playground instead if we built something on top of the graphql explorer and that we can use any of the languages we support.

I also like the idea (that we did for 0.2 as well) where we put tabs with languages as well, even if they were not ready yet (it was for CI providers)