#How to mix SDKs from multiple languages in a team setting?

1 messages · Page 1 of 1 (latest)

nova talon
#

There was a talk from a community call recently about this. the tl;dr for that is that each team writes their own, and then the ops team writes another dagger pipeline that invokes the others, dagger-calls-dagger

My understanding is that CUE is legacy Dagger and only the new SDKs work with the more recent engines.

What I don't think this gets at is sharing at a more granular level across languages. Here, I have had the same question, for Pulumi, and wonder what the solution is. It's not like it really makes sense to import a JS file into a Py file and call that. Maybe a intermediate representation can be created? After all, the SDKs all end up calling the same GraphQL API

#

I think a reduced user story is something like, as the ops team, I would like to provide some building blocks for other teams, but they work in N languages. How can I avoid having to write and maintain the same building blocks for each of the N languages?

frank saddle
#

this is correct, As i understood, we can provide the base blocks, but dev teams should be able to build on that foundation in their own language. We will provide what currently exists in our pipeline, and devs can then extend that functionality.

tawny grotto
#

@frank saddle @nova talon That feature is what we call "extensions" that we've had working in POC form since the inception of the multi-language approach, but decided to focus on the SDK DX and then bring out extensions when a few other important facilities were in place 🙂

nova talon
#

is there a separate plugins proposal?

tawny grotto
#

The end result is that you can write code once and then use it from any SDK in any language as easy as you would call the core API today. Extensions would also benefit from the same caching you get in the core API too.

frank saddle
#

Its unfair I suppose to expect this to be that much better than terraform in that regard, but If I can export my project ID, commands, etc through the python sdk - there must be a way for another language to access that project id? In any case - I'll happily contribute to making this happen once I can grapple with the basics. The api is doing something to help this in the future. But I suppose the need to make a good core comes first.

formal compass
#

@stoic thicket @dense dirge @shell wharf and I are working on design this week together in SF. Extensions is one of our key topics.

#

(we're literally talking about extensions right now)

frank saddle
#

If I build something in one language that is the base of all of our apps, At the very least I would like to share that across our gitlab projects somehow - WITH the ability to allow node devs to run the whole pipeline locally (even if SRE wrote them the setup script) . I cant for the life of me remember who presented in Toronto but they really made it seem like it was the answer to the whole companies yaml mess, not just per team.

tawny grotto
# frank saddle If I build something in one language that is the base of all of our apps, At th...

That was our own @burnt raven !

Today, with a single language choice for the whole org, you can do just that. Say if you pick Javascript.

In the near future (with extensions) you can have each team write in their preferred language (one team in go, another in javascript, another in python, etc) . In that scenario, you could have one team define the platform and others consume it, extend it.

frank saddle
#

Well props to kyle - he was very enthusiastic. And made it seem like this is all I needed in my life. Single language looks like thats the way Im going to go. Removing all the yaml I can and substituting dagger calls into our gitlab pipeline. We will have to give devs a script to setup the pipeline locally. Still exctiing. And I hope I can make kyle glad he took time out to do that talk.

tawny grotto
#

What are the two languages between SRE and dev teams?
Go Python (SRE) and Javascript (devs). I read your comment again above.

#

That's super common too. In that approach, the SRE/Platform team can write everything in Python and then the devs can have a script in their package.json like

...
"scripts": {
    "build": "dagger run python ci.py"
},
...
#

that they run with npm build or yarn build

nova talon
#

The platform team almost always lacks the knowledge and context to write the entire pipeline, there are devils in the details and the dev team should be able to fill those in. Today, we provide hooks, places for extension, or just starter templates.

The idea here is to write some building blocks that the dev teams can use in their pipelines for common tasks like "deploy this application to the staging cluster"

#

So if the platform team writes that in Go, how would the dev team use it in JS?

hexed relic
#

I regularly come back to this thread to see how it's going.

So if the platform team writes that in Go, how would the dev team use it in JS?

Replace JS with Python and Rust and you get the problem that I'm in. I write deployment utilities in Golang and then the expectation is that developers consume that utility library within their own Go based pipelines for their non-Go project. It works, kinda, but it's harder to push Dagger internally for adoption.

tawny grotto
hexed relic
# tawny grotto Thanks <@140339723112677377>. Definitely something we're working on actively. Ap...

Yeah, I've been pushing for Dagger within my company because it's what I've been using to create a generic build and deploy pipeline for applications being pushed out to Nomad clusters. The clusters orchestrate a heterogenous fleet of edge hardware. One of the hardest parts of these types of operations platform initiatives is creating a good UX experience for internal developers when you're stitching together a myriad of bespoke tooling. It's hard to approach Node developers and say "in addition to everything you have to learn for this stack... You also need to learn Go."

#

This isn't a complaint. I know y'all are working hard and I'm super excited for the roadmap. Pretty much everything being worked on is something that I want: better project UX, extensions, and GPU access for pipelines.

frank saddle
#

I watched the youtube in the Demos channel called Monorepo Example. I think that's how I will plan to move forward for now. I too am writing my code in Python, but watching that video - I can clone a teams CI folder that uses any language and run it through python Dagger -> dagger, running the js through a container. The only drawback is that is still something that will need to be implemented by the SRE team, rather than allowing "import" from javascript. Its not ideal but its something.

nova talon
#

I have a project called Harmony, really more of a pattern. The main idea is to create a Dagger pipeline that has several building blocks and makes it easy to test downstream projects. It makes it easy for users to register their projects for testing mainly, and also requires they follow your blueprints, however you set them. It needs updating to the latest engine (still CUE based, so old). I'm looking at making it work for both

  1. test code on user projects before releasing
  2. drive the tests in user projects, this is the same as the Monorepo example
    These two are basically the same thing in Harmony (https://github.com/hofstadter-io/harmony), just looking at it from different directions