#Dagger & MacOS virtualization

1 messages · Page 1 of 1 (latest)

quick venture
#

@worn cloud how would your tool compare to other tools like Tart?

worn cloud
#

The technical part will be similar. The difference will lay in the license, and the alignment with Docker's API, which will make it feel more familiar to developers.

#

I remember you mentioning in a podcast that Dagger doesn't use Docker itself but some internal component of it. Where can I read more about it?

quick venture
#

Yeah we use buildkit. You can think of Dagger as a specialized container engine, made for containerizing pipelines rather than regular applications.

#

So in a way it's an alternative to the Docker engine, although it's not really competitive (because the two engines can co-exist peacefully, and we target different use cases)

#

How would you see Dagger and your virtualization tool integrating with each other?

#

Dagger orchestrating Mac VMs alongside containers?

worn cloud
#

So in the Apple ecosystem there's a tool, Fastlane (Ruby), which popularized as the automation tool to make CI pipelinees more lightweight. However, it's become clear that the community would rather write the pipelines in Swift. So in that sense, Dagger's idea is amazing for that, I think it'd address a big need in that ecosystem. A the concept of functions and modules plays very nicely with it, because developers could encapsulate functions that are very unique to that ecosystem.
Some of those functions can run in a Linux environment, but others require macOS-specific toolchain. So for those, it'd be awesome if I could express to Dagger, use this macOS container. I'd expect everything else to be 1-to-1 to what you can do with Linux containers:

  • Mount directories
  • Expose environment variables
  • Forward results and standard pipelines
#

At the infrastructure level, things a bit more complex. Since images are so heavy, you need to have a pool of hosts that are warm with images. Typically you have images per Xcode version, which is the toolchain that most developers need.

#

You mentioned that's a different engine. Does that mean that both reuse the same core concepts and technologies but differ in high layers in some way?

#

For context, we'd like to build a CI service in https://tuist.io, and would like to bet on Dagger as the language instead of throwing another proprietary format out there.

#

So we'd like to support in making that happen, and do so in a way that's beneficial for the entire ecosystem. So would like to chat with you too about what's best in terms of license. I heard in multiple interviews that you had a lot of learning from your Docker experience 🙂

quick venture
#

cc @hot viper 👋

#

@worn cloud I think the most important limitation is that, for the foreseeable future, Dagger can only orchestrate Linux containers. It's a limitation we plan on removing eventually, but it's at least 1 year away. This is a limitation that we inherit from buildkit, around which Dagger is built. To lift it, we need to either 1) expand buildkit support beyond linux containers, or 2) support mutiple runtimes beyond buildkit. Both are major engineering projects and it doesn't make sense to take them on at this stage.

#

BUT that doesn't mean a good integration is not possible

#

The solution IMO is to integrate higher up in the stack - with modules

#

Dagger is highly extensible (via modules) and has great networking APIs. These two features combined give us a lot of options.

#

For example:

  • You can expose your native Mac virtualization tool as a network service. Then pass that service endpoint as an argument to your Dagger functions. So you have a Dagger pipeline, containerized the usual way, and some functions in that pipeline are orchestrating Mac VMs via API calls over the network.

  • There's also good old SSH. There are already modules out there that focus on remote-control from a Dagger function. Same general idea: run a Dagger pipeline, and some of the functions don't actually do the work, but instead remote-control the work over ssh.

#

Combine this with a Swift Dagger SDK (which doesn't exist but could be developed for sure), you start getting a nice solution for hybrid Mac/non-Mac workflows

#

I emphasize hybrid because, if you're in a pure Mac silo, using only a Mac toolchain to produce Mac applications, you're unlikely to be excited about Daggerizing that.

But, in my experience, usually there are several siloes. And integrating the Mac silo with the rest of the pipeline is a source of pain. In that situation, using Dagger + this integration can really help.

#

By the way this can be done on either a local Mac dev machine (assuming you have docker or a clone installed), or a shared Mac CI server. I've seen both patterns.

worn cloud
#

Oh, I didn't know a networking API existed. For some reason I assumed a strong coupling between a pipeline and a virtualized environment, but if that's not the case, then the possibilities are endless.

#

Quick question regarding having an SDK in a compiled language.

#

Does the engine take care of compiling the pipelines before running them? I see that there's a Go SDK, so I assume that's how it works?

quick venture
#

Yes there is a strong coupling in the sense that your Dagger Functions (the building blocks of a Dagger pipeline) can only run in a linux container, run by the Dagger engine. But that container can connect to the outside world in lots of cool ways

#

and "the outside world" includes the local host system (possible Mac) and also remote systems accessible via the host network

worn cloud
#

Gotcha

#

I need to further familiarize myself with all the concepts.

quick venture
worn cloud
#

Does it cache the compilation?

quick venture
#

Yes everything is always cached

worn cloud
#

And... do you generate those SDKs? I was looking at Elixir's yesterday and I noticed how well structured it is despite being beta

quick venture
#

Also, SDKs themselves are just Dagger modules, made of regular Dagger functions. So you can create your own custom SDK, or fork an existing SDK.

quick venture
worn cloud
#

It's taking me some time to assimilate all the concepts. It might be the consequences of so many years embracing a particular mental model 😆

quick venture
worn cloud
quick venture
#

Here’s the answer: we were cheating. Our SDKs don’t actually run your pipelines! Instead they send pipeline definitions to the Dagger Engine, which runs them and returns the result in a language-agnostic way.

quick venture
worn cloud
#

What's a runtime builder in this context?

quick venture
#

So there's a client component (generate Swift client library from Dagger API schema) and server component (built special runtime container from Swift server code)

#

You write custom Dagger Functions and types in Swift -> the Swift SDK builds that code into a runtime container -> the Dagger Engine loads that container and extends the Dagger API with your functions and types

worn cloud
#

Let me see if I understood this correctly:

  • An SDK provides an API to describe the pipeline in your language of choice.
  • The compilation of the pipeline is managed by Dagger, and the output of that is something standard that's then pass to the engine.
  • The pipeline is a set of steps, some of which are defined by you, and some of which can come from other modules and functions that can live in other repositories. Some of the steps can be "run this in a container" but also "run this ssh command or command in the host"
quick venture
#

cc @quasi steeple who is better at explaining this stuff than me 🙂

#

(the Dagger SDK master)

worn cloud
#

😆

quick venture
#

Sorry I have to run afk but will be back! Thanks for being curious and patient

#

--> this conversation will be distilled in to a docs article at some point

worn cloud
#

How are you folks thinking about presenting this terms to developers? I think it's fascinating, but also transitioning from the current mental models seems a bit challenging.

#

No worries, I really like diving into tech that can evolve ideas that even though accepted, are clearly not ideal. I'd like to contribute some of my time to bring Dagger awesomeness to the Swift ecosystem, so for that I need to fully understand it well myself

clear wasp
#

The configuration language is somewhat obtuse XML, but there are graphical tools (eg virt-manager) that will generate the XML for you or you can steal a template from someone else who’s done it before (see below). the benefit is that it can also be run on a Linux host in addition to MacOS. This gives you a consistent API over both Hypervisor.Framework and KVM

quasi steeple
clear wasp
worn cloud