#Networking multiple computers

1 messages · Page 1 of 1 (latest)

chrome salmon
#

Hey, there. I'm trying to do some basic networking with dagger.

I have two containers that need to be set up with networking between them. I couldn't see any networking examples in the snippets section of the cookbook.

My docker-compose.yml file looks something like this:

version: "3.0"
services:
  boudi:
    build: .
    container_name: build-your-own-internet-002-boudi
    image: build-your-own-internet-002-boudi
    hostname: boudi
    networks:
      caternet:
        ipv4_address: 10.1.1.3
    cap_add:
      - NET_ADMIN
  pippin:
    build: .
    container_name: build-your-own-internet-002-pippin
    image: build-your-own-internet-002-pippin
    hostname: pippin
    networks:
      caternet:
        ipv4_address: 10.1.1.2
    cap_add:
      - NET_ADMIN

networks:
  caternet:
    name: build-your-own-internet-002-caternet
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 10.1.1.0/24

As you can see, I'm trying to build a network that connects these two containers. Is this possible? Been hunting around the dagger documentation without much success.

GitHub
dagger/docs/current_docs/cookbook/snippets at main · dagger/dagger

An engine to run your pipelines in containers. Contribute to dagger/dagger development by creating an account on GitHub.

proud bronze
echo tendon
chrome salmon
#

Thanks for the feedback. It's not at all clear to me how I would use this to connect two VMs together. My naïve attempt looks like this:

    const container1 = dag.container().from("nginx:latest").withDirectory("/usr/share/nginx/html", source)
    const s1 = await container1.withExposedPort(80).asService().withHostname("web1")
    const container2 = dag.container().from("nginx:latest").withDirectory("/usr/share/nginx/html", source)
    const s2 = await container2.withExposedPort(80).asService().withHostname("web2")

But all I want to be able to do with these machines is bring them up and then exec bash on them and then ping one from the other. Is dagger just the wrong tool for the job here?

proud bronze
#

@chrome salmon do you mean 2 separate machines (physical or vm) or two containers possibly on the same machine? The distinction is important because each dagger engine is single-node. If you want networking between 2 containers across 2 machines, dagger doesn't handle that transparently. Similar to docker-compose.

#

If we're talking about 2 containers on the same machine (like docker compose but code-driven) then yes that is easy and well-supported

#

the way dagger would handle cross-machine networking, is indirectly: orchestrate a dagger pipeline which deploys containers to a distributed system like kubernetes or docker swarm.

chrome salmon
#

I think what I'm trying to do falls firmly in the "2 containers on the same machine (like docker compose but code-driven)" use-case. In fact— you might think this is a little insane, but — I'm ultimately trying to orchestrate about fifty VMs being set up, configuring about twenty different networks in between them, and setting up various of those VMs with special roles (such as DNS routing, etc).

i'm mostly doing this with docker-compose today and lots of bash scripts and the like, which is why I thought dagger might be an interesting tool to replace that infrastructure.

I'm not super excited about the idea of involving another complex tool like k8s or docker swarm, but could try that on. Do you have any snippets for this by chance?

proud bronze
#

ok this is a great use case for dagger and it looks like you will not need to involve an additional layer of kub etc

echo tendon
#

@chrome salmon here's a brief example that might give you an idea on how to glue all this together:

  @func()
  async test(): Promise<Container> {
    const src = dag.directory().withNewFile("index.html", "Hello, world!");
    const container1 = dag
      .container()
      .from("nginx:latest")
      .withDirectory("/usr/share/nginx/html", src);
    const s1 = await container1
      .withExposedPort(80)
      .asService()
      .withHostname("web1")
      .start();
    const container2 = dag
      .container()
      .from("nginx:latest")
      .withDirectory("/usr/share/nginx/html", src);
    const s2 = await container2
      .withExposedPort(80)
      .asService()
      .withHostname("web2")
      .start();

    return dag
      .container()
      .from("alpine")
      .withServiceBinding("web1", s1)
      .withServiceBinding("web2", s2)
      .withExec(["apk", "add", "curl"])
      .terminal(); // this line is optional to create a shell in the container
  }

^ if you call dagger call test sync, you'll get a shell in the alpine container at the very end where you can curl or ping the web1 and web2 services. Additionally, because web1 and web2 are service with withHostname, they're able to resolve each other without the need of withServiceBinding as the alpine container needs

proud bronze
#

picking up kids at school but will make a snippet if nobody (nevermind lol)

echo tendon
#

re: if you need to exec separate things inside the web1 and web2 running services after the service has started, there's no current direct API for that given that this specific use-case hasn't really been requested.

The way that users handle this today is via programatically spawning new services with the correct configuration as needed instead of having to exec things withing them to mutate the state. That's one of the benefits of Dagger, which makes it extremely easy to programatically parametrize and dynamically launch as many permutations of the service is needed within your pipeline

chrome salmon
#

Okay, yeah, that fundamentally works, thank you!

One thing that I'm going to need to be able to do is to assign IP addresses to individual machines and to the networks between them, especially as I build out larger and larger networks. I'd like to be able to say something like, "spin up this VM, and it has three network interfaces, one on X network, one on Y network, and one on Z network, and here are the IP addresses you should assign to each of those interfaces."

I'm also going to need to be able to set low-level Docker permissions on VMs, such as NET_ADMIN, SYS_ADMIN and sysctls such as net.ipv4.ip_forward=1 and net.ipv4.conf.all.rp_filter=0

Any chance dagger's API is sophisticated enough to let me do these things?

proud bronze
#

I might need more details on how the VMs fit in the picture. If you really need VM-level control (different machines with different kernels) then you might need to combine Dagger with a VM management tool (for example kvm or libvirt). If on the other hand we're talking about controlling the networking stack of the container, then it boils down to API features. At the moment we don't have the API to configure container networking at this level of detail. We could potentially add it but it would require discussing design tradeoffs (specifically making sure the pipelines remain portable)

chrome salmon
#

I'm using strictly settings that currently appear in a docker-compose.yml (I'm not sure if that means they have different kernels or not?). But yes, in general I'm talking about being able to control the networking stack of the container pretty precisely, especially with regards to being able to make administrative changes to routing (using, e.g. the ip route command), and turning off reverse-path filtering and turning ip forwarding on. (All very non-standard requirements for most builds!)

Is this functionality I could add myself through the daggerverse, or is this more core API functionality that I would need to wait for?

proud bronze
#

OK that makes sense. It's just the use of the wording "Docker permissions on VMs" which threw me off. I'm with you now.

#

Is this functionality I could add myself through the daggerverse, or is this more core API functionality that I would need to wait for?

Unfortunately I don't think you could do this in a module...

  • CAP_SYS_ADMIN is supported already. Container.WithExec(insecureRootPrivileges: true) boom done

  • Controlling the IP is either a medium-difficulty core change, or a XL-difficulty core change. @keen pumice will probably have a good sense of which. The issue with giving direct control of the IP address, is that it's hard to virtualize the IP address space. As soon as your pipeline hardcodes an IP address, there's a risk that you'll need a README "before using this module, make sure to run the following setup commands on the host"". Or even worse, "make sure to configure your machine's network in this way". So it could become a portability nightmare. It definitely did with Docker. But perhaps there is new tech now that makes it easier. Let's see what @keen pumice says.

  • The sysctl control I think is XL-difficulty no matter what. I believe that is not supported by buildkit. Therefore Dagger cannot support it without messing with buildkit. @lucid beacon, @sleek rampart or @glass steeple might be able to gauge difficulty there

glass steeple
#

you should be able to change sysctls in a privileged container (with insecureRootPrivileges)

#

i think that should probably just work? though the changes likely won't persist between WithExecs

#

there's no underlying buildkit functionality to help set them, but if you just need to set them, i think they should all be isolated correctly?

lucid beacon
#

(going off memory of sysctls here, take with grain of salt)

glass steeple
#

here's a better source: https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/#safe-and-unsafe-sysctls

Kubernetes
Using sysctls in a Kubernetes Cluster

FEATURE STATE: Kubernetes v1.21 [stable] This document describes how to configure and use kernel parameters within a Kubernetes cluster using the sysctl interface.
Note:Starting from Kubernetes version 1.23, the kubelet supports the use of either / or . as separators for sysctl names. Starting from Kubernetes version 1.25, setting Sysctls for a ...

#

ah in the above:

The following sysctls are known to be namespaced

#

Those net.* that can be set in container networking namespace.

#

theoretically are aren't too difficult to propagate if we wanted to allow configurin those from the dagger api?

echo tendon
glass steeple
echo tendon
#

@chrome salmon so in summary, you can achieve what you need in Dagger with the current version it's just that you'll have to use a little bit of Service entrypoint networking setup instructions so services are configured as needed

#

One thing you can't do though is setting custom IPs for services. 😬 . Hopefully you can stil make your use-case work without that

chrome salmon
#

Thanks very much for giving this some hard thinking. I really appreciate all of your contributions.

In the end, I am building a network like this with docker-compose, so I absolutely must be able to control network addressing for each machine and between each machine. This entire system is for the purpose of building a large simulated internetwork where people can get hands-on with basic networking technologies like DNS. Really looking forward to the day when I can rip out all this tooling and replace it with dagger, but it sounds like that day is not today.

Thank you!

BYOI_-_DNS_-_4b._Recursive_DNS_1.png
echo tendon
# chrome salmon Thanks very much for giving this some hard thinking. I really appreciate all of ...

I see.. that's an awesome project BTW and I really see us supporting this use-case in the future. Having said that, we are effectively missing some primitives to achieve something like that which will take time to implement specially around fleshing out design implications 🙏 .

Hopefully we'll come back to you sooner than expected with a working version where you can express all this beautifuly by using Dagger modules daggerfire

chrome salmon
#

Thanks, y'all — really appreciate talking this through with me. I love your project and will stay tuned

grizzled vine
#

I'm ultimately trying to orchestrate about fifty VMs being set up, configuring about twenty different networks in between them

VMs, or containers?

Sorry to butt in, but what about Kubernetes? This sounds like a Dagger in k8s use case to me, where you build out the infra (via something like Terraform) inside k8s, then Dagger runs whatever container images that need the building, testing and releasing using the resources built out previously. A lot of development platforms turn to k8s for the reason of needing the orchestration. 🤔

proud bronze
grizzled vine