sshd service - exited before healthcheck

[young ember]

👋 I'm trying to do something straightforward with Dagger and failing miserably! I'm trying to run sshd as a service, here's the code I'm using:

package main

import (
    "dagger/rde/internal/dagger"
    "fmt"
)

type Rde struct{}

func (m *Rde) SSH(
    // Optionally override the username
    // +optional
    // +default="dagger"
    username string,
) *dagger.Container {
    return dag.Container().
        From("ubuntu:latest").
        WithExposedPort(2222, dagger.ContainerWithExposedPortOpts{Protocol: dagger.Tcp}).
        WithExec([]string{"apt-get", "update"}).
        WithExec([]string{"apt-get", "install", "-y", "openssh-server"}).
        WithExec([]string{"mkdir", "-p", "/var/run/sshd"}).
        WithExec([]string{"useradd", "-m", "-d", "/home/" + username, "-s", "/bin/bash", username}).
        WithExec([]string{"/bin/bash", "-c", fmt.Sprintf("echo \"%s:%s\" | chpasswd", username, username)}).
        WithExec([]string{"sed", "-i", `s/#PermitRootLogin/PermitRootLogin/`, "/etc/ssh/sshd_config"}).
        WithExec([]string{"sed", "-i", `s/#PasswordAuthentication yes/PasswordAuthentication yes/`, "/etc/ssh/sshd_config"}).
        WithExec([]string{"sed", "-i", `s/#PermitEmptyPasswords no/PermitEmptyPasswords yes/`, "/etc/ssh/sshd_config"}).
        WithEntrypoint([]string{"/usr/sbin/sshd", "-D", "-p", "2222"})
}

This is the error:

✔ Container@xxh3:db65564e8d33de66.asService: Service! = xxh3:040f0eba9a8be9cb 1.8s
  ✔ start apt-get install -y netcat-traditional 1.8s
    ✘ 2222/tcp 1.8s
    ! checking for port 2222/tcp: context canceled
    ┃ 20:07:28 WRN port not ready host=4keuslrt534gu.29an7v1ee93ta.dagger.local error="dial tcp 10.87.0.52:2222: connect: connection refused" elapsed=76.389169ms
    ┃ 20:07:28 WRN port not ready host=4keuslrt534gu.29an7v1ee93ta.dagger.local error="dial tcp 10.87.0.52:2222: connect: connection refused" elapsed=144.08476ms
    ┃ 20:07:29 WRN port not ready host=4keuslrt534gu.29an7v1ee93ta.dagger.local error="dial tcp 10.87.0.52:2222: connect: connection refused" elapsed=243.41302ms
    ┃ 20:07:29 WRN port not ready host=4keuslrt534gu.29an7v1ee93ta.dagger.local error="dial tcp 10.87.0.52:2222: connect: connection refused" elapsed=472.685814ms
    ┃ 20:07:29 WRN port not ready host=4keuslrt534gu.29an7v1ee93ta.dagger.local error="dial tcp 10.87.0.52:2222: connect: connection refused" elapsed=690.216982ms
    ┃ 20:07:30 WRN port not ready host=4keuslrt534gu.29an7v1ee93ta.dagger.local error="dial tcp 10.87.0.52:2222: connect: connection refused" elapsed=1.364113982s
✘ Service@xxh3:74da8cdd935b7aaf.up: Void = xxh3:db4766ebfa4eaa27 1.8s
! failed to start host service: start upstream: service exited before healthcheck
  ✔ host: Host! = xxh3:ff73c85214b8f75c 0.0s
  ✔ Host.tunnel(
      native: true
      ports: []
      service: ✔ Container@xxh3:db65564e8d33de66.asService: Service! = xxh3:040f0eba9a8be9cb 1.8s
    ): Service! = xxh3:3c90486513784082 0.0s

Setup tracing at https://dagger.cloud/traces/setup. To hide: export NOTHANKS=1

Error: response from query: input: container.from.withExposedPort.withExec.withExec.withEntrypoint.asService.up resolve: failed to start host service: start upstream: service exited before healthcheck

If I open a terminal on the container I can run the entrypoint just fine and the config looks good.

There's no indication that sshd is exiting early. I'm a bit stumped as to what to try next.

[timber valve]

@young ember what command do you run?

dagger call ssh up ?

[young ember]

dagger call ssh as-service up

[timber valve]

ah 👍

(as a convenience you can skip the as-service but irrelevant to your problem)

[young ember]

TIL! I'm so used to calling AsService() in Go 😄

[timber valve]

this is where sharing a trace link would be useful 😇

maybe with --no-exit you can explore the TUI and find additional clues?

[zenith bluff]

I think it's fixed by changing the last WithEntrypoint to WithExec. The service will run whatever the last withExec is so setting the entrypoint doesn't impact it in this case (though I get intuitively why that would have been expected)

[young ember]

That fixed it! As you said, I just assumed running it as a service would run the entrypoint for the container 🙂

Thank you both

[zenith bluff]

and in case you hit anything like this in the future, if you run at -vvv you'll see under Container.asService the actual command that's being run, which previously ended up being sed -i ...

[timber valve]

I got bit by the exact same thing last week. We really need to change that API imo

[young ember]

So there's an interesting thing here - if I use WithExec([]string{"/usr/sbin/sshd", "-D", "-p", "2222"}) then I can't use terminal on the cli, because that layer never completes. That kind of makes me think that WithEntrypoint should be what gets run when I invoke a container as a service.

Because I now need to write an AsService() function which just adds that final layer to the container, but that feels janky. If I leave the sshd WithExec as the last layer, then I can't actually pass that around as a usable container because it's always going to hang.

I think that when you run a container as a service, it should use the entrypoint if there is one defined

[timber valve]

@serene flicker @dapper plaza @lean leaf @fast shore I know this 👆 is already on your radar. Maybe we should just bite the bullet and make the change?