MySQL failing to start when running on GitHub Actions

[queen wind]

Hi, we are trying to run our build on GHA, but there seems to be some issue with permissions when MySQL starts. Strangely enough, when running locally (Docker Desktop on macOS) everything works fine.

Replication Code: https://github.com/j-martin/dagger-mysql/blob/main/main.go
Failed Run: https://github.com/j-martin/dagger-mysql/actions/runs/8335539081/job/22811205062

Logs:

Error: -19T00:26:34.130429Z 0 [ERROR] [MY-010338] [Server] Can't find error-message file '/usr/share/mysql-8.0/errmsg.sys'. Check error-message file location and 'lc-messages-dir' configuration directive.
...
Error: -19T00:26:34.173702Z 0 [ERROR] [MY-010174] [Server] Can't change data directory owner to mysql

We are not sure how to proceed.

[split hearth]

Hey @queen wind! I'm singing off now, but I'll take a look at your issue tomorrow early morning!

[queen wind]

Update: using a non-ancient image of mysql solved the issue. https://github.com/j-martin/dagger-mysql/pull/1/files?w=1

[split hearth]

Hey @queen wind! Sorry for the late reply, wasn't able to take a look at the issue yesterday, it was a bit of a crazy day. Glad it fixed it! Quick question, when you ran the CI locally using the same command, that worked? Just checked your message and you mentioned it did, ignore me 🤪

[opaque swan]

Encountering a similar Issue with a custom MySQL8 image, it's strange because the same Dagger workflow runs fine in Codespaces and on a Mac laptop but for whatever reason the mysql container dies after <1 s when used as a Service and .Start'd manually as in the example above. I even tried changing the entrypoint to []string{"/entrypoint.sh", "bash", "-c", "mysqld", "2>&1"} but don't see any output.

The entrypoint.sh is necessary for $reasons but essentially just does some setup and then execs the arg

[velvet willow]

I thought we had a thread somewhere...
this works for me

package main

import (
    "context"
    "fmt"
    "os"
    "dagger.io/dagger"
)

func main() {
    ctx := context.Background()
    // create a Dagger client
    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
    if err != nil {
        panic(err)
    }
    defer client.Close()

    if err != nil {
        fmt.Println("Error opening docker-image-list-ghe")
        return
    }

    testResults := client.Directory()
    mysqlContainer := client.Container().
        From("mysql").
        WithEnvVariable("MYSQL_ALLOW_EMPTY_PASSWORD", "true").
        WithMountedDirectory("/test-results", testResults).
        WithExposedPort(3306)

    database := mysqlContainer.AsService()

    src := client.Host().Directory(".")

    // Run application tests
    out, err := client.Container().From("golang:1.21").
        WithServiceBinding("db", database).      // bind database with the name db
        WithEnvVariable("DB_HOST", "db").       // db refers to the service binding
        WithEnvVariable("DB_PASSWORD", "test"). // password set in db container
        WithEnvVariable("DB_USER", "postgres"). // default user in postgres image
        WithEnvVariable("DB_NAME", "postgres"). // default db name in postgres image
        WithDirectory("/src", src).
        WithWorkdir("/src").
        WithExec([]string{"go", "version"}). // execute go test
        Stdout(ctx)

    if err != nil {
        panic(err)
    }

    fmt.Printf(out)
}

No need to start() manually

[opaque swan]

Thanks for digging into this... I was provoking the Start() eagerly to trigger the failure before encountering it in the client expression. I think this is some eccentricity with our internal container -- I am trying to mount /var/log/mysql (where we config the mysql logs to emit to) to a Host().Directory so I can look at them now. But is there generally a way to inspect a container after a Dagger session ends (with docker logs $CONTAINER_NAME) or are all the containers --rm'd after the Session exits?

[velvet willow]

They all run inside the dagger-engine so there isn't a way to access them from the outside via docker.
I know we've got plans for making that sort of debugging easier.
cc @deep hatch @latent shore

[opaque swan]

Mounting the logs didn't cook for me for some reason so I thought I'd replace the entrypoint wholesale and install strace in the container and flip the entrypoint to run strace mysqld in the foreground (it was exec-ing before). It seems like mysqld in my case starts up and is running for <1s but never opens a socket before dying 🤔

[opaque swan]

tried mounting /var/log/mysql/error.log two ways: once with a Host().Directory() and a second time with the bare file itself.. both times there're no logs. This seems to imply the mounting mechanism isn't working or that mysql crashes before any logs at all are emitted (semi-hard to believe). This was done in Codespaces

[latent shore]

Unrelated note, I recommend using the dag global client in examples, to make the transition to functions easier

[opaque swan]

The difference between functions and module functions(?) was a little fuzzy to me. I started off as a module function but found it difficult to access the Host() and had no case (yet) for re-use so I went with a function/main

[latent shore]

when we talk about Dagger Functions, or "functions" for short, it's always the one kind (containerized, loaded from modules).

We don't have a good name for the other way. I call it "custom client" since that's what your tool is - a custom dagger client

[amber rivet]

Yeah, there's an issue for that but since we moved to new docs I thought it wasn't worth the time to go back and change what we had to dag, but some times I wished we did since many users still depend on those.

[velvet willow]

Yep. I just edited elee's existing code example, but that would have been smart.

[latent shore]

@fallen schooner I think accessing anything from a service container after it has been started, is a bit of a blind spot at the moment (but @chilly delta may know of a way).

One thing you can do, is mount /var/log to a cache volume before executing the service. Then you can write another function that also mounts the same cache volume, but in an interactive container for you to inspect

[latent shore]

Something like yourContainer.WithMountedCache("/var/log/mysql", dag.CacheVolume("mysql-logs")).AsService()...

Then in another run, you can yourContainer.WithMountedCache("/var/log/mysql", dag.CacheVolume("mysql-logs")).Directory("/var/log/mysql").Export("./logs-export")

[queen wind]

Oh... I think this should be documented somewhere: How to get logs or files out of a service.

I was going to post earlier that it was not possible since I didn't know about the mounted cache trick.

[latent shore]

Yes I agree

My guess is that the most practical way, when possible, is to redirect all service logs to stdout or stderr, then you can rely on the dagger engine logs

[queen wind]

Hopefully, the cookbook (https://archive.docs.dagger.io/0.9/cookbook/) gets updated to > 0.11.0 docs soon. It's a hidden gem, but you need to know about it.

[latent shore]

A lot of that content is already in the new docs

There's a user manual, for CLI-oriented tasks: https://docs.dagger.io/manuals/user (a lot of what you had to do with custom code, the dagger CLI now handles for you)

And for code patterns: the developer manual. https://docs.dagger.io/manuals/developer

For example, here is the section on services in the Go developer manual: https://docs.dagger.io/manuals/developer/go/742930/services

[queen wind]

I did a few spot check, you are right, there is a lot of stuff that has been ported. There's this gem that must be re-added in the docs: https://archive.docs.dagger.io/0.9/cookbook/#continue-using-container-after-command-execution-fails 🙂

(it would be even better if there was native way to do it)

[old schooner]

@opaque swan just checking if you managed to sort this out. Might have some bandwidth today to check this out if you're still having issues 🙏

[opaque swan]

@old schooner appreciate you checking in. Unfortunately I have not. I think the suggestion to use the cache is something I’m going to pursue next. My current suspicion is that it has to do with how I am running the dagger engine, runs the same container create the bridge network with docker and then use a bare docker run and that works. Is there something about the engine doing all these operations that is privileged is my thinking 🤔

[latent shore]

@opaque swan to confirm, you are running the dagger engine container yourself with docker, and not running it privileged? Or did I misunderstand

[opaque swan]

As an aside: I have really enjoyed kicking the tires on Dagger and even with my current container entry point murder mystery I’ve found the tool a joy to learn. Thanks to everyone who’s chimed into the thread so far 🙏

I am running it by:

curling the installer into my Action workflow and doing a simple go run main.go of the above. This seems to be what the official Dagger Action did before I took it out just to get closer to the problem with the official action that had the same issue

[latent shore]

Ah ok. yes that is fine. Under the hood it runs the engine as needed, using docker by default

[old schooner]

yes, probably getting the mysql logs will be the best. I find it a bit surprising that it works everywhere except in GHA

[old schooner]

I'd assume you can't share that image with us, can't you?

[latent shore]

@opaque swan FYI after thinking about it, I think getting mysql to stream everything to stdout/stderr, then getting the dagger engine logs, is a more reliable route than trying to access log files via a cache volume. At least in your case where you have other variables to worry about already.

[opaque swan]

I'm afraid I cannot :<

[opaque swan]

Doing a little fiddling with the mysql.cnf to emit logs/errors to stdout yields some interesting output: https://gist.github.com/elee/12ef7da9442501f5c923c3d9b87ed022

[old schooner]

it's extremely weird this happens in codespaces since AFAIK they're pretty much the same underlying hardware, correct?

this message results somehow weird: mysqld: File './mysql-bin.index' not found (OS errno 13 - Permission denied)

I was thinking that it could be related to the source architecture that you've used to build your image

which could end up resulting in some files having a different target arch

[opaque swan]

I believe they're relatively similar yes, maybe some userland tooling is different but the container is already baked in both cases I'm just yanking it down from containers.pkg.github.com.

But that Permission Denied message is what I'm staring at right now. I think it has to do some sort of race condition between the custom entrypoint chmod-ing the custom datadir and the mysqld process racing to load it. Laggy filesystem sync in Actions (or something) given filesystem permission mutations are non-atomic in general

[quiet mauve]

@opaque swan this line in your strace catches my eye
2024-04-18T20:19:14.3320775Z stat("/data/user/mysql/mysqld-auto.cnf", 0x7ffd1c144b80) = -1 ENOENT (No such file or directory)

If this is a file needed, and it doesn't contain secrets, you might consider baking into the image instead of mounting it

[opaque swan]

I ended up working around this. I believe it's some strange race condition in how the overlay mounts are solved in Actions. I have a TODO to make a dummy public repo to try and reproduce it in the open, but the TL;DR is we do some fancy footwork in the container I'm working with's default entrypoint to copy /var/lib/mysql into /data/user/mysql and then chmod that to mysql:mysql for $reasons. The permissions error creating mysql-bin.index implies our chmod is not being applied in the container when mysqld setuid's to mysql from root -- the base image is constructed in a funky way and I may just upstream that very change @quiet mauve (among others)