#@vito we're in a live demo with @Lev
thin hound
- Test container depends on a temporal service
- Temporal takes too long to start
- "not ready before deadline"
Is there an escape hatch for that?
young stag
is that the literal message? "not ready before deadline"?
thin hound
no
young stag
iirc we have an absurdly long health check timeout so wondering if that's come from somewhere else
thin hound
(sorry here is the actual error message - I know there is no dagger errors in the trace) 👆
(zoomed in span) 👆
young stag
hmm, just taking notes
- temporal service started and passed checks in 1.3s: https://dagger.cloud/levs-test-org/traces/a08c50910a0fd599d6d1adae49c27b26#7401c772e09b2f5b
- mongo too (1.5s): https://dagger.cloud/levs-test-org/traces/a08c50910a0fd599d6d1adae49c27b26#cf25f08dbf95c450:L6
it even says it connected to the db, but then it errors 10 seconds later 🤔
oh maybe that's a different connection (looks like grpc)
thin hound
ah got it, the dagger healthcheck passes, but the service is not actually ready
so the client fails (the deadline error)
we're looking at implementing a custom healthcheck
young stag
or maybe the client's address to the server is misconfigured?
thin hound
(they have the custom healthcheck logic, looking for a way to plumb it into the dagger pipeline)
(sorry I'm following the rapid fire debugging session unfolding around me)
young stag
lol np
i don't see any configuration pointing it to the temporal service, so maybe it's trying localhost or some default and timing out
mongo's address is configured, but not temporal's: https://dagger.cloud/levs-test-org/traces/a08c50910a0fd599d6d1adae49c27b26?span=d2b59c054b31a1dd#8174d7d99b589741
thin hound
It;s coming from the doppler token
(some config lookup happening via the doppler client)
they're adding a sleep 10 for now
young stag
ah ok
thin hound
currently re-running with the sleep 10, to see if it helps
(once again wishing for self calls 😛 )
sleep 10 didn't help. They're now trying a custom healthcheck
It feels like Container.withServiceBinding could take an optional healthcheck argument which could be an interface maybe? or perhaps more simply, a Container. Maybe a Syncable interface? 🙂
young stag
i think we have a handful of issues around service health checks at this point, one of them proposes respecting the Dockerfile HEALTHCHECK
whole iron
Happy to add more eyes, but sounds like y'all have it covered...
young stag
thin hound
i think we have a handful of issues around service health checks at this point, ...
Ha ha they Lev just asked about that 🙂
@young stag is there a stopgap / escape hatch at the moment? I guess just add the custom healthcheck command in the client command?
whole iron
just noticed there is no search box in Dagger Cloud on a trace...wanted to search for temporal everywhere in that trace
young stag
i think we have a handful of issues around service health checks at this point, ...
guess that might mean something like .withHealthCheck(checkCtr)? to match Dockerfile semantics?
young stag
<@108011715077091328> is there a stopgap / escape hatch at the moment? I guess j...
you could manually start the service and keep running the healthcheck in a loop yourself before continuing
or add a withExec that checks in a loop, if you want to keep it in the call chain
haughty vessel
<@108011715077091328> is there a stopgap / escape hatch at the moment? I guess j...
Late to the party but there is ExperimentalSkipHealthcheck on ContainerWithExposedPortOpts.
https://pkg.go.dev/dagger.io/dagger#ContainerWithExposedPortOpts