tall shadow
There are times when my pipelines appear to run the same exact actions multiple times. The internal container log output is identical, but certain values such as timestamps are different.
Would this occur in the case of multiple calls to export or other "resolution" methods (ie. Stdout, ExitCode, etc)?
final prism
Hi @tall shadow ! Will you please share your Dagger code (if you can)?
tall shadow
The code is pretty complex because I have created a bunch of common action execution code that is used by a number of actions.
The process is as follows:
At the end of the action execution I export a number of directories out of the dagger container for review on the filesystem.
I am calling a number of the following export methods on every action.
// ExportArtifacts exports the container artifacts to the pipeline
// host artifacts.
func (p *Pipeline) ExportArtifacts(name string, container *dagger.Container) error {
if _, err := container.Directory(exportArtifactsDir).Export(context.Background(), filepath.Join(p.Artifacts[name])); err != nil {
return err
}
return nil
}
// ExportContainer exports the container as OCI compliant tar.
func (p *Pipeline) ExportContainer(name string, container *dagger.Container) error {
_, err := container.Export(context.Background(), filepath.Join(p.Artifacts[name], "image.tar"))
return err
}
// ExportReports exports the container reports to the host
// filesystem.
func (p *Pipeline) ExportReports(container *dagger.Container, path string) error {
if _, err := os.Stat(filepath.Join(p.reports, path)); !os.IsNotExist(err) {
return fmt.Errorf("path '%s' already exists", path)
}
_, err := container.Directory(exportReportsDir).Export(context.Background(), filepath.Join(p.reports, path))
return err
}
I also have a "trap" handler that captures commands with non-zero exit codes to prevent the container from becoming "bricked" because I need to export the reports on failure for review.
That code is in a few of the actions as well and is as follows:
// WithCatchExec runs the command with error trapping to prevent
// failure from bricking the dagger container.
func WithCatchExec(container *dagger.Container, cmd []string) (*dagger.Container, error) {
container = container.WithExec(
[]string{"/bin/bash", "-c", fmt.Sprintf("set -o pipefail; %s | tee /tmp/output || echo $? > /tmp/code", strings.Join(cmd, " "))},
)
ctx := context.Background()
code, err := container.File("/tmp/code").Contents(ctx)
if err != nil {
// if the command succeeds /tmp/code will not be created
if strings.Contains(err.Error(), "no such file or directory") {
return container, nil
}
return container, err
}
if code != "0" {
output, err := container.File("/tmp/output").Contents(ctx)
if err != nil {
return container, err
}
return container, errors.New(output)
}
return container, nil
}
tough bison
@tall shadow from those logs, that may just be different times during download of the same thing. You'll get better visibility by looking through the TUI, rather than these engine logs.
tall shadow
Is there way to display the TUI with the Go SDK?
tough bison
You use dagger run go run ...
So you need the CLI, not the auto provisioning in the SDK.
Btw, I'm curious as to why you store the exit code in a file. Is it to avoid a trimmed error output?
tall shadow
I am storing the exit code to a file to prevent the container from encountering a non-zero exit code
Maybe it has been fixed, but early on when the container errored due to a status code greater than 1 I could no longer run Export on the container to get the reports from the failed action (ie. test reports, lint reports, security scan reports) to identify why the command failed.
tough bison
I see, thanks.
Btw, have you tried using ContainerWithExecOps{ RedirectStdout: "/tmp/output" } instead of using tee in the command?
tall shadow
I don't think that I did. I can definitely try it out and see if it works though
tough bison
Back to the exit code, there's some discussion in https://github.com/dagger/dagger/issues/4979#issuecomment-1557046765 to add an option to withExec to allow it not to fail. Use case there would be to allow calling .Stdout() even on "error", but your use case fits too. Problem would be detecting if it's an error or not, but now that I think of it, .ExitCode() would work too.
tall shadow
That would be an awesome feature. I honestly haven't thought much about it since I had a reasonable workaround. The main issue with my workaround is dealing with containers that don't have bash installed by default like Alpine.
tough bison
Are you aware that you're caching failures? So if you have an unexpected error like a network issue, it'll be cached.
tall shadow
How so? Is that a side effect of my workaround?
tough bison
Yes. Since you're returning zero, buildkit considers it a success, thus caching it.
tall shadow
I see. It shouldn't be a major issue because of the way I am using this function.
I only use it when I expect a failure that wouldn't be considered a legitimate command error.
One use case is, when running security scans, I catch failures to export the security reports with the expectation that the pipeline will continue on and give the engineer a chance to determine if the vulnerabilities have significant impact
It may still be something to think about though
tough bison
So if there's a network problem during the vulnerability scan if it needs to check in with some service, then you'd see that in the report, or export itself would fail if it couldn't even have been generated, right?
tall shadow
Technically yes. There is the side effect that if a legitimate error occurs during the command execution (ie. not a unit test failure or scan result error) it would be caught by the function.
I think the only way around that would be to detect if the error was expected or not. It would require a bit more inspection in the wrapper code to detect.
As far as caching goes, wouldn't the cache be invalidated if the source directory in the container were different?
tough bison
Yep
tall shadow
I guess I could still have issues if there was an error and the pipeline needed to be rerun with no change. Perhaps a (albeit ugly) way around it is to disable cache on actions that use this catch function to be safe
That definitely isn't ideal though
tough bison
What I can think of is you can export something to your source dir when you detect an error with if code != "0" {.
Or something smarter to rule out expected errors like a found vulnerability.
I'm wondering if the flag to allowFailure in withExec is worth it for some use cases, even though failures are cached.
That would allow simplifying your code a bit, but as you said, maybe it's not too bad a workaround.
tall shadow
Yea it would be a nice feature, but I definitely can see the caveats. I am going to dig back into that code at some point and do some more testing and refactoring
I will give that redirect flag a try as well
nocturne path
If you’re interested in early access to Dagger Cloud, I think the pipeline visualization would help with drilling down and debugging this. Also you could share specific run URLs with us, and we could look at it together
hidden wind
@tall shadow +1 to what Solomon mentioned above 👆
I can help with getting you a sneak peak at Dagger Cloud.
tall shadow
@hidden wind That would be great.