[SOLVED] It seems that `with_exec()` didn't like pipe character

[next tartan]

Hi all

I'm coding using Python and I try to find all Bash scripts in my repo and, then, run /bin/basn -n script.sh to make sure there is no syntax error.

When I'm in my console, the command find . -type f -name *.sh | xargs -n 1 /bin/bash -n is working.
In my snippet below, I've add .terminal() and, yes, it works.

But .with_exec() throws an error telling that the pipe character is not recognized by find

command: str = r"find . -type f -name *.sh | xargs -n 1 /bin/bash -n"

return (
    await (
        dag.container()
        .from_(self.env.images("DOCKER_ALPINE"))
        # [...]
        .terminal()
        .with_exec(command.split())
        .stdout()
    )
)

Did you have any tips?

Thanks!

[unkempt cobalt]

Try this.

command: str = r"sh -c find . -type f -name *.sh | xargs -n 1 /bin/bash -n"

#... your code
.with_exec([command])
#..rest of your code

Or if you do not like this you can break your command and manually take the output of the command to the left of pipe and then call the command to the right. I have seen that dagger doesn't work so well with pipe operators.

[next tartan]

Thanks Jain ... but I can't make it working right now...

By jumping in my terminal; (so by adding .terminal()) ; I can do find . -type f -name *.sh | xargs -n 1 echo and then I see all my linux .sh files.

By using your suggestion sh -c find . -type f -name *.sh | xargs -n 1 echo, I get all files (no more just the bash scripts)

By using double-quotes : sh -c "find . -type f -name *.sh | xargs -n 1 echo" then it works again as expected in my terminal session.

And when I put that command in my Python Dagger script, it didn't work anymore (stupid escape problem I think)

command: str = r'sh -c "find . -type f -name *.sh | xargs -n 1 echo"'

return (
    await (
        dag.container()
        .from_(self.env.images("DOCKER_ALPINE"))
        # [...]
        # .terminal()
        .with_exec(command.split())
        .stdout()
    )
)

The command is then translated like this:

! process "sh -c \"find . -type f -name *.sh | xargs -n 1 echo\"" did not complete successfully: exit code: 2

✘ .withExec(args: ["sh", "-c", "\"find", ".", "-type", "f", "-name", "*.sh", "|", "xargs", "-n", "1", "echo\""]): Container! 0.2s

[royal talon]

Hey, try this instead:

command: str = "find . -type f -name *.sh | xargs -n 1 echo"

return (
    await (
        dag.container()
        .from_(self.env.images("DOCKER_ALPINE"))
        # [...]
        .with_exec(["sh", "-c", command])
        .stdout()
    )
)

[next tartan]

Once again, many many thanks for your help. It's a nice thinking out-of-the-box working alternative. It's works like a charm.

[SOLVED] It seems that with_exec() didn't like pipe character

[next tartan]

It was too difficult to maintain a single find ...command with all my criterias (like ignoring a list of folders) and , too, to properly manage the different scenarios i.e.

• if find didn't retrieve any files, the job should be considered success (nothing to do)
• if findretrieve files:
  • scan all files one by one and run a command like shellcheck on each.
  • If at least one fail, return an exit code 1
  • If none, return an exit code 0

So ... I've tried another approach:

ctr: Container = await (
    dag.container()
    .from_(self.env.images("DOCKER_SHELLCHECK"))
    .with_mounted_directory("/app/src", self.source_directory)
    .with_workdir("/app/src")
    .with_new_file("/tmp/dagger-ci-script.sh", self.script(), permissions=0o750)
    .with_exec(["/tmp/dagger-ci-script.sh"], expect=ReturnType.ANY)
    .sync()
)

return StepResult(stdout=await ctr.stdout(), exit_code=await ctr.exit_code())

Inspiration comes from https://docs.dagger.io/cookbook/#continue-using-a-container-after-command-execution-fails

The self.script() method is for getting a string; the content of the script I'll create in the container.

And, I think, this approach will make things easier to maintain.

Once again, thank you for your valuable help