#elixir

1 messages · Page 1 of 1 (latest)

obtuse cosmos
#

https://elixirforum.com/t/using-dagger-elixir-sdk-to-refactor-a-script/56748

Elixir Programming Language Forum
Using dagger Elixir SDK to refactor a script

In this post I’d like to share my experience working with Elixir SDK for dagger. dagger positions itself as a “programmable CI”, but I used it to re-write a shell script: using Elixir SDK for dagger, I was able to simplify my write-only fish-shell script significantly. I’m impressed with the results, and intent to use it as my go-to approach fo...

#

Thank you again @tawny escarp for contributing the Elixir SDK 🙏

obtuse cosmos
#

Thank you @quiet sentinel for the article above ☝️ !

prisma wraith
tawny escarp
#

We’re missing the dagger logo on hex.pm dagger user. cc @prisma wraith

IMG_9194.png
prisma wraith
tawny escarp
#

@prisma wraith thank you!

tawny escarp
knotty badge
#

context: there's a new input PortForward { ... } type, in one case it's specified like ports: [PortForward!]!, and in another API it's ports: [PortForward!] (nullable). That's the only difference I can think of from BuildArg

tawny escarp
tawny escarp
#

@knotty badge It needs a fix to make services v2 works. Do you mind if I push a fix, including a new codegen to your PR? Screenshots show the result from testing with container-to-container and host-to-container networking.

Screenshot_2566-08-30_at_22.57.44.png Screenshot_2566-08-30_at_22.59.29.png
#

I really like the host-to-container feature. 💕

tawny escarp
knotty badge
tawny escarp
ripe fable
last hamlet
#

Discovered https://gleam.run today which runs on the bean vm - looks like it can use elixer, or hex modules - would that work out of the box with the elixir dagger bindings at all? ( more curious than any serious query )

Gleam

The Gleam programming language

tawny escarp
#

@wingyplus Anything I can do to assist

#

Gleam

tawny escarp
ripe fable
west haven
#

@tawny escarp You shared the following with me a while ago (https://gist.github.com/wingyplus/ffa1f7b8f6a548715b9a11501f632c14 ) Would there be an updated version of this to for default elixir phoenix application (to do something similar to the generated dockerfile for the release)? Or is this example still up-to-date?

Gist
ci-next.exs

GitHub Gist: instantly share code, notes, and snippets.

tawny escarp
#

ci-next.exs

tawny escarp
#

If anyone use :dagger with Phoenix that uses swoosh and ran into this error:

$ mix deps.get
Resolving Hex dependencies...
Resolution completed in 0.025s
Because the lock depends on absinthe_client 0.1.0 which depends on req ~> 0.3.0, the lock requires req ~> 0.3.0.
And because "the lock" specifies req 0.4.8, no version of the lock is allowed.
So, because your app depends on the lock, version solving failed.
** (Mix) Hex dependency resolution failed

The workaround solution is adding {:req, "~> 0.4", override: true} to your mix.exs file should solve the problem.

ripe fable
#

@tiny mist Got your email. I'm here again; where is the fun I was promised? I joke I joke 😄

tiny mist
#

@tawny escarp has really led the way on the Elixir SDK so far. Looking forward to seeing more folks join in, use it, improve it. We're also working on a new way to write cross-language functions packaged in modules. I'm hoping we can get the Elixir SDK ready for #daggernauts 🚀 and https://daggerverse.dev

Daggerverse

Find modules built by the Dagger community, or publish your own.

tawny escarp
tepid drum
#

Very exciting @tawny escarp !

tight seal
#

Hello there 😄
Small question, but when trying the button "Running in Livebook" from the documentation, it seems the $PATH isn't fully complete 🤔

I don't have Docker inside for instance, which make the rest of the documentation not work

(on mac M2)

#

otherwise it seems to work outside livebook!

#

Can't wait to try it ❤️

tawny escarp
#

Unfortunately, livebook cannot be run because of it requires dagger and docker engine. But it can be run on local by clone dagger source code, change working directory to sdk/elixir and running the livebook server. 🙂

tight seal
tight seal
#

well it's pretty amazing 🔥

tight seal
#

Little question, if I run something like that:

sobelow =
  dev
  |> Container.with_file(".sobelow-conf", Directory.file(project, ".sobelow-conf"))
  |> Container.with_exec(~w"mix sobelow --config")
  |> Container.stdout()

format =
  dev
  |> Container.with_file(".formatter.exs", Directory.file(project, ".formatter.exs"))
  |> Container.stdout()

test =
  source
  |> Container.with_env_variable("MIX_ENV", "test")
  |> Container.with_exec(~w"mix test")
  |> Container.stdout()

are those different steps parallelized?

#

Or do I need to actually use Elixir processes (through a task for instance) to run this in // ?

#

Or put the Container.stdout/1 call elsewhere in the pipeline 🤔

cloud pine
tight seal
#

And okay it shows that you have to create a container per process, I see

#

At least they're getting cached after the first run 🤔

#

Thanks 👌

#

I need to change a bit my code around then 🙂

tawny escarp
#

Oh sorry, the hex doc didn’t get improve since the sdk was born. I think it is the time to improve it. 😀

tight seal
tawny escarp
#

@knotty badge Hi, I just realize that env_variables function is now breaking changes during rewrite the codegen. What I understand is make it returns the list of EnvVariable that had name and value contain in it but currently its broken because EnvVariable has an id now.

Or we should change the direction for this kind of function/method to give the develop are free to get id, name or value freely?

knotty badge
#

So instead SDKs need to just return full-blown objects, which they can do now that even objects like EnvVariable have an id

tawny escarp
tawny escarp
tepid drum
tawny escarp
#

I just hit 2 issues last night (or maybe my bad because coding at midnight T_T)

When using sdk with local path mode, it returns TODO error from Directory.diff

$ ~/src/github.com/dagger/dagger/hack/with-dev dagger init --sdk=../elixir-sdk --name=first-module --debug first-module
Error: failed to generate code: input: resolve: moduleSource: withName: withSDK: withSourceSubpath: resolveFromCaller: asModule: failed to create module: failed to update codegen and runtime: failed to diff generated code: TODO: cannot diff with different relative paths: "/" != "/src"

When using sdk with git path, it returns 500 internal server error. Report it on https://github.com/dagger/dagger/issues/6689.

GitHub
Issues · dagger/dagger

Application Delivery as Code that Runs Anywhere. Contribute to dagger/dagger development by creating an account on GitHub.

tawny escarp
tawny escarp
tight seal
#

niceee

prisma wraith
#

Love it @tawny escarp 💪

image.png
tawny escarp
#

Hi, I would like to update the progress of the module support. I start working on a library called dagger_module_runtime which help translate an Elixir module into a Dagger module by calls use Dagger.ModuleRuntime and annotate a function with @function for any function that want to expose to Dagger functions.

The demo is here https://github.com/wingyplus/daggerverse/tree/main/wttr if you would like to try but it's lack of documentation and many features are missing, so don't use it in production. 😂

Screenshot_2567-03-19_at_00.16.56.png
GitHub
GitHub - wingyplus/dagger_module_runtime

Contribute to wingyplus/dagger_module_runtime development by creating an account on GitHub.

GitHub
daggerverse/wttr at main · wingyplus/daggerverse

My Dagger modules. Contribute to wingyplus/daggerverse development by creating an account on GitHub.

tawny escarp
prisma wraith
#

Hi! We were just talking about this with @quiet sentinel at KubeCon in Paris. Will make time this week to get it in!

IMG_3686.jpg
#

This was right now!

quiet sentinel
#

Hi, @tawny escarp! Nice timing, as @prisma wraith mentioned! I was just telling Gerhard how I am stuck with my article describing the Dagger Elixir SDK. One issue with the article I had is debugging. In particular, I was wondering if it's possible to enter a shell of the pipeline... The conversation made me realize it might make sense to wait a little longer with the article & update it when Elixir SDK supports modules...

#

if it helps, I could check your PR too 🙇

tawny escarp
#

Helping review is very appreciate. 🙂

tawny escarp
#

Just get the module up and running without doing extra steps.🚀

Screenshot_2567-03-23_at_01.09.52.png
#

Here's the sample code, generated by the elixir module. It's work only primitive type (string, integer, etc.), the Dagger type are not support yet but it'll support soon. 🙂

Screenshot_2567-03-23_at_01.12.26.png
tawny escarp
#

Accepting and returning id type are work now. It's automatic cast the id into a type that defined in @function. 🙂

Next step is write a guide on it and share it in this channel to get the feedback.

Screenshot_2567-03-24_at_00.13.43.png Screenshot_2567-03-24_at_00.15.03.png Screenshot_2567-03-24_at_00.15.31.png
prisma wraith
tawny escarp
wicked herald
#

currently looking into it now, but is there a way of removing this release from hex.pm?

#

cc @abstract dagger

#

published v0.11.1 now 🎉

image.png
prisma wraith
#

Retiring v0.0.0 now

tawny escarp
#

Sorry, i falling to sleep last night. 😭 And thank you for fixing it. 🙇‍♂️ @wicked herald @prisma wraith

tawny escarp
tawny escarp
tawny escarp
#

👋 The Elixir SDK is now added basic support for Dagger Module. You can initiate the project with

$ dagger init --sdk=github.com/dagger/dagger/sdk/elixir/runtime <name>

The Dagger will fetch the SDK runtime directly from main branch. 🎉

The feature still lacks compared to 3 main SDKs (Go, Python, TypeScript) and can have breaking changes. Please try and tell me what you think about it. 🙏

tawny escarp
barren agate
#

Just getting started with Dagger. Have a question about with_mounted_cache. I expected that similar to a Dockerfile, if I moved a directive earlier in the chain, then cache would be ignored, but that's not the case.

below is what i had; i had forgotten to put |> Container.with_env_variable("MIX_ENV", "test") before the deps.get/deps.compile.

source_container =
  elixir_container
  |> Container.with_mounted_directory("/app", source_dir)
  |> Container.with_mounted_cache("/app/deps", deps_cache)
  |> Container.with_workdir("/app")

deps_container =
  source_container
  |> Container.with_exec(~w"mix deps.get")
#

when i inserted the above with_env_variable before with_mounted_cache, i expected the deps_cache to recalculated. it looks like we have to manually invalidate our Client.cache_volumes?

#

This may be a silly question. I was expecting the Elixir pipeline to act much like a Dockerfile itself with respect to invalidating steps after it. I'm still trying to build intuition for how Dagger sees the world.

tawny escarp
#

As far as I know, the mounted cache act as a directory but store inside the engine which’s similar to mount type cache in Dockerfile.

So to invalidate mounted cache you may need to stop and remove engine container or use different cache volume key.

barren agate
#

Dagger docs mention several times that it 'aggressively caches', but without with_mounted_cache the deps are rebuilt every time (which makes sense, nowhere else did i config that dir as being cacheable). Am I caching the deps folder the wrong way by using with_mounted_cache?

tawny escarp
#

It is not wrong but it is because of mix deps.get fetch all dependencies. When you run pipeline again, the mix will not do anything because the deps are already presented.

barren agate
#

i had been assuming some old cache was my problem, but even if i change the deps_cache variable to one with a different key, it still complains that the deps folder is empty. i found your repo 'https://github.com/wingyplus/hello_dag' and am trying to use it, to understand what i might be doing wrong, but even after changing {:dagger, path: "../dagger_sdk"} -> {:dagger, "~> 0.11.8"} and trying to run mix dagger.invoke in the sub-app hello_dag/dagger/hello_dag, it throws the error '%Dagger.QueryError{errors: [%{"message" => "resolve: no current module: main client caller has no current module", "path" => ["currentFunctionCall"]}]}'.

#

I'm not trying to use Dagger.Mod, and I noticed that your repo does make use of it, so I attempted to follow its example.

tawny escarp
#

The mix dagger.invoke should not use directly but simply run dagger call <function> instead. So in that example, you should run dagger call build --source=.:default.

#

@barren agate Do you get the error like

resolve: /app/_build/prod/rel/<app>: cannot retrieve path from cache

I just see the example above and try it with hello_dag example. But if you move the with_mounted_cache to before with_mounted_directory the pipeline will be run normally.

barren agate
#

ok, yes. that seems to have gotten me past the the cache errors, thank you. the error i was getting was acting as if i had not run 'mix deps get', it was that list of deps, saying it can't find it. it's scrolled out of my buffer, so i'm describing it instead of copy-pasting

i had guessed it was related to my MIX_ENV, but i missed that MIX_ENV=test was a second time high up in the stack, so dev vs test wasn't the problem.

barren agate
#

We've managed to keep moving forward. The only way we got caching to work was to define a cache in another directory, we used root /deps, /build and /node_modules. then we told elixir and yarn where to find their deps/build/modules. trying to use with_mounted_cache directory that was inside a with_mounted_directory just didn't seem to work.

tepid drum
barren agate
#

maybe we're forcing an edge case by using include:

client
|> Client.host()
|> Host.directory(".",
  include: [".credo.exs", ".formatter.exs", "mix.exs", "mix.lock", "assets", "checks", "config", "lib", "packages", "priv", "rel", "test"]
)
wintry nimbus
#

Would you recommend adopting Dagger in an Elixir project?

prisma wraith
#

Would you recommend adopting Dagger in

wintry nimbus
#

I'm trying to add Dagger to an Elixir project, and I'm having a hard time getting a directory mounted. Can anyone spot what the issue is here?

container = client
  |> Dagger.Client.container()
  |> Dagger.Container.from("hexpm/elixir:1.14.4-erlang-25.3-debian-buster-20230227-slim")
  
working_directory = Dagger.Container.directory(container, "")

{:ok, output} = container |> Dagger.Container.with_mounted_directory(root_directory, working_directory)
  |> Dagger.Container.with_exec(["mix", "deps.get"])
  |> Dagger.Container.with_exec(["mix", "test"])
  |> Dagger.Container.stdout()

Dagger.close(client)
#

I wonder if I'm getting too meta with this.

  • An Elixir script, .exs, is the one responsible for calling Dagger
  • Dagger then uses containers to mount the current project with a version of Elixir.
  • Containers run Elixir commands with the project from where the script is running
#

Nevermind... I found the solution myself

chilly pecan
#

@tawny escarp, are you around?

tawny escarp
#

feat(sdk/elixir): relocate module runtim...

tawny escarp
#

Hi Elixir community,

I'm working on rework the function syntax to make it shorter and easy to remember. I had opened a PR to tackle this at https://github.com/dagger/dagger/pull/8094. So the syntax will reduce from

@function [
  args: [name: :string]
  return: :string
]
def hello(self, args)

to

function [name: String.t()], String.t()
def hello(self, args)

But I discover the alternative way to define a function by annotate type into function signature like this:

defn hello(name: String.t()) :: String.t() do
  ...
end

Combine with https://github.com/dagger/dagger/pull/8099 which make Dagger client be global so we can do something like this:

defn container_echo(string_arg: String.t()) :: Dagger.Container.t() do
  dag()
  |> Dagger.Client.container()
  |> Dagger.Container.from("alpine:latest")
  |> Dagger.Container.with_exec(~w"echo #{string_arg}")
end

The prototype is on https://github.com/wingyplus/dagger/tree/elixir-positional-function that I just manage to make it work.

I have questions to ask you:

  • What do you think about this?
  • What do you think of using typespec instead using atom to define type?
GitHub
Build software better, together

GitHub is where people build software. More than 100 million people use GitHub to discover, fork, and contribute to over 420 million projects.

GitHub
💥feat(sdk/elixir): make dag instance be global by wingyplus · Pull ...

NOTE: This requires #8094 to be merge before review this PR.
Instead of setting dag to the struct of the object type, use GenServer
to store it during invoke the function. And add import statement ...

GitHub
GitHub - wingyplus/dagger at elixir-positional-function

A programmable CI/CD engine that runs your pipelines in containers - GitHub - wingyplus/dagger at elixir-positional-function

fervent ether
tawny escarp
dusky narwhal
tawny escarp
tawny escarp
#

@dusky narwhal @chilly pecan I just opened a PR to use context directories on the Elixir runtime here: https://github.com/dagger/dagger/pull/8290. Once this PR is merged, I can help make the change on the PHP runtime.

GitHub
Build software better, together

GitHub is where people build software. More than 100 million people use GitHub to discover, fork, and contribute to over 420 million projects.

rich fractal
#

Hi all, I'm giving Dagger + Elixir a try. I've written a module to build my app and I keep running into an issue. Roughly every ~5 calls I get this error about running out of space.

➤ dagger call elixir-build --source-dir .. --admin-ui-dir ../admin-ui --destination prod --elixir-version 1.16.3 --erlang-version 26.2.5 --linux-version debian-bullseye-20240612-slim
✔ connect 0.2s
✔ initialize 9.0s
✘ prepare 37.7s
! failed to get value for argument "source-dir": input: moduleSource.withContextDirectory.resolveDirectoryFromCaller resolve: failed to import local directory module arg: failed to mkdir /var/lib/dagger/worker/snapshots/snapshots/209/fs/uploads/a7/85/0d/f2: mkdir /uploads/a7/85/0d/f2: no space left on device
● ModuleSource.resolveDirectoryFromCaller(path: ".."): Directory! 37.9s
✔ upload /Users/dan/Workspace/explorations from 4a8y38radupvs0z8y51tremuh (client id: mz49xrqen37hbzh0fdrvtdx9j, session id: qc3i84l39eolcnvox0a5d6djz) 16.7s

#

I've been deleting the dagger-engine container to get unblocked.

#

Perhaps there's something about my codebase causing it? Or maybe it is a known issue?

#

Has anyone else encountered this issue?

#

Any thoughts or insights would be appreciated. 🙏🏽

rich fractal
#

Hi all, I'm giving Dagger + Elixir a try

tawny escarp
#

👋 I found a bug on Elixir SDK that the Dagger.Directory.with_directory/4 always crash due to variable collision. I opened an issue to track it here https://github.com/dagger/dagger/issues/8610. And working on it to fix the issue. 🙏

GitHub
Issues · dagger/dagger

An engine to run your pipelines in containers. Contribute to dagger/dagger development by creating an account on GitHub.

wicked herald
#

@tawny escarp any insight into the following flake we've started seeing?

📎 message.txt
tawny escarp
#

@Win any insight into the following

tawny escarp
#

I submit a PR to relocate mix task for invoke a module here https://github.com/dagger/dagger/pull/8769.

Since this PR we're no longer need mix task in user defined module (<module>/lib/mix/tasks/dagger.invoke.ex) anymore. 🙂

GitHub
Build software better, together

GitHub is where people build software. More than 100 million people use GitHub to discover, fork, and contribute to over 420 million projects.

tawny escarp
#

@chilly pecan @wicked herald I have a question about TUI. I execute the ContainerEcho function I notice that the Elixir SDK show Void instead of Container type but still returns correct result. What's SDK should do to make result correct again? Need to implements OTel?

Screenshot_2567-11-04_at_09.26.33.png
chilly pecan
chilly pecan
#

I can't even see the calls for the module registration. I think you should at minimum implement the trace parent propagation, like this:

Basically, you need to to check for a TRACEPARENT env var, and pass it on as an HTTP header when you make API requests.

Docs:

Enabling custom spans (and logs, metrics...) is a whole other thing, but the trace id propagation is more important.

tawny escarp
tawny escarp
quiet sentinel
#

@tawny escarp in Elixir, is it possible to specify optional parameters to functions? Can't quite figure this out or find this in the docs for the SDK

quiet sentinel
wicked herald
tawny escarp
#

CI failure

wicked herald
tawny escarp
wicked herald
#

really no rush on this, currently digging into dagger shell also doing it slightly wrong

tawny escarp
obtuse cosmos
#

@tawny escarp , I love that I get to ping you here and on dotnet channel 🙂

Thank you for all of your work on the Elixir experimental SDK.

As I mentioned in the dotnet channel, we've worked on clarifying the various levels of SDKs, and added a "Community SDK" level. This level requires a certain level of maintenance from SDK maintainers and has a docs requirement.

See announcement here:
https://dagger.io/blog/php-sdk
https://dagger.io/community-sdks

If you or anyone here is interested in doing that for Elixir, please let me know! We'd be happy to move Elixir up.

If not, no worries! We are very thankful to have Elixir as an experimental SDK.

tawny escarp
obtuse cosmos
tawny escarp
wicked herald
#

👍

wicked herald
#

lgtm, sorry for the delay 😄

tawny escarp
#

Thank you. 🙏

Still retrying CI jobs. 😢

wicked herald
#

(note there's a #php failure that's probably unrelated)

#

discussing in that channel

tawny escarp
#

Oh, just saw that test-module-runtimes run TestPHP . That's probably a root cause. 👍

tawny escarp
wicked herald
#

hey @tawny escarp, noticed that the elixir tests are pretty consistently failing on main?

wicked herald
tawny escarp
#

Oh. Sorry. I will fix in 2-3 hours

wicked herald
#

no worries!

tawny escarp
#

@wicked herald Is it possible to break a module-runtimes ci into each SDK? I ask this because I feel like it is hard to find which runtime is broken until you send a trace (and I can't access a trace by myself 😦 ).

tawny escarp
tawny escarp
#

Its now pass. 🎉

I add a local test to make sure the SDK is working and temporary skip other tests because of the main failure.

wicked herald
#

merged 🙂

tawny escarp
#

Thanks. 🙏

wicked herald
#

they should be public, we added that feature a couple months back

#

yeah, just checked, i can access that link in incognito

tawny escarp
#

My fault, I didn't know that we have a trace url in the summary, so I thought that it cannot accessible. It can access now.

sturdy stratus
#

Hello, I'm currently playing around with the Elixir SDK. Quick questions on testing.
I have a test set up like this:

defmodule HelloDaggerTest do
  use ExUnit.Case
  doctest HelloDagger

  setup_all do
    pid = start_supervised!(Dagger.Global)
    {:ok, pid: pid}
  end

  test "greets the world" do
    input = "Hello, Dagger!"

    {:ok, output} =
      input
      |> HelloDagger.container_echo()
      |> Dagger.Container.stdout()

    assert "#{input}\n" == output
  end
end

When I run them, I get a lot of output from Dagger that I don't really care about.
Is there any way to silence that output?
Is there a better pattern I should be following for testing?

sturdy stratus
#

Another question, this time about organization.
I have a monorepo, with a frontend, backend, and other misc projects.
My generated main entrypoint is at:
monorepo/dagger/monorepo/lib/monorepo.ex, and functions there work fine.
But when trying to split into separate modules, like monorepo/dagger/monorepo/lib/frontend.ex, I'm not entirely sure how to wire things up so they're listed in dagger functions and can be called.

tawny escarp
#

👋 Unfortunately, the Dagger.Global doesn't support suppress log but it should not be hard to do it.

tawny escarp
sturdy stratus
#

Okay, for now I'll just stick everything in the main .ex file.
Last quick question - is there a way to set a default directory with the Elixir SDK?
Like I wanted to omit directory_arg and have it default to src/.
Or maybe an easy way to hardcode it?

  defn grep_dir(directory_arg: Dagger.Directory.t(), pattern: String.t()) :: String.t() do
    dag()
    |> Dagger.Client.container()
    |> Dagger.Container.from("alpine:latest")
    |> Dagger.Container.with_mounted_directory("/mnt", directory_arg)
    |> Dagger.Container.with_workdir("/mnt")
    |> Dagger.Container.with_exec(["grep", "-R", pattern, "."])
    |> Dagger.Container.stdout()
  end
tawny escarp
# sturdy stratus Okay, for now I'll just stick everything in the main .ex file. Last quick questi... 
defn grep_dir(directory_arg: {Dagger.Directory.t() | nil, default_path: "."}, pattern: String.t()) :: String.t() do
    dag()
    |> Dagger.Client.container()
    |> Dagger.Container.from("alpine:latest")
    |> Dagger.Container.with_mounted_directory("/mnt", directory_arg)
    |> Dagger.Container.with_workdir("/mnt")
    |> Dagger.Container.with_exec(["grep", "-R", pattern, "."])
    |> Dagger.Container.stdout()
  end

The | nil make argument be optional. The type can also annotate by using tuple with the :default_path to make default directory works.

sturdy stratus
#

Awesome, thanks for the help

tawny escarp
#

You're welcome. 🙂

sturdy stratus
#

Is there a recommended pattern for getting things to run in parallel?

Tried a few different ways like this, but it doesn't ever want to do build and test at the same time:

  defn ci(source: {Dagger.Directory.t() | nil, default_path: "./frontend"}) :: boolean() do
    operations = [
      fn -> build(source) |> Dagger.Container.exit_code() end,
      fn -> test(source) |> Dagger.Container.exit_code() end
    ]

    operations
    |> Task.async_stream(fn operation -> operation.() end, timeout: 600_000)
    |> Enum.all?(fn
      {:ok, _} -> true
      _ -> false
    end)
  end

  defn build(source: Dagger.Directory.t()) :: Dagger.Container.t() do
    source
    |> build_env()
    |> Dagger.Container.with_env_variable("NODE_ENV", "production")
    |> Dagger.Container.with_exec(["npm", "run", "build"])
  end

  defn test(source: Dagger.Directory.t()) :: Dagger.Container.t() do
    source
    |> build_env()
    |> Dagger.Container.with_env_variable("TZ", "utc")
    |> Dagger.Container.with_exec(["npm", "run", "test"])
  end
tawny escarp
#

The Task.async_stream should run it concurrently. So your code looks correct.

Could you try the Dagger Cloud to visualize the pipeline?

tawny escarp
sturdy stratus
#

still no luck, very likely that I'm misunderstanding something with how Dagger works

Screenshot_2025-03-12_11-23-52.png
sturdy stratus
#

i was able to get it working with something very similar using the typescript sdk, though
🤔

sturdy stratus
#

would it be helpful if i put together a small example repo that can reproduce?

tawny escarp
#

Sure that would help a lot.

sturdy stratus
#

https://github.com/pikdum/dagger-elixir-concurrency
dagger call ci
What I'm intending to do is have these two blocks run in parallel, but instead they're running sequentially:
Confirmed that a similar pattern in typescript using Promise.all works to get them in parallel.

20250312_18h40m28s_grim.png
tawny escarp
#

@sturdy stratus Thanks for the reproduce repo.

I guess there is something wrong with the httpc in grapql client that block us from doing concurrent. I will open an issue for investigate and resolve the issue. 🙏

sturdy stratus
tawny escarp
# sturdy stratus Wonder if it could be needing some different options set? https://www.erlang.org...

I opened a PR at https://github.com/dagger/dagger/pull/10104 to see if the Req could solve the issue.

If you want to try this, just set sdk.source in dagger.json to github.com/wingyplus/dagger/sdk/elixir@4271e6e0c2f7a6e7c8c0f2ffe8223dbdc4b53a18, add a :req dependency to your mix.exs and add config/config.exs with a configuration below:

import Config

config :dagger, client: Dagger.Core.GraphQLClient.Req

Let me know if it's works for you. The PR needs to polish PR and investigate more about the httpc a bit. 🙂

GitHub
feat(sdk/elixir): add req graphql adapter by wingyplus · Pull Requ...

TODO: add me

sturdy stratus
wicked herald
#

👋 @tawny escarp question! would you be interested in doing something like https://github.com/dagger/dagger/pull/9123 for elixir? we've got support in go/python/typescript
not the sourcemap part, but the bit where we're reading the custom directives from the introspection api - we're able to attach a bunch of interesting metadata there (today, sourcemaps + experimental notes, but in the future, enum values, etc)

#

the idea is it'll always be optional but it'll have that extra level of info

wicked herald
tawny escarp
wicked herald
#

nope! no hurry at all ❤️

#

just something for your radar

tawny escarp
#

Okay. Add to my backlog. 🙂

tawny escarp
tawny escarp
# sturdy stratus Yep, can confirm that worked for me to get things running in parallel

The PR https://github.com/dagger/dagger/pull/10104 is ready for review now.

Please note that the Req is experimental at the moment. It require the :req dependency and needs to enable through the config. Once it stable, it'll become a default backend and the :httpc will be dropped.

GitHub
feat(sdk/elixir): add req graphql adapter by wingyplus · Pull Requ...

This changeset add the Req as a new
HTTP client backend for GraphQL aims to solve the concurrent issue on
the :httpc backend. [1]
This backend can enable by adding a :req as a dependency to the mix...

tawny escarp
astral quarry
#

hi folks! was pleasantly surprised to join this Discord and learn that there's an undercover elixir sdk in the works! we use Erlang+Elixir extensively at work and this might make adopting Dagger easier at my workplace.

is there a good place to look to check in on how far along the elixir SDK is compared to the official SDKs?

tawny escarp
#

hi folks! was pleasantly surprised to

snow solstice
#

Is someone working on the Otel support right now ? I'll be glad to take it if no one is on it

tawny escarp
wicked herald
tawny escarp
wicked herald
#

no worries! just wanted to check!

tepid drum
tawny escarp
#

address(): a unified address to load con...

fallen dirge
#

Hey there, out of curiosity is the elixir SDK stable and good for prod use?

In your experience are there any warts? I'm looking for a CI solution for elixir projects and just wanted to vibe-check the compatibility with elixir.

Of course I could use the go/python sdk in the project but it would be really dope if I could use the same language

tepid drum
# fallen dirge Hey there, out of curiosity is the elixir SDK stable and good for prod use? In ...

👋 I haven't personally used the Elixir SDK, but I can vouch for its maintainer @tawny escarp, he is super active in the Dagger community in general, and a very reactive maintainer.

The SDK is community-maintained, but it's in the core Dagger repository, and hooked up to our CI to avoid drift. Since all SDKs are generated, changes to the Dagger API are continuously propagated to all SDKs. Merging PRs is gated on all SDK tests passing; etc.

fallen dirge
# tepid drum 👋 I haven't personally used the Elixir SDK, but I can vouch for its maintainer ...

Amazing thanks Solomon! I've dug in a bit and the code is great :), cheers to all involved in the implementation and maintanence of it!

There's a really interesting usage pattern here in elixir due to the BEAM. Specifically, running a server that can execute arbitrary dagger functions/pipelines.

From the outset I can't really tell if there are use cases for this kind of thing, or if it makes sense to do at all haha. But I'm toying with the idea of a static site hosting app that builds/generates the underlying sites using a repo-defined interface in the form of a dagger module which you can run directly within the lifecycle of an elixir app

west haven
#

Would there be any 'full' example for a dagger file to use with phoenix? I'm struggling a little to connect the dots currently, especially with the (generated) code that uses the defn to be able to do dagger something. Is it the current workflow? Or should I just make a dagger_ci.exs file to do everything? Everything: build the application with assets, run tests, copy build artifacts to a new image and publish to a registry.

fallen dirge
# west haven Would there be any 'full' example for a dagger file to use with phoenix? I'm str...

Think of dagger as replacing dockerfile + build scripts/make files/etc.

So you basically have to port over the build/test/etc. logic that you have in a script/ci yaml/etc. and move it into dagger (i.e. moving the execution of tasks into containers, instead of natively on your host).

Everything: build the application with assets, run tests, copy build artifacts to a new image and publish to a registry.
And what you should do depends, but you definitely can do all of these things!

Personally before digging into the code sdk, I found the interactive mode mode to be a great way to start if you like the bash pipe style syntax, and you can always end with a | terminal to exec into the place. Kinda like a REPL for docker, it's very neat!

Also keep in mind you don't have to use elixir sdk for the pipelines, you can use another sdk if you prefer as well for you build/ci stuff

#

I have some code below that I made as an example when I was playing with dagger a while back that does some checks within an elixir repo

west haven
#

I am currently using earthly, but since that's sadly end of life in would be nice to migrate to dagger.

Having a full example would be nice to see the best practices and save myself many hours trying to figure out the minutia about dagger (see the above questions about the defn with dagger <something> versus an .exs script

#

Just odd I couldn't find any such examples on the web

fallen dirge
#

Not sure if what I did above is 'best practice' but it seemed reasonable, and only has 'checks' really, I didn't get to the build and push, but the idea is you use the 'ci_env' (or 'build_env') as the builder, and pull out the mix release into another runtime container with limited deps

fallen dirge
#

@west haven I realized that there was no build/publish in there, today I took a little time and made this flow with the minimal chainguard wolfi-base image. I haven't tested with phoenix but it does let you start the release into an iex session with e.g. start_iex. I've commented out the phoenix specific pieces.

This is an adaptation from the Dockerfile in the phoenix container documentation https://hexdocs.pm/phoenix/releases.html#containers using volume cache to make things quicker instead of layer caching

fallen dirge
tawny escarp
west haven
#

I've finally managed to get something working, but didn't manage to do so with the defn's for now

#

Using an output directory for mix release is stolen from Mac's example, without I had difficulties getting the final build out of the _build directory somehow

fallen dirge
#

Yeah it's b/c of the volume caching on the _build directory (for the compiled deps mainly) IIRC

west haven
#

Yeah, but I didn't find a way around it, the whole point of dagger/builkit is to have good caching imo 🙂

tawny escarp
#

I can get it working with the function but needs to be tomorrow because I’m out of computer now. 🙇‍♂️

fallen dirge
# west haven Yeah, but I didn't find a way around it, the whole point of dagger/builkit is to...

The release still should be cached though via layer/function caching though if it is run with the same inputs, and if it is not run with the same inputs/code then a new release should probably be created haha.

Of course the way the above is implemented departs from the phoenix docs Dockerfile in the sense that I lean on volume caching through dagger instead of only layers/multistaged builds

terse matrix
#

Hey 👋 , release for the Elixir SDK will be manually triggered today, we have a hiccup atm 🙏

terse matrix
#

Release done ✅

snow solstice
tawny escarp
tepid drum
tawny escarp
#

✨ elixir: support checks · dagger dagger...

tawny escarp