Step cache is not used - is this normal?

[compact bobcat]

Hi!

When I initialize dagger inside the folder of my project, then, every time I re-run my pipeline, the step cache appears to not be used at all. Is this normal? Here's what I mean.

I initialize dagger inside my project like this:

cd my-app
dagger init --sdk=elixir .

The make my module looks like this:

defmodule MyApp do
  use Dagger.Mod.Object, name: "MyApp"

  @doc """
  Just a demonstration of an issue
  """
  defn just_install_deps(src_dir: Dagger.Directory.t()) :: Dagger.Container.t() do
    dag()
    |> Dagger.Client.container()
    |> Dagger.Container.from("hexpm/elixir:1.14.0-erlang-25.1.1-ubuntu-focal-20211006")
    |> Dagger.Container.with_mounted_directory("/src", src_dir)
    |> Dagger.Container.with_workdir("/src")
    |> Dagger.Container.with_exec(~w"mix local.hex --force")
    |> Dagger.Container.with_exec(~w"mix local.rebar --force")
    |> Dagger.Container.with_exec(~w"mix deps.get")
    |> Dagger.Container.with_exec(~w"mix deps.compile")
  end
end

Now, every time I call - even without any modifications to files inside my app or dagger module...:

dagger call just-install-deps --src-dir .

...the step cache is not used, as if something invalidates it.

I thought this might be related to the fact, that .dagger is part of my project's folder. However, attempting to exclude it does not seem to be helping either:

defn just_install_deps(src_dir: Dagger.Directory.t()) :: Dagger.Container.t() do
  src_dir = Dagger.Directory.without_directory(src_dir, ".dagger")

  dag()
  |> Dagger.Client.container()
  |> Dagger.Container.from("hexpm/elixir:1.14.0-erlang-25.1.1-ubuntu-focal-20211006")
  |> Dagger.Container.with_mounted_directory("/src", src_dir)
  |> Dagger.Container.with_workdir("/src")
  |> Dagger.Container.with_exec(~w"mix local.hex --force")
  |> Dagger.Container.with_exec(~w"mix local.rebar --force")
  |> Dagger.Container.with_exec(~w"mix deps.get")
  |> Dagger.Container.with_exec(~w"mix deps.compile")
end

To completely side-step the issue, I initialized dagger module at the same level as all of my project are, e.g.:

dir-with-all-my-projects
├── an-app
├── some-other-app
├── ...
├── my-app
└── my-app-dagger

This way I can call my function like this without an issue:

dagger call --mod my-app-dagger just-install-deps --src-dir my-app

But... That's not how I should be doing things, right? E.g. I am probably doing something very wrong here? Like, going against a best practice or something. Because it kind of natural to place all things related to a project - inside the project's folder, like with .github folder, for example.

@solar pecan tagging you, since examples here are in Elixir, but I think I am hitting some kind of general misunderstanding on my part / not following a good practice 🙃

[wicked coral]

@compact bobcat how do you determine which step gets cached?

[compact bobcat]

@wicked coral by steps here I mean layers ofc. So I kind of rely on the fact that layers are cached automatically, and re-runs should be instantaneous... because nothing changes in any file in the project, including in .dagger folder 🤔

[wicked coral]

OK I see. One thing that might help get more resolution while debugging, is setting up tracing in dagger cloud (you can get a free account if you don't have one already). Then you can press w while the call runs, and get a web view of the ongoing call.

To further bisect the problem: which steps seem to be invalidated:

1. Loading the module?
2. or executing the module's implementation, aka your code?

[compact bobcat]

Actually, both loading the module and module implementation. E.g. in my teminal output, I first see the process module being compiled, then my code being compiled (e.g. steps with "mix" on above snippets). This happens over and over again, without me modifying anything - just re-running dagger call

I will set up tracing and will get back here with more info

[solar pecan]

I think it is https://github.com/dagger/dagger/issues/8283

[compact bobcat]

I think we might be talking about 2 issues here:

1. first, module itself gets recompiled,
2. the project layer cache is not used

For issue number 1, @solar pecan the GH issue you linked might be it 😦

For issue number 2 - my original issue - I created two traces, but sadly can't quite able to share them - seems public sharing can only be enabled for GH orgs?

https://dagger.cloud/gmile/traces/e14d03b98f34ef98886c6bde7ed04a85
https://dagger.cloud/gmile/traces/0fe9725f5f86e4e162e7fee37810e3e2

Anyway, in each trace I see that the directory is getting uploaded each time, even though I'd suspect dagger should see that the contents hasn't changed - and find the layer with directory - in the cache 🤔 (though I am not 100% sure I get the basic principles right yet)

[amber dune]

Not sure if this is the issue about your dagger directory being copied in your container but without_directory won't ignore the directory, it will remove it after it has been copied, invalidating the cache anyway I suppose. You need to use pre-call filtering (See https://docs.dagger.io/api/filters/#pre-call-filtering) instead, in Python this is working fine however I don't know how to do it in Elixir, sorry 😕

[solar pecan]

For 2, not sure if build directories (_build, deps , LSP directories, etc.) make the pipeline invalidate. You might need to add ignore option to your src-dir argument:

defn just_install_deps(src_dir: {Dagger.Directory.t(), ignore: ["_build", "deps", ".elixir-tools", "*whatever you want to ignore*"]}) :: Dagger.Container.t() do
  ...

The ignore pattern is like the +ignore in Go SDK but the list is the native Elixir syntax. 🙂

[compact bobcat]

@amber dune thanks, that's actually quite nice! I didn't know pre- and post- filtering are a thing, and also that they are documented