#v0.19.9: rpc error: code = NotFound

1 messages · Page 1 of 1 (latest)

storm otter
#

Upgraded to v0.19.9 and getting a flood of these errors now:

! failed to receive stat message: rpc error: code = NotFound desc = get full root path: rpc error: code = NotFound desc = eval symlinks: lstat /.../output/src1402935557: no such file or directory

#

Tried to fully scrap and relaunch a new EC2 runner and still getting the same error.

#

Downgrading...

#

Downgrade seems to work fine

#

Dang, now getting this on v0.19.8 too all over the place. Nothing else has changed. The only thing I've changed was going from v0.19.8 to v0.19.9 and then back. But in between the entire EC2 was relaunched from scratch, so there's nothing remaining. The PR itself that is failing to build is extremely tiny, just upgrading a GitHub action "checkout" from v4 to v6.

vague lodge
#

@storm otter are you re-using the state between dagger versions by any chance? Like re-attached EBS storage or something?

storm otter
#

Not at all

#

I wish 😄

storm otter
#

@vague lodge Caught another one

eternal gyro
#

@vague lodge is that a known issue ?

#

i'm having the same issue after upgrading

#

it's very weird. I have a file passed to a module function. if i leave it untouched. The function completes successfully. If i do a change to the file (as simple as adding a whitespace) I encounter this error

! failed to receive stat message: rpc error: code = NotFound desc = get full root path: rpc error: code = NotFound desc = eval symlinks: lstat /source: no such file or directory```
#

if remove the whitespace it runs fine again

#

can replicate on two different workstation on 0.19.9

slow gate
eternal gyro
vague lodge
eternal gyro
#

is the workaround to revert back to 0.19.8 ?

vague lodge
#

Fix will go out in the next release

slow gate
# eternal gyro cache should be invalidated as the file changed right ?

Not necessarily, when removing the space again, your file could be in the same state as it was in your first run, resulting to the same content addressed hash and thus not even recompute. As it was successful once, it would work, whilst it might have failed on the other case (due to above bug).

But glad to see that we fixed it ahahah

eternal gyro
#

hmmm getting the same issue on v0.19.8 now. Help 🤣

vague lodge
#

what version were you using before?

eternal gyro
vague lodge
#

just checking if you might have to prune the cache

eternal gyro
vague lodge
#

when downgrading locally the cache gets cleared automatically

eternal gyro
#

i just ran it with

// +cache="never"

and it worked

#

just works once then fails

slow gate
#

can you please try: dagger core engine local-cache prune

eternal gyro
slow gate
eternal gyro
slow gate
#

do you mind nuking your dagger volume ?

eternal gyro
#

sure thing

slow gate
#

do you want commands ? or is it ok ?

eternal gyro
#

all good i nuked the docker engine container and delete the associated docker volume

#

i'm running the steps now just takes a bit of time (building an app)

#

same as before

  • 1 st run post pruning worked OK
  • changed file content
  • 2nd run failed
slow gate
slow gate
twin cradle
#

I repro'd it locally, ty both for the info ty

#

We are gonna do a release tomorrow to get some other emergent fixes needed out, I will see if I can sneak in a fix for this one too but hard to say until I start looking into the cause

eternal gyro
#

@twin cradle Guillaume mentioned that it was only when using external module (git )

#

so that's part of the reason we couldnt replicate locally

twin cradle
#

ftr (also in case I lose this), repro is

dependency module code (pushed to a git repo):

package main

import (
    "context"
    "dagger/dep/internal/dagger"
)

type Dep struct {
    Dir *dagger.Directory

    Ref *dagger.GitRef
}

func New(
    dir *dagger.Directory,
) *Dep {
    return &Dep{
        Dir: dir,
    }
}

func (m *Dep) WithRef(
    ctx context.Context,
    ref *dagger.GitRef,
) (*Dep, error) {
    m.Ref = ref
    return m, nil
}

func (m *Dep) Fn(
    ctx context.Context,
) (*dagger.Directory, error) {
    return m.Ref.Tree(), nil
}

main module code with dependency on the above through a git dep:

package main

import (
    "context"
    "dagger/test/internal/dagger"
)

type Test struct {
    //+private
    Ref *dagger.GitRef
    //+private
    Dep *dagger.Dep
}

func New(
    // +defaultPath="."
    ref *dagger.GitRef,
    //+defaultPath="crap"
    source *dagger.Directory,
) *Test {
    return &Test{
        Ref: ref,
        Dep: dag.Dep(source),
    }
}

func (m *Test) Fn(
    ctx context.Context,
    //+defaultPath="config/config.local.js"
    configFile *dagger.File,
) (*dagger.Directory, error) {
    return m.Dep.WithRef(m.Ref).Fn().WithFile("config.js", configFile).Sync(ctx)
}

Then run dagger call fn, modify config/config.local.js, run dagger call fn again and it fails

eternal gyro
#

tricky repro !

#

thanks @slow gate and @twin cradle much appreciated !

twin cradle
#

Fairly sure this is the exact same problem as was fixed here: https://github.com/dagger/dagger/pull/11350, except that PR missed the specific case of "contextual GitRef that's actually sourced from a local dir". So hopefully I can just follow the same pattern as the previous fix but applied to this case

twin cradle
slow gate
trim prairie
eternal gyro
eternal gyro
trim prairie
eternal gyro