aws

[naive cloud]

https://docs.dagger.io/183109/aws-codebuild-codepipeline

https://docs.dagger.io/183109/aws-lambda

https://docs.dagger.io/7442989/cookbook/#aws-cloud-development-kit

[still vigil]

🤩

[vital latch]

People who are not following the #java channel may have missed that: I made a demo of the WIP Java SDK during the last community call where I deployed a serverless application with AWS SAM. Easily convertible to any other SDK (ping me if you have any question)
https://github.com/jcsirot/aws-serverless-with-dagger

[zealous vapor]

some of the examples seem to be broken, e.g. aws-sdk gets this:

# main
./aws.go:126:3: client.Container().From("samalba/aws-cdk:2.65.0").WithEnvVariable("AWS_REGION", c.region).WithEnvVariable("AWS_DEFAULT_REGION", c.region).WithDirectory("/opt/app", cdkCode).WithDirectory("/root/.aws", awsConfig).WithExec(cdkCommand).ExitCode undefined (type *dagger.Container has no field or method ExitCode)
./main.go:44:21: client.Container().WithNewFile("/secret", dagger.ContainerWithNewFileOpts{…}).File("/secret").Secret undefined (type *dagger.File has no field or method Secret)

compiles if i do this:

git checkout v0.8.8

still get this when i run it though, despite cdk bootstrap already executed and other stacks working just fine:

#2 20.24  ❌ Building assets failed: Error: Building Assets Failed: Error: DaggerDemoECRStack: SSM parameter /cdk-bootstrap/hnb659fds/version not found. Has the environment been bootstrapped? Please run 'cdk bootstrap' (see https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html)
#2 20.24     at buildAllStackAssets (/usr/local/lib/node_modules/aws-cdk/lib/index.js:342:116743)
#2 20.24     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
#2 20.24     at async CdkToolkit.deploy (/usr/local/lib/node_modules/aws-cdk/lib/index.js:342:142824)
#2 20.24     at async exec4 (/usr/local/lib/node_modules/aws-cdk/lib/index.js:397:51795)

[zealous vapor]

read through the code and found AWS_REGION

region is hard coded in main

[oak echo]

What's the suggested strategy for allowing Dagger containers to use an IAM session from the host?

ah. It seems like a simple solution would be to use the outputs from the configure-aws-credentials step
https://github.com/aws-actions/configure-aws-credentials/blob/69786f15aefa0b940de5c7d1b7155038b71037a0/action.yml

outputs:
  aws-account-id:
    description: The AWS account ID for the provided credentials
  aws-access-key-id:
    description: The AWS access key ID for the provided credentials
  aws-secret-access-key:
    description: The AWS secret access key for the provided credentials
  aws-session-token:
    description: The AWS session token for the provided credentials

thanks for being my ducky, y'all

[fiery tapir]

happy to help 😁

[oak echo]

I did eventually get this working: it's important to note that there is a flag in the Configure AWS Credentials action that turns on the output, and it needs to be set to true in order for these items to cooperate. You'll also have to give the step an ID

        uses: aws-actions/configure-aws-credentials@v4
        id: configure-aws-credentials
        with:
          aws-region: us-west-2
          role-to-assume: arn:aws:iam::<acct>:role/ci-role
          output-credentials: true

[fluid lance]

I would like to see the full example of this. I realize that might be a big ask, I think I can partially get this setup.

[oak echo]

You bet!

[oak echo]

here's the pipeline and the IAM trust policy

# Assume Role Policies
data "aws_iam_policy_document" "github_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${var.github_oidc_provider_name}"]
    }
    condition {
      test     = "StringLike"
      variable = "token.actions.githubusercontent.com:sub"
      values   = [var.github_repo_subject_allowed]
    }
    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }
  }
}```

I believe that variables for subject looks like

variable "github_oidc_provider_name" {
  type = string
  default = "token.actions.githubusercontent.com"
}

variable "github_repo_subject_allowed" {
  type = string
  default = "repo:owner/repo_name:*"
}```

@fluid lance

I followed this guide to set up the OIDC provider https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html as mentioned here https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services

that :* in the variables allows any branch to have access - you can and should tighten it up

Let me know if that helps or if you need more context! I'd have loved to get the OIDC provider terraformed as well but just didn't make the time

One important note is that the OIDC UI setup will have you create a role for use with the provider - but you really don't need it. You just add the trust policy to the role you actually care about. I assume there's CloudFormation or something you can use also

[manic wagon]

For my part, I used the AWS OIDC action but without the 'output step', and then used some pipeline code (in this case, boto3 and Python) to set the env vars. The advantage with that approach is that it also works locally for users who have their aws credentials setup; boto3 will find credentials using it's internal logic, and then just inject them into dagger, whether running locally or in github actions

[fluid lance]

this is the coolest thing I have seen all day thank you

[dusky sable]

Hey, interested about this solution, I am using python and boto3 and I feel like I have to pass the AWS id/secret/token as an argument every time i call my function even if I have them set as env variables. I end up with something like this:
dagger call my-function --aws_id=$AWS_ACCESS_KEY_ID --aws_secret=$AWS_SECRET_ACCESS_KEY --aws_token=$AWS_SESSION_TOKEN
And then I can create my boto client inside my function is there a better way ?

I would need something which set my AWS env variables as container variables at the start

[oak echo]

My sense was that there isn't really a good way to do this at the moment without an IMDS server running

The Zenith design intentionally makes it challenging to read env variables from the caller - see this issue https://github.com/dagger/dagger/issues/6112

If you do have IMDS running then it's not challenging - you just need to tweak IMDS to allow an extra hop

I have successfully used AWS Vault for this

And in an EC2 environment it's there by default also

I've not tried IMDS in GHA

I suspect it's easier to allow explicit tokens but fall back to IMDS when the token args are not present

[dusky sable]

Ok thanks for the informations ! I will stick with my arguments as i'm just trying to do a little demo ahah

[manic wagon]

Hmm, yeah I haven't messed with modules/functions yet, this is 'old style' dagger code; just pipelines and control code!

[wanton elm]

There is some work happening to make the DX around this a bit better, this issue is a good one to follow https://github.com/dagger/dagger/issues/6723

[flint solar]

any <@&1122942803266261084> fans going to AWS re:Invent?

@south crater and @wanton elm will be there, so we'd love to schedule some time to meet up with our fellow Daggernauts.

[fiery tapir]

I'll be there (briefly) also 🙂

[runic tapir]

Moreso, I'd say I'm a Dagger fan going to AWS. 😅 Do you folks have a booth?

[flint solar]

No, we don't, but @wanton elm and @south crater would be happy to meet up with you. I just sent a DM, so I can help coordinate.

[wanton elm]

We’d love to meet with you! Here all week 🙂

[runic tapir]

Will DM, thanks all

[woeful badge]

Hello All - hope everyone had a lovely holiday and new year !!

I've been playing around with Dagger for a while (after seeing it at kubecon last year), and got something working in jenkins (which is hosted inEKS within AWS), I've got dagger running as a daemonset. The issue i'm having, is we use AWS's CodeArtifact repo, which annoyingly, the only way to pull from it, is generating a token (max life of this token can only be 12hrs) , so with my dagger code, it runs that step, caches it which is great, of course, when the token changes, it runs that step again.

My question is, is it possible in dagger (using the python SDK) for it to ignore certain changes ? Stupid question I know, unfortunately, this blocker is really stopping me from getting dagger properly off the ground at my company.

Snippet of code below:
source = (
client.container()
.from_("XXX.dkr.ecr.eu-west-1.amazonaws.com/jenkins-java:0.0.8")
.with_directory("/home/jenkins/", src_download)
.with_workdir("/home/jenkins/")
.with_env_variable(name="MAVEN_REPO_URL", value=f"{maven_repo_url}")
.with_env_variable(name="MAVEN_LIB_RLSE_REPO", value=f"{maven_lib_rlse_repo}")
.with_env_variable(name="MAVEN_LIB_SNPST_REPO", value=f"{maven_lib_snpst_repo}")
.with_env_variable(name="MVN_REPO_USER", value=f"{mvn_repo_user}")
#.with_env_variable(name="CODEARTIFACT_TOKEN", value="XXX"
)
download_dep = source.with_workdir("./").with_file("settings.xml", src_download.file("settings.xml")).with_exec(["sh", "-c", "mvn -s settings.xml dependency:go-offline > /dev/null 2>&1"])

If the env_variable CODEARTIFACT_TOKEN changes, don't run it again ? Many thanks !

[wanton elm]

My question is, is it possible in dagger (using the python SDK) for it to ignore certain changes ? Stupid question I know, unfortunately, this blocker is really stopping me from getting dagger properly off the ground at my company.

Def not a bad question!

caches it which is great, of course, when the token changes, it runs that step again.

Right now the caching system works purely based on input. I think the only way to get something like this to work would be to figure out a way to get the token without having it be an input to a function.

Something that comes to mind is getting the token via some sort of HTTP request. Not sure if this is possible for you.

But can you help me understand more about why you want to keep this cached when the token changes? Is it a performance thing or something else?

[woeful badge]

Hi Lev ! Many thanks for responding, some of our main microservices are done in java, and we use Maven to build the '*.jar', and we use AWS CodeArtifact to download the deps for each micorservice that they specify via their own 'pom.xml'. there is also a settings.xml so Maven knows about where the CodeArtifcat is... the issue here, is unfortunately with AWS, you must use a token to auth against it, and the max life span of this token is 12hrs...

Within my dagger python code:

src_download = client.host().directory(".", include=["pom.xml", "settings.xml"])
        source = (
            client.container()
            .from_("571738151596.dkr.ecr.eu-west-1.amazonaws.com/jenkins-java:0.0.8")
            .with_directory("/home/jenkins/", src_download)
            .with_workdir("/home/jenkins/")
            .with_env_variable(name="MAVEN_REPO_URL", value=f"{maven_repo_url}")
            .with_env_variable(name="MAVEN_LIB_RLSE_REPO", value=f"{maven_lib_rlse_repo}")
            .with_env_variable(name="MAVEN_LIB_SNPST_REPO", value=f"{maven_lib_snpst_repo}")
            .with_env_variable(name="MVN_REPO_USER", value=f"{mvn_repo_user}")
        )
        download_dep = source.with_workdir("./").with_file("settings.xml", src_download.file("settings.xml")).with_exec(["sh", "-c", "mvn -s settings.xml dependency:go-offline > /dev/null 2>&1"])

The reason i would like this to be cached, is because currently, our pipelines download all the deps every build which isn't needed, and was hoping dagger to cache that part. Thanks

to add further, i can take away the '.with_env_variable' as that variable is within the 'settings.xml', so before dagger executes, the token in that file has been updated, and thus, dagger re-runs it.

[woeful badge]

When the token is 12hrs, first run in my pipeline takes a lot longer because it downloads all of the deps for offline, once that is there, following runs are within a few mins

[oblique onyx]

Hi guys, hope you are all doing well. I am contacting you because i am having an issue... Since i updated from 0.12.0 to 0.15.1, my aws get-role command stdout is not working as with 0.12.0, if i downgrade it works well.

for more detail, i am catching the error to check if in err string there is the word "NoSuchEntity"

with 0.12.0 err returns An error occurred (NoSuchEntity) when calling the GetRole operation: The role with name

but with 0.15.1 err returns input: container.from.withMountedDirectory.withMountedDirectory.withWorkdir.withEntrypoint.withExec.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withExec.withEnvVariable.withServiceBinding.withEnvVariable.withSecretVariable.withSecretVariable.withSecretVariable.withExec.withExec.stdout process "sh -c aws iam get-role --role-name role --query 'Role.RoleName' --output text" did not complete successfully: exit code: 254

so my check no longer works with 0.15.1, the idea is to check if role exists to create it if it is the case, and to skip creation if it exists

thanks in advance for your help

i tested it using stderr but same problem

i can still see in the logs An error occurred (NoSuchEntity) when calling the GetRole operation: The role with name

but when i print out and err the text is not present

example:

        "aws iam get-role --role-name %s --query 'Role.RoleName' --output text",
        roleName,
  )
  out, err := ctr.WithExec([]string{"sh", "-c", getRoleCommand}).Stdout(ctx)
  println("DEBUG OUT", out)
  println("ERROR ERR", err.Error())```

364 : Container.stdout: String!
365 : setSecret(name: "AWS_SESSION_TOKEN"): Secret!
365 : setSecret DONE [0.0s]
366 : setSecret(name: "AWS_SECRET_ACCESS_KEY"): Secret!
366 : setSecret DONE [0.0s]
367 : setSecret(name: "AWS_ACCESS_KEY_ID"): Secret!
367 : setSecret DONE [0.0s]
357 : Container.stdout DONE [4.4s]
362 : Container.withExec DONE [0.2s]
363 : Container.withExec DONE [2.4s]
363 : [2.6s] | 
363 : [2.6s] | An error occurred (NoSuchEntity) when calling the GetRole operation: The role with name role cannot be found.
368 : Container.withExec(args: ["sh", "-c", "mkdir run-artifacts"]): Container!
368 : Container.withExec CACHED [0.0s]
369 : Container.withExec(args: ["sh", "-c", ETOOBIG:sha256:22d4470ef683198d7daae65e44c06dea39775776f540a96f8bb969311df08aee]): Container!
369 : Container.withExec CACHED [0.0s]
370 : Container.directory(path: "./run-artifacts"): Directory!
370 : Container.directory DONE [0.0s]
364 : Container.stdout ERROR [2.8s]
364 : ! process "sh -c aws iam get-role --role-name role --query 'Role.RoleName' --output text" did not complete successfully: exit code: 254
330 : CloudPlatformEngine.runPipeline: Directory!
330 : [14.0s] | DEBUG OUT 
330 : [14.0s] | ERROR ERR input: container.from.withMountedDirectory.withMountedDirectory.withWorkdir.withEntrypoint.withExec.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withExec.withEnvVariable.withServiceBinding.withEnvVariable.withSecretVariable.withSecretVariable.withSecretVariable.withExec.withExec.stdout process "sh -c aws iam get-role --role-name role --query 'Role.RoleName' --output text" did not complete successfully: exit code: 254```

you can also see that in line 363 there is the text An error occurred (NoSuchEntity)... i used to catch it in err variable with version 0.12.0

[flint solar]

Thanks for reaching out @oblique onyx , and sorry that you are hitting these issues. Can you create a post in our help forum? We have someone take a look there - https://discord.com/channels/707636530424053791/1030538312508776540

[fiery tapir]

@oblique onyx are you able to share the relevant code, where you catch that error? It's possible that you are hitting a small API breaking change. See "exit code API" in 0.14: https://dagger.io/blog/dagger-0-14

If you relied on the contents of stderr being available in the withExec error, I believe that behavior changed in 0.14. Instead there is a more explicit API:

withExec([]string{"sh", "-c", "..."}, dagger.ContainerWithExecOpts{Expect: dagger.Any}. Then even if the executed command terminates with a non-zero exit code, withExec will return without an error. Then you can check Container.ExitCode(), Container.Stderr() etc. at your leisure. You can read the blog post for more details.

Note: it's possible that my assumption is wrong, and that this is not the cause of your error. But it seemed worth a shot!

[oblique onyx]

Sorry fro my late reply, i will test and let you know. Thanks a lot for the help

[oblique onyx]

    ctr := cpe.BaseAWSContainer(ctx).WithExec(
        []string{"sh", "-c", "envsubst < ./aws/iam-assume-policy-template.json >> iam-assume-policy-tfe-role.json"},
    )

    getRoleCommand := fmt.Sprintf(
        "aws iam get-role --role-name %s --query 'Role.RoleName' --output text",
        "role-name",
    )
    _, err := ctr.WithExec([]string{"sh", "-c", getRoleCommand}).Stdout(ctx)

    if err == nil {
        // Role exists
        return fmt.Sprintf("Role %s already exists. Skipping creation.", "role-name"), nil
    } else if strings.Contains(err.Error(), "NoSuchEntity") {
        // Role needs to ne created
        }
}

this is the part of code failing with 0.15.1, error string does not contain any longer the text `"NoSuchEntity" but in 0.12.0 it does

even though it appears in the logs

[oblique onyx]

the other problem is exit code 254 could also mean access denied

[unique bobcat]

Hi,
Im trying to test a module i created (which triggers a lambda in my aws account).
However im unsure what the ideal way is to test this locally. Is there a way to simple pass the host env vars (acces key, id, sts-token) to the dagger container?

[young panther]

Yep, construct your module something like this:

type MyModule struct{
  // +private
  MySecretVar *dagger.Secret
}

func New(
  // Secret arg passed from host
  myVar *dagger.Secret,
) (*MyModule) {
  return &MyModule{
    MySecretVar: myVar,
  }
}

then use the secret in a module function something like:

func (m *MyModule) HelloSecret() *dagger.Container {
  return dag.Container().
    WithSecretVariable("MY_VAR", m.MySecretVar).
    WithExec([]string{"echo", "hello"}) // Use the secret env var.
}

then you must pass in the secret from the host when calling your module function:

export MY_VAR=blah
dagger call --my-var env:MY_VAR hello-secret

So you'd want to express however many module constructor arguments and fields appropriate for injecting your aws credentials. For Github Actions or similar OIDC federated environment, you could be doing assume-role-with-web-identity so you'd need to inject an OIDC_TOKEN secret to assume a role with.

If you don't need the secrets available to the whole module as the above example expresses, you can just have them as args to individual functions instead of New().

[oblique onyx]

Hi, hope you are doing well, i tested the example with latest version (0.15.4) but am always having exitCode 0 even though exit code is different than 0 ```func (cpe *CloudPlatformEngine) TestAwsContainer(ctx context.Context) (string, error) {
exitCode, err := dag.Container().From("alpine:3.20").
WithMountedCache("/var/cache/apk", dag.CacheVolume("apk-cache")).
WithExec([]string{"apk", "add", "aws-cli"}).
WithExec([]string{"sh", "-c", "exit 255"}, dagger.ContainerWithExecOpts{
Expect: dagger.ReturnTypeFailure,
}).ExitCode(ctx)

// Get the precise exit code
if err != nil {
    fmt.Printf("Command exited with code: %d\n", exitCode)
    panic(err)
}

return "", nil

}```

[fiery tapir]

You need to use Expect: dagger.ReturnAny I think

Then all exit codes will be received without catching an error

[oblique onyx]

same result ```! process "/runtime" did not complete successfully: exit code: 2
│ ✔ container: Container! 0.0s
│ $ .from(address: "alpine:3.20"): Container! 2.8s CACHED
│ ✔ .withMountedCache(
│ │ │ cache: ✔ cacheVolume(key: "apk-cache"): CacheVolume! 0.0s
│ │ │ path: "/var/cache/apk"
│ │ ): Container! 0.0s
│ $ .withExec(args: ["apk", "add", "aws-cli"]): Container! 0.0s CACHED
│ ✘ .withExec(args: ["sh", "-c", "exit 255"], expect: ANY): Container! 0.1s
│ ! process "sh -c exit 255" did not complete successfully: exit code: 255
│ ✘ .exitCode: Int! 0.1s
│ ! process "sh -c exit 255" did not complete successfully: exit code: 255

✔ parsing command line arguments 0.0s

Error logs:

✘ .testAwsContainer: String! 3.2s
Command exited with code: 0
panic: input: container.from.withMountedCache.withExec.withExec.exitCode process "sh -c exit 255" did not complete successfully: exit code: 255 ```

[oblique onyx]

the only way i have managed to get the exitCode and error containing error text (An error occurred (NoSuchEntity)...) is with this code _, err := ctr.WithExec([]string{"sh", "-c", getRoleCommand}).Stdout(ctx) var e *dagger.ExecError if errors.As(err, &e) { fmt.Printf("exit-code: %d. error: %s", e.exitCode, e.Stderr) }

i can create an issue on github if you want

[unique bobcat]

Still looking at this tbh. I am wondering if i cant read the host env vars automatically with something like process.env, this way we dont need to pass them everytime

[frozen blade]

this is expected - Failure+Any only catch return codes up to 128

there's some logic for this explained in this thread here (and specifically the linked comment) https://github.com/dagger/dagger/pull/8466#issuecomment-2438095236

which catches 0-127 exit codes (not higher, those are magical signal exit codes as mentioned above, catching these by default is really not a good idea)

we don't allow doing this because caching signals can cause bizarre caching issues - e.g. suppose you run out of memory and get SIGKILLed - you should not cache that response

[fiery tapir]

https://tenor.com/view/cat-kitty-surprised-shocked-oh-my-god-gif-15440994307613885016

I had no idea

[unique bobcat]

Im wondering if there is something i can do to speed up my runs in aws codebuild.
What i do now is simply run a command that calls an api. ( so a simple function, no build logic etc).
So what i was thinking i could do to possibly speed up the dagger run (it takes a minute on codebuild and 2 seconds when running locally)

• S3 caching for the codebuild instance
  -Maybe finetuning of the dagger container? (Since i just run the api call i dont do anything with the dagger container itself).

Are there any other options that im missing? Not using dagger cloud btw since i read this has another form of caching.

[woven seal]

The best option would be local caching, on reserved-capacity instance(s) that are always-on, so you'll always have that cache locally when builds run.

[fiery tapir]

We're working on adding (way) more options for persisting and distributing cache, bear with us just a little bit longer 🙂

[azure crypt]

Anyone using localstack with dagger? particularly around .publish(..) to a localstack ecr

[abstract iron]

Nope, but I do something similar with a registry.

The problem is Publish cannot use services at the moment: https://github.com/dagger/dagger/issues/6411

This is how I work around it for the moment: https://github.com/dagger/dagger/issues/6411#issuecomment-2354072245

[azure crypt]

Yeah i looked at doing something like this. I'm not using localstack as a service so I actually think i'm dealing with fun docker networking issues more than anything else. e.g. pinging container registry 000000000000.dkr.ecr.us-west-1.localhost.localstack.cloud:4566: Get \"https://000000000000.dkr.ecr.us-west-1.localhost.localstack.cloud:4566/v2/\": dial tcp 127.0.0.1:4566: connect: connection refused" . changing localhost.localstack.cloud to host.docker.internal doesn't work either

[flint solar]

https://www.youtube.com/watch?v=S9BNa4Hn4-0

[civic spire]

Anyone self-hosting Dagger server here on AWS, but not on K8s?

[fiery tapir]

Are you thinking individual EC2 VMs? Or something like ECS?

<@&1122942803266261084> 👆

[civic spire]

I’m open to any.

But I think ECS won’t work as it’s not possible to have elevated privileges

[fiery tapir]

Indeed ECS will not work. EC2 works just fine. But it's up to you to setup ingress connections to the engine, and cache persistence.

I know we have users running a fleet of engines on EC2, and load-balancing connections to them via a standard tcp/https proxy, and persistence via EBS or equivalent. It works, but worth noting:

• We don't provide a recipe for the load-balancing, so it's DIY. But there's nothing special at the network layer, each engine listens on a tcp port, TLS and auth are up to you

• With cache persistence, you will get cache reuse, but within the bounds of what's possible with a random allocation of clients. If you want better cache use, that's unchartered territory and will require some configuration on the client. For example, configure different clients to connect to different engines (or different load-balancers) based on the workload. On this point, we are working on a much better solution, but not ready yet. If you're interested in early access, let me know in DM with some details on your use case 🙂

Hope this helps. We're happy to answer questions along the way/

Also, if you're in the market for a contractor with experience setting this up for you, we can make a call to the community for power users with dagger operational experience who are looking for contract work.

[civic spire]

Thank you. What you are saying makes sense. It’s sounds very similar to self hosted BuildKit basically.

Can’t cache be reused by mounting a shared network storage?

[fiery tapir]

Thank you. What you are saying makes sense. It’s sounds very similar to self hosted BuildKit basically.

Yes, after all it is buildkit under the hood 🙂 Although a heavily modified version.

Can’t cache be reused by mounting a shared network storage?

Yes. But be careful to avoid concurrent writes. Each cache is tightly coupled to the engine state directory, which cannot be shared across engine instances.

But, you can reuse snapshots to run more instances, as long as you take the snapshot after the engine is done shutting down & cleaning up.

Also, Dagger is very disk-intensive. Avoid esoteric distributed storage solutions that introduce IO bottlenecks. It will destroy performance.

[civic spire]

Would you classify EBS as esoteric? 😅

[fiery tapir]

No

🙂

But if you have access to, say, direct NVMe, that will make a big difference on big resource-intensive builds

EBS works just fine though

I was thinking of things like EFS or other "magic distributed filesystem" with lots of hidden tradeoffs

[civic spire]

EFS is very slow indeed. EBS too though. I’ve benchmarked it against NVMe and it’s just not even comparable 😂

I think that’s why companies like Depot using esoteric cache syncing strategies.

[fiery tapir]

Depot used EBS for a long time. I believe now they use Archil (a startup building a robust distributed S3fs as a service)

(disclaimer I am a happy Depot investor 😛 )

But their workloads are different, it's predominantly Github Actions so they have the benefit of a cleanly decoupled remote cache protocol over HTTP. They can probably hit S3 directly for that (I don't know what they do to persist that, but they have way more options than with buildkit/dagger)

The block-level shenanigans are for people like us, stuck with software that tightly couples compute and storage 😅 Something we are actively working on solving btw.

[icy totem]

EFS is very slow indeed. EBS too though