#general
1 messages ยท Page 13 of 1
oh sick
I need to figure out the state of docker on WSL then I can start playing around with blocklayer
Last I heard, Docker works very well with WSL. Docker and Microsoft have always been good partners
That was a journey, but I got docker working working in my WSL
I need to update windows more
@muted gale were you able to activate your BL account?
I needed to figure out docker first
I think the login failed when I tried b/c of that
or that might be bl up
@winter linden when after logging in through the browser I get failure since it tries to open a port on localhost
http://localhost:38932/callback?code=08ed38a0-7a96-43c4-aad7-bed79547aa24&state=38c36e81a8e95638f0a102df0403927c5d6c62ed70ce280f54d77110cabbecef
Unsure if this is b/c of WSL but it should be able to connect if it is listening
I think b/c it errors here it the process is not available to listen after the login:
info Logging into your Blocklayer account...
info Opening ...
fatal exec: "xdg-open": executable file not found in $PATH```
I think that opens the browser, so it should direct the user to open the link manually continue to listen for the login callback
Unclear to me whether I can use the local agent without logging in first
correct, you need to login first
printing the link so it can be open manually is a good idea
another thing you can do, with the current version, is generate an API key
you can run bl apikey create once on a session where auto-opening a browser will work; then pass the key with bl --api-key which will not require a browser
Oh yeah that sounds like a good fallback
Ill try to unblock myself with that
For the login flow, other than priniting the link for manual navigation I think displaying the token so that it can be pasted back into the terminal would help as well. I actually do significant amount of development on remote machines where I would not have bl listening on my localhost. Need to way to login with that scenario as well. I remember the concourse CI CLI having that kind of flow
(I think setting BL_API_KEY as an env variable will also work)
Oh yeah that sounds like a good fallback
Ill try to unblock myself with thatFor the login flow, other than priniting the link for manual navigation I think displaying the token so that it can be pasted back into the terminal would help as well. I actually do significant amount of development on remote machines where I would not have
bllistening on my localhost. Need to way to login with that scenario as well. I remember the concourse CI CLI having that kind of flow
Yeah you're not the first one to encounter this. We will fix it.
I was able to workaround the login with an API key, thanks
bl domain claim google.com
success Successfully claimed domain google.com
bl domain delete google.com
success Successfully deleted domain google.com
Was curious if this would work lol
ha ha ๐
yeah at the moment there is no ownership check on a domain
(but claiming a domain yields no particular privilege of course)
@muted gale to make it easier to develop a configuration from scratch, we implement a local mode: see bl local
Hmm, hitting some issue when following the advanced usage docs:
โฐโ bl push cedriccharly.com
fatal build constraints exclude all CUE files in . (ignored: example.cue)
Unclear to me what this error means
Oh I bet I need to set up the domain first
Should not have skipped to a CUE example
That error manifest in a number of situations, most commonly where you have a cue.mod defined but the .cue file(s) does not have a package clause at the top
So in this case I would guess example.cue doesn't have a package clause
Oh, when I copy/pasted the example vim cut off the first few letters so it only had ckage at the top of the file
Hello !
Howdy @gusty thorn ๐
I'm guessing you caught that final notice on the Telegram group?
Yeah! Also I have been thinking about you lately, I feel like a bunch of things at my new job could be nicer with something like CUE.
You've come to the right place ๐
We have a new demo to show you
Warning: it may give you dotCloud deja-vu
TLDR dotcloud control plane + runs on your own infra + write your own services in cue
(invitation open to anyone else who wishes to see the demo of course)
Is blocklayer in stealth mode rn or is this something I can talk about publicly?
kind of in between ๐ talking about it to other people is fine but it would be better not to write about it in a blog post. It'll be much better/more interesting when we launch the beta.
That sounds reasonable
I was thinking this would be a good tweet to respond to, but I will hold off: https://twitter.com/marcosnils/status/1290356751846592512
@alexandrosM @ibuildthecloud Any references to projects using cue in an interesting way?
@urban cliff all your Cue questions this week are making me curious about what you're building ๐
Welcome @red spruce ! Feel free to ask any devops-related questions here, this is basically devops nerd central. And let us know when you're ready for a demo of Blocklayer ๐
Thanks, @winter linden! Good to be here. Hi everyone ๐๐พ
Heads up we're doing a live deployment session here. @fallow bloom is deploying 2 apps to https://vercel.com (serverless JS platform) and take notes on the experience as a deployment beginner. Vercel is probably in the top 5 best deployment UX today, and we want our beta release to be that way too.
Nice
So @fallow bloom if you have any question along the way just ask them here, this is a channel about devops & deployment ๐
Maybe someone here already has hands-on experience with Vercel? Otherwise we'll learn it together
Okay sure thing, thank you!@winter linden
( Vercel is the new name of Now I think)
Their quickstart guides are looking good ๐คค https://vercel.com/docs#quickstarts
Looks like the deployment worked ๐
@fallow bloom do you have a screenshot of the errors youโre getting?
Iโm out grocery shopping at the moment I will send you a screenshot of them when I get to the office
This is the error thatโs coming up
@fallow bloom is that on the vercel deployment? Or running locally on your laptop?
From the vercel @winter linden
It looks like your app is trying to connect to something (I guess your API?) on localhost, which is what itโs supposed to do on your laptop, but when deploying it wonโt work; you need to configure your deployed app to connect to your deployed backend.
basically your app still thinks itโs running on your laptop (I think)
Okay I will figure this out ๐ค
Which guide did you follow on the vercel documentation?
The one that they have on vercel
Did you use one of the quickstarts? Thereโs a bunch of them: https://vercel.com/docs#quickstarts
No I was never informed on a QuickStart I was only following the instructions Thinkful gave me
Can you share the link to what they sent you? If I understand the steps you followed so far it will help figure out what you shod do next
Okay Iโll send it to you
Sign in to advance your career with 1-on-1 mentorship from industry experts.
Iโm not sure if your going to be able to access it.
no unfortunately
Figured... maybe tomorrow I can screen share on a zoom call or something so we can look at it together
sounds good
Thanks for the help so far appreciate it ๐
Youโre welcome, you are also helping me by sharing your real-world deployment experience. We want to learn from other deployment productโs mistakes for Blocklayer.
๐ค๐ค
I think the error yesterday was because this step had not been done, or something went wrong when you did it: 2. Update the React client with the URL of the server.
If everything works now, it means you probably followed that step correctly which fixed it
So now your React client (running on Vercel) can fetch data correctly from your server (running on Heroku)
Yes thatโs correct
Hi Devs, is there a way to view currently deployed workflows from the UI with the bl trial version?
I'm refreshing my workflows and the I can see alot has changed in the past few months
Weird bl error. I thought this was because I was configured behind a VPN, but different workarounds don't seem to work
I configured the BL_API_SERVER to http://localhost:8080 and that didn't do the trick. I'm curious if there's something here I'm doing wrong
apparently the local command is deprecated ๐ฆ
Hi @cunning yacht ! Just emerging from international travel. I will look into your error in a little bit
So youโre right that bl local is deprecated in the upcoming Beta release (more on that soon!). But the alpha release is stil available, and bl local should work just fine with that.
So Iโm guessing that you are using the wrong version of the bl command. That may be our fault, Iโm not sure. Can you share details on the version you are using?
- What is the output of
bl version? - When did you last update your
blinstallation?
Thanks & sorry for the rough user experience!
@cunning yacht looks like you have an alpha version of bl, but not the latest, which explains that bl local is not available. If you update to the latest alpha, it should resolve the problem.
Here are the update instructions: https://alpha-3.blocklayerhq.com/getting-started.html#local-machine-setup
Hey Solomon I know itโs getting pretty close to the dead line. And Sydneeโs been pretty stressed so I think Iโm ready to step and help out with the project.
Sounds good ๐
Important: if anyone gets authentication issues on alpha-3, please upgrade the CLI to the latest version: https://alpha-3.blocklayerhq.com/getting-started.html#local-machine-setup. If you have any issues, let us know
If you are curious why there is more activity here all of a sudden, it's because we are getting ready for beta ๐
Thanks @winter linden I'll get on this later today
I think I'm on the latest bl version
authenticated multiple times thinking it must have been a expired auth token, still did not work
hey @cunning yacht, can you confirm you use the latest version? We rolled out a breaking change on the auth 2 days ago that requires you to run the latest cli. If then you try bl logout and bl login, let me know what you have.
@wraith niche my version is 9c2a9b45 I just updated this last week
I'll try those changes and see if it fixes it
@cunning yacht we pushed a newer version after you reported your issue this week. So if youโre having issues authenticating, that might be why.
I shared the version number above
I'm trying to reinstall a new version of bl to see if it fixes this issue
Another thing is the docs will need to be updated with this update
Thanks @wraith niche @winter linden it works now, I can see bl fetching the containers now
info fetching latest version of containers
info starting the containers
Creating network "bl_default" with the default driver
Creating bl-registry ... error
ERROR: for bl-registry Cannot start service registry: Ports are not available: listen tcp 0.0.0.0:5001: bind: address already in use
ERROR: for registry Cannot start service registry: Ports are not available: listen tcp 0.0.0.0:5001: bind: address already in use
ERROR: Encountered errors while bringing up the project.
fatal cannot start the local environment: exit status 1```
here is the error message
hey @cunning yacht, looking at the error now. It seems the port 5001 is already used on your local machine (host machine). It's not possible to tweak those ports at the moment, sorry... If you find out which process is binding on 5001 and stop it, then run bl local up, it should work as expected. Let me know what you get.
Welcome @modern rampart ๐
hey guys, having some errors pop up on a basic RDS creation
The maximum number of addresses has been reached. (Service: AmazonEC2; Status Code: 400; Error Code: AddressLimitExceeded; )
Hey @modern rampart, let me check that
looks like an elastic ip limit that you reached on your account. What does your stack look like? Only RDS? It would make more sense for ECS to raise since it provisions 2 ec2 instance for the cluster. I am checking if rds needs an EIP
the limit is per region. I had a very very low limit for my certs for instances (10) despite aws saying the default was 1000. It's very possible you have a very small limit for EIP. Increasing the quota is very simple.
Will dm w/ larger logs if you want?
yes please, it would help confirm my guess
update for anyone interested in the issue: there is a default limit of 5 ec2-VPC elastic IP per AWS region by default, and blocklayer allocates 2 for the NAT gateway for AWS-components. Asking a quota increase is needed if you encounter that error.
Having an issue logging into the CLI, won't let me even though the PW is correct
info Opening https://blocklayer.auth.us-west-2.amazoncognito.com/oauth2/authorize?client_id=44tbd1f9754vs8dvc9nt5d5v7n&redirect_uri=http%3A%2F%2Flocalhost%3A38932%2Fcallback&response_type=code&scope=openid&state=3e70ed959efd00e6c9a45c942d18687d14f58f414b964e70f04b0240a8128fb4
Tried uninstalling/reinstalling, no-go
If I try to reset password, I get this error "Could not reset password for the account, please contact support or try again"
hey @modern rampart, checking this now.
@modern rampart can you give me a bl version output, looks like you have an old version
โ ๏ธ CLI update - the syntax changed for pushing source code to a component
Example with a component named web:
Old syntax: bl push web ./
New syntax: bl push web=./
This now allows atomic changes when pushing multiple sources: eg. bl push backend=./code/api web=./code/web
There is a current outage on AWS Cognito and some authentication queries from Blocklayer are timing out. When this happens, both the cli and dashboard are unaccessible. We're tracking the issue and will update the status here.
The service is back, everything should work now. Although the issue is not resolved on the AWS side, so some errors may happen again.
FYI itโs AWS Reinvent today, Corey is live-tweeting it here: https://twitter.com/quinnypig/status/1333797449685553153
Weโve pretty much defaulted to ECS to run containers on the Blocklayer โstarter stackโ
This is cool too: https://twitter.com/quinnypig/status/1333816368441921537
And a little bit of competition for Blocklayer ๐ https://aws.amazon.com/proton/
Proton's problem statement seems pretty close to BL's messaging indeed, but the templating format looks meh (https://docs.aws.amazon.com/proton/latest/adminguide/ag-svc-template-bundles.html)
cloudformation yaml + jinja templating. Meh, indeed
Yeah. Blech.
God, as if yaml was not already enough ๐คข by itself
Container support for lambda is a great feature, Iโm sure it will help grow Lambda usage even faster. Itโs also great for Blocklayer, because the more you use Lambda or other FAAS, the less you can rely on traditional app platforms to deliver everything. If you use lambda youโre going to want to glue it together with other stuff.
Congratulations @grim dawn for graduating!!!!! Well deserved.
Whoo! I missed that. Congratulations @grim dawn
Welcome @swift barn ๐
Turns out people have a lot of opinion on PaaS, who knew??? https://twitter.com/solomonstre/status/1336160531258855424?s=21
dotcloud 2.0?? ๐
thanks kema!! game time now ๐
TIme will tell ๐
I will say that Docker was meant to be an amplifier of Dotcloud. We wanted them to be more than the sum of their parts. But we never got around to that.
And nobody else did either
very interesting - I do remember when we first started working on Fuzzbuzz, we didn't want to build/manage infra - we just wanted to put our dockerfiles on something that just ran docker but couldnt find anything (app engine took 30+ mins for 2 containers, heroku docker support is ehh) - wouldve been interesting to see a platform for docker like you mention
What does your current deployment stack look like?
hmm - much more complex than it used to be, but we try to keep it as simple as possible. We deploy debian VMs that just run our go binaries. We pull updates from a private package server we update directly from github actions. all of this sits behind nginx, and i think we also use rds. I think thats about it - postgres, nginx, go and of course docker ๐
not as close to that side of things as I used to be though
@soft jay it would be nice to do a quick call for discovery and also if you're interested to see a demo of the blocklayer Beta
๐ Released today: new provider for Heroku. Now you can include containers/heroku apps to the rest of your BL Stack
haha we're on AWS now, but sure - I'm up for a call
well, we support ECS and RDS as well right now ๐
basically with BL you assemble your PaaS on your own infra
here is the link: https://calendly.com/blocklayer/demo
๐ค interesting - I hadn't thought of it that way. I'll pick a time
Yeah the BL philosophy is: build a paas on top of what you have. Then you can change it in a million ways later, very easily - and everyone keeps the same tools to deploy
so if you run binaries in VMs today, then your BL paas would run the same binaries on the same VMs. Until you decide to change, which BL helps you do easily
So you get the productivity benefits of a paas without the cost or rearchitecturing your stack
The current generation of paas cannot do this because they are not programmable
interesting - I'm really curious to see what this looks like in practice.
Also, I made the call for Friday and added Everest + an engineer but realized they can't do friday - I'll cancel that and pick another time - sorry!
no problem, you have a reschedule link in calendly
Hi, I'm interested in experiments of a build definition syntax in cue. Is there some code somewhere I can look at?
weโre going to open-source our own def soon
i'm more just interested in the syntax itself, brainstorming about it
so you can look at the whole code soon ๐
you can look at the current (alpha3/beta1) syntax in github.com/stackbrew/stackbrew but we are making major changes in the next (soon to be open-sourced) version
in alpha-3 the only supported build is via dockerfile. In next version you can hit the full buildkit API , complete other end of the spectrum ๐
Hello awesome Blocklayer team!
I very recently got an invite to Blocklayer beta, thank you for that ๐
unfortunately I can't login as I'm getting an error incorrect username / password combination
hey @swift barn, I'll set you a new pwd manually. I will pm the new one.
thx!
@swift barn in case you need to chat another day: https://calendly.com/samalba
Just saw this: https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-ecs-announces-the-general-availability-of-ecs-deployment-circuit-breaker/ - it'll greatly improve the deployment error handling on our ECS provider
@urban cliff I found the cause of the error. โcopyโ was indeed not implemented...
update: โcopyโ is now implemented ๐ Your example config works.
Happy new year everyone, thank you all for investing your time in testing the many iterations of Blocklayer prototypes. Your feedback was invaluable. We promise to finally launch this year :)
I wish you all happiness and good health.
You're welcome, and you know, Blocklayer will certainly help us to launch this year too! ๐ฌ โจ
happy new year blocklayer !
today I managed to start my api server in a blocklayer stack ๐ small victories \o/
now to clone the stack in order to rename it let's just say I wasn't thinking clearly when I created the stack
@swift barn I have ๐
bl set <component> <key> [value] [flags]
It's the json representation of the key
You can see what it looks like using bl get
bl get -S backend -E staging
{
"Component": {
"api": {
"Settings": {
"Settings": {
"autoProvisionInfrastructure": true,
"awsConfig": {
"accessKey": {
"value": "redacted"
},
"region": "eu-west-3",
"secretKey": {
"value": "readacted"
}
},
"cluster": {},
"container": {
"dockerBuildArgs": {},
"dockerfilePath": "api/Dockerfile",
"healthCheckPath": "/api",
"healthCheckTimeout": 10,
"memory": 512,
"port": 3000
},
"environment": {},
"initContainer": {
"environment": {
"ANALYTICS_KEY": "",
"ANALYTICS_SERVER_SECRET": "",
"ENV": "",
"FIREBASE_CLIENT_EMAIL": "",
"FIREBASE_PRIVATE_KEY": "",
"MAILJET_API_KEY": "",
"MAILJET_NL_API_KEY": "",
"MAILJET_SECRET_KEY": "",
"PASSPORT_SECRET": "",
"STRIPE_ENDPOINT_SECRET": "",
"STRIPE_PRIVATE_KEY": ""
}
}
}
},
"ComponentDomain": "backend-399-api.microstaging.io",
"ComponentName": "api",
"EnvName": "staging",
"EnvSlug": "staging-863",
"StackSlug": "backend-399"
}
}
}
I thought this would let me set individual envionment variables like I don in the website
but now I'm not sure what a key is ๐
It's the key within the json -- in this case there'a s environment in Settings
should I try to set Component.api.Settings.environment
So bl set api environment '{"foo": "bar"}'
I think Component.api.Settings is implied
ok
Let me double check something
I think though you should be able to do that from the dashboard directly
I wish it would take a file ๐ my "value" is quite large
Are you trying to set this for a stack or an environment?
If you click on the environment then Settings -> API, there should be an environment thing you can change
Oh I see
So you'd rather do that from the CLI at once
I hoped I could script it to avoid setting them one by one ๐
yeah yeah
this should work then
I did it in 2 steps
foo=$(jq -rc '.Component.api.Settings.environment' env.staging)
bl set api environment $foo -S backend -E staging
I am "renaming" a stack (so dumping config from the initial one, creating the new one, reloading config)
Oh I see
You should be able to do that with bl get api > api.settings, then bl install api ecs -f api.settings
Well, bl install is for the stack, not the environment
Yeah I see ๐ Then what you're doing is the right way
though i used bl get --stack-admin --stack staging | jq .Component.api
to get the component settings
error Change failed: bl.Directory "context" is not concrete
the bl set failed ๐ฆ
oh yeah ! i need to push the code
of course
๐คฆโโ๏ธ
something didn't go that well : error Change failed: Component.api.run.infra.ecsAMI.environment.AWS_DEFAULT_REGION: failed to resolve task command: Component.api.run.infra.ecsAMI.environment.AWS_DEFAULT_REGION: non-concrete value string
on the push
uhm, let me check ...
it's kind of weird :
bl get --stack-admin --stack backend
correctly lists the component with the aws credentials
Looks like the region is missing
but in the website the credentials are missing
@wraith niche ^^^ this was reinstalled "manually"
is it possible we don't do proper config checks on CLI installs?
I'm guessing the CLI install went wrong
like, maybe a different key was set
Do you have the command you used?
I assume the bl install from the settings.json didn't pick up the credentials that were dumped
yes and the files
Yes the cli install does not check that the settings are concrete. Youโre missing an arg in the aws config.
Maybe install the same component from the dashboard and compare the get with your set payload
Youโll see what differs
Based on what is missing (e.g. complaining about the region, but also you cannot see the credentials on the website), I would think maybe all settings are missing
As in, maybe they were set under the wrong key? Since we don't do type checking on the CLI it's possible
bl install --stack backend api ecs -f component.json
@wraith niche does the json file need to have a Settings key at the top? I don't remember
@swift barn if you do a bl get --stack-admin, what do you see under Settings?
Like, do they look alright, or do you see "Settings: Settings: { ..."
yes
?
I can't do that querynow : I used the website to force the values in the mean time sorry
Oh yes great, thanks
but the bl get was returning things I redacted the secrets manually
Yep, our fault for not validating
they were present and correct
You can see in the settings above it shows up as "Settings: { Settings: {"
(see the extra "Settings" embedded)
bl install takes a settings file as an input, there's no need to have "Settings: {" inside that file
In this case, it created one giant setting named "Settings"
Sorry about that ๐
no problem
so I deleted the component and the env using the website
I recreated the component after updating the json to have the expected shape
I recreated the env and its values
I tried to push the code
new error :
error Change failed: failed to execute task: failed to solve: rpc error: code = Unknown desc = illegal base64 data at input byte 8
weird, secrets looks invalid, can you share your config in pv?
invalid secret is a much better error message ๐
I manually updated my settings file to redact the secrets before uploading it to discord
promptly forgot and used the file with redacted secrets to recreate the component
I was sure I updated them through the website but looks like not
Ha! Good, that's an easy fix ๐
not so fast captain'
this time I am certain I updated them through the website
and when I bl get --stack-admin -S backend
I still get the old ones
nop I only get the old
Can you try to do a get, then set the secret to anything else through the dashboard, then get again?
I created a component and did that : the second get returns the same as the first (and I can't share it right now cuz I used the fixed file with my actual secrets in it so the get returns my secrets ๐ )
ok I forgot to click on update in the bottom right when editing my component sorry about that
I deleted the old stack through the dashboard
- dropped all the components
- dropped all the envs
- deleted the agent
- deleted the stack
in that order, however the ec2 instances running the stack were not removed
the stacks ecs cluster is also still there
I think I managed to track most of the resources there may be leftover vpcs and nat gateways and stuff
just to confirm:
- for each stack I get an ecs cluster with its own set of resources (network, disk, instances etc)
- all envs within the same stack are deployed on the same cluster but with different settings
so creating a new env on an existing stack is "free" only increases pressure on existing cluster but if the cluster has enough capacity that's not an issue
I intend to use BL to trigger auto deployment of test environments from gitlab ci, my current plan is to create a new environment with a name derived from the branch name
ideally I would be able to easily "clone" a reference environment if the env doesn't exist (I haven't looked into it yet but I think I can achieve that using bl cli)
now I have to find a way to delete the env once the MR is merged to avoid polluting my stack with hundreds of leftover envs ๐
my first attempt at cloning an env to create another env
I tried to set the custom domain too through the dashboard but it failed
info Fetching logs for change-id 1af34633-80d7-4d18-9301-306218645227
error Change failed: component "" has incomplete Settings: cue compile: Component.api.#ECRImage.buildImage.context: structural cycle
Component.api.#ECRImage.buildImage.image: structural cycle
Component.api.#ECRImage.ecrCreds.build.image: structural cycle
Component.api.#ECRIma
yes it's correct
also correct, you'll need to increase the size of cluster if you end up reaching the limit of running tasks on the ECS cluster
every env generates a cloudformation stack for the container running and associated resources (ecs service, task definition, lb target group, etc...), so if you delete that stack, it'll cleanup all resources associated with the env. Not perfect, but it's a stopgap until we support auto-destroy.
I think @jolly plank had this error as well before. I was tracking to track this. What comment did you run? Just a push / apply?
I simply defined the custom domain for my env through the dashboard
ok, I think you trigger a race condition that is unrelated to custom domain. I think if you retry it should go through. I still need to fix this race though...
for custom domain, the first apply will generate a certificate and you'll receive an email to validate it. The apply will fail saying the cert is not validated. Just validate the cert from the email received and re-apply, it'll work.
hmm I did not receive an email but that my be because my domain is not hosted at aws and it used CAA (which I discovered when I tried to manually create the certificate)
i'll delete what I created manually and retry
even if the dns is not on route53, you still need a tls cert on your aws LB, that's why it goes through this generation. AWS usually send emails to "admin@yourdomain.com" or "postmaster@yourdomain.com". If there is no MX on the domain, you may need to generate one manually in the corresponding region, with validation type "DNS". I can show you how to do from the console, then you'll have a txt record to add to your domain so aws can create the cert.
I think it failed because of the CAA records which did not list aws as a trusted emitter
I'm cleaning up to retry
awesome ๐
now I'm going to play tester and try some dumb things ๐
Oh NOW youโre going to play tester? ๐
well ... until now I was simply failing to follow normal behaviour
now I tried something really risky ... and it seems I did break everything ๐
fyi, I am testing the ecs circuit breaker right now, so the container crash loop on error should disappear. Ecs will auto-rollback...
that sounds great, make sure to check on an initial deployment where the container doesn't work so there is nothing to rollback to :p
since sam said there are no wildcard domains anymore, I tried to bypass that by using a wildcard domain in my env's custom domain
it sees the dashboard didn't like my idea :p
oh its back (maybe it was just a small outage)
damn, it looks like sam really didn't want us to use wildcards : in the validation email I got, the *. wilcard prefix was stripped
oh but in the cert manager the wildcard is there !
well you can still add a wildcard manually to the LB, I don't think BL will mess with it
no but it will likely stil ask me to validate for the new domain I create right ?
actually yes, but those certs are free. You can still validate it and use the wildcard instead.
i'll try that in about 18 minutes ๐
ouch
the I can still validate works
not sure how to use the wildcard instead but that's going to be for another day
I fixed this race condition, it should not happen anymore. Let me know if it does. Also the circuit breaker (preventing container crash loop on ECS) is live. It will auto-enable on your next bl apply.
We just released a new Agent version that increases stability on code deploys. It should remove most of the intermittent errors.
We also released a new CLI yesterday that improves the error readability and overall logging. Feedback welcome!
just saw the popup warning of unsaved settings tonight ๐
๐
No, why?
hmm , I had initially installed bl as a binary, since I don't like manual update chores I made a script to use the docker container instead and shared the ${HOME}/.config/bl directory with it. today when I try to run commands, it complains about xdg-open not being in the path
so I assume that my credentials are somehow not valid anymore but I am unsure why
Yes itโs usually the login. Probably your token that expired. If you want to avoid it, you can use an api key (generate it from your user settings in the dashboard)
I guess i'll just use an api key
And give the key using the env var โBL_API_KEYโ
Probably your token that expired
shouldn't there be a refresh token for such cases ?
it will automatically pick it up if it is named BL_API_KEY ? (that would be nice)
Yes
Sharing what went to prod today: new logs activity in the dashboard (see the screenshot). It's much much nicer than the previous implementation. Now it's possible to follow the whole app deployment from the dashboard (without the cli). And it's searchable. Feedback welcome.
Also working to upgrade our Runtime to the latest version of Cue, it should increase the performance a lot (early tests show that is twice as fast for evaluating our whole production config).
Lastly we are working on open-sourcing our engine and SDK. If you would like early access let us know!
@here https://beta.app.blocklayerhq.com/ isn't loading for me, has the url of the dashboard changed?
looks like the right url. checking
There is an outage on Netlify right now, bringing down our dashboard. Our API (running on AWS) is not impacted so the cli works fine.
โ The dashboard is back. We changed the dns config to avoid using Netlify's A record (which is faulty), and use the CNAME instead. Sorry for the downtime.
Looks like itโs official, congratulations @cosmic zinc ๐
A big congrats to @justincormack who has been named CTO of #Docker! Justin joined in 2015 and was a senior engineer in security. He's been a maintainer of Docker's #opensource projects & is a member of the @CloudNativeFdn TOC. Well deserved! ๐๐ฅณ๐โญ๏ธ๐
150
Nice!! Congrats ๐
(I donโt think he checks discord very often ๐
Awesome news!
Awesome indeed! Congrats @cosmic zinc ๐ฅณ
Welcome @swift barn ๐ Whatโs your github ID so we can give you access to the future opensource repo?
danpopsd
thanks for the issue @swift barn !
i really really really like what i see!
see the promise of this
I replied with a suggestion for one-line PR if you feel up for it ๐ (just have to remove one line)
oh yeah you need to do git commit -s
nah my syntax was off half asleep ๐
all good ๐
didnt add '<>"
have the same problem sometimes with falco on my mac editor vs my linux desktop.
appreciate the patience.
aaaand merged @swift barn thanks ๐
Welcome @frigid canyon ! We discuss the Blocklayer beta here, and our (soon to be) open-source project, in #dev . You should have received a github invite.
Long time, @frigid canyon. Welcome!
please be gentle
will try my best :)
indeed!
didn't receive anything yet, but will patiently wait!
ok I double checked
The request for me took about 45-60 minutes
Does dagger look for cue.mod/... relative to the directory argument?
meaning that the following (sharing of cue.mod/...) is not supported currently?
project/
cue.mod/dagger.cloud/dagger/...
app-1/
dagger.cue
app-2/
dagger.cue
@stoic knot currently yes, the directory is packed and shipped to buildkit, it doesn't see anything else
Thinking of adding something like this:
$ dagger compute โsource-dir project/ app-1
We don't have a stdlib as of yet
then app-1 could be replaced by any valid cue eval argument
for example:
$ dagger compute โsource-dir project/ dagger.cloud/dagger/foo
(computing package foo)
Iโll do it now actually
๐
optional cue arguments
for specifying what configuration to build and compute. By default it just builds the source directory as is (same as cue eval)
But in a directory layout like the one described by @stoic knot , where the source directory is a project with multiple configurations, then you would specify a path within the source: in this case either app1/ or app2/
Itโs kind of like docker build -f
the pattern of [context is current working dir, arg is path to inputs] is common. Docker, Go, and Cue all use it
For the examples, it would be nice to remove all of the repeated cue.mod dirs
Yeah agreed, weโre going to load a default stdlib from a public repo. Itโs on my todolist ๐
Another cool use: cat test.cue | dagger compute โsource-dir project/ -
also loading from remote sources: dagger compute โsource-git https://gitlab.com/shykes/myconfig#prod
Yeah, don't pay too much attention to those cue.mod in the examples, they're just experimentations
Perhaps we could start by embedding a cue.mod overlay in dagger itself ... at first the code and the stdlib are going to be tighly coupled, it's kinda neat if they go hand in hand at the beginning
(so we can make PRs that change code/spec/stdlib at once, we test them together, ship them together, etc)
I'm thinking that --source-dir should default to "." like other tools. Is it correct to say that today it defaults to the argument supplied? That always has to be a directory as well?
Is there anything on how multiple envs will work?
w.r.t. --source-dir defaulting to cwd, I typically have a ci dir as a peer to my source dirs, then run all commands from the root of the repo, where I keep go.mod / cue.mod / etc.
or so that commands have the entire repo as the working context
+1
makes sense to me
^^ @winter linden
(I think that's what @winter linden meant with --source-dir)
@stoic knot ^^ but yeah, what I meant by this, is we could embed our stdlib in dagger itself, which would be overlaid on top of cue.mod ... the use case you described would work. You'd have your cue.mod in your repo with your packages, and dagger would overlay the built-in stdlib on top
basically, import "dagger.cloud" would magically work because of the overlay, and import "myowncuemod/something" would also work
(ideally at some point we could piggyback on cue's module support)
I'd say that Cue modules load like one would expect, and that it does not manage dependencies yet (versions / vendoring / fetching)
my hof tool uses Cue mods, as they work mostly out of the box
yup
^^ this is basically what cue itself does for its stdlib
it works for non-stdlib as well
I have a few Cue modules already for hof code generators
Oh yeah we use them a lot too
Just for daggerโs stdlib we canโt expect everyone to copy paste those files in their cue.mod
Is the stdlib just Cue? or does it include Go code in the dagger cli?
Right, you can think of dagger as providing a cue distribution in the same way say Red Hat provides a Linux distribution
Would you describe hof as providing a Cue distribution? @winter linden wondering how you are defining that? (context the user-functions discussion we had on the Cue GH, which coincidentally came up again)
You want the ability to individually build and install anything straight from source - cue mods. And you also want a stable universe where everything is known to work together - the dagger stdlib
I just mean a bunch of cue packages shipped & tested together - the stdlib weโre discussing
๐ Heading out for the day
Is there support for GCR (google container registry)? (or AWS's version, i.e. private registry auth)
Today, I run gcloud auth configure-docker, can dagger reuse docker's auth config?
Supporting docker's current config seems like it could smooth transitions
I've been referring to these as "cue modules." (like go mods) Though there is more when it comes to things like dagger & hof, because they have a sort of DSL on top of Cue
Weโve developed it at some point. It might still be in the stackbrew repo (not currently ported to our beta, but could be rather easily)
you can date it by the old-style :: notation for cue defs ๐
Welcome ๐
Thanks! Some familiar faces here โบ๏ธ
More to come I hope!
For sure! Having a quick glance at the repository (while scanning photos for a friend ๐ yay for multitasking)
Good to see you @fresh mesa ! ๐
You too! ๐
Latest update on the Blocklayer Beta ๐
We just released a new Agent version b7c3ab93. It's strongly recommended to upgrade. Along with bugfixes, it embeds the latest version of Cue which speeds up the overall deployment.
We re-enabled the Agent's cache (which was disabled in a previous version because of several invalid cache issues, those are now fixed). With the latest Agent, every time a Stack's component does not need to run, it'll re-use the cached version, avoiding unnecessary steps.
We also fixed a bug in RDS, some users were impacted. If you see an "invalid password" error on the DB deployment, just ping us, we'll help you fix it manually on RDS.
Feedback is welcome, as usual ๐
A new provider is available on Blocklayer Beta ๐ฅณ
based on the config we've been using to monitor our production endpoints for months - so it works ๐
we're also preparing a new agent release that adds telemetry on the codebuild activity (more data about what's going on on your aws account while the code is deployed) - will share more details in a newsletter this week.
Blocklayer Beta upgrade:
- New agent that includes the latest runtime, it'll increase the speed and stability
- Fixed a cache issue that causes builds to fail in some situations: if you get weird cache errors during a deploy, ping us and we'll fix the cache manually (it won't happen again after upgrading to the latest agent)
- The Netlify provider now has the ability to disable the build (useful to push static assets with no JS to build)
๐
Welcome @lime sierra ๐ Youโll probably be most interested in #dev where we discuss open source development. Iโll give you access to the github repo
@winter linden Yes that's it ๐
Hello! ๐
Hi @patent stirrup good to see you here ๐ Whatโs your github account so I can give you access to the future open-source repo?
fkautz
Invite sent ๐
OSS dev discussions are in #dev , itโs a mix of maintainer/contributor discussion, and user questions and feedback. Make yourself at home ๐
Thanks!
@winter linden I can try driving this for edge and service provider use cases
They are going to have issues here, terraform and helm just donโt cut it
they have a meta-orchestration problem
everyone focuses on the k8s cluster, but the real challenge is above that
Whoโs โtheyโ? Iโm guessing your employer or client?
Seems very relevant to Dagger ๐
ah, i should be more clear, i'm working with service providers (think ISPs, telecoms, edge) in the open source space
i should put together some slides on edge computing and 5G to describe how this all works and where some of challenges are
from a business perspective, target enterprise though... sales cycles in service provider sales cycles take too long
they should be long term goals, they won't solve short term solvency issues
@patent stirrup we got another request for KinD today ๐
think it would be possible to make a generic kubernetes type and specify the implementation underneath?
e.g. gke, eks, aks, baremetal provisioned via kubeadm, openshift, k3s, kind, minikube, microk8s, docker desktop k8s... all result in a kubeconfig
yes absolutely. we did just that in an earlier prototype. Cue is a perfect language for it
Yes itโs what we started to do in stdlib/kubernetes. It aims to be a generic client to apply a config to any k8s cluster.
nice
It works but itโs just a start, contributions or ideas are welcome of course
oh, also forgot... being able to just pass in a kubeconfig is useful too
Yes, stdlib/kubernetes takes a generic kubeconfig. In the eks example, the kubeconfig generator is aws specific. But the kubernetes pkg could take one that was generated from gcloud for example.
you're forcing me to learn yet another language...
i picked up go because of docker ๐
Itโs honestly addictive ๐
I donโt see myself doing things differently, although I am obviously biased ๐
yes blocklayer was the original name, the blocklayer beta will become Dagger Cloud
did you know Gordon is actually a GIRL???
Hello, thanks for the invite ๐
Welcome! You should have received a github invitation
I only write cool code
My first react app deployed with dagger up ๐ https://60663fd11f216500b3fda262--dagger-example-react.netlify.app
I followed the new React example, live here: https://github.com/dagger/dagger/blob/main/examples/README.md#react-deploy-a-simple-react-application
Please try it out and let me know how it goes
More updated examples:
- Deploy a React application
- Provision a Kubernetes cluster
- Deploy an application to Kubernetes (with or without Helm)
- Add HTTP monitoring to any app
Got the invitation, thanks!
Hello ๐ Thanks for the invite
Welcome ๐
github invite underway
Great to see some familiar names already in here
Hi Solomon. Thanks for the invite. My GH handle is lachie83. I look forward to kicking the tires
Thanks. Invite received. Taking a look around now
Just read through the examples. Pretty neat. Working on getting my head around how it works
Yeah lots of work ahead on explaining it succintly
Basically you write your deployment plan in a declarative language called Cue. If is basically JSON meets Go. So you have a full blown developer experience including reusable packages.
In that declarative layer you (or someone else if you reuse their packages) can embed processing pipelines to run arbitrary code as part of applying the configuration. The pipelines are really buildkit DAGs: basically anything you can do in a docker build, you can do in a pipeline.
Once I've looked at it a little more I'll ask some questions. Right now I'm trying to figure out how I would point it at any Kubernetes cluster (not specifically one on AWS) assuming I have a kubeconfig (I know it doesn't flow with the narrative in the examples). I was going to see if I could retrofit with Kind (https://kind.sigs.k8s.io/). Really just wanted to experiment so that I "get it"
The result of those 2 things together is a complete programming model for application delivery as code, end to end.
The declarative CUE is like the HTML layout; the processing pipelines are like JS handlers.
Weโre working on that example right now. We got this request a lot ๐
Great! That's really neat.
Literally 3d request for KinD in 24h
The killer feature is the programming environment - we think we have solved the problem of expressing your app delivery as real code. Everything else is packaging for that.
You can also take a look at our stdlib: https://github.com/dagger/dagger/tree/main/stdlib
It gives a sense of what it looks like to develop reusable Cue packages for dagger
It's pretty easy to read at a quick glance
That remains true even as complexity increases. Weโve used it in production and the perceived complexity stays close to the actual complexity. Which is not the case for most deployment scripts and other โxxx as codeโ things like Terraform, Cloudformation...
Most of the credit goes to Cue for that. We are working closely with the Cue developers to take full advantage of it. It is a marvel of a language.
That's neat and is incredible value if it turns out to be true
Hey solomon, thanks for the invite. My github username imumesh18. Looking forward to get my hands dirty๐
Welcome Umesh! I just sent you an invite. Weโre here to answer any questions you may have
I appreciate you taking the time to test this out everyone
Be KinD! :p
@abstract prairie not sure if you saw it, but thereโs a section in the docs called โDagger programmer guideโ that tries to address some of those questions of โhow does it really work?โ
Thanks Solomon for the invite. My GitHub username is wondenge.
Welcome ๐ Sending invite now
thanks for the invite! my github is abdulajet ๐๐ฟ
Welcome! I just sent you an invite. Ask any questions here, the docs are still rough as you can see from all the questions above ๐ Any time you have to try it out is appreciated.
bank holiday weekend here so iโll give it a go. looks great!
so this in cue at first glance looks fantastic:
CUE can currently extract definitions from:
Go code
Protobuf definitions.
I remember when I could code on the weekend... those were the days... have fun and let us know how it goes!
yup ๐ Cue is ridiculous
any experience with protoc-gen-validate?
int32 port = 1 [(validate.rules).int32 = { gte: 5000, lte: 10000 }];
nope sorry but maybe someone else here
doesn't look like valid proto syntax to me, but perhaps there is a preprocessor or a language feature i'm unfamiliar with
one example os that cue can auto-generate the full kub resource schema by simply pulling the kub go source code
yea, that's what i was thinking
thereโs a kub tutorial on cuelang.org
i've been developing most of my APIs as grpc endpoints which begin life as a protobuf
What is invalid syntax? The "options" you see with protoc-gen-validate is "valid" syntax, AFAIK.
so i'm wondering how grpc integration works, if at all
behold the mastermind behind Cue ๐
the master himself
@spice valve yea, i may just not know proto well enough, most of the options i've used i recall were annotations
It is just that protobuf doesn't really have a spec, and where it does it ad-hoc and not reflecting state of the art (implementations are leading spec in proto land mostly).
i don't remember that syntax in the docs
ah, there it is...
[deprecated = true];
in the docs :p
I'm currently working on interpreting and generating proto values as well (at first just textproto and jsonpb). This is quite a complicated feat, as it is impossible to parse even JSON pb properly without having a schema. But getting there.
they don't specify how field options work in the language guide located at https://developers.google.com/protocol-buffers/docs/proto3
deprecated is the only reference to their existence
literally that line i posted
Yeah that spec is not reflecting the full reality. Also textproto isn't specified at all. And the protobuf to JSON mapping is just plain weird.
so it's a knowledge gap on my side (and i'll hapilly blame that doc)
I do think these type of options are covered in the spec. Though not sure.
i've also used this for generating REST endpoints in grpc (via grpc-gateway)
service EchoService {
rpc Echo(EchoMessage) returns (EchoMessage) {
option (google.api.http) = {
post: "/v1/echo"
body: "*"
};
}
}
that's really nifty though ๐
yea, lang spec is probably elsewhere, and they just don't cover these in the guide
Example of mapping weirdness: integers can be represented as strings or integers, and when converting to JSON, some integer types are converted to ints and others to strings (by choice, not a bug).
and to further complicate things... javascript also treats it asa float :x
CUE currently doesn't allow specifying message signatures, but we're thinking about it. In the mean time one can use attributes for arbitrary annotations.
hello: โworldโ @myannotation(goes here,foo=bar)
i'll definitely ask questions here, i came cross cue a couple days ago with this community being my first exposure
I'm here only sporadically. If you want cue-specific answers to be lastingly documented, the best forum is github discussions on cuelang/cue.
ah, thanks for the pointer
just tried importing in k8s api as mentioned in the docs
really nice!
I started a thread to collect preferences on kubernetes setup : https://github.com/dagger/dagger/discussions/257
i'm happy to help there
i'm involved with a few open source projects that'll benefit from this as well
as dagger matures, i'm going to spend some time (or convince others) to integrate ๐
Iโm off to bed, thank you all for participating, I look forward to hearing from your first experience with Dagger.
good night!
doesn't appear to be object oriented
If I recall properly, OASIS Tosca is object oriented which ends up being a source for much complexity
the telecom industry is on tosca :p
What is not object oriented? If you're referring to CUE that is correct. We've found that overrides especially are what causes the complexity. This holds for GCL/Jsonnet-style languages, as well as file overlays, like those in Kustomize. We've been pretty adamant in the CUE design of not copying that practice.
yea, cue not bieng OO is good
tosca is OO and that causes a lot of complexity
telecom uses tosca to handle deployment and configuration of their infra, though some of that is changing as they move to helm in some limited circumstances
but they are even looking at lifting tosca into k8s to replace helm
i think that's a mistake though
Hello! Just joined! Thanks for the invitation!
they need a way to represent a service and pod in k8s
interestingly, they assume the k8s cluster already exists
Just wanted to say hello now so I donโt forget. Iโll be back on later on today, but my GitHub username is codeherk.
Sounds good. No problem at all!
@spice valve here's an example of tosca: https://github.com/oasis-open/tosca-community-contributions/blob/master/profiles/cloud.puccini/kubernetes/1.0/interfaces.yaml
shifting these over to dagger (and cue) i think would be beneficial ๐
Thanks for sharing! Iโll look into them
@copper light those are not part of dagger ๐
i think it would be interesting to see these implemented/replaced by dagger though
ah you did notice that!
sorry, it's late here:p
@patent stirrup lol no worries! I appreciate the early intro
Hello from Dublin Ireland! It's great to be here. My GitHub ID is tomwillfixit.
Welcome @civic lodge ! Sending you an invite now.
Welcome! ๐
Thanks Solomon. Hi Sam ๐
yay more friendly names! How are you Tom >?
@wraith niche @cloud canyon my impressions of new cli commands after 48h: weโre missing a way to get a quick overview of the current deployment state: whatโs the deployment plan, what are the inputs? Kind of like git status gives an overview of the current repo state
not sure what to call it
@winter linden ya, I wanted to at least know which deployment the current directory is mapped to, if any
dagger status doesn't sound too bad?
deployments plural - it can be more than one
and yes that would be good to know too
Would anyone be interested in a live demo / tutorial for those just getting started with Dagger, with open Q&A?
yes i am definitely interested in a live demo of dagger
Ok weโll set it up next week! Iโll share info here
ok sounds good
@distant pebble hello!
github: spullara
so it looks like something i may be able to use for packaging instead of serverless.com? i have a pretty complicated one i am using right now that has a lot of cloudformation.
yes you can compose the steps of your delivery: build, packaging, upload artifacts, cloudformation templating, push new cloudformation stack etc - in one dag
not cloudformation specific but there is a cloudformation package, most packages weโre still porting over from our earlier non-open source prototypes. There is a ECS+Cloudformation stack which got some good production mileage. Itโs a good fit
the thing that serverless does for me is mostly just packaging up lambdas and docker containers and uploading them. most everything else I could do with cloudformation. a few plugins are also useful.
If youโre a CF expert and enjoy usinf it directly of course you can use dagger and cf side by side. Or as a convenience you can call cf from your dagger deployment. Whatever works - itโs glue ๐
sadly, i am now a CF expert by necessity
ha ha I know what you mean.
The guiding principle is that dagger adapts to your stack and not the other way around
so serverless.com generates your cloudformation config?
I see
their value is in packaging, plugins for common tasks, some wiring, but at the end of the day, CF is huge.
If the yaml is getting painful you could import it as cue, and output yaml at will
and there are tons of things that they don't handle directly
but they also do stupid things
like reupload unchanged lambdas
every deploy
which wouldn't be so bad if AWS SDK v3 for node wasn't 100M of dependencies lol
yeah that adds up
We built some config that we use internally to deploy lambdas from compiled code and deploy them using cloud formation. All of this with dagger. Could extract some of this to create some examples config. I remember some of the setup you showed me a while ago @simple jackal
Also gained knowledge on cfn, not by choice ๐
But itโs handy to manage the infra state efficiently
Hey Victor, keeping well. We are still in Level 5 lockdown but we have our health and enjoying life ๐
When dagger is public, would be good to put a simple demo up on https://github.com/cncf/podtato-head
Or do something similar
good idea
we can both contribute to this, and create something similar as an โencyclopediaโ of everything you can deploy with Dagger
the little dagger book of deployments
assassins screed
@winter linden i still use https://github.com/docker/gordon ๐
i need to fix it to default to main and fallback to master
Hello everyone! Lots of exciting new things happening here! Could I get an invite to the GitHub project? Username is โborjaburgosโ, thanks!
Welcome back @twilit flax ๐ adding you now
Merged experimental config to dev dagger with dagger. To use:
$ git clone https://github.com/dagger/dagger
$ cd dagger
$ dagger new
$ dagger input dir source .
$ dagger up
Overall flow still feels awkward. After a few days of use we will probably adjust UX & DX. Feedback & suggestions welcome.
@winter linden I've been thinking this for a while and would like to know your idea: how should we describe this approach? "Developer Experience Enabled by Modularized IaC Components"?
DevOps Superglue really resonates with me as a DevOps practitioner. I think of it as Terraform meets Pulumi from local dev to production. I also come at it from CUE for DevOps, as a way to get rid of YAML and also have imports / modules.
hmm... OX? (Operations eXperience)
Weโre still searching for the โbestโ explanation. We have tried different versions. Currently in the README we describe Dagger as a โprogrammable deployment systemโ. We also tried โapplication delivery as codeโ, which is a superset of infrastructure as code. Delivery includes the entire workflow from source to production: build, test, artifact management, infrastructure provisioning, configuration management, staging etc.
Yo yo yo! Would love to try out the project, gh handle: n4ss
Welcome ๐ Invite sent
๐ howzit? would love to have an access to the repo (https://github.com/dagger) to have a look ๐ too many exciting things happening recently! ๐ฆ https://github.com/AdrienFromToulouse
Never been to Toulouse ๐
huge mistake
Iโd like to fix that after COVID-19
I heard the architecture is amazing
Told my wife we need to go visit and she said itโs already on the list ๐
you married a very smart woman then ๐
Hi Adrien! I just sent you an invite.
sweet! ๐
https://gregoryszorc.com/blog/2021/04/07/modern-ci-is-too-complex-and-misdirected/ (thanks @stoic knot )
Saw this hilarious DevOps video today https://www.linkedin.com/posts/mushegh-davtyan_devops-automation-cicd-ugcPost-6784040740038176768-jvbI
hi everyone ๐ and here we go again for some history making ๐ฅ
um... loving the react example (netlify...). very straight forward.
great job!!! 
aws -provision. -- โ
aws-helm (deployed falco...). โ
love it
dagger input SO FUCKING INTUITIVE....
pardon my french ๐
Nice you cleared those examples IRL?
Attempted them. React had an error but elements deployed.
Aws still testing into today.
There is a bug at the moment where the subdir operation doesnโt work properly. As a result instead of deploying the result of the react (yarn) build, netlify probably deployed (or tried to deploy) the root filesystem of the container that did the react build
i will have time later today to play around
๐ thanks for the early access peek! I'm intrigued and need to make time to dig more. My mind immediately jumped to comparisons to Terraform, so I'm curious to discover how it differs/improves on that ๐
Welcome! Dagger does not replace terraform, but all the custom scripts you usually write to glue all your tools together for your app delivery automation. Itโs quite trivial to use some terraform or cloudformation to provision a piece of infra in the middle of your config.
looks like golang is breaking single file go apps which have imported dependencies if there is no go.mod present
go 1.16 breaks it by default, but you can use GO111MODULE=off for GOPATH to work again
go 1.17 will break it entirely
GOPATH is going away
anyway... wrote up a quick workaround to this... https://github.com/fkautz/rungo ๐
Thread to ponder what Dagger in CI is like.
Some questions:
- How do I use inputs, secrets, and environments? Currently, dagger knows about things because of a directory in the users
$HOME/.dagger. Do I need to rundagger neweach time? How does this work across my CI stages? How does this work when I have a single VM building multiple commits (concurrent PRs) (i.e. in the Jenkins world)? - Currently, I get a nicely drawn DAG (i.e. GitHub Actions, Jenkins Blue Ocean, CircleCI)....
Grafana just merged a PR with Cue support ๐ https://github.com/grafana/grafana/pull/32527
Hi there, thanks for the early access ๐ Glueing our ecosystem tooling is something I was looking for a long time. Can't wait to test dagger ๐
๐ same as Elie thanks a lot for the invite ๐
Welcome ๐
hey guys - could you shoot me an invite please? ๐
thanks @winter linden ๐
nice find on grafana!
Credit to @turbid tulip ๐
I so love this. It's gonna be awesome.
The current dagger input looks like this:
โ dagger input
Manage a deployment's inputs
Usage:
dagger input [command]
Available Commands:
container Add a container image as input artifact
dir Add a local directory as input artifact
git Add a git repository as input artifact
json Add a JSON input
secret Add an encrypted input secret
text Add a text input
yaml Add a YAML input
Should dagger secret be its ...
This is a thread to discuss Daggerโs terminology (the choice of specific words to designate specific things), our experience with it, suggestions for changing it, etc.
Daggerโs terminology is still changing rapidly. Product terminology is part of product design, and the only way to find the right design is to try lots of bad ones. So, weโve been trying lots of words to describe the important parts of Dagger. As we use the product, we get a better idea of which words work and which donโt.
...
I have observed 2 distinct patterns for how to use Dagger in combination with an application repository:
-
Embedded. Sometimes one wants to embed a Dagger workflow directly in an application repository, in the same way one might embed a Dockerfile or docker-compose.yaml.
-
Standalone. Sometimes one wants to develop a Dagger workflow separately, from any application repository.
How Dagger supports these two patterns, and how to make sure each user follows the correct pattern for...
For #2, what if the encrypted secret is an X509 cert or something else not json compatible? We could base64 encode it before passing it in, though that seems like an extra step that may not be necessary when cue supports bytes naturally.
SGTM
Do you want to go ahead with the renaming or still need some time to see how it feels?
Requesting feedback on terminology: should call the cue configuration executed by Dagger: a) a โplanโ or b) a โworkflowโ ? https://github.com/dagger/dagger/discussions/360#discussioncomment-650989
Whatโs a good example of a tool which manages its own state/data file in the current directory? Like for example node_modules. What are other examples, and do they work well?
Context: if dagger could create a modify a file in the current directory, it would make it easier to map dagger global state to each directory
git? ๐
Ha ha thatโs true ๐
We could piggy-back on .git when itโs there, to avoid requiring that everyone manually add a file in their .gitignore
.git/dagger/
or we could piggy-back on .git/config with custom keys
but then you need a fallback when youโre not in a git repo
This is an unfinished proposal to solve the issue of organizing dagger environments in general, and organizing them in relation to the current directory in particular.
Problem
On the one hand, each dagger installation has state which must be centralized in one location on the system. Currently that location is $HOME/.dagger. On the other hand, each dagger installation is typically used for different projects. Therefore the user will use the same dagger installation in different wa...
It would be neat if dagger could optionally launch a companion webapp to monitor and configure an environment. This companion webapp would not implement a specific feature, but rather provide a platform for delivering various features that are more appropriate for a web environment. This would allow the CLI to focus on what it does best, and relax the requirement to do absolutely everything in a POSIX terminal, which is very challenging.
In short: if dagger always has access to 2 modes...
You could even store state as references in git ๐
They donโt need to tie to the standard refs/heads or tags
refs/dagger/state-obj
Youโre right thatโs another option
Maybe itโs ok to make git a special case. Realistically 99% of dagger commands will be called either 1) inside a git directory or 2) in a completely unrelated directory that should not be messed with
So we could look for a git repo in local dir; and if we cannot find one, then use a โdefault contextโ which could be an empty git repo in ~/.dagger/default or something
git
most language environments (node, go, python, ...) โ> but thatโs maybe different since the whole purpose of the directory is the language ...
Most IDEs (e.g. .vscode) youโll find a list of ide files in most boilerplate .gitignore (including githubโs defaults for new repos)
Slightly different since itโs human written and not tool written: node projects have a ton of companion files in the project directory (.prettierrc, tsconfig, cypress.json, jsconifg.json are a few that we have on our own dashboard) that are written by humans, read by tools
(by language environments I mean go.mod, Gemfile, etc)
Iโve mentioned it before, I think environment is a perfect fit for the first concept ๐
Iโm not a fan of workflow, which is also pretty overloaded and sounds like something sequential (which dagger isnโt). Iโve always like Ansibleโs terminology of playbook, and more generally something like plan or recipe sounds good to me too.
PS: I feel bad for bike-shedding on naming without taking the time to participate on deeper topics.
VueCLI is an excellent example of this
Donโt feel bad. I donโt consider this bikeshedding, more like structuralwalling :) Youโre a busy guy and any contribution is appreciated.
Personally I am still hesitating between workflow and plan. @samalba and @icecrime : do you think there might be confusion with the concept of terraform plan and pulumi plan?
I will check it out, thanks!
request for comments on contextual UI that would greatly improve how dagger integrates in a local git workflow: https://github.com/dagger/dagger/discussions/365
Update: I opened a PR to replace deployment + plan with environment + plan. I think โplanโ could work, I agree it is less overloaded than โworkflowโ.
How does everyone feel about possible conflict between the concept of a Dagger plan, with the actions terraform plan, pulumi plan and (perhaps one day?) and equivalent dagger plan ?
The user experience would be greatly improved, especially for beginners, if Dagger allowed selecting a plan from a catalog of ready-to-use plans.
For intermediary and expert users, Dagger should also searching for reusable components, to make it easier to develop your own plan.
A catalog of plans would be immediately useful for Daggerโs tutorials and examples. Currently they rely on dagger plan dir which assumes users have a local copy of the Dagger repository - this is an acceptable ...
๐ hey everyone, I'm Jimmy, @jzelinskie on GitHub and @jimmyzelinskie on Twitter
Hi Jimmy, welcome! Iโll add you to the repo right away
Hi! I'm pnathan - @shrewd cargo on github and @p_nathan on Twitter.
Welcome Paul! I just added you
Done!
Most of the activity is still in #dev and #github-feed - thatโs where we actually develop dagger itself. As more testers start coming in, #general and #help-old-do-not-post will get more lively. Make yourselves at home ๐
Very interesting system.
I think this is the largest cue codebase I've ever seen, haha
we use it for our k8s manifests and github actions right now
Ah youโve played with Cue before? Cool ๐ What was your experience so far?
An over-simplified description of how Dagger works under the hood might be: a DAG compute engine powered by Buildkit and programmable in Cue
Question. When will this be public?
We are hoping to launch by June at the latest. It wonโt be production-ready... But weโll continue to improve it in the open
I'm going to be moving to a new role and this has resonances with likely early tasks.
Weโre here to help ๐
Do you anticipate this to exist with a distributed cluster scheduling backend, i.e., "airflow" or similar systems?
We have private support channels with a small number of testers who are also using our commercial cloud service
We've found cue great at reducing boilerplate. Sometimes incompatible upgrades have been annoying
Under the hood we use buildkit as a compute engine. So any buildkit infrastructure is also Dagger infrastructure. That means it works out of the box on top of Docker. But also you can get fancy with a clustered buildkit + containerd setup, like the folks at Netflix are doing for example. Dagger will use that seamlessly
Hi all! happy to be here. My github is "FaisalAbid"
Welcome Faisal! I just added you to the repo. Let us know if you have any questions, weโre here to help.
I honestly think that DAG from something like BuildKit is probably what's lacking in our current usage of cue for gcloud infra/k8s manifests
We're actually experimenting with using Pulumi right now for GCP infra since DeploymentManager has been so bad
Weโre being very careful to remain 100% compatible with upstream cue, so you can reuse all your existing cue configurations in the context of dagger
DeploymentManager is the GCP "CloudFormation"
Our docs are pretty sparse at the moment, so some of the important concepts which make dagger โclickโ may not be obvious from just reading the docs. There is a very basic โprogrammer guideโ linked from the README
but reading the examples is probably a must
The general idea for Dagger is to be the โglueโ that you can integrate all the parts of existing stack in a unified DAG. Then from there you can refactor, simplify, etc.
For example you can wrap Pulumi in Dagger
as well as, say, cloudformation
amazing, already have a lot of usecases in mind as I need to do this almost everyday. Will share feedback during this week
So if youโre stuck with a stack which uses both, you have a common framework for configuring them together
Thatโs great! Looking forward to hear more. Just remember itโs still early, so it might not work perfectly on the first try, or it might behave in a confusing way. The more you tell us about that, the better we can fix it ๐
a cloud service will host the execution and secrets?
Hosting will always be on your infrastructure. The cloud service will be optional, and provide unified management, audit log, access control etc.
If it turns out to be helpful, the cloud service may also help automate infrastructure setup, for example by giving you a ready-to-use Cloudformation template to set up everything you need on AWS, exactly the way you need it. But that is also optional, you can also just install it yourself
The cool thing is that Dagger only needs buildkit under the hood. So it can be embedded anywhere that has a docker daemon. In other words you can use your existing CI as infrastructure ๐
the service you're describing sounds similar to how Anisible Tower is positioned in the Ansible market
food for thought
Yes, I think thatโs an accurate comparison. Although Ansible has an on-prem version right?
yeah, that's true tower is not hosted
but the centralized coordination & enterprisey features fits the bill
Ansible is definitely one of the products weโre drawing inspiration from. Also: terraform, bazel, github actions... each in their own ways
hi Jimmy ๐
yeah, i honestly haven't used ansible -- i just worked at Red Hat so I know how the product is positioned
so you'll have all the latest cue schemas for cloudformation/k8s/etc packaged at dagger.io?
๐
Knew this was going to be in cue
ha ha ๐ Hi @stuck wyvern glad you could make it
yeah, i have to say, i did see you in the cue slack, solomon
btw the Cue developers are lurking here too, we work closely together
cool, i'll avoid talking trash then ๐
@stuck wyvern have you been using cue?
Not at all. I've looked at the docs a few times.
HOLY i never knew about transforms
Iโm hoping they add sugar on top so you can write foo: mytransform(bar)
๐ I'm Brian, @bscott on Github, @brainscott on Twitter
Welcome Brian! I just added you to the repo. Itโs still rough so please donโt hesitate to report issues and ask anything here! Any feedback is greatly appreciated.
Will do, Thanks again!
Hello ๐ Iโm hugo on GitHub. Pleased to be here with all you fine folk.
Hi, I am Rick. Very early adopter of Docker and having the privilege of meeting Solomon back then. Very curious to see what has been cooking. My name on GitHub is rickpeters
Hello from ๐ฌ๐ท - really excited to be here! My GH/Twitter username is akalipetis
Hello @summer oriole @warm flame @nova knoll, Welcome! I just added your github handles to the repository. Have fun! The docs are not complete yet, so feel free to ask questions here. Lots of useful answers on this channel in the last 12 hours ๐
Hello everyone from France ! Little curious about dagger after a call with @wraith niche. My GitHub username is wbeuil.
You should have access now ๐
Hey @wraith niche, mine is eliecharra ๐
done!
Thanks ๐
thank you for the PR @lyric notch , that was fast ๐
Haha just discussed about it with Sam in his yesterday demo ๐
thanks @lyric notch for the PR, I suggested a change on the PR (to make always optional)
Ok ! I will fix it, I was wondering myself if it make sense to be optional as always is enforced in other aws pkg
Welcome @torpid linden!
I think it makes sense for s3 simply because it could be used for another use case that requires caching. But for something like a kubctl client or something that deploys, it makes sense to force always: true to make sure the underlying infra is reconciled.
Welcome @little cairn!
@little cairn I tracked you down on github & sent you an invite ๐
Awesome. Thank you! On mobile right now but will accept as soon as I get back to my desk ๐
Ongoing debate on code vs. configuration in CICD pipelines: https://news.ycombinator.com/item?id=26986493
Hello! Thanks for the invitation @winter linden. I look forward to playing with the code & contributing. I'm unclejack on GitHub.
Welcome @mint axle!
Are there any known cache miss issues? My base alpine image is getting re-built more often than it should.
This is what gets rebuilt too often: https://github.com/dagger/dagger/blob/main/.dagger/env/dev/main.cue#L23
Welcome ๐ You should have received an invitation to the repo
Yes, Iโve accepted it a few seconds ago.
Fixed it ๐ The problem was that I mounted a persistent cache directory (go cache) even for commands which didnโt need it (apk add). https://github.com/dagger/dagger/pull/393/commits/f81c16cef642c8ee0f9d5d9315ac5c04fb210468
@winter linden there's a dagger.io/go package, why not put those changes there?
I tried but felt too constrained. Now that I used it a bit more, Iโm looking at how I can reconcile
eg. wanted to customize my container image, experiment with exporting log files to analyze test results etc
Oh I see what you mean: youโre suggesting upstreaming the go stuff into dagger.io/go. Yes completely agree. Iโm workgin on it now.
Latest stdlib goodness:
import (
โdagger.io/dockerโ
โdagger.io/ioโ
)
ctr: docker.#Container & {
command: โecho โhello world!โ > /etc/motdโ
}
// Read the raw contents of /etc/motd
motd: {
io.#File & {
from: ctr
path: โ/etc/motdโ
read: format: โstringโ
}.read.data
// List the contents of /etc
etc: {
io.#Dir & {
from: ctr
path: โ/etcโ
}
}.read.tree
io.#File has fixed a lot of headaches for me, when mixing composition of containers + wanting to export files to the config
I donโt have to mess with low-level exports anymore, itโs completely hidden from me in stdlib
And docker.#Container with defaults that just work is nice too ๐ I just set a command and Iโm good to go
Also fun: quick and easy chaining of containers without messing with low-level ops:
ctr1: docker.#Container & { command: โfooโ }
ctr2: docker.#Container & { image: ctr1, command: โbarโ }
@cloud canyon pushed here (non-breaking): https://github.com/dagger/dagger/pull/393/commits/07f5e19ac01e5348845a1e170a0d6b984690719d
New and improved Dagger Programming Guide: https://github.com/dagger/dagger/blob/main/doc/programming.md
Writing good documentation is hard... ๐
Double welcome, impressive
@vernal igloo good to have you here, thanks for joining! should I send the invite to @josephjacks on github?
sent
Welcome @elfin stirrup!
Hello I am Walid Shaari, platform engineer, mostly operations, my github is walidshaari
I am excited, and exhausted, just over midnight in here, and do work tomorrow morning, so I will check laterz
Invite sent ๐ Happy hacking, let us know how it goes!
skimmed through the README.md quickly, did not see Dagger vs waypoint https://www.waypointproject.io/ ??
Good point Walid... Here's a general thought I had on these lines: Cue itself is introducing a really fundamentally different paradigm for manipulating infrastructure as data (vs. code). I think there is a lot of education needed to set the stage correctly for newcomers to this approach instead of comparing Dagger to the wide range of IaC tools.
What category would you put waypoint in?
I would not put it as IaC, it is more abstracting the deployment and infra it runs on
So paas?
most likely, Yes
Ok then weโre good, thereโs a section on Dagger vs paas ๐
I am reading it now
Welcome @crystal jasper!
(Sorry, @crystal jasper is me. Need to merge. Logged in from iPhone app)
that bot really likes you
Haha
Welcome @feral timber!
Wow lots of familiar faces here ๐
I'm rothgar an github. I see that's the first thing I should give
Welcome ๐ Invite sent
Welcome @swift barn!
Hello everyone and thank you @winter linden for the invite
Hello. I think my Github user still needs access (aloisreitbauer). Thx!
Welcome @swift barn! You should have access now ๐
Thanks
welcome @swift barn ๐
Welcome @wind yoke!
Hi Alex, thanks for joining us!
Hello, lots of familiar faces ๐ Thanks for the invitation @winter linden
My github user is AlexsJones ๐
Invite sent ๐ Let us know if you have questions or issues. Big UX changes (hopefully good) coming in a couple weeks
Welcome @ruby gale!
Cheers!
Hey everyone ๐ thanks for the invite! my github username is sundowndev
Welcome, invite sent ๐
Welcome @timid quiver!
Woah!! This looks exciting. Thanks @winter linden for the invitation. My GitHub ID is ajeetraina
Hey Ajeet, welcome! I just gave you access to the project on GH. Let us know what you think!
Thanks @wraith niche
Can't wait to test this. BTW, I am Ajeet Singh Raina, DevOps evangelist & author/blogger at collabnix.com, currently based out of Bengaluru, India.
@wraith niche whatโs the query command you used in yesterdayโs demo to only show outputs?
query -I -P ?
yep correct
For some reason itโs printing an empty dict in .dagger/env/dev...
Even though it definitely has computed values... Maybe something in my config causes some output values to be not recognized as outputs?
Hi @everyone . How do you all feel about doing a mini-meetup here on discord next week, to show you our progress and take questions? If anyone wants to show their dagger experiments that could be fun too. Let me know.
would love to participate
Likewise!
Ok weโll organize something and share more details tomorrow ๐ Let me know if some days & times work better than others
While I test drove the first hello-world example with dagger new/up/query I felt like I am in the world of Docker Compose. Quite impressive. Can't wait for Dagger Cloud ๐
Still lot to explore and experiment with.
@timid quiver you can actually include your existing compose files in your dagger environment ๐ Since cue has builtin yaml support
I am going through the cue tutorial. Surely, I wanna try out compose file.
The API for file access needs a little work, but you can do:
import (
โencoding/yamlโ
โdagger.io/osโ
โdagger.io/daggerโ
)
source: dagger.#Artifact
compose: os.#File { from: source, path: โdocker-compose.yamlโ }
services: yaml.Unmarshal(compose.read.data)
www: services.www
dagger input dir source ./path/to/project
or dagger input git source https://github.com/path/to/repo master
Will try it out.
Welcome @warped meteor!
Iโd join, too. Itโd be great to learn more about the use cases.
Looking at https://github.com/dagger/dagger/pull/332, I do see we have dagger input yaml <TARGET> [-f] <VALUE|PATH> [flags] that I can leverage directly for Docker compose . Right?
If you want the compose file to be a user input to your deployment environment , yes. You can also take an input directory, then get the file from there
Welcome @candid pasture!
Thanks for the invite.. my github user is stevenbower
Welcome @upper roost!
Hey .. lewismacdonald on github would be cool to see more !
Hi @candid pasture @upper roost , welcome! Your invites are on the way.
Welcome @wanton thorn!
Thanks for the invite, my github username is samj1912
Welcome @mossy turtle!
Welcome @crude pollen!
Hey! danyalprout is my GitHub username - excited to check out what youโve been working on ๐
Welcome @wanton thorn @mossy turtle and @crude pollen! I gave you access to the github repo, have fun ๐
Hello, I'm @mogul on GitHub.
Hello, invite sent!
Hi ๐ I'm @Oursin on GitHub
Hey, you're in!
Welcome @lethal breach!
Hey ๐ thx for the invitation @wispy tapir I'm PoC Innovation ex-president and a colleague of @hot shuttle. My github is loicttn :)
Hello and welcome! I just gave you access to the repo.
@here We now have binary releases of dagger: https://github.com/dagger/dagger/blob/main/doc/install.md
I strongly encourage everyone to switch to it (instead of compiling from source), the latest release aims to be more stable and will automatically notify you when a new version is available.
ok whos buying the domain and suffix get.dagger.sh ๐
works awesomely @wraith niche
tested on mac/linux/wsl
oh nice, thanks for testing this out!
i got you fam
@wraith niche Tested it on Catalina and the binaries worked like a charm
Thanks for the feedback, really appreciated!
Tested it on $59 Jetson Nano and it worked great on ARM platform too ๐
Linux pico1 4.9.140-tegra #1 SMP PREEMPT Fri Oct 16 12:32:46 PDT 2020 aarch64 aarch64 aarch64 GNU/Linux
pico@pico1:~$ dagger version
dagger version 0.1.0-alpha.3 linux/arm64
pico@pico1:~$
looks up jetson nano
Do we have an example how to pass configuration variables for Heroku? For example, I want to leverage marketplace.json https://github.com/redis-developer/basic-rate-limiting-demo-python/blob/master/marketplace.json ```import (
"dagger.io/git"
"dagger.io/github"
"dagger.io/heroku"
)
repo: git.#Repository & {
remote: "git clone https://github.com/redis-developer/basic-rate-limiting-demo-python"
ref: "master"
}
backend: environment: {
REDIS_URL: database.url
REDIS_HOST: database.hostname
REDIS_PASSWORD: database.password
REDIS_DB: database.dbName
REDIS_PORT: "(database.port)"
}
backend: heroku.#App & {
// Heroku configuration values
}```
cue import marketplace.json looks really cool
Do you mean passing environment variables to the heroku app?
Welcome @fallow solstice!
Hello ๐ Thanks for the invite ๐ My github username is alexisrobert
Hi Alexis, invite is on the way ๐
We released 0.1.0-alpha.4 today. Mostly bug fixes, and a new package dagger.io/os for simple system operations.
Next big change will be a revamped CLI flow that wors better with your existing git flow. Itโs pretty slick ๐
Also in the pipe:
- Native encrypted secrets
- Multi-tenancy for sharing settings across environments (infra team configures the infra; app team deploys without needing any infra credentials)
- Documentation site
- Gitops mode for experts. Review plan & input changes in a pull request, merge, boom theyโre applied across all envs
Thank you all for your feedback, itโs very appreciated!
white house just published this: https://www.whitehouse.gov/briefing-room/presidential-actions/2021/05/12/executive-order-on-improving-the-nations-cybersecurity/
so the question is... how can dagger help? ๐
Interesting! Not surprising as the US government is under heavy attack online atm
I guess Dagger would be relevant as a framework to upgrade existing CICD pipelines with devsecops capabilities, without requiring rip and replace. In a government setting that is a game changer because there are so many overlapping layers of legacy and IT jurisdictions. Rip & replace is doomed to fail.
Dagger is a good fit for incremental upgrades since you can glue it to any system
yea, predictable change
super important
we need automation to secure things
there's also a knowledge issue
as we innovate, we lose institutional knowledge because older knowledge becomes irrelevant
this is a direct threat to building and securing systems, we must constantly upskill and reduce gatekeeping to the industry
Upgraded to Alpha 5 today. Deployed a static page to AWS S3 bucket using dagger and it worked like a charm.
I badly need dagger delete now
The dagger help version has nothing to do with Global flags. ```dagger help version
Print dagger version
Usage:
dagger version [flags]
Flags:
--check check if dagger is up to date
-h, --help help for version
Global Flags:
-e, --environment string Select an environment
--log-format string Log format (json, pretty). Defaults to json if the terminal is not a tty
-l, --log-level string Log level (default "info")
ajeetraina@Ajeets-MacBook-Pro simple-s3 %
Glad to see that https://github.com/dagger/dagger/pull/462 gonna taken care of this. Thanks @wispy tapir
The pleasure is mine ๐
Welcome @frank fossil!
Ooh la la happy to be here! Hi everyone I'm Angie from San Francisco. Not a coder (yet, I'm just learning!) but I do write words! And I'm a super fast learner so soon I'll be up to speed with you veterans ๐