snow dock
I want to store state of my pipeline in a sqlite3 database, but I'm getting this error:
Error opening database at: Binary was compiled with 'CGO_ENABLED=0', go-sqlite3 requires cgo to work
I tried exporting: CGO_ENABLED=1 but I'm still getting the same error. My guess is that the environment variable is not being exported to the build environment. Is there a way to fix this?
thanks!
strange lynx
I want to store state of my pipeline in a sqlite3 database, but I'm getting this...
I think this will require a custom Go SDK given that the base go SDK runtime is based on an alpine and does not have the knobs to change that. cc @quasi ivy
snow dock
I don't have any problem customizing the SDK runtime, I've checked on the github repo but I didn't find any documentation on how to do this. Could you point me to a resource with this info?
Also, I don't really need sqlite, I just need to store my state in some kind of database
thanks!
strange lynx
@snow dock does KV work in your case? I was thinking Badger might be a good fit for this (https://github.com/hypermodeinc/badger)
additionally, do you need persistence across function invocations? or just within the same dagger call session?
snow dock
Yes, I can use a KV database!
And also, yes, my pipeline can benefit with persistence across functions; I'm achieving this by using attributes in a struct
I also want to save the database file in my host to have persistence between dagger calls
strange lynx
Yes, I can use a KV database!
And also, yes, my pipeline can benefit with persi...
Gotcha. By function invocations I was thinking more about multiple dagger calls
strange lynx
I also want to save the database file in my host to have persistence between dag...
This basically
Perfect
Let me think about it for a bit
snow dock
thanks!
snow dock
I started using badger but I would like to create my database file on the host but I can't find a "oficial" way to do this. Is there a workflow to do it or should I just hack inside the engine container?
strange lynx
I started using badger but I would like to create my database file on the host b...
sorry, had to take care of parenting duties 🙂
yeah.. Dagger doesn't allow mounting files from the host to the function calls as these functions could potentially run in remote engines. (https://github.com/dagger/dagger/issues/9326)
so the current possible ways to make this work is either using passing the sqlite files to your module on each call and exporting them back afterwards which has some caveats around concurrency and efficiency or simply use a traditional network database and pass the *dagger.Service or *dagger.Socket in
It'd be nice to know better about the use-case. Do have demanding needs for this DB to be safely persistent? Or is that something you could start as a dagger service and put in a cache volume which could potentially get garbage collected or wiped if the engine container gets recycled
snow dock
sorry, had to take care of parenting duties 🙂
yeah.. Dagger doesn't allow mou...
No worries! I have a couple of gremlins that sometimes take all of my time and energy 😉
I was thinking on sqlite, badger or any type of database stored in a file to minimize the moving pieces but this project might grow to use a more robust DBMS like postrgres which I guess might be easier to access over the network so I don't discard this option.
The goal of this project is to create a pipeline to build a Linux distribution from git repos that contain spec files (right now CentOS dist-git) and generating: RPM packages -> RPM repos -> ISO, qcow, container images, etc...
I want to be able to maintain persistent states like package versions so the same version of an RPM package doesn't get build over and over, sources don't get re-downloaded and assets don't get copied if the have alredy been generated.
The main goal is to use a modern tool like dagger to create Linux distributions from scratch in a modular, observable and reproducible way.
I for example I want to integrate AI agent/tools to check CVE's on the packages if there's one, go and check if there's a new version, if there is a new version, create a PR with the version upgrade to the git repository and notify the owner of the package that there is a security PR for the package.
That's why I picked dagger for this project, the sky is the limit!! (and no yaml ohh yeahh!! no yaml!!!)
strange lynx
No worries! I have a couple of gremlins that sometimes take all of my time and e...
great! thx for the summary. Yes, Dagger definitely fits very nicely in this use-case. Going back to the Database topic, I'm thinking that eventually you might probably want some other members in your team or CI job to be able to access the same database so they can all verify what artifacts to build.
In this context, for testing purposes you could simply use a Dagger service to spin up a postgres, mysql or any remote DB you prefer and once you're ready to move this to a more consolidated environment, you can simply point it to your "prod" DB and that should be pretty much it.
WDYT?
snow dock
Since the goal of the project is to define software supply chains that converge in Linux distributions makes sense to define them into a relational model and using a Postgres server that can be accessed from diferent endpoints can improve traceability and observability in a more broader way.
I'm going for that, thanks for the insights!! now I'm going to be bothering you guys on proper workflows to access remote services
thanks!
strange lynx
! now I'm going to be bothering you guys on proper workflows to access remote services
the services redis example is quite along the lines of what you need to start https://docs.dagger.io/api/services/#persist-service-state
you can spin up a postgres or any DB you prefer in a service and connect to it either directly from your functions or another container in the pipeline
here's how you can use endpoints to connect directly from your function using service endpoints https://docs.dagger.io/api/services/#use-service-endpoints
another thing I'd suggest is creating a function that returns your DB service like func (m *MyModule) DBService() *dagger.Service so then you can do something like dagger call db-service up from your local machine so it maps the DB ports locally and you can connect from your machine to check whatever is in the DB.
snow dock
So what is recommended?
create de Postgres service inside the pipeline or in the host?
strange lynx
So what is recommended?
create de Postgres service inside the pipeline or in the...
Inside the pipeline is better. That way you have a fully reproducible setup without the need for local depedencies
Once you're ready to move to production you just switch the service DB with an external Postgres hosted endpoint and that's it
snow dock
I'm not sure if Ishould create a new thread so I'll ask here: I'm starting my postgres service inside the dagger pipeline like this:
dagger -vvv -c 'database-service env:POSTGRES_PASSWORD | up'
With this I can access the database from my host with no problem. The only inconvenience is that this command does not go to the background. I can just put '&' to send it to the background but I want to know if dagger offers an up variant that sends the service to the background.
Thanks!
strange lynx
There's no command to send it to the background currently. Given that Dagger is mostly designed to run software workflows, the DX is not currently optimized for background processes a-la docker.
What I generally do is to run up in a separate tmux pane so I can watch the logs of the service I'm running there
cc @snow dock
snow dock
That's what I'm currently doing.
thanks!
snow dock
FYI turns out that is better for my use case to run my module outside the dagger sandbox. With the Rust sdk I can connect to the database and interact with the engine
thanks for the help!