Hello! I have a basic setup with k8s node, dagger engine daemonset, and a runner connecting over buildkitd.sock. I noticed that after my functions are completed / errored out, engine's RAM usage doesn't always decrease, and if it does then not fully. I'm not sure if its due to my constructor module calling another dagger modules or something and i'm willing to figure it out, but I need to know the expected behavior.
#Is dagger engine supposed to build up RAM usage?
ebon grove
sonic ridge
Hello! I have a basic setup with k8s node, dagger engine daemonset, and a runner...
👋 @ebon grove is this RAM usage constantly going up to the point that your pods OOM? The engine does indeed store some things in memory to speedup future executions so some RAM increase after each function invocation is indeed normal. Having said that, it should eventually stabilize and it should definitely not OOM your daemonset
ebon grove
I had another pods scheduled on the same node engine is running on, so that caused OOMs which got me looking on the engine consumption and worried. I removed the scheduling and will continue to observe. Thank you for clarification! It does seem like it's constantly increasing, but I might have skipped the part where it hits GC trigger or something due to another pods meddling and causing OOM. Again, will keep monitoring for now.