i'm real close to done with the initial | Dagger | Page 1

smoky olive Jan 13, 2025, 9:26 PM

#

I'd default to some sort of warmup run where we run the full suite, maybe tagged differently so we can still have data for it just in case, and then run it again tagged 'warmed up' or something, and mostly care about the already-warmed-up case since it'd probably have the least noise

#

downside is ~2x longer total runs obviously (though probably much less with caching)

raven iron Jan 13, 2025, 9:32 PM

#

also much less because a lot of the runtime here is building & starting the dev engine

#

just gotta decide if it makes more sense to do the warming inside or outside of go test

kind carbon Jan 13, 2025, 10:32 PM

#

Do those tests not get run by default anymore? You have to manually trigger them?

raven iron Jan 13, 2025, 10:33 PM

#

yeah, at least pre-merge they only run if you've labelled the PR "benchmark"

#

that's not hard to change, fwiw, if we find that we're missing perf regressions frequently or that we turn them red a lot

kind carbon Jan 13, 2025, 10:38 PM

#

Yeah iirc you said it makes a pretty big diff in terms of runtime, so I'm okay starting here even if not getting the coverage is a bit of a downside. Just double checking I was following

raven iron Jan 13, 2025, 10:46 PM

#

to temper expectations a bit, right now it helps a shit ton locally but I have more digging to do to make it speed up CI runs significantly

#i'm real close to done with the initial