#test runner improvements

1 messages · Page 1 of 1 (latest)

solar fiber
#

🥇 run tests in parallel
🥈 log tests that take a long time (> 1s)

#

hard mode: compile each test block to an object file so that the build system can run them in parallel

craggy hare
#

^ THIS

#

Wouldn't this be improved by incremental compilation too?

solar fiber
#

it would make building tests faster, not running them

craggy hare
#

Wouldn't the incremental compilation generate artifacts the test runner could use?

#

I'm just speculating, I am not smart enough to fully understand.

solar fiber
#

incremental compilation helps the case where say you change a single line in a test block, then only that one test block gets recompiled instead of the whole test binary

craggy hare
#

Right, so it's not just on a module level. That makes sense.

agile osprey
#

Running tests in parallel might be tricky/need a process per simultaneously running test since those tests could be doing (naughty) things like accessing/mutating global shared state

quaint yew
#

if your tests rely on global state, you are probably doing something wrong

buoyant jewel
#

or at least what numbers/use cases he needs to make a decision.

#

processes would steer zig into more complexity (module tests), but are necessary for panic tests.

buoyant jewel
rare coral
buoyant jewel
# quaint yew if your tests rely on global state, you are probably doing something wrong

Most system test do this and it is a necessity for testing security related properties due to suid bit mandating absolute paths.
Personally I'd be in favor of annotating the test property to make usage most simple, which boils down to "affecting local, process or system state" per usage in a Kernel and on default running system tests sequentially with the rest parallel.

#

So something like test local "testname1", test process "testname2", test system "testname3".
The test runner implementation can the decide what to do with each test type.

#

Without this, one needs hacks with the return type as error for process local tests and system tests are always separate.

rare coral
#

zig the language model doesnt have a concept of "system" i dont think; only local and program/"process"

buoyant jewel
#

The other not very nice thing is that expected output communication to the server for spawned runner processes relies on the subprocess with the current design.
For example this is not possible:

    #[test]
    #[should_panic(expected = "Divide result is zero")]
    fn test_specific_panic() {
        divide_non_zero_result(1, 10);
    }

So one or the other way, one has to run into the test, detect "expected result" and that its a child process based test and then somehow write the memory for either 1. global usage or 2. threadlocal usage (passing return values over stack not available in test blocks afaik).

#

So from my point of view panic tests would belong into the build system, but this makes debugging very much not nice.

light pewter
solar fiber
#

warn and err show

agile osprey
#

or you can set std.testing.log_level to whatever level you want

#

(on a per test basis, it gets reset for each test)

buoyant jewel