Approaches to testing concurrent code execution

[grand jungle]

have a background function that runs on a separate thread. At startup, this function atomically sets a flag that disables flushes during a rewrite. Instead, all data is written to a buffer and then merged at the end of the rewrite.

I’d like to write tests for this multithreaded scenario. Specifically, I need to detect the exact moment the code sets the flag, verify that it’s active, and then continue execution. During that process, I plan to append data to the AOF buffer and expect the rewrite algorithm to correctly merge it at the end.

Does this testing approach make sense, or would you recommend a different strategy?
For synchronization, I initially tried using std.Thread.ResetEvent, but encountered a Bus error at address 0x100e24b0c.

https://zigbin.io/a32acd

Approaches to testing concurrent code execution

[silk spade]

You can't detect exact moment an atomic is set, the timing of writes and reads happens inside the CPU.

To test parallel code you'd also need to spawn multiple threads in test itself

[tranquil crag]

you could inject test implementations of your threading primitives that perform additional checks maybe?

testing multithreaded code is very hard though, because it's inherently extremely nondeterministic

i've been trying to learn formal verification techniques for exactly this reason

[grand jungle]

I meant like to stop execution for a bit to simulate the process takes long time to in meanwhile be able to do few appends and then remove this stop execution lock and check at the end if it was merged properly

But I need to have control over what's happening in the tests to know where exactly the rewrite ended and to wait for it.

So I figured I'd do it using std.Thread.ResetEvent, until a reset occurs, the code will stop right after setting the atomic flag, then I'll do a few appends, checking if there's no flush to disk from the buffer during rewrite, reset again to let the code continue executing and check the effects of rewrite and if atomic flag was properly released and after that flushing works.

[grand jungle]

I feel like i need just synchronization over whats happening inside the function and outside (in test) to properly test it.

[tranquil crag]

ideally, concurrent code should be tested by using a fuzzing scheduler that preempts the code at important points (eg. between the read and write of a shared variable) in a deterministic way

but idk if that exists

it's something that's on my infinite list of project ideas ^-^

[grand jungle]

yeah i basically want to do something like that but more simpler, but why i got Bus error at address 0x100e24b0c?

[tranquil crag]

impossible to know without more context

run it in a debugger and get the stack trace

[grand jungle]

I didn't even set the flag from the test yet, just used set and wait in rewrite function

[tranquil crag]

😭

that's where the bus error comes from

do not use constCast

ever

[grand jungle]

why not?

[tranquil crag]

because it does not do what you think it does

[grand jungle]

umm, what it does then?

[tranquil crag]

there are exactly two cases where it is correct to use constCast:

• when you have a const pointer to memory you know is non-const. this should be fixed by taking a non-const pointer instead
• when you're calling a function that takes a non-const pointer, but you know it will not mutate that memory. this should be fixed by changing the function to take a const pointer (if it never mutates it and is just wrongly typed) or by creating an alternative version of the function that guarantees non-mutation (if it will only not mutate in some cases)

[grand jungle]

oh then rest of my @constCast usages are good i guess

[tranquil crag]

no, constCast usages are never good

there is always a better way

imo constCast should be removed from the language again. you used to have to use @ptrFromInt(@intFromPtr(x)), which is much more obviously Doing Something Evil

[grand jungle]

Btw i fixed the Bus error by getting rid off the constCast

Thank you for pointin me up

[tranquil crag]

:)

[grand jungle]

yeah i need to learn how to avoid using that thing, i didn't know its evil

[tranquil crag]

it's the second most evil cast builtin, after @ptrFromInt ^-^

[grand jungle]

but how then i'll could do that another way removing const cast in that case?

return .{ .sstr = @constCast(string[0 .. string.len - 1]) };

[tranquil crag]

though it's more commonly used for evil, by virtue of being less obvious

[tranquil crag]

why is string const?

[delicate brook]

@constCast should be removed from the language, or at least ZLS to give warning.

[tranquil crag]

at bare minimum, the language reference should be much more upfront about how dangerous it is

[grand jungle]

w8 i'll give the entire context

[delicate brook]

Const memory access is just dangerous and many newbies just use @constCast

[grand jungle]

https://zigbin.io/a6982d line 75

[grand jungle]

I just use it to get rid of anoyin errors and make my code compile 🙏

[tranquil crag]

incorrect use of constCast is one of the most common newbie mistakes i see, which is absolutely insane because it's not a feature even advanced users should be touching most of the time

@import("root") is kinda similar in that regard

[delicate brook]

Reasonable

[ocean surge]

The fact that code compiles dosen't mean it's correct

[grand jungle]

And how about that?: .{ .sstr = @constCast("OK") } ?

[ocean surge]

Illegal

[grand jungle]

i return a string literal

[tranquil crag]

readlineAlloc should return []u8

not []const u8

[delicate brook]

I just trying to figure out the best way to use TCP sockets and I/O in stable, so I can do too

[tranquil crag]

either allocate the string or make sstr: []const u8

[ocean surge]

Literals are const

Well

[tranquil crag]

because either you need to mutate it, in which case you must allocate it, or you don't, in which case just don't take a mutable slice

[ocean surge]

Why ypu need to const cast a literal, ever

[tranquil crag]

there are cases where it's legal. none of them are a good idea though

[grand jungle]

i feel like allocating string literal doesn't make sense

[grand jungle]

oh yeah i just could make it oneliner with return and it will be good

[tranquil crag]

well, if you need to mutate it, you need a mutable copy of it

[grand jungle]

Its ain't mutated never, just returned as a response to the client

[tranquil crag]

then make sstr const

[grand jungle]

well in that context is not mutated, but i can't make this sstr const cause it could be mutated but not in that case when i return literals

[delicate brook]

$ zig run src/main.zig
/home/ci/actions-runner/work/main.zig:9:9: error: you used @constCast and your code is ass, compilation terminated
    _ = @constCast(color)
        ^~~~~~~~~~~~~~~~~

[tranquil crag]

can you show me the definition of ZType?

[grand jungle]

https://github.com/sectasy0/zcached/blob/master/src/protocol/types.zig

[tranquil crag]

what is the difference between str and sstr?

[grand jungle]

just at protocol level, simple string don't need a size

[tranquil crag]

also, can u show me some code that mutates an sstr value?

[tranquil crag]

oh, is it 0-terminated or smth?

if so u should encode that in the type

[grand jungle]

Its the same as redis protocol RESP

or maybe wait, maybe simple string could be a const

[tranquil crag]

ah, crlf terminated

[tranquil crag]

yeah, i'm not seeing it be mutated anywhere

[grand jungle]

simple string is more like, response format than actuall type

but in my database it is also a datatype

but yeah i guess it could be const

[tranquil crag]

yea just make it const

do u even mutate str values anywhere?

that should prolly be const too

[grand jungle]

I can't remember

I'll check for it later

[tranquil crag]

just make it const and see if it errors

[grand jungle]

It works, so it could be const

[tranquil crag]

yay! ^-^

another mutability footgun removed

[grand jungle]

Now i need to get rid off the rest of those guys

[vital sphinx]

On the topic of @constCast there is an example in std (more than one) that use it and I have no idea why. Ostensibly it's wrong, what is the explanation for this one? Is it because of some knowledge about how things work in Windows specifically?

e,g

❯ rg '.Buffer = @const' -C3 --vimgrep
std/posix.zig-5090-    var nt_name = windows.UNICODE_STRING{
std/posix.zig-5091-        .Length = path_len_bytes,
std/posix.zig-5092-        .MaximumLength = path_len_bytes,
std/posix.zig:5093:9:        .Buffer = @constCast(sub_path_w),
std/posix.zig-5094-    };
std/posix.zig-5095-    var attr = windows.OBJECT_ATTRIBUTES{
std/posix.zig-5096-        .Length = @sizeOf(windows.OBJECT_ATTRIBUTES),
--
std/process/Child.zig-1101-        var app_name_unicode_string = windows.UNICODE_STRING{
std/process/Child.zig-1102-            .Length = app_name_len_bytes,
std/process/Child.zig-1103-            .MaximumLength = app_name_len_bytes,
std/process/Child.zig:1104:13:            .Buffer = @constCast(app_name_wildcard.ptr),
std/process/Child.zig-1105-        };
std/process/Child.zig-1106-        const rc = windows.ntdll.NtQueryDirectoryFile(
std/process/Child.zig-1107-            dir.fd,

...etc

Generally related to this windows.UNICODE_STRING struct

[tranquil crag]

yeah, i guess the windows API is just badly typed here or something

interacting with poorly designed C APIs is one of the acceptable usecases for constCast

(though generally you should try to fix the API upstream, if you can)

[grand jungle]

How should i fix something like that?

 const ctx_ptr: ?*anyopaque = if (self.ctx) |ctx| blk: {
                break :blk ctx.ctx;
            } else blk: {
                break :blk @constCast(@ptrCast(&self.handle));
            };

[tranquil crag]

needs more context. what is self and why is it const? how is ctx_ptr used and why is it non-const?

also just fyi you can do this, instead of wrapping the expressions in labelled blocks: ```rs
if (cond)
foo
else
bar;

[grand jungle]

https://zigbin.io/4148b4 - Its basically stream implemenetation with generics i need that cause i have 2 ways of compiling this with tls support or without and in tests i need both.

I don't even understand why self is const

[tranquil crag]

this should not be generic

you have two implementations, chosen by a compile option. just check that compile option whenever you need to pick between the two different behaviours

[grand jungle]

Previously i had like that, and i introduced e2e tests to my project and during those tests i really need those two.

[tranquil crag]

oh, so you want to test both of them without changing the compile options?

[grand jungle]

only during e2e tests step and it works that way now

[tranquil crag]

why not run the e2e tests twice, once with tls and once without?

[grand jungle]

Dunno, I wanted to have one step for all my e2e tests

[tranquil crag]

having multiple steps will mean you can run different configurations in parallel

[grand jungle]

So it's const cause its generic?

[tranquil crag]

it's const because it's taken as an arg, but the way you've written it is what's forcing you to make it var