#Best practice for creating faster testing option

15 messages · Page 1 of 1 (latest)

bronze plover
#

I have a library written in rust that was made for a research project. I wrote some tests to gather baseline statistics for message timings and sizes for each of the protocols in the library. Each of the tests runs under various conditions. One of the conditions is an assumption of 10000 users. This figure is necessary to ensure that our statistics are highly accurate with low standard deviations. However, due to this figure and the other conditions, the tests take a very long time to complete. This isn't a problem for our results but our code is supposed to be tested for peer review and I was hoping to create a "fast" test mode where a flag or something could allow us to run each of the tests for 10 users instead of 10000. This would give us less accurate results but would allow the people trying to verify our code/results to get a sense of the correct range without maxing out their cores for a day+. I assume there is a better way to do this besides copying each of the tests and adding a "fast" prefix to each of their function names but I can't seem to find a good solution. I would very much appreciate any tips or pointers to the appropriate documentation.

arctic oar
#

A compiler directive could be used for that to my knowledge, such as #[cfg(test)] only applying to cases where the test flag is included in the compiler arguments...

rapid sigil
#

It's probably best to make a function that takes the number of users as a parameter, then make test functions that call it and mark the big one with #[ignore]. You can also use features similar to the above message.

bronze plover
# rapid sigil It's probably best to make a function that takes the number of users as a parame...

I was thinking about that but all of the tests are big when the number of users are big. Ideally, I want to run each of the tests exactly as they are with default 10000 users but be able to toggle the number of users depending on some config the person testing can specify. I am imagining, for example, passing a flag at test time for "fast" that would set the user size variable to 10 for each of the tests--but I don't know if such a thing is possible. I haven't figured out how to do it with #[cfg(test)] as @arctic oar has suggested but maybe that's a way?

rapid sigil
#

Yeah just do something like

#[cfg(feature = "slow")]
const USERS: usize = 10000;
#[cfg(not(feature = "slow")]
const USERS: usize = 10;
bronze plover
#

yesss this looks like exactly what I want!

arctic oar
bronze plover
#

So how does one actually call the correct feature? I have added

#[cfg(feature = "fast")]
const USERS: usize = 10;
#[cfg(not(feature = "fast"))]
const USERS: usize = 10000;

to the top of the tests.rs file and call:
cargo test --release --features=fast -- --nocapture <testname> but get
error: Package <our_code> (/home/<our_code>)does not have the featurefast`
do I need to add the feature to the Cargo.toml?
I also tried an if else using cfg-if crate but got the same error

rapid sigil
#

Ah, yeah you have to add it to Cargo.toml:

[features]
fast = []
bronze plover
#

That worked! Thanks so much for your help ferrisLove

rapid sigil
#

No problem! Last thing: if you want people to run it with the small number by default, you need to add it to the default feature:

default = ["fast"]
```That's why I made the feature "slow" before, because it's off by default.
bronze plover
#

That's good to know. I'm a bit conflicted about the default. Technically, the slow version is the correct version to replicate our results--but assuming in this particular case our reviewers might want to do a sanity check rather than generate the full results, they might opt for the fast option

#

but I'll keep that in mind and know how to change it if need be 🙂

rapid sigil
#

Maybe the name "full" would be better than "slow", but yeah up to you guys