I'm having an extremely weird issue where tox -e py reports two tests as failing if I add any new file to tests/data/cases/.
It's really:
$ rm -rf .tox
$ tox -e py
$ echo "pass\n\n# output\npass" > tests/data/cases/some_test_case_name.py
$ tox -e py
FAILED tests/test_black.py::TestFileCollection::test_get_sources_with_stdin_filename_and_exclude - AssertionError: assert [] == [PosixPath('_...xclude/a.py')]
FAILED tests/test_black.py::TestFileCollection::test_get_sources_with_stdin_filename_and_extend_exclude - AssertionError: assert [] == [PosixPath('_...xclude/a.py')]
It's those two tests, consistently. When tested with a debugger, they don't fail.
The test session info:
platform linux -- Python 3.12.1, pytest-8.1.1, pluggy-1.4.0
cachedir: .tox/py/.pytest_cache
rootdir: /home/sumezulike/PycharmProjects/black
configfile: pyproject.toml
plugins: cov-4.1.0, xdist-3.5.0
The issue doesn't appear when running tox -e ci-py, but I don't want to push a potentially broken state.
Any suggestion how to proceed here?
...
I've found some correlation to the pytest --numprocesses parameter...
Those two tests fail consistently with the numprocesses values auto (4 for me), 4, 6, and 8.
They don't fail with --numprocesses 1, 2, 3, 5, 7, 9, 10, 11, 12.
...
I've found something different: My last change had added tests, bringing the total pytest collected items to 368. If I remove any one of the test parameter sets, reducing the test item count to 367, everything runs smoothly. If I remove another, I can add my test case file and all is good. Seems like an issue with multiprocessed testing but 367 is such an arbitrary number.
I'll try to make it as simple as possible to reproduce and write an issue but if it's not only weirdness on my machine, that's probably going to become an issue very soon anyways.