Tests pass individually but fail when run with tox or alongside others - ChannelsLiveServerTestCase

[floral ginkgo]

Hi,
I know the issue title might sound a bit strange, but I’m not sure how to better describe what’s going on.

I'm currently working on test cases to improve django-eventstream.

Here are the related tests:
tests/test_stream.py

When I run them individually using:

pytest tests/test_stream.py

everything works fine. But when I run them with:

pytest

or through:

tox

they start failing. It seems like when the tests are executed along with others, or maybe under different process contexts, something changes — perhaps the test priority or isolation — and some of the parsing logic in the code fails to retrieve all the events that were sent.

Is there a way to make sure my tests remain stable and work even when executed with others or through tox?

Thanks in advance!

[steep glen]

I don't really understand what you're trying to test (I just am not familiar with the domain, so I apologize ahead of time), but one guess is that you can't benefit from django's testing infrastructure resetting databases via transactions. That can only be achieved using setUpTestData(...), which is a classmethod, and using the test command of manage.py, not pytest, AFAIK.

In other words, it could be that the data that stays around from one test may be affecting your others, which wouldn't happen if you tested one at a time. I've had this problem as well, and when I saw that you weren't using manage.py and setUpTestData(...), it seemed familiar.

Again, I don't know what channels is, but so my knowledge is limited.

[floral ginkgo]

I'm using pytest

[steep glen]

I understand that, but you can't use transactions without using manage.py

and setUpTestData(...)

[floral ginkgo]

There is no transaction?

I'm not using setUpTestData

[steep glen]

The django testing infrastructure uses transactions to isolate tests

Exactly

setUpTestData supports testing using transactions

as does setUp, as long as the test class is a subclass of django.test.TestCase

[floral ginkgo]

I don't understand your point.
In muy investigate it seem like with other tets or under tox, the test are not working on the async/threading side as they would when I execute just them

[steep glen]

All I am saying is that maybe your tests are failing because data from one test is leaking into your other tests, which wouldn't happen in your scenario of doing one test at a time (the interpreter quits, any in-memory databases are deleted as a result)

[floral ginkgo]

But there is no data used in my tests

[steep glen]

You don't write to the database at all?

Not even in ChannelsLiveServerCase?

[floral ginkgo]

In the ChannelsLiveServerCase not at all. It fail when execute with pytest and some storage test are runs but when trying to do tox and in tox define tests/test_stream.py and then tests/test_storage.py it fail

So it's look's like that if the tets is not the main process of the test that are going to be done, it does not work

[steep glen]

Then I don't know the answer. All I know is that testing involving some amount of Django's ORM requires using django.test.TestCase, unless you really hack at it, and inconsistency between runs seems to often be because the testrunner doesn't clear the database before the next test. I would check to make sure you know that isn't the case, but that's the most I can help, unfortunately.

Have you looked at https://channels.readthedocs.io/en/latest/topics/testing.html, specifically "Setting Up Async Tests"? It discusses things that you may not have done to make it work with pytest. Also, the rest of the page is worth looking at, if you haven't already.

They discuss decorators that are required, etc.

Again, I don't know the specifics of what you're doing, I just know what I've seen at a glance

I guess your tests themselves may not be async, so maybe that section doesn't apply.