worthy leaf
They are complaining in new PRs for code that is not related to them at all.
The tests should be runnable in the same environment I use for development, correct? I am happy to rerun tests for each commit in a PR (IMO all commits should build and test clean to maintain bisect-ability). Do we have docs on how to run them?
And as another side question, much of the code in a provider would require interacting with a source, do we have a preferred way of mocking this input to drive tests?
elfin warren
Yeah, I noticed it as well. Its due to the fact the models changed a tiny bit - It is in fact a clear signal that the tests do what they should do. I will fix it
worthy leaf
I am also curious about testing infrastructure. It would be nice to have a set of tests for the subsonic provider. I can hand roll tests that create the objects from py-opensonic for consumption, just was curious if there was an easier way.
elfin warren
well have a look at the jellyfin provider because @glad radish started using sysrupy there for the tests using snapshots
glad radish
I started out down the path of building fakes in my “upstream” library
So I capture real json from my jellyfin but have a fake implementation of the api pagination, searching, etc that returns those real json fragments
The syrupy stuff uses those same json snippets to make sure my jellyfin to MA model parsing works. If I want to add a new edge case to the tests I just capture JSON from my jellyfin instance and drop it into the fixtures folder and regenerate the syrupy stuff.
Then I have fixtures to spin up an MA instance but using the fakes
I got 20% of the way to what I wanted to do before real life took me for a ride…, hope to get back to it soon
Again, based on what HA does.
Syrupy is great for testing you don’t regress your util funcs for converting from the upstream version of an object to your MA version of it, takes quite a bit of manual effort out of it, and the diffs are reasonable for reviewing the impact of something does legitimately change
There are fixtures for the end goal of spinning up an MA instance with your provider (connected to a fake client library) so you can do an end to end test that everything syncs and is usable. I think adding test players was next on my list.
This is basically the testing style HA uses, we found unit testing to be really good at testing out of date mocks, and we can’t test against real stuff either. So good fakes and controlling the test HA instance through the public part of its API rather than calling internals.
Eg I don’t want to test asking my provider directly for stream details, I want to test making MA ask my provider for stream details.
Unfortunately faking the interaction with the source depends a lot on the client library. Ideally your client lib provides the testing fakes and has its own tests to verify the fakes match a real one. Eg with pytest you can parameterize your tests and run the same test against your fake and real source.
For jellyfin it’s a really thin wrapper around a JSON RESTish api that I’m only using a fraction of so the fakes are super easy
drifting quail
I think I'll also be poking around at your tests 😉
glad radish
Oh my
worthy leaf
This is awesome, thanks for writing all that up. The subsonic API is much the same so the fakes should be pretty easy and I can wrap them into the provider library which can verify.
worthy leaf
This is basically the testing style HA uses, we found unit testing to be really ...
I think that first failure mode is really important and jives with my experience on teams that prioritize unit testing.