unit-testing

[maiden pawn]

This meme is about coding without unit testing

[brazen onyx]

That is perhaps the reason why there is so much code done without unit test. Im not buying into everyone is unskilled though.

[maiden pawn]

Tests serve as additional layer of self documentation to the code, which are 100% of time always up to date
Because code does not lie and auto validated
U have not a bad chance easily continuing project even after year of abandining

[maiden pawn]

I do want to point out that prototype code mostly is the one that goes with very few tests because its discovery
it is common practice during development with tests, writing exploratory tests for discovery
they are just quickly refactored or removed (if they are for some reason not adding any value) after feature is complete

Im not buying into everyone is unskilled though.
😅 u are very optimistic then

[brazen onyx]

I think this is again relating to pressure from different things or laziness. I just see people not as machines.

Example is you get horrendous code that was not refactored and you are asked to add a feature. How much time do you spend adding unit tests to a pile of crap?

[maiden pawn]

i mentioned this usage case. it can take horrendous amount of time and of great difficulty fixing applications without unit tests.
Than more code was written, than more tangled it became, and adding/or fixing anything in it will involve propagating problems across whole application
At this point any change to this is difficult, be it adding unit tests, adding new feature, or fixing a bug
horrendous code is horrendous code. that's why having tests is nice. they ensure minimal plausable quality (even if the code underneath potentially horrible too, at least it is usually not more horrible than some certain level)

Working in commercial dev environment without unit tests is like working without git today.
Surely some do and use zip archives or copy folders for backing up code 😅

I would have refused to join such company though. i would have discovered it during interview hopefully.

https://www.youtube.com/watch?v=CkFuimg1umY

[brazen onyx]

I once got a project that involved a lot of store procedure work from a legacy system that i was going to replicate to a new database. I added unit test to the slew of store procedures and found numerous bugs in the legacy system before i wrote my 2 way replication. The pressure for me to get that right was what made me do that. Some people will wing it though. But it feels good when you do it. With something less important i probably wouldn't spend that much time. I listen to what the CTO thinks, and do that, whatever that is. 😄

[proper wind]

@brazen onyx @maiden pawn Thank you so much for this conversation. I learned some great stuff here.

[rigid ridge]

I'm working on this Django project that was started by someone else. They wrote many tests, all of which fail for reasons I cant make sense of because Im not very experienced with writing tests.

https://paste.pythondiscord.com/YOIQ

it has something to do with the setup

unless anyone has any ideas what im doing wrong, im just going to go back to not writing tests for anything 🥹

[rigid ridge]

I could easily delete all these tests and this would not be a problem anymore lol

[proud nebula]

You should have django-fastdev installed so you can see what the query params were. Or step through with a debugger. I would guess "random_id" is null or something stupid like that.

[rigid ridge]

I wouldnt even know how to run a debugger on djangos automated testing

[maiden pawn]

Try git bisect to find if they were ever working
Or working in greater amount in general

[proud nebula]

In pycharm there's an icon next to each test case

[proud nebula]

Also, just add breakpoint() in the code?

[hexed cloak]

you will have to paste /app/src/captcha/views.py too

[rigid ridge]

https://github.com/divSelector/divchan/blob/add-auth/src/captcha/views.py

Its probably not that interesting tbh. There's also this one

https://github.com/divSelector/divchan/blob/add-auth/src/captcha/interface.py

This code apparently works though from outside the tests

[rigid ridge]

also yes maybe breaking out pycharm is worth it i remember the debugger is very good buts its been a while since ive used it. As for git bisect, I don't really know how to use that. I guess the idea is basically just go back and look for a point when the tests passed but because I didnt write the code that could be extremely time consuming. It is maybe worth it to look around the timme the tests were written

[hexed cloak]

Oh that's not how you select a random row

The way the tests work is that each test runs in a transaction that is rolled back

But the consumed ids don't get reset to 1

So the test is failing because it's not the first test in the suite

You want order_by("?").first()

[rigid ridge]

So you're saying try to replace

    random_id = random.randint(1, captcha_count)
    captcha = Captcha.objects.get(pk=random_id)

with

    captcha = Captcha.objects.order_by("?")[0]

[hexed cloak]

You'll need to catch the exception from .first()

But yeah I reckon that will fix it

[rigid ridge]

let me try it out

[proud nebula]

That... is not good code

[rigid ridge]

Welp, I didn't write it

[hexed cloak]

Do the tests pass now?

[rigid ridge]

No, but it stopped that error. Very good at spotting that, I would have never figured that out.

It did solve that problem

[hexed cloak]

What's the new error?

[rigid ridge]

when you say its not good, what aspects of it jump out at you? The interface.py thing seems unconventional for Django. There are other things about it that are weird like storing image data in the database. Other than that, this part of the code ive barely looked at because I dont need the captcha system that much. I modified the code so that users cant posts without auth, which makes it easier to manage spamming without a captcha imo. Ive already modified to optionally use them. Figuring out why these tests fail is the first time ive looked at this part of the code.

[rigid ridge]

the most common one is tthis

======================================================================
FAIL: test_posting_timeout (imageboard.tests.test_new_post.NewPostTestCase.test_posting_timeout)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/app/src/imageboard/tests/test_new_post.py", line 153, in test_posting_timeout
    self.check_redirect(response_01, '/t/0x000000/')
  File "/app/src/imageboard/tests/test_new_post.py", line 70, in check_redirect
    self.assertEqual(e, None)
AssertionError: BoardNotFound('Board not found') != None

[hexed cloak]

Same problem as before ids don't start at 1 in tests

https://github.com/divSelector/divchan/blob/add-auth/src/imageboard/tests/test_new_post.py#L49

[bitter wadi]

src/imageboard/tests/test_new_post.py line 49

'board_id': '1',```

[hexed cloak]

Should be self.board.pk and self.thread.pk

@rigid ridge ^

[rigid ridge]

Okay I need to look for everywhere he has hardcoded 1.

thanks

[hexed cloak]

And 2

https://github.com/divSelector/divchan/blob/add-auth/src/imageboard/tests/test_new_post.py#L121

[bitter wadi]

src/imageboard/tests/test_new_post.py line 121

self.check_image_created(1, filename)```

[hexed cloak]

And https://github.com/divSelector/divchan/blob/add-auth/src/imageboard/tests/test_new_post.py#L122

[bitter wadi]

src/imageboard/tests/test_new_post.py line 122

self.check_image_created(2, filename)```

[hexed cloak]

Any more errors?

[rigid ridge]

I mean there is a lot of them... https://paste.pythondiscord.com/JQKQ

The first one about users.exceptions.PostAsNonAuthUser that happens because of code I added that is not accounted for in tests, the rest are issues with the previous maintainers tests

This is before fixing all the issues above you found

[hexed cloak]

Ah well fix those first

[rigid ridge]

I believe there might be an issue with hardcoding things like '/t/0x000000/' ... because if the pks are not rollling back, I don't see why the hex ids would roll back either.

To some extent I wonder like... if his tests are so fundamentally broken, are they even wortht anything to me?

[hexed cloak]

Once you fix the id issues they're probably valuable again

[rigid ridge]

nice I got rid of all the errors that are not associated with my code :)

The last one I actutally found myself but I wouldnt have if you didntt tell me the cause

[hexed cloak]

Nice

[rigid ridge]

I wonder why he would have written tests that failed and just left them that way. I checked that they failed on his branch before I touched it and they did.

one thing that is weird is that his CI runs ./manage.py test which finds 0 tests, you have to do ./manage.py test imageboard to get it to find them

[hexed cloak]

You'll have to ask him

[rigid ridge]

Actually the only one left is this one

    def test_new_post_many_images(self):
        filename = os.path.join(os.path.dirname(__file__), 'noise.png')

        with self.settings(MEDIA_ROOT=str(settings.STORAGE_DIR / 'test')):
            with open(filename, 'rb') as fp1, open(filename, 'rb') as fp2:
                post_data = self.base_post_content.copy()
                post_data.update({'images': [fp1, fp2]})
                response = self.client.post('/create/', post_data)

                self.check_redirect(response, '/t/0x000000/')
                self.check_post_created()
                self.check_image_created(1, filename)
                self.check_image_created(2, filename)

This one is confusing because he's referencing an image id without even attempting to create an Image object at all.

I do kinda have a lot of questitons for him. Maybe I should reach out. He hasn't looked at the code in 5 years

[hexed cloak]

The code itself creates the images

You'll have to check im1, im2 = Images.objects.all().order_by('id')

[rigid ridge]

that makes sense.

$ docker exec -it hexchan_app ./src/manage.py test imageboard
Found 33 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
.................................
----------------------------------------------------------------------
Ran 33 tests in 8.760s

OK
Destroying test database for alias 'default'...

The last thing Im wondering about, is I would really like to be able to say docker exec -it hexchan_app ./src/manage.py test and it runs all the tests in every app. At present I have to specify an app directory for it to find tests

[hexed cloak]

I don't know maybe try pytest-django?

[hexed cloak]

Yay

[rigid ridge]

Posting that code in here was really a shot in the dark I didn't expect anyone to look at the code and see the problem. I think if I had more experience writing tests I would have known that's how it works.

[proud nebula]

Yea it's messy for no reason.

Plus that it was broken ;)

[rigid ridge]

yeah the tests were very broken and i dont know why... my only guess is they were neglected, but the project as a whole is pretty salvageable and works as intended for the most part. Modifying it to be what I want is very doable.

github dumpster diving is pretty fun. Its rare that you find something that you actually want to salvage tho

[proud nebula]

8.7s is a lot

[pearl cliff]

setting up a test database and inserting test data can be slow, especially with an OLAP database that has long overhead for each request and you can't run it locally (Snowflake)

my integration tests w/ Snowflake take a few minutes, of which ~90% of the time is just CREATE TABLE and COPY INTO

[proud nebula]

That's crazy!

[pearl cliff]

i agree but i don't have an alternative choice

[proud nebula]

Could you maybe have that run lazily after the last test run or something?

[hexed cloak]

I used to work somewhere where the tests took 3 hours with xdist

[proud nebula]

Still. Startup time is particularly bad. It makes dev on single tests hell.

[pearl cliff]

most of it is setup, not teardown. and it is lazy via pytest fixtures, and i marked all the slow snowflake tests with snowflake so i can deselect them for faster dev iteration

[maiden pawn]

What if to parallel process of uploading data

[proud nebula]

Sure. But you can maybe do the setup ahead of time?

[proper wind]

whsit

[pearl cliff]

ah yeah. some of it can definitely be optimized, but it has to be done very carefully because then you're sharing resources between tests that aren't necessarily related

[pearl cliff]

i could definitely try that

[proud nebula]

I was thinking maybe at the end of a test you start the setup for next time somehow so that the test run isn't affected. Then at start of tests you check if that setup is done and if not then you do it. Best case it's way faster. Worst case should be the same as what you have now.

[pearl cliff]

Ah, that would require a total re-engineering of how we do it now, which uses temporary tables that don't persist between db sessions

[river cobalt]

Hey!
Is specifying @classmethod decorator on top of each test function is bad or no ? Or self is using somehow by pytest even if I don't create any instances for my test class ?
Code snippet :

class TestUpdateCreatedBooking:
    """
    Test class checks PUT and PATCH update APIs of booking object
    """

    @classmethod
    def test_full_booking_object_update(cls,
                                        session_creation,
                                        create_booking_object,
                                        booking_object_data_generation
                                        ):
        """
        Test checks PUT update of the created booking object with a valid generated data
        """
        authorized_user = session_creation
        booking_response, booking_id = create_booking_object
        booking_data = booking_object_data_generation

        response = AuthorizedAPIs.booking_object_update(authorized_user,
                                                        booking_id,
                                                        booking_data
                                                        )

        assert response.status_code == requests.codes.ok
        BookingDataPayload.model_validate_json(response.text)
        # Check that created model != updated model,
        # but some fields can be generated with the same value
        assert booking_response.json()["booking"] != response.json()

    @classmethod
    def test_partially_update_without_session(cls,
                                              create_booking_object,
                                              partial_update_data_generation
                                              ):

[proud nebula]

If you are using pytest, you should probably remove the class and just have functions

[river cobalt]

But what If I wanna group my tests in logic group to have a test suit ? I can make custom marks instead but visually it's hard to check to what group such test is related

[proud nebula]

DIfferent files?

[river cobalt]

Yep, but I am new to the pytest and saw different approaches in the repos, someone using classes, someone not and wonder what is the guidline/best practice for the lib ?

[proud nebula]

I have never really seen much use of grouping except a 1-to-1 mapping between file and corresponding tests. We do this in iommi where form.py and form__tests.py are in the same folder. I find that nice...

I personally don't like misuing classes like this...

[river cobalt]

Ok, ty for the response, I will think about creating separated file for each test group, but also wanna hear what other pytest users think about grouping tests in classes

[solemn mason]

Hey all. Quick question about testing in FastAPI (they use Starlette's testing lib from what I understand):.I have some strings that are used in multiple tests. Can I make a fixture out them, even though they're only strings?

[proud nebula]

I suggest also thinking about if that grouping serves any real purpose or if it's just the sorting squirrel inside you wanting order :P

[proud nebula]

Can't you just put them in a module and import them like any other python symbol?

[solemn mason]

that was my instinct, but I haven't seen this in the tests that already exist, so I thought maybe it was some kind of smell.

[proud nebula]

I don't think so. Tests are just code like any other code.

Your team might have some other opinion though, so talk to someone on the team about it

[mystic beacon]

what are y'alls thoughts on robot framework?

[pearl cliff]

i use classes for grouping in pytest, it's fine and there is nothing wrong with it

but i usually use instance methods, not class methods

[hexed cloak]

Do you have a URL?

[mystic beacon]

https://robotframework.org/

imo i'm not a fan of it but would love to change my mind to it

[proud nebula]

It runs an entire browser? Then my advice is the same as cypress and selenium: avoid at almost all costs. It's super slow.

[mystic beacon]

well it can but it's mostly a framework for testing

[hexed cloak]

Selenium's great

[proud nebula]

I liked cypress much more than selenium personally. But I like not having a browser spun up more.

BS4 > jest > selenium/cypress/etc

[zinc coral]

E2E tests take ages 😴

[proud nebula]

If you can get away with jest it's basically e2e but way way faster.

[mystic beacon]

Playwright is the best

But those are tools

Framework wise i love pytest

Just got done with a POC for robot

And it sucks

[somber hamlet]

I will fight back and say Cypress is better!

[pearl cliff]

i had an immensely negative experience using jest for snapshot testing

this partly had to do with the company i worked for having no idea how to do testing at all, but while i had plenty of success building a testing culture for the python code, i found myself at a total loss when it came to even making sense of how to use jest effectively

[proud nebula]

I've used it for normal e2e testing. Basically like selenium/cypress but faster. But limited unfortunately so I had to bail on it.

[proud nebula]

This isn't the place

[cinder dagger]

has anyone seen this before? codecov complaining about a commented line as uncovered

[dense bough]

May be it is a weird way of highlighting branch coverage.

[proud nebula]

It rings a bell, but I can't see any examples of it from a random look through of a big project of mine 🤷‍♂️

[dry valley]

I am creating a pytest playwright setup for my client as poc

Would like to know if any idea on conftest.py setup

We are doing e2e testing for a ecomm website

[vocal sage]

From having friends in industry, playwright is generally overtaking cypress now and they've all switched away

[somber hamlet]

Why? From people i know they still use Cypress and see no need

[vocal sage]

I shall ask!

[vocal sage]

So "just does it better" is the summary but their gripes about cypress are how it handles async, but his favourite is the trace tool which he sent me this - https://www.youtube.com/watch?v=cgj-eHricIc

Obviously this is dependant on project size, this is from enterprise setting, personal projects cypress might do all you need

[proper wind]

Quick question: I am a manual QA tester. There seems to be a growing need for a manual tester to also create auto test scripts. I am looking at tools such as Selenium, Cucumber, and testim. Do I need to have software developer skills to create auto test scripts to help do some manual testing such as logging in?

[maiden pawn]

More no than yes.
Scripting is scripting. Not very far from regular macrosing.

U just need communicating skills
Debugging skills
Limited scripting skills
Preferably managing your dependencies and ability to configure attached browser for those automations.
Ergh... It requires some skills but it does not have big depth to work with.
It will require extra effort to figure it out how to launch it from CI though

Modern kids/hackers/student level people manage doing that

[proper wind]

@maiden pawn Hmmm....I am looking at selenium and using Python. I know some Python 101.

I assume with scripting I don't have to know OOP?

But, I do need to use libraries?

[late robin]

I'd question if there is a need for python here. If it's just opening + logging into a website, (on windows) that can be done in autohotkey, or pure batch with more effort.

[proper wind]

It's more then just logging in. I was just trying to say I am looking to script UI stuff.

[maiden pawn]

i guess not. for scripting u need at most reusable functions and testing fixture usages

[maiden pawn]

But, I do need to use libraries?
yes u do. selenium alone is the library

also pytest is somewhat preferable over unittest.
pytest is installable library while unittest is std lib already available
u could work around and just use unittest though, not super big deal

[proper wind]

I took a Python 101 class a few years ago. I'm updating PyCharm community edition.

[proud nebula]

I think yes absolutely. Automated tests are just code.

[proper wind]

I am trying to figure out how much I need to know a programming language just to do some automated testing. So, let's start by taking out needing to do unit tests that are typically done by the web development team (from what I understand). That leaves doing auto tests to replace some of the manual testing. For that context, do I need to know Python well enough to be a full fledged developer? Or just know the language at a basic knowledge level?

[proud nebula]

I think you can probably muddle along with a basic understanding with help from the dev team. Learning on the job is a great way to keep focused and motivated.

And python is pretty easy to get started with

[proper wind]

If I were to use PyTest and Selenium, would I be mainly calling methods in libraries? I mean, I will look at coding examples. I do know the basics of Python and thought it was easier to learn then JavaScript and way easier to learn then Java. I never got past the basics in either language. So, just trying to get a feel for what more I need to learn.

[proud nebula]

Oh yea, for sure. You probably want to learn something about CSS selectors (or god forbid xpath 😱 ) too.

[proper wind]

LOL! I already know HTML and CSS. I found CSS selectors pretty easy to grasp. 🙂 THough, I haven't learned about xpath.

[proud nebula]

You can do a lot with a tiny bit of knowledge. And don't be afraid to ask the dev team for help :P You will write horrible code and that's absolutely fine. Accept it and just ask for help.

Horrible code can still provide huge value.

[proper wind]

I can do that. Hopefully, I don't write horrible code. Happily, I have a developer happy to help me learn selenium.

I assume I can have my coding checked in here as well.

[proud nebula]

I mean... I've been a programmer since the mid 90s, full time since 2001. You better believe I write horrible code still from time to time :P

[proper wind]

That is good to know. I won't be hard on myself then. 🙂

[proud nebula]

https://www.zainrizvi.io/blog/the-impostors-advantage/

This is one of the best blog posts about development work I have ever read. I re-read it every once in a while.

[willow galleon]

hello guys, I am using pytest and I don't know why I am getting an import modul error. Can someone explain?
My tests are in a folder that is in the same folder as the files containing the functions.

[maiden pawn]

U have ecommece, u try to import ecommerce
One letter difference - r

[willow galleon]

@maiden pawn
Yeah I messed it up trying to repair it but even after I rename it it doesn't work

[maiden pawn]

Can u publish it to GitHub? Hard to see from single file

Oh wait, I see

[maiden pawn]

Delete __init__.py at the root path and never put it there again.
This file should be only in all sub folders with python code, but not at the project root

[willow galleon]

Thanks a lot, I didn't know, I am a beginner in test driven developement and still learning about all that stuff.

[maiden pawn]

Not really problems related to test driven development.
More like problem about figuring out how python module importing works... I figured it out only after two years of using python 😅

Not the most beginner friendly subject in python

Many people never figure it out and use completely hacky project configurations with multiple PYTHONPATH overrides (p.s. never touch it)

[willow galleon]

yeah I agree and I started to learn to program last year. I get confused between packages and all that stuff. normally I was always working on a single file

[proper wind]

Is PyTest only for unit testing? Or can you use PyTest to do UI testing such as logging in?

[maiden pawn]

You can do everything including UI testing.

for example by using Selenium from pytest

[hexed cloak]

"The pytest framework makes it easy to write small, readable tests, and can scale to support complex functional testing for applications and libraries."

[proper wind]

Thanks. 🙂 I am using Selenium webdriver to get started with Python. I figure to learn PyTest later, once I get good with just webdriver now that I understand I can use PyTest for UI testing.

[languid sentinel]

I'm runing my unitest with command pytest -vv --cov=. --cov-report=xml --ignore test_e2e/ 4 of the tests are failed yet pytest stll exit with code 0

[ember maple]

More context is needed, test failures typically don't result in no error

[languid sentinel]

Sorter this out, our pytest were run within docker container. Pytest corectly exited with exit code 1. But because there was no failure in container docker itself were exiting with 0. This combined with CICD we're creating false positive scenario were CICD was green despite failed tests

[maiden pawn]

should not have happened such situation if u ran container with docker-compose run
should be working with docker run -it as well

as long as you did not obfuscate Entrypoint of course

u should have used as Docker Entrypoint pytest directly preferably

although with shell stuff for entrypoint should have worked too

[timber swift]

How to do testing on functions which modify system files? For eg my program has a function which modifies a file in /sys virtual file system, how to emulate the actions of the function without actually modifying the system file?

[proud nebula]

There are systems to fake filesystems. I've used one bit never really got it to work and gave up.

Imo the best is to make pure functions that takes some inputs like filenames and contents and then returns some output. Then you can test that. And the imperative function that wraps that can be tested separately on a temporary directory.

[timber swift]

Thanks for the info.

[pearl cliff]

one option is to make the / prefix configurable in your app. then you can set up a temporary directory with whatever fake sys files you want, and operate on that temporary directory. and of course setup/teardown is easy

!d tempfile

[bitter wadi]

tempfile

Source code: Lib/tempfile.py

This module creates temporary files and directories. It works on all supported platforms. TemporaryFile, NamedTemporaryFile, TemporaryDirectory, and SpooledTemporaryFile are high-level interfaces which provide automatic cleanup and can be used as context managers. mkstemp() and mkdtemp() are lower-level functions which require manual cleanup.

[timber swift]

Thanks for the feedback, I'm trying to do testing with Mock class in unittest.mock

[pearl cliff]

i'd strongly suggest making your code configurable instead of mocking. but yes you can use unittest.mock.patch if you really cannot modify the design at this point.

you probably don't need a "mock" at all, patching with some kind of stub might be sufficient

[timber swift]

well, the thing is, the function uses read_file(path) and write_file(path, arg) functions to modify files in /sys file system. So depending on the value of path the output of read_file may change, while write_file only returns true or false. So I find it quite easier to work with mocking behavior of these two functions.

[proud nebula]

You can change the function.

[timber swift]

Any particular reason to not use mocking?

[maiden pawn]

in this particular instance, it is better not using mocking or patching at all because filesystem is thing under our control as it is.
it is not third party accessable resource
u can just run test again temporal folder with some content and cleanup after that if necessary

as for the point of using patching instead of mocking as preferable i will kind of disagree.
i use it as last resort measure too often... but ideally code should be using proper mocking for better code architecture 😅

for reasons of laziness and bad code, we go with just patching though... and it is kind of often enough i guess in dynamic patched language

the power of dynamic runtime patchable language. everything is possible to make easier unit testable by patching on runtime.

[pearl cliff]

your tests now depend on arbitrary implementation details of your code. not what you want. also it's generally a lot harder to reason about mocks and patches than just changing a setting. you also introduce more complexity and mutable state into the test code itself.

[proud nebula]

"functional core, imperative shell" >>>> mocking

If you can make that choice of course...

[timber swift]

Can anyone link me some resources to get started with proper testing? @pearl cliff @proud nebula

[proud nebula]

I think you shouldn't worry too much about "proper" and just go for it. A kinda silly test is better than no test. Pytest makes it easier and more fun to write and run tests to you will wrote more tests which is good. Look at coverage to see how much of the code has been run at all.

[timber swift]

Thanks, I'll give pytest a try.

[maiden pawn]

https://www.amazon.com/Test-Driven-Development-Kent-Beck/dp/0321146530 from practical point, head on instruction
https://www.amazon.com/Unit-Testing-Principles-Practices-Patterns/dp/1617296279 for best pratices and avoiding pitifals leading to unit testing dissapointments
otherwise, to practice go for pytest indeed

[proud nebula]

Some code is well suited for testing. Some is not. You will learn in time the difference. Don't listen to zealots.

[pearl cliff]

but sometimes the zealots have good ideas because they've thought long and hard about their topic of zealotry 🙂

so do listen to them, but don't take them too seriously

[ember maple]

as test-ability is a often overlooked dimension of system architecture/design "proper" is very dependent on the starting point - starting a new system makes it way easier to test early and often

when there is a organically grown codebase hostile to testing smaller units, pain will follow and one will have to make painful tech debts to start the process

[placid dock]

Btw, if you want those question marks to dissapear and actually render the characters, use a nerdfont

[fathom nacelle]

Hello. I'd like to create a program to test a procedure. Is this the right place to do so?

[proud nebula]

Sure

[fathom nacelle]

Okay, so here's what I want to do.

I have a program that does operations on binary words. I have a set of word inputs and their respective outputs, and I want to create a program to test formation laws.

Where do I start?

[proud nebula]

Pytest?

[fathom nacelle]

What do you mean?

[proud nebula]

Do you know what pytest is?

[fathom nacelle]

I know now. But isn't that more for testing errors or something?

[proud nebula]

If your program fails the tests that will be errors.

[fathom nacelle]

I see, but just to make sure that we're on the same page.

What I want to do is kinda like brute-forcing a password, except without illegal implications.

[proud nebula]

That sounds not even close to what you said above... Maybe you should this time explain the real situation.

[fathom nacelle]

Okay, let's try again.

I have a bunch of inputs and outputs (binary words). I know that there is a formation law, AKA a series of operations that turn the input into an output. I want to create a program to find out the formation law.

[proud nebula]

Ok, then that's not unit testing, that's code breaking.

Or cryptoanalysis maybe

[fathom nacelle]

Okay. Thanks.

[amber fulcrum]

Is there a way to stop pytest completely from within pytest_generate_tests(). E.g. when having custom configuration options and the given option is critically wrong. raising an Exception or calling pytest.fail() or pytest.exit() only aborts the current test and keeps going it seems, not stopping pytest immediately.

[proud nebula]

Maybe you could validate whatever that is earlier?

[weary quarry]

A bit unclear as to what you are looking for. It sounds like you just want to run pytest with the --exitfirst flag.

[tender rose]

I am trying to test my output of a telnet/socket test tool. It mass connects to IPs/FQDNs on the ports specified. Its for network operations teams to test reachability of ports before an after network changes.

https://github.com/ryanmerolle/netops-utils/

I am not quite sure how to approach a test for testing some ports and the responses or lack of response

any thoughts would be awesome. You do not need to look at the code, I just need ideas on how to approach the normal connection success, refused, timeout

Regardless of if anyone has ideas. I appreciate your time

[faint mauve]

@civic field hey man, thanks for helping me earlier

@civic field I solved it by actually installing nmap from the website

[torpid osprey]

    response = requests.get(f"{SERVER}/usercredentials",headers=headers)
    assert response.status_code == 200
    data = response.json()
    return data
@pytest.mark.asyncio
async def test_server_websocket():
    headers = get_authorization_header("Blaziken","1234")
    user_information = get_user_information(headers)
    token = headers["Authorization"].split(" ")[-1]
    username = user_information["username"]
    profile = user_information["profile"]
    async with websockets.connect(f"ws://127.0.0.1:8000/ws/server/{token}") as ws:
        await ws.send(json.dumps({"chat":"dm","dm":6,"type":"text","text":"hello world","username":username,"profile":profile,"date":datetime.now().isoformat()}))
        data = await ws.recv()
        assert data
        await ws.close()``` Is there any improvements that can be made and anything specifically in a web application I should test for?

[drifting sorrel]

If I understand the test correctly it looks like it expects a service already running? It looks like an integration test and I know many prefer those kind of end-to-end tests. An alternative is to unit test the layers below the endpoints, where you can write tests for the individual parts. Otherwise, add more integration tests, to test corner cases (maybe passing unexpected, malformed data or something like that).

[steep pebble]

I'd mock the open connection call and set the side_effect to the exception you want to raise.

https://docs.python.org/3/library/unittest.mock.html#unittest.mock.Mock.side_effect

[ember maple]

More context needed, error in generate-tests should trigger collection errors and halt pytest after collection unless it's specifically requested to run even after collection errors

Additionally it's recommended to raise usage errors in pytest- configure in case configuration is wrong

More context needed, error in generate-tests should trigger collection errors and halt pytest after collection unless it's specifically requested to run even after collection errors

Additionally it's recommended to raise usage errors in pytest- configure in case configuration is wrong

[lucid bone]

Resources

Bot

[pearl cliff]

you have two options in general:

1. mock
2. run an actual server in a background thread and make requests

i'd be tempted to go for option 2, using pytest fixtures for setup and teardown. mocking tends to involve patching and patching tends to rely on implementation details, which is best to avoid if possible.

[river pilot]

it's too bad that we are not all in the habit of distinguishing between the various forms of test doubles more. It would be cool if the mock library gave us components with different names when they are different test doubles.

[pearl cliff]

fair. that said, what is a stub but a particular kind of mock?

[river pilot]

well, i would say, what is a stub but a particular kind of test double. A mock is another kind of test double.

[pearl cliff]

i'll take that. from an API point of view, what would unittest.mock.Stub look like?

[river pilot]

tbh, i have to review the differences. I try to use fakes where possible, which the mock package couldn't provide, because they have implementations of the methods, but super-slim.

[pearl cliff]

yeah, whatever the name of the particular variety of test double, i tend to use patch much more than i actually set up mocks. and even then, whenever i find myself using patch i make a mental note that it should probably be refactored

[steep pebble]

Same, it almost always patch. I try not to ever mock a class. It's caused too much trouble in the past when refactoring.

I would love an option to check that the return value or side effect matches the target function's signature. Maybe that's more of a mypy wish though.

[lament heath]

@proud nebula are there any other automated metrics besides coverage and mutants that we can use to asses the "quality" of our tests?

[proud nebula]

Not that I can think of right now. In theory mutation testers can also find tests that are "shadowed", ie that can be deleted without affecting the mutation score. But that is a pretty minor thing imo. And mutmut doesn't have this feature.

[lament heath]

Btw how is the effort with mutmut3 going?

It honestly feels like these kinds of tools could change how many people test if they were fast and easy to use. And mutmut feels quite fine on the second criterion ;)

[proud nebula]

Basically not at all honestly. I have a full time job, kids, and other hobby projects. I don't need mutmut for work so 🤷‍♂️ Right now I'm working on my iOS weather app. A totally new design is coming out soonish :)

[lament heath]

Oh well. Sad but I get it :)

[civic field]

If i were to make a post for today-i-teach lets say for pytest, what would you all think is worth including?

[proud nebula]

Basic test functions and fixtures seems like 99% of what you need. Parametrize is the next step from that I think.

[civic field]

All right, any ideas for an example project to pytest?

Someone mentioned algorithms are good, i did Luhns last time with unittest, however i got no comments back :S

[crimson frigate]

Hey everyone, I'm currently learning Django and looking to improve my understanding of code testing. I've found the "Test-Driven Development with Python" book helpful so far. Are there any other resources you would recommend about code testing?

[tall brook]

I loved this one - https://www.obeythetestinggoat.com/

[proud nebula]

If you're learning Django, I recommend the Unofficial Django Discord https://unofficial-django-discord.github.io/ which I'm on. Also install django-fastdev :P

[maiden pawn]

https://www.amazon.com/Test-Driven-Development-Kent-Beck/dp/0321146530

This book teaches practice behind unit testing. On a specific example it walks you through how to have unit testing as part of your development cycle at every moment. It teaches you feeling how much gap between tests is allowed in your working code.

https://www.amazon.com/Unit-Testing-Principles-Practices-Patterns/dp/1617296279

This book teaches theory behind unit testing. For which goals to aim for, how to escape pitifals. It will explain you importance of unit testing and what kind of testing exists. This book is important to weaponize your skills to an average commercial level development.

[crimson frigate]

Yeah it is wonderful and I mentioned this book. For now, the Deployment process is terrible, but there is this disclaimer.

[crimson frigate]

I joined it, and I see self promotion haha. I am said that it is not a updated version of mypy that works with django 5

[crimson frigate]

1. I see the classic one. Idk why I haven't it on my to do list
2. Sad that it is in C# Will it be relevant if I do not know C#?

[maiden pawn]

it will be more correct to say it is written in Java.
I would say it is not important at all in this case for whole book except like... around its 3% related to design patterns may be.
Usage of more complex code structures in interpreted dynamic typed languages and static typed languages has significant difference in my opinion

[modern gull]

Hello everyone, I want to setup an in-memory PostgreSQL database for my tests. I found testing.postgresql package but it is 7 years old. Are there alternatives to this package?

[river pilot]

doesn't postgres have a native syntax for connecting to an in-memory database?

[modern gull]

Sorry I am not aware of that. Could you point me in the right direction.

[proud nebula]

You could first try this old package to see if there's any point to doing this at all :P

[maiden pawn]

Just use docker composed Postgres and create temporal logical databases on tests start. Destroy after that

Pretty much similar result

[modern gull]

I think ChatGPT was recommending this method. But isn't calling an actual DB slower?

[maiden pawn]

Optionally don't use auto commit in addition and reset connection between tests. It will make sure tests do not share data

[maiden pawn]

That is reliable method that will make sure everything was unit tested with least amount of infra and coding effort

[proud nebula]

He's obviously trying to speed up his tests. Recommending to start a docker container on each test seems counterproductive.

[maiden pawn]

No no, don't launch at every test

Single db is enough

Single applying migrations is enough

[proud nebula]

...so then docker or not is irrelevant?

[river pilot]

@modern gull are you using postgres-specific features?

[maiden pawn]

Docker is for easy temporal database install of necessary version, recorded into your git repo and CI. Documentation as a code + easy install of dedicated db for this repo only

[modern gull]

Well I am using raw SQL commands.

[river pilot]

ok, but are you using anything but standard sql?

[modern gull]

Well it's mostly standard SQL, I guess. But I am not sure, there may be some syntax variations from other DBMSes.

[river pilot]

i woulld have thought postgres would have an in-memory option, but I don't know if it does.

[hexed cloak]

You could put the database on a shm mount

[proud nebula]

Last I heard using memory disks ends up being less efficient than letting the OS do the disk stuff the normal way. 🤷‍♂️ Don't know if that's true but I'd test in on a small scale and make sure you're getting some gains!

Also, if the tests are slow, my recommendation is to first fine the 10 ten slowest tests and run a profiler on them.

I've seen huge gains with just this type of stuff, and sometimes joining 10 tests that are written with "one assert per test" philosophy but actually have the exact same setup.

[river pilot]

@modern gull boxed has a good idea to find the real slow-down. Are you using pytest?

[modern gull]

Yes

[river pilot]

perhaps you have repeated work that could be moved to a sesssion-scoped fixture, for example.

[modern gull]

I think I wasn't clear in my communication, I haven't yet started writing tests, I was asking how I can setup my PostgreSQL connection, so that I may use that during my tests.

[river pilot]

oh, you mentioned in-memory.

[proud nebula]

You are over thinking this a lot then. Just get it to work.

[modern gull]

Yeah, that maybe, but the problem is I am not sure how to do it. I am doing it for the first time.

I have done it before with MongoDB which comes with an in-memory version as well.

So unit testing was easy

[proud nebula]

First make sure you need it

[modern gull]

I think I was not clear with my question. I am writing tests for my application which uses PostgreSQL, I am wondering how my tests will use PostgreSQL in a way that is not affecting the actual DB.

For instance I am currently using DDD to write my code, so I have repositories, unit of work and commands which use the repository.

I want to write tests for my commands which are calling repository which in turn is executing SQL queries.

[proud nebula]

Connect to another db. Call it "tests"? This is what you get by default in django tests.

[pearl cliff]

does anyone here have experience writing unit tests for custom airflow operators? i'm trying to avoid relying entirely on manual QA for testing. would appreciate any advice on this.

[crimson frigate]

Good to know! I will check it out

[hidden root]

Hello!

I'm looking for simple open-source test reporting tool for my project. I mainly care about graphical representation of the results. I've found few options - first is Allure framework and second is Robot framework . Does anybody have any experience with it and could advise if they are worth considering?

[river pilot]

can you say more about what you need? What kind of graphical representation of the results

[hidden marten]

A big question is what behaviors of the operator you want to test.
A while back we added unit tests, and mocked out any remote calls with expected responses, but generally it was an instance of an operator class so we could test methods and properties.

[maiden pawn]

I prefer just using straight best unit testing framework language has (pytest for python)

And then if we need representation better for results, just export results to junit xml or Smh and use tools from that ecosystem to show something flashier

I dislike usage of robot, because it is not debuggable code as far as I know. No point not to use python for testing if your app is in python

[proud nebula]

Why isn't just "all test successful" enough? :P

[hidden root]

I'm learning azure pipelines - running my py tests from there. There is some test results in azure but i think i need to configure some xml test results and then publish it...

I was thinking about Allure because as i was reading doc i can run web on machine and get all tests passed and failed

[hidden root]

i wish 😉 i think i need to know any reporting tool as person who want to write tests

[proud nebula]

I didn't ask "isn't?" I asked "why?"

I have seen many fancy reporting tools. All of them have been useless in practice I find. In my experience it's a "wouldn't this look cool?" thing and people get distracted by a sales pitch, while there isn't anything real there.

[hidden root]

Or azure pipelines are "great and widly using tool" for getting test results in this area

[proud nebula]

I am getting the feeling there is a language barrier or something, because that made no sense either.

[pearl cliff]

it's a custom spin on ExternalTaskSensor, 99% of our logic is sqlalchemy queries against the airflow database.

[pearl cliff]

you want to show test results in some kind of dashboard? like "CI: passing (207/207, 0 skipped)"

[pearl cliff]

you definitely don't need a reporting tool in order to write tests. very few teams have that. write the tests first, then get some kind of CI set up, then worry about reporting later

[hidden root]

Indeed

[pearl cliff]

unless you're some kind of full time QA engineer and you're on a large team working on a large codebase with a large number of tests, i don't think this is a high value project compared to actually writing tests and setting up a CI pipeline

then again i'm not your manager so 🤷

@hidden marten my plan was to do something like this:

1. run the airflow standalone runner in a background thread/process
2. using a dedicated dag directory for my test dags, let the dagbag fill up + let some dag runs complete
3. make assertions using the airflow CLI

but that's a lot of scaffolding and it would be nice to do something simpler

[hidden marten]

Yeah, that sounds like a lot of scaffolding. I've previously mocked out the airflow hook, and constructed an instance of the operator and he asserted against the expected behaviors, so as to remove the need to run airflow at all

[pearl cliff]

i see, i don't think there is any hook to mock out here unfortunately. i guess i could mock out airflow.config.session and point it to a dedicated test database that is cloned from a snapshot of prod

[hidden marten]

For me, it comes back to what exactly are you looking to assert with the test. I'd previously mocked out airflow.hooks.base_hook.BaseHook, "get_connection" so that when I construct the custom operator in my test, I don't get much airflow bits running as a result

[pearl cliff]

well again, this is an ExternalTaskSensor. i don't even think their implementation uses any Hook at all, it's just the poke method hitting airflow.settings.Session() directly

but otherwise that's a great idea and i will definitely try it

[quartz meadow]

Omg just noticed there's a place for unit testing conversation 😃

[tawdry minnow]

Hi everyone, I want to learn about testing what should i learn first?

[maiden pawn]

Test Driven Development: By Example

This book teaches practice behind unit testing. On a specific example it walks you through how to have unit testing as part of your development cycle at every moment. It teaches you feeling how much gap between tests is allowed in your working code.

Unit Testing Principles, Practices, and Patterns

This book teaches theory behind unit testing. For which goals to aim for, how to escape pitifals. It will explain you importance of unit testing and what kind of testing exists. This book is important to weaponize your skills to an average commercial level development.

[tawdry minnow]

Oh yeah I already saw it

Thanks

[river pilot]

what do you already know?

(about it)

[tawdry minnow]

Nothing but i know i can use it to know if my code is working correctly aside from that nothing

[proud nebula]

It's just running code to check known results for known inputs.

[rocky comet]

Is there a clear winner between tox and nox? Or does it boil down to preference

[maiden pawn]

jobs:
  linter:
    runs-on: [self-hosted, linux, x64]
    name: unit testing:python=${{ matrix.python-version }},django=${{ matrix.django }}
    strategy:
      matrix:
        django:
          - "3.2"
          - "4.0"
          - "4.1"
          - "4.2"
        python-version:
          - "3.8"
          - "3.10"
    steps:
      - name: Set up Python
        uses: actions/setup-python@v2
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
          architecture: x64
      - name: Install Task
        run: sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d -b ${{ github.workspace }}
      - name: Install requirements
        env:
          DJANGO_VERSION: ${{ matrix.django }}
      - name: Show current requirements just in case
        run: pip freeze
      - name: Mypy
        run: ${{ github.workspace }}/task mypy
      - name: Pytest
        run: pytest --cov --cov-report=xml:coverage.xml -v
      - name: Django migrations
        run: django-admin makemigrations --settings=testdjangoapp.settings --pythonpath=. --check --dry-run
      - name: Display coverage
        if: ${{matrix.django=='4.2' && matrix.python-version=='3.10'}}
        continue-on-error: true
        uses: ewjoachim/coverage-comment-action@v1
        with:
          GITHUB_TOKEN: ${{ github.token }}
          COVERAGE_FILE: coverage.xml

[hidden marten]

No clear winner - for me it comes down to familiarity and preference. From Nox's docs:

Unlike tox, Nox uses a standard Python file for configuration.

[rocky comet]

Fair enough

[proper wind]

I am new to selenium webdriver. The test I am running will:
go to the login page.
Locate the "Forgot Password?" link and click on it.
The reset password webpage opens.
I have an assert that wants to verify that the reset password webpage is open.
However, unless I use the time.sleep(3) code, the assert fails because it is still reading the login page.

I tried a bunch of different ways to use selenium sleep, but can't get it to work. Here is my coding. How do I use selenium to keep the reset password page open long enough for the assert to work?

import time
import pytest
from selenium.webdriver.common.by import By
from page_classes.base_page import BasePage
from page_classes.login_page import LoginPage


class TestLoginPageLinks:
    __forgot_password_locator = (By.LINK_TEXT, "Forgot password?")
    __not_member_button_locator = (By.XPATH, "//button[@class = 'submit_btn']")
    __forgot_password_webpage_locator = (By.XPATH, "//h2")
    __registration_webpage_locator = (By.XPATH, "//h2")
    __password_url = "https://music.duetpartner.com/forgot-password"

    @pytest.mark.parametrize("locator", [(__forgot_password_locator), (__not_member_button_locator)])
    def test_links(self, driver, locator):
        login_page1 = LoginPage(driver)
        base_page1 = BasePage(driver)
        login_page1.open()  # opens the login page

        base_page1._click(locator)
        time.sleep(3)

        if locator == self.__forgot_password_locator:
            # assert base_page1._get_text(self.__forgot_password_webpage_locator) == "Forgot your password?"
            assert base_page1.current_url == "https://music.duetpartner.com/forgot-password"
            print("I am at the forgot password page.")
        elif locator == self.__not_member_button_locator:
            assert base_page1._get_text(self.__registration_webpage_locator) == "Start your two week free trial!"
            print("I am at the registration page.")

[proper wind]

I found a solution. Instead of using time.sleep(3), I am just using

driver.get(url) 

at the beginning of the if and elif coding blocks. It's not my favorite solution, but works.

[proper wind]

Here is another solution too. I'll play with it later: https://www.selenium.dev/documentation/webdriver/interactions/windows/

[fierce flare]

You may find that https://github.com/seleniumbase/SeleniumBase simplifies your pytest + selenium testing. There's even a Recorder that can generate your test for you, like the script below:

from seleniumbase import BaseCase
BaseCase.main(__name__, __file__)

class DuetPartnerTest(BaseCase):
    def test_duetpartner(self):
        self.open("https://music.duetpartner.com/login")
        self.type('input[placeholder="Email Address"]', "email")
        self.type('input[placeholder="Password"]', "password")
        self.click('button:contains("Log In")')

        breakpoint()

[bold rampart]

Recently did a series of updates and swapped to poetry,

Now my pytest is returning an error found below on almost every test when I try run all tests, but not inside my ci/cd so I didn't notice right away and am not sure what update specifically altered this behaviour.

I saw some people saying implementing a pytest fixture with manual event loop handling would fix it, but it hasn't seem to changed anything.

Another unusual thing is that I am actually using starlette TestClient and all my tests are synchronous

RuntimeError: Event loop is closed
============ 81 failed, 3 passed, 1 skipped, 39 warnings in 21.70s =============

[bold rampart]

fwiw I fixed it and it was a total red herring, an env variable was set wrong causing the lifespan event to hang, but my test runner addon and ci/cd had the env variable set in its config

[deft orchid]

I have a doubt regarding use of mock objects in unit tests. Please see this: https://discordapp.com/channels/267624335836053506/1216093148409892915

[proud nebula]

A question. A doubt is something else.

[deft orchid]

Okay...a question probably...?

[river pilot]

"Doubt" for "question" is a common usage in India.

[verbal jasper]

and in LATAM as well.

tengo una duda

[pearl cliff]

i think that's just a mistranslation though. i'd still translate that as "i have a question" or "i'm uncertain about xyz"

[ashen pond]

hello! i posted a question morning about testing/mocking, but i think i posted too early to get a reply perhaps? https://discord.com/channels/267624335836053506/1216356386905264249

it seems like maybe im just not quite understanding what can be mocked. why can i mock a class that's been imported, but not mock it at it's source?

[river pilot]

here's how I explained it: https://nedbatchelder.com/blog/201908/why_your_mock_doesnt_work.html

[ashen pond]

thank you! reading now. was just playing by putting breakpoints in a bunch of places, and seeing that my test imports from the tested module which imports the shared thing all before the fixture is created, which is likely the issue

[ashen pond]

thank you! that explains it nicely. i really apprecaite it. so what it doesn't say is - what's the best practice for this kind of repeated code? just define the fixture in every file that needs it? or still move it to a conftest file, but use some kind of logic with the request object to figure out what needs to be patched?

[river pilot]

i'm not sure what's the repeated part. a fixture is usually a good way to do repeated test-prep work.

[ashen pond]

the answer may be just that i need to restyle my tests, but at least right now, i have this in 5 different test files:

@pytest.fixture
def mock_action_response(mocker):
    """Fixture to mock ActionResponse"""
    mock_resp = mocker.MagicMock()
    mocker.patch(
        "project.actions.action1.ActionResponse",
        autospec=True,
        return_value=mock_resp,
    )
    return mock_resp

the problem is, the one line (project.actions.actionX.ActionResponse) changes (at the X) in each file

[river pilot]

hmm, this might be a place to write a function that takes X and returns the mock to use.

[ashen pond]

something like this seems to be working... feels a bit...ugly, but...

module_mapping = {
    "test_action1": "action1",
    "test_action2": "action2",
}

@pytest.fixture
def mock_action_response(mocker, request):
    """Fixture to mock ActionResponse"""
    mock_resp = mocker.MagicMock()
    mocker.patch(
        f"project.actions.{module_mapping.get(request.module.__name__)}.ActionResponse",
        autospec=True,
        return_value=mock_resp,
    )
    return mock_resp

or even better...make a fixutre:

@pytest.fixture
def target(request):
    """Fixture to get the target action"""
    return request.module.__name__.split("_", 1)[1]

[river pilot]

Hmm, I'm not sure I like that implicit detection of where the test is from...

[ashen pond]

why is that?

[river pilot]

it just feels like too much implicit magic

[pearl cliff]

agreed, i would push back on that if a colleague suggested it

[ashen pond]

Better to use an explicit match statement?

[pearl cliff]

the problem for me is the magic module autodetection. i'm not a fan of that

tests/_testutils.py:

def _apply_actionresponse_patch(mocker: MockerFixture, module: str) -> str:
    mock_resp = mocker.MagicMock()
    mocker.patch(
        f"{module}.ActionResponse}"
        autospec=True,
        return_value=mock_resp,
    )
    return mock_resp

tests/test_moduleX.py:

from _testutils import _apply_actionresponse_patch

@pytest.fixture(scope="function")
def mock_action_response(mocker: MockerFixture) -> MagicMock:
    return _apply_actionresponse_patch(mocker, "project.actions.moduleX")

another option might be to define the fixture to take its patch target from the fixture's parameter, and then use pytestmark to mark the entire module with pytest.mark.parametrize("mock_action_response", "project.actions.moduleX", indirect=True) ... but i'm not sure that actually works

let me try that

yup, it worked

working example:

conftest.py:

import pytest

@pytest.fixture(scope="function")
def silly(request: pytest.FixtureRequest) -> int:
    param = request.param
    match param:
        case int():
            return param * 3
        case _:
            raise TypeError("Parameter must be an integer.")

test_thing.py:

import pytest

pytestmark = [
    pytest.mark.parametrize("silly", [2, 3], indirect=True)
]

def test_indirect(silly: int) -> None:
    assert silly % 3 == 0

Then pytest test_thing.py passed with 2 tests passing.

so you can use that technique to parameterize a fixture at module level, in your case parameterizing with a particular module name to patch

[ashen pond]

nice, thank you

[drifting sorrel]

I usually want the actual unit test to be in control of what is mocked or patched, and the return data from it. So I would probably do something like this, instead of writing fixtures or using the MagicMock thing:

def test_something(monkeypatch): # using the built-in monkeypatch fixture/feature of pytest
    expected_return_data = "hello world"
    monkeypatch.setattr(the_namespace.the_module, "the_function", lambda *args, **kwargs: expected_return_data)

    ...
    assert some_result == expected_return_data # or any other assert that makes sense here

[pearl cliff]

i normally agree, but i do occasionally find myself writing groups of tests that all use the same fixture (sometimes with expensive setup and teardown that i don't want to run a lot of times), so in that case i'll use class-scoped fixtures and group them together in a class

i think it's OK to use the implicit pytest magic as long as you're disciplined and organized

[jolly tangle]

In python flask I am trying to pytest the verify_token method but I am getting an error.

Please ping on reply.

Here is the docs for itsdangerous and flask. https://itsdangerous.palletsprojects.com/en/2.1.x/concepts/

When I pytest the code the error is located below.
https://paste.pythondiscord.com/X2RQ

I am basing the code on https://github.com/CoreyMSchafer/code_snippets/blob/master/Python/Flask_Blog/11-Blueprints/flaskblog/models.py

My guess is the error might be caused by yield.

I googled the error and found this https://stackoverflow.com/questions/30140437/how-to-fix-itsdangerous-badtimesignature-signature-error

The problem is when I try to import app from app/app.py I get a circular import error in tests/models.py

Here is my code.

conftest.py

https://paste.pythondiscord.com/GF3A

test_tokens.py
https://paste.pythondiscord.com/PPWQ

models.py

https://paste.pythondiscord.com/O4VA

I am using blueprints where I use different configs and then in init.py I create the app from different configs in config.py. Then I call create_app in a file in app.py.

Thanks for the help

[hexed meadow]

I'm struggling to get pytest to pick up a conftest.py file where I added a command line option

we normally run pytest as pytest -d -n 5 foo, where foo is a dirctory that contains all our python code. Tests are adjacent to whatever they're testing, so theyr'e spread out all over foo

(usually nested another level, like foo/bar/baz.py, foo/bar/test_baz.py

I've added a conftest.py with

def regression_path(parser):
    parser.addoption("--blub-path")

Inside foo

but when I run say pytest -d -n 5 foo --blub-path it says unrecognized arguments

oof I didn't realize that it had to be named exactly pytest_addoption 🤦‍♂️

disregard sorry

new question: is it possible to leverage argparse properly with pytest? It seems like pytest has its own "parser" that is, well, I'm not sure what.

but I can see it has addoption instead of add_argument, and it doesn't seem to have add_mutually_exclusive_group

[river pilot]

can you say more about what arguments you want to add? there might be a better way.

[hexed meadow]

sure. Basically, I just want to be able to pass a certain path explicitly, or pass an option that will cause teh path to get picked up automatically from an env variable we often set

[river pilot]

what's the mutual exclusion?

[hexed meadow]

we're kind of quasi-abusing pytest to also run some unit tests, that aren't very "unit-testy". They require some additional files that aren't in the normal repo.
We have a special regression testing repo where I want to put those files.

well, it makes no sense to specify both of those right?

if you pass pytest --regression-path foo/bar/baz then it's goign to use foo/bar/baz as the regression path

[river pilot]

you sometimes want neither option?

[hexed meadow]

if you pass pytest --regress then it will try to look at e.g. $TEAM_REGRESSION_PATH env variable and use that as the regression path

[river pilot]

every argument parser eventually has a limit where you need to do manual validation

[hexed meadow]

yeah, if you pass neither option then it will basically skip those tests

yeah, the limit just came earlier than expected this time I guess, I'm spoiled by argparse 😛

I guess there's no real plan for pytest to simply adopt argparse for this, is there?

[river pilot]

probably not, certainly not in the timeframe you need to use it!

[hexed meadow]

😛

How do I manually validate then?

not a big deal of course, just struggling a bit to find documentation for this

[river pilot]

i guess wherever you were going to use those options, check for the conflict.

[hexed meadow]

that doesn't really make sense though, the options get used many times

I'd want to check immediately

and have a failure immediately

the problem is, inside pytest_addoption I'm not actually getting back the parsed arguments, just adding options

https://docs.pytest.org/en/7.1.x/example/simple.html#pass-different-values-to-a-test-function-depending-on-command-line-options
Like, there's some interesting examples here about different functions you can put in pytest_

sorry, in conftest.py

just really struggling to find documentation about everything that can be in there

I think that I will probably just define

def pytest_collection_modifyitems(config, items):

And since I'm also planning to use this ot skip tests, I guess I can just do the command line enforcement there

which sort of feels wrong but it's not easy to figure out a better way

[river pilot]

i would have thought the code using the option would be factored into a function or fixture, but i don't know the code

[hexed meadow]

yeah, it seems that way. I'm honestly struggling a bit because there's so much magic and the initial examples don't make it exactly clear what is magic or not

I had no idea originally that even argument names would be magic

I'm curious is pytest considered like, idk, one of the top/best most modern options in the python unit testing space? Or is it more that it's been around for a long time sort of thing?

[river pilot]

pytest is easily the most used framework. hard to separate out 1) most powerful 2) best designed 3) been around longest, etc.

argument names matched to fixture names is definitely surprising at first.

but i find that all test frameworks have some kind of outside-the-language machinery helping to write this odd code we call tests.

unittest wants you to write classes you never instantiate with methods you never call.

[hexed meadow]

all test frameworks have you write functions you don't call

[river pilot]

yes, that's kind of my point

[hexed meadow]

and using classes to organize is also super common

[river pilot]

yes, true

[hexed meadow]

the surprise factor of parameter names is a lot higher.
I'd also say that while this is "cute" it seems in my eyes to scale really poorly because now there's no notion of "scoping"

it's basically just a giant single namespace of legal python identifiers for anything you want to pass to unit tests

[river pilot]

you can define fixtures at multiple levels in your hierarchy, just as you have conftest.py in multiple places.

[hexed meadow]

i mean my needs are very simple so I don't care much but, idk, I'm not loving my dive into pytest so far I guess.
Maybe you don't expect to get both "easy" and "rigorously well thought out software engineering wise" but at least 1/2 I would hope for

[hexed meadow]

yes, but that's going to be some very confusing code to read if names actually collide

The same name could refer to a totally different fixture in different parts of your code

[river pilot]

you don't have to use fixtures if you don't want, you could use imported functions that you call, which i guess is what unittest would have you do.

[river pilot]

isn't that the whole point of scoping, which you seemed to want?

[hexed meadow]

unittest does have fextures at the test suite level, but I admit I have no idea how I would do what I'm doing now in unittest

[hexed meadow]

I mean usually scoping would be explicit. Like, you can do from foo import bar in one test file, and in another test file I'd do from glug import bar
bar means different things in different files, but I can figure out from that one file, how to follow a "trail" I guess to figure out what bar actually is

[river pilot]

i see

[hexed meadow]

with this, I mean, it seems like you basically need to run around and grep and hope for the best

to track down the actual meaning of stuff

it's very very "magic string" oriented

[river pilot]

yes, it is.

[hexed meadow]

I guess I would be less grumpy if it was easier to grasp. I feel like a more explicit solution would have been a little more typing, yes, but I would have already figured this out an hour ago, and it would be done.
And it would still be much easier to read and figure out for people on the team.

[river pilot]

it's true that concision and discoverability are often at odds

[hexed meadow]

is concision intrinsically a goal though? often concision is used as a standin for "easy to write" - and that' susually valid.
in this case, yes, the final code will be very concise. but it's not at all easy to write, and it also won't be particularly easy to read.

[river pilot]

i don't understand why it's not easy to write? I use a fixture name as an argument, and I get the fixture i need, with the desired scope.

[hexed meadow]

It wasn't easy to figure any of this out. and even now, I have a new problem: I want to put the logic for actually getting hte regression path in a fixture, right, because it's not "just" a command line argument

but now I want that value passed to pytest_collection_modifyitems - but I cannot, because that enforces a specific signature

[river pilot]

"figure this out" is reading, not writing. I'll grant that you have to be able to read it before you can write it.

[hexed meadow]

🤔

[river pilot]

you're using pytest_collection_modifyitems because you need to includ/exclude tests to run?

[hexed meadow]

yeah

so, I have my reg_path fixture

if it turns out that a regression path is not supplied, then I want all functions that take reg_path to be skipped

[river pilot]

you could let them be included in the collection, but skipped.

[hexed meadow]

This is how the docs recommend skipping tests based on a command line option it seems: https://docs.pytest.org/en/7.1.x/example/simple.html#pass-different-values-to-a-test-function-depending-on-command-line-options
How else could I do it?

[river pilot]

pytest_collection_modifyitems gets passed config, so you can use config.getoption() to read your option.

[river pilot]

the other way is in your reg_path fixture, raise pytest.skip

[hexed meadow]

reg_path fixture gets called over and over again?

[river pilot]

it kind of depends if you prefer the result to be 100 tests, 100 passed, or 120 tests, 100 passed 20 skipped

[river pilot]

that depends how you scope it.

[river pilot]

by default, yes.

[hexed meadow]

that's the thing, something like a command line argument isn't really a "fixture" in the way I think of it. fixtures are usually run repeatedly per test or set of tests.

this is a global thing.

[river pilot]

ok, but you can make it run for each test if it's useful.

there are two different models here: the tests are not even considered, or they are but then skipped.

(sorry, have to be afk for a while)

[hexed meadow]

sure, np, thanks for your help

sorry if i was bombing you with questions a bit, just getting a bit frustrated here

(with pytest of course, not you)

wow, and the parser seems to be unable to handle certain argument orders 😦

--regression-path foobar --regress errors in parsing

have to do --regress --regression-path foobar

[pearl cliff]

addoption is probably the worst thing about pytest i've encountered

that is: the rest of the framework is better. nowhere to go but up!

[hexed meadow]

that's good to know

I'm still mega struggling here though. it seems like there's two different ways to approach skipping tests, it's totally unclear where to put my configured value

[pearl cliff]

personally i avoid addoption and use marks for fine-grained test selection

[hexed meadow]

i mean I need a command line option

I don't understand how you can accomplish that with just marks?

[pearl cliff]

you can mark all of your regression tests with a custom regression mark, then select with -m regression or deselect with -m 'not regression'

not sure if that addresses your actual requirements though

[hexed meadow]

no

I need a path

[pearl cliff]

it has to be configurable? 😛

[hexed meadow]

yes

I mean.... this isn't really asking the world is it? This feels mega basic

[pearl cliff]

does this example help at all? https://stackoverflow.com/a/77014798/2954547

[hexed meadow]

should I be using that or using pytest.skip as Ned suggested?

I have no idea

[pearl cliff]

it depends. do you want it to show up as "skipped" or not?

[hexed meadow]

let's say "yes"

[pearl cliff]

pretty sure you can just do for item in items: item.add_marker("skip") or something like that

https://stackoverflow.com/a/58493116/2954547

[hexed meadow]

i'm so lost

if I call pytest.skip from inside my fixture, will it show up as skipped, or not?

[pearl cliff]

wait, in a fixture? i thought we were talking about the pytest modifyitems hook here

[hexed meadow]

there's two ways to do it, aren't there?

no?

[hexed meadow]

here

[pearl cliff]

yeah, either you call pytest.skip() inside your fixture, or mark the test as skipped (or remove it from selection entirely) in the modifyitems hook

https://stackoverflow.com/a/28198398/2954547 example of skipping from inside fixture

how you want to actually do it is entirely up to you

[hexed meadow]

and how do I make sure my fixture is only called once?

[pearl cliff]

set pytest.fixture(scope="session")

[hexed meadow]

sigh my brain just hurts. Someone on my team switched us over from unittest to pytest, but most things are still written in the unittest style

So I can't even pass pytest fixtures to it without changing it

[pearl cliff]

are you trying to get the value of that --regression-path option inside a fixture, so you can load data from that path?

[hexed meadow]

i have it in a fixture

I'm trying to get the fixture into a unit test

[pearl cliff]

ah

https://docs.pytest.org/en/8.0.x/how-to/unittest.html#pytest-features-in-unittest-testcase-subclasses

The following pytest features work in unittest.TestCase subclasses:

Marks: skip, skipif, xfail;

Auto-use fixtures;

The following pytest features do not work, and probably never will due to different design philosophies:

Fixtures (except for autouse fixtures, see below);

Parametrization;

Custom hooks;

not sure if that narrows things down at all

[hexed meadow]

yeah, I can do the autouse thing. but it's just pointless suffering

I don't really get it

like why this person decided to use pytest at all

but keep writing everything with unittest

man, still debugging these magic functions 😦

[pearl cliff]

i mean, pytest is great. lots of features, lots of customization and control, less boilerplate than unittest for a lot of things. but the point of pytest + unittest is so you have an upgrade path for legacy test suites, not to use pytest as a test runner for unittest in perpetuity. that's going to be the worst of both worlds, which you seem to be finding out.

[hexed meadow]

I mean, yeah, I agree with the last part

but having trouble agreeing with the first part

the argument parser for pytest is terrible for multiple, independent reasons

[pearl cliff]

i'm pretty sure it's just a wrapper for argparse.ArgumentParser.add_argument

[hexed meadow]

and it's also super magical, and hard to learn, and later on it will be difficult ot use an IDE/tooling to quickly track down a fixture

it is not

it has the same API but it's not using ArgumentParser

[pearl cliff]

https://docs.pytest.org/en/8.0.x/reference/reference.html#pytest.Parser.addoption

attrs (Any) – Same attributes as the argparse library’s add_argument() function accepts.

i won't vouch for its implementation, but it's supposed to work the same way at any rate

[hexed meadow]

I mean it just doesn't, I showed a trivial example of its parsing problems

it can't parse --regression-path foobar --regress

you have to revese the order

[pearl cliff]

what's the error? does it assume nargs=1?

[hexed meadow]

it also doesn't support any of argparse's other features

like mutually_exclusive_group

[pearl cliff]

huh, it is actually just a custom Parser thing.

[hexed meadow]

The error is not very informative, it just says the arguments are not defined

i think someone said it's a wrapper around the old optparse

[river pilot]

Fwiw, I would use env vars instead of command line options

[pearl cliff]

let me see if i can reproduce in a minimal test suite

[hexed meadow]

i mean I'm not a fan of using an env var exclusively, personally

[river pilot]

I'm not a fan of having the pytest command be the center of everything

[pearl cliff]

it's possible that all options are assumed to have nargs=1

[hexed meadow]

the idea here is taht there's two options, an explicit one and a convenience one that uses an env var

[pearl cliff]

i think it's pretty reasonable for something that's test-specific

[river pilot]

I don't like how pytest-cov jams everything into the command line for example

[hexed meadow]

I'm so confused

i'm doing

print("f{reg_path}, {reg_env_bool}")

and it's not printing the f-string correctly

ohhhh

sigh

I probably should have stopped for lunch an hour ago but this is making me too mad 😛

[pearl cliff]

do they not also support it as an ini option?

[pearl cliff]

--regress pulls from env var, --regression-path loads from a specific path?

[river pilot]

Well, I do. But why are there coverage reporting options on the pytest command line?

[pearl cliff]

because it's calling out to coverage.py, no? i like the ability to control code coverage along with the test invocation

[hexed meadow]

yeah

[pearl cliff]

@river pilot it's useful if you want to do something like split your tests into integration tests and unit tests and run coverage separately on them

[pearl cliff]

what if you simplified it by accepting a single --regression option, and if the arg is empty, fall back to the env var? like how --cov= is valid for pytest-cov.

doesn't solve the parsing issues but maybe doesn't make them necessary.

[river pilot]

But why not run the report after you run the tests? The reporting isn't a party of running tests

[pearl cliff]

partly for convenience, and partly so you can fail the test suite if coverage is below threshold

[hexed meadow]

I think I basically have it figured out I"m just annoyed at how much time has gone into this, and I'm also really annoyed that my co-worker decided to switch to pytest without talkinga bout it

but then didn't even do it properly

[pearl cliff]

yeah... that's not something you just "do"

[hexed meadow]

well honestly maybe I'm mis-remembering.

[proud nebula]

Imagine if he switched to hammet 🤣

[hexed meadow]

I guess maybe it could have just been treated as "personal preference" since the test suite runs with either? I don't think we use any pytest specific features

[proud nebula]

Or. Ward? Is the other weird one?

[hexed meadow]

what's the main reason people actually use pytest, out of curiosity?

(over unittest)

[river pilot]

Power and convention

[hexed meadow]

can you explain those? It seems like unittest has all the same standard features. Like, when I look online people talk about fixtures being a reason why they prefer pytest, but unittest obviously has fixtures

[river pilot]

Does it have fixtures?

Unittest has added things it didn't used to have, maybe parametrization?

[hexed meadow]

of course, it's always had them afaik

it has per-function, per class, and per module level fixtures

i guess a difference here is that you can "mix and match" fixtures more easily between different sets of tests? Pretty minor difference IMHO

[river pilot]

Perhaps plugins?

Xdist is useful

[hexed meadow]

hm, so here's a quesiton

the pytest help shows htis code

@pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")
class TestPosixCalls:
    def test_function(self):
        "will not be setup or run under 'win32' platform"

as it happens, I have a file with a unittest class and two test functions

and it's using skipif

I remove the inheritance from unittest.TestCase or whatever

but now the tests are not being run at all?

[river pilot]

Why did you remove the base class?

[hexed meadow]

because the whole point is to move away from unittest, and the example doesn't have it?

Why would pytest documentation show examples of how to write tests in a way that they're not actually run by pytest?

[river pilot]

I'm not sure. I thought methods would only be run if they were in a TestCase

[hexed meadow]

pytest docs have been a pretty rough experience too ngl

[river pilot]

I can see you've had a bad afternoon

[hexed meadow]

yeah for sure

https://stackoverflow.com/questions/50016862/grouping-tests-in-pytest-classes-vs-plain-functions

here's another example

people writing tests for pytest

as methods of a class

so the class name has to start with Test 🙂

magical

[river pilot]

That is configurable

[hexed meadow]

thanks :-). Don't mean to sound ungrateful or irritated towards you or salt, I really want to emphasize.

you and @pearl cliff have saved me time and I'm grateful

[proud nebula]

Nothing close to how nice fixtures in pytest are afaik

[hexed meadow]

FWIW I'm totally fine with the Test* convention. Just that's like the 3rd or 4th time I've read an example and I didn't realize exactly which parts were "magic" and I wasted a bunch of time

[proud nebula]

Totally different ballgame

[hexed meadow]

idk, only difference I really see is that you can mix and match fixtures more easily, so it's more of a free-form graph rather than having to conform to a tree

[proud nebula]

The nicer asserts and nicer errors are also a huge deal imo

[proud nebula]

"Only" heh. It's a HUGE benefit!

[hexed meadow]

the flip side is that when you use these magic string fixtures, there's no obvious way to track them down; you have to figure it out by grepping basically

[proud nebula]

Cmd click in pycharm to jump to them.

[hexed meadow]

not using pycharm

[proud nebula]

You should :)

[hexed meadow]

nah

it's cool that pycharm has figured this out, but magic strings are just not a tooling friendly approach, compared to something explicit like imports 🤷‍♂️

[river pilot]

There's apytest option to list the available fixtures

[hexed meadow]

like when you run the test suite?

[proud nebula]

The lookup isn't that hard to code. I have implemented it for hammett. 🤷‍♂️

Same file or conftest.py and they are decorated clearly. Not hard to make tooling for that.

But yea, it does require specific tooling. For sure.

[river pilot]

It's a separate option that doesn't run the suite

[hexed meadow]

even easier to re-use existing tooling, like imports. Idk, it's preferences in the end. I guess the python community is much more into dynamic, reflective things because python has a lot of that

I use python pretty statically for the most part; there's aplace for reflection and magic but for me it's going a bit too far with little benefit. YMMV.

[pearl cliff]

i totally understand your frustration, no worries at all.

the reason i use pytest is mainly for: low boilerplate for easy things, lots of control over test collection/selection, the powerful fixture system (despite my dislike of the param name introspection and some other wonky details), the plugin system & ecosystem, and the huge user base

pytest is the perl of test frameworks

[hexed meadow]

LOL

that's probably a great way to sum it up, in the sense that people that view perl and everything it stands for negatively, like me, will feel similarly here.

[pearl cliff]

the difference is that, unlike perl, nothing feature-comparable has really come around that has also gained wide adoption

there is "no python of test frameworks" that is also widely used and actively supported

[hexed meadow]

right, I got that. I'm a practical person and since my team is ahlfway onto pytest and that's the main thing in python, i'll just use that.

[pearl cliff]

it's an acquired taste 😆

[hexed meadow]

i definitely rethought even more of these features when I started using rust unit tests tbh

Like, isn't the simplest way to "use a fixture" just

foo = my_fixture()

🤷‍♂️

I used unit test frameworks in C++ and unittest and they all have a fixture feature, but i just wonder how much value it actually adds

[pearl cliff]

it's because pytest builds a DAG of fixture usages and deduplicates therein, invoking each fixture exactly once per scope

[river pilot]

Do you know about the scooping of fixtures?

[pearl cliff]

so it has to be injected somehow

personally i'd have preferred a decorator

[proud nebula]

I use that stuff in custom code to great effect quite a lot.

[hexed meadow]

right, larger scoped fixtures is a good counter point

[pearl cliff]

it could also be pytest.get_fixture(...) but that's worse than arg name matching, not statically analyzable

a pytest-alike hipster framework could use Annotated

[hexed meadow]

you could do it using types

one of the things about pytest is that it feels very reactionary to unittest copying java unit tests

so it's very "don't use classes ever!"

but like, actually types are great because they're analyzed

[pearl cliff]

nah, it just lets you use classes for optional grouping/hierarchy rather than being mandatory

it also far predates type hints

[hexed meadow]

class MyFixture:
    ...

from some_other_file import MyFixture

def some_function(f: MyFixture)

now everything is easily to analyze, navigate with tooling, and it's still extremely concise

[pearl cliff]

yeah, that would be the hipster next-gen approach (and i agree with you that it would be very useful)

[hexed meadow]

that's hipster?

Then I'm too old to do it 😞

[hexed meadow]

even before type hints, IDE's were decent at navigating around using types, and imports though. Obviously, not as good as with typehints.
there were doc-based type hints as well, long before typehints

[pearl cliff]

that's pretty far afield from modern type hints, and imo would have been even more annoyingly magical than just looking at parameter names

plus not everyone is convinced about "typed python" in 2024, even though you know that i am

[hexed meadow]

yeah for sure, it' snot the only way. Anything that involves explicit imports, IMHO.

[proud nebula]

Unique types for all fixtures? Hmm. Then what? Unpack to get the real data?

[proud nebula]

I just find the tools kinda stupid. https://kodare.net/2023/02/02/names-can-be-so-much-more.html

[hexed meadow]

Idk what you mean by unpack

You define a class, and an init method for it

It has data members and everything

[river pilot]

then you can't have a function as a fixture?

[hexed meadow]

Any unit test that wants that fixture just asks for its type

Right, fixtures would be types

[river pilot]

so it's magical method calling based on types?

[hexed meadow]

Yep

[tired pendant]

[hexed meadow]

Similar to how a lot of dependency injection frameworks work

[river pilot]

i'm not sure why magic by types is better than magic by names

[hexed meadow]

The difference is that types are much different than strings

Hah

I think it's pretty obvious

[river pilot]

not to a python programmer it's not

[hexed meadow]

Try going to definition on a type versus a strong : shrug

[river pilot]

do i have to import the type before using it?

[hexed meadow]

Yes, they're just normal types

[river pilot]

but i don't need to import types for annotations necessarily

[hexed meadow]

Using a fixture you don't import is exactly what makes things more magical and then requires specialized tooling to make accessible

With the approach I suggested the amount of magic is much less

[river pilot]

I guess there's a little less, but there's still invoking the fixture, scoping it, etc.

and it runs counter to Python's duck-typing style

[proud nebula]

If I need an admin User and a standard User in you scheme, that's two classes? And I have to read an attribute to get the thing I want from inside the object?

Types for everything is the product of confused minds imo. It makes no sense if you think about it for even a little while.

[hexed meadow]

another benefit is that of course, the types will be correct

when you have a pytest test, and you are passing it a fixture, you can annotate the fixture but nothign enforces you actually use the correct type

[hexed meadow]

I don't really see the point of general and insulting statements here 🤷‍♂️

it's an engineering problem, the goal is to maximize certain things and minimize others.

we want to minimize boilerplate, maximize readability, maximize reuse of existing tooling, minimize errors at runtime that could be caught earlier and more easily, and so on

the approach I'm suggesting adds very small amounts of boilerplate compared to what pytest is doing, but it's also a lot more explicit and it helps a lot with preventing certain kinds of errors

[proud nebula]

🤷‍♂️ We are all confused by default. It's not an insult.

[hexed meadow]

obviously, it's still a trade-off, and obviously, this approach will never be for people who want to ignore type annotations

[river pilot]

requiring fixtures to be a class sounds like adding boilerplate

[hexed meadow]

what you said came off quite insulting, I think if you didn't intend to insult then you should probably phrase it differntly.

[river pilot]

to be fair, "if you think about it for even a little while" is a bit of an insult.

[hexed meadow]

i was just suggesting using types quite extensively for a unit testing framework, so when you say "Types for everything is the product of confused minds"
it's hard not to extract from this that you're saying that I have a confused mind 🤷‍♂️

and yes, I consider that an insult

[proud nebula]

Your alternative scheme will be quite a lot of code for my trivial example above. Try writing it out.

[hexed meadow]

it is, but I think you can probably recover the boilerplate in the typical case. E.g. (and this is off teh top of my head)

from typing import NewType
import blubtest

FixtureDB = NewType('FixtureDB', DataBase)

@blubtest.fixture
def make_db_fixture() -> FixtureDB:
    ...

so this is what, one extra trivial line of boilerplate, if you can call it that?
but not really, because that line is helping ensure correctness

[river pilot]

definitely a downside to pytest's "alternative" semantics are that other tooling doesn't understand what's happening

[hexed meadow]

(okay, and if you prefer, from typing import NewType... although we could also have a blub.NewType to save that one line 😛 )

yep

with something like this, you wouldn't need any special tooling

you would write, in a completely different file

from some_other_module import FixtureDB

def my_test(db: FixtureDB):
    ...

a) you can just go to definition on FixtureDB in any python IDE and it will work
b) my_test is guaranteed to be type annotated correctly relative to how it's called. If you do def my_test(db: FixtureDB) in pytest, all that matters is the db so there's no guarantee from anything that the FixtureDB type annotation is even correct. The actual call to my_test happens deep inside very dynamic pytest code and won't be type checked

[maiden pawn]

😛 Developer should develop with the language and not by the language.
If language is not supporting proper things for code quality like static typing, then libraries or new conventions should be made to bend the language to a will and make a quality code.

[hexed meadow]

(fwiw, I"m not sure what trivial example you meant)

[maiden pawn]

Imagine Javascript developers developing everything by javascript only rules. we probably would not have stuff like Vscode or Discord and etc

[river pilot]

it sounds like you would like the new test framework: quicknir-test

[hexed meadow]

would you like it, ned, is the question 😛

[river pilot]

i am a late adopter in general, and well-versed in pytest

[hexed meadow]

I'm curious if you think that e.g. this NewType approach compares favorably to what pytest is currently doing for fixtures

Of course. but i'm not asking if you would use it.

Only how you feel about the snippet above.

[river pilot]

honestly, the type being so meaningful runs counter to my intuition in python

[maiden pawn]

Not easily googlable 🤔

[hexed meadow]

This is certainly a fair point

[river pilot]

it doesn't exist, quicknir is brainstorming ideas here.

[drifting sorrel]

Maybe I'm missing something, why making unit tests so complex - and to take it even further - why even use fixtures? I get the need for some "setup/teardown" functionality - but putting a bunch of fixtures in nested conftest modules or somewhere else away from the actual tests seem like bad Develeoper Experience to me. I've worked in code like that is described here, and the main reason that I think is causing it is that the actual code under test - the source code - is so complex. (end rant)

[river pilot]

often you have helpers that you want to use throughout your tests, and you want pytest to manage their lifetimes. fixtures are good for this.

[hexed meadow]

using context managers work well when it's a per-test fixture

but if you have say ten tests, and you want them to share some kind of fixture

then it's a lot harder to do. I mean effectively you'd need to have one big test function, that calls each of the individual little ones

passing in the argument

[drifting sorrel]

I understand that. My solution would be to at least put the fixtures in the same module as the tests needing it.

[hexed meadow]

right, so you can certainly do that

[river pilot]

what if you have tests in multiple test files that need the same fixture?

[drifting sorrel]

That would indicate some duplicated testing, and opportunity to actually delete tests (yes, deleting unit tests is allowed). 😃

Or, that the tests aren't really unit tests but maybe integration tests.

[hexed meadow]

i don't see why having multiple tests need the same fixture automatically proves that there's duplication

or that they're regression tests

that certainly might be the case but it almost might not

[river pilot]

have you ever had helper functions that you used in multiple modules?

[drifting sorrel]

I'm pro TDD and all that since way back but really dislike what seems to be very common usage of fixtures and "magic" with pytests.

[river pilot]

i haven't seen your tests, but I guess you import helpers where you need them, or duplicate helping code in tests.

[hexed meadow]

well, there's degrees of magic. pytest is the most magical test framework I've used by a very large margin.
but fixtures supported somehow have been standard practice in unit test frameworks since forever

[drifting sorrel]

I probably write low level-kind of tests with few external dependencies - or, dependencies that can be monkeypatched (and do that in the test function body usually). I use fixtures sometimes, but use it like setup/teardown in other languages such as JS and C#.

[drifting sorrel]

I'm really picky with not duplicating code 😃 But yes, sometimes writing helpers close to the tests and initialize them with data when needed.

My point is that unit tests should be really simple and really easy to understand when looking at them - and it should be really simple to refactor or change an existing unit test. The fixture magic makes all of that a lot more blurry from my point of view.

[pearl cliff]

the only thing bad about this is that you are kind of lying about what db "is", unless you expect something like a .get_value() to actually get the fixture data

[river pilot]

perhaps it's just something to get used to? Tests often have a DSL-like feel to them.

[hexed meadow]

why am I lying?

[pearl cliff]

well, your code is lying

[hexed meadow]

I mean I wasn't taking it personally, don't worry 🙂

I'm just asking where the lie is

[drifting sorrel]

I'll try! 😃

[pearl cliff]

well before i go there, what are the intended semantics?

[hexed meadow]

Are you familiar with NewType?

[pearl cliff]

oh, i see above

hm... annotating any parameter as FixtureDb automatically invokes the fixture that returns FixtureDb?

[hexed meadow]

Yes. it's similar to pytest's current magic

just using types, which are understood by the entire tooling ecosystem

instead of strings

[pearl cliff]

i think that's mildly worse, because what if i have 3 fixtures that all return the same type?

[hexed meadow]

that's the point of NewType here?

[pearl cliff]

yeah but nothing is stopping me from returning the same type from 3 different fixtures

unless you want (name, return type) to be unique pairs

[hexed meadow]

nothing is stopping you from naming your fixtures the same in different files

the answer is presumably the test framework itself will error out

[pearl cliff]

yeah, so at that point why have two different names that need to be unique?

[hexed meadow]

huh?

pytest requires different names

this approach requires differnt types

pytest uses names... this approach uses types

[pearl cliff]

yeah but what if i do this?

@fixture
def fixt1() -> FixtureDb:
    ...

@fixture
def fixt2() -> FixtureDb:
    ...

def test_foo(db: FixtureDb):
    ...

which one is called? fixt1 or fixt2?

if you wanted to go the types route i think you'd need to do this

@fixture
def fixt1() -> Db:
    ...

@fixture
def fixt2() -> Db:
    ...


FixtureDb: TypeAlias = Annotated[Db, Fixture[fixt1]]

def test_foo(db: FixtureDb):
    ...

[hexed meadow]

what if you do

# one file
@fixture
def db():
   ...

#another file
@fixture
def db():
   ...

in the pytest approach?

[proud nebula]

It's the old Haskell statement that types saves you, which falls apart as soon as a function needs two of the same type and they can't be flipped.

[pearl cliff]

let's say in the same file

[hexed meadow]

but why in the same file?

[pearl cliff]

because there's nothing stopping you!

and there's no semantic indication that it's bad

[hexed meadow]

please, if you want to discuss constructively, can we not reduce everything to hyperbole?

[proud nebula]

Local overrides conftest. As you would expect.

[hexed meadow]

i wouldn't expect that

I would expect an error

🤷‍♂️

what if it's defined in two local files?

[proud nebula]

Shadowing is pretty universal as a thing. It's quite hard to keep names unique in the entire code base otherwise.

[proud nebula]

Then they don't interact at all.

Same as local variables.

[hexed meadow]

@pearl cliff i don't really find this a big issue personally. If you do this, you immediately get an error when you try to run your tests.
If you are using pytest and define functions with the same name twice in the same file, then the second just overwrites the first.
Overall this approach is still going to catch more issues before tests are run, than pytest is.

[maiden pawn]

def fixt1() -> Db:
    ...

def test_foo():
    db = fixt1()
    ...

Simple fix to typing connecting problem. not using pytest fixtures unless u really need them
For fixture scope of function, we can use regular functions

[pearl cliff]

i think my proposal with annotated makes more sense, or injecting a callable as darkwind i think was suggesting

[proud nebula]

You could write this test framework you imagine in an afternoon if you want to test it out. Maybe we're super wrong.

[hexed meadow]

I don't think it does

[pearl cliff]

"autoscope"?

[hexed meadow]

This is well trodden ground, fwiw

[pearl cliff]

i don't see why you're being so dismissive here. it's not helpful.

[hexed meadow]

look at any dependency injection framework

[maiden pawn]

brain mush. i mean for fixture scope of function / per test

[proud nebula]

Errr. I just said I was open to being wrong and encouraging him to prove me wrong. That's the opposite of dismissive.

[hexed meadow]

@pearl cliff fwiw I have nothing against your approach either. Honestly it's probably a little better, because that way fixit is returning the "true" type.

[maiden pawn]

That's rather cool 🤔

[proud nebula]

Discussing back and forth without an actual implementation is not getting anyone anywhere. While writing a test framework like this is pretty fast to do. I wrote my first hammett prototype in an hour or two.

[hexed meadow]

also keep in mind that this is just a shortcut for cases where there happens to be a type that happens to already be exactly what you need.
in a lot of cases, you'll be writing a new class anyway, IMHO. Like if you're doing some kind of mocking; you're more likely to have a Database protocol/interface, and write a MockDataBase for your test, for example

in that case you could skip having functions; as long as MockDataBase has a zero argument init, functions could simply annotate arguments as MockDataBase

[proud nebula]

What about a test that needs an admin user and a non-admin user? How do you handle that?

[pearl cliff]

using fixtures is a good idea. the dependency graph is useful

[hexed meadow]

At any rate there's obviously a lot of details that could be worked out - my main point is that using types it's very easy to do something that's minimal additional boilerplate compared to pytest
but far more type enforced, and far more compatible with existing tooling

[proud nebula]

In pytest that's very clean as it's the name that is the key.

[hexed meadow]

seems like a pretty easy win to me (of course, someone has to sit down and write all this stuff, feature parity, adoption... so not actually "easy" 😛 )

[hexed meadow]

I think the examples above show it?

[proud nebula]

Hmm. Maybe I missed it. There was a lot of messages.

You would have two types?

[hexed meadow]

I assume that these are both of type user; if you want to go salt's approach it would just look like

import blubtest
@fixture
def non_admin() -> User:
    ...

@fixture
def admin() -> User:
    ...

AdminUser: TypeAlias = blubtest.FixtureFunc[non_admin]
RegularUser: TypeAlias = blubtest.FixtureFunc[admin]
...

def test_me(user: RegularUser, admin_user: AdminUser)

that's it

two lines of boilerplate to get proper typing and fit in with standard tooling seems to me a very very cheap price to pay

I dont' know off the top of my head how to implement FixtureFunc, I'd have to look into that

[proud nebula]

Does that work with standard tooling? As in the test function sees an instance of User as input?

[hexed meadow]

That wouldn't be "working with standard tooling" technically here, because the type annotation says they are instances of RegularUser and AdminUser.
but presuming you do it via NewType internally, then every RegularUser is an instance of User.

(that's how NewType) works

[proud nebula]

I think you might be arguing for a different type of custom tooling... so that seems to negate the advantage of the approach.

[hexed meadow]

what custom tooling?

[proud nebula]

This part?

[hexed meadow]

no, i'm saying that your notion of "works with standard tooling" isn't correct

if something is type annotated as RegularUser, then it needs to be of type RegularUser

not of type User

but every instance of RegularUser is a User, and everything will work out correctly

this isn't "custom tooling" - this is stuff that's already part of typing which is the standard library

[proud nebula]

I asked a question. I didn't make a statement.

[hexed meadow]

okay. Yes, this works normally

[proud nebula]

So then "yes it will work with standard tooling"?

The test function sees a User instance and the arg name can be whatever?

And hmm... all such fixture types must be unique right?

I think this might be a fun toy project to try out for sure.

[hexed meadow]

file1

class User:
   def do_thing(): print("hello world")

file2

import blubtest
@fixture
def non_admin() -> User:
    ...

@fixture
def admin() -> User:
    ...

AdminUser: TypeAlias = blubtest.FixtureFunc[non_admin]
RegularUser: TypeAlias = blubtest.FixtureFunc[admin]

file3

from file2 import AdminUser, RegularUser

def test_me(user: RegularUser, admin_user: AdminUser):
    # Prints hello world twice
    user.do_thing()
    admin_user.do_thing()

test_me will print out hello world twice, and test_me will type check properly with mypy

and the person reading file3 can just put their cursor on "RegularUser", hit their "goto definition" shortcut

And it will bring them to file2

[proud nebula]

This was interesting but I gotta go sleep:)

[hexed meadow]

cheers 🙂

[drifting sorrel]

Maybe you already discussed this earlier: would the outcome be about the same if the fixtures to use would be imported into the test module? (and not magically read by pytest during runs)

[maiden pawn]

that's a cool stuff to see. never knew about that 😅
Technically should be not needed feature with static typed connection to fixtures though
Because u can just easily go to definition of the fixtures

[hexed meadow]

You could definitely do it that way

[pearl cliff]

that works for me, although personally i'd prefer Annotated, that's where the "lie" comes in, because admin_user: AdminUser is actually a FixtureFunc, not a User

so AdminUser: TypeAlias = Annotated[User, FixtureFunc[admin] was my suggestion

i think it accomplishes the same thing, but more transparently and in a way that both IDEs and static analyzers can all work together on

[hexed meadow]

I don't know quite how to express what I want; basically I'd have each invocation of blubetest.FixtureFunc return a NewType

with the underlying type being the return type of the passed function

[pearl cliff]

yeah, i think i know what you mean. not sure that's possible with python's static type analysis

that's a #type-hinting question though

personally i think that's what Annotated is good for so i'd still return an Annotated, whether it's a TypeAlias or a NewType

[hexed meadow]

yeah. I mean, i guess my overall point here was: there's probably lots to discuss about how exactly to do it but it's not goign to make an enormous difference

the basic idea is the same; using types rather variable names to connect your fixtures and your tests

and reap all the benefits from that

I just don't want to do Annotated[User, FixtureFunc[admin]]

because User is being repeated basically

admin returns a User so this shouldn't have to be specified

[pearl cliff]

oh i see, yeah that's true

[hexed meadow]

but honeslty I'm not sure any of this is really worth it compared to the simple initial version I gave

[pearl cliff]

well the initial version you gave i think has unclear semantics 🙂

[hexed meadow]

i mean, respectfully, it doesn't.

it's completely clear, it just has a failure mode that you don't like.

I don't really think that failure mode is much of an issue, it's not like it's a subtle thing

[pearl cliff]

it's unclear from looking at the code what's going to happen, and the failure mode is unusual with respect to other things in the python world

but i see where you're going with it, and it does make sense

[hexed meadow]

I disagree with 1, and as for 2, everything happening here is unusual with respect to the rest of the python world

it's far more magical than any "normal" python code

[pearl cliff]

yeah, although a lot of frameworks are doing more magical type things now

pydantic of course, as well as fastapi beyond what pydantic provides, and now sqlalchemy

[hexed meadow]

are they? More to avoid I guess 😛

[pearl cliff]

pydantic is very magical

[hexed meadow]

Yeah I really dislike pydantic

Wish they had been more aggressive about reserving : for type annotations

and not dumping random stuff there

even the relatively simple magic in pandas gives me headaches

@pearl cliff btw have you ever used a dependency injection framework?

[pearl cliff]

not any serious one

[hexed meadow]

@pearl cliff if you understand the concept, then you can basically see that fixtures -> unit tests is exactly the same problem conceptually as as DI frameworks. With DI frameworks, it's objects -> other objects, without manually routing all the different calls

here, we want every fixture to reach every unit test that needs it, and we don't want to call all the unit tests with those arguments, or otherwise route things around

DI frameworks generally focus primarily on type

[drifting sorrel]

Isn’t DI primarily there to make (mostly OOP) code testable? What’s the benefit of introducing DI into the tests themselves? 🤔 Even if it has been a bunch of years since I coded C#, where DI is basically how you develop things, I cannot remember the unit tests really needing that.

[river pilot]

DI is used by tests instead of mocking. maybe that's not what you mean by "DI in tests"

[drifting sorrel]

Seems a bit backwards to me, but maybe I’m old school 😆

[river pilot]

DI is just a fancy word for passing in things to functions that need them, rather than functions creating them or finding them themselves.

[drifting sorrel]

I agree with that. But would avoid that in a unit test. If there’s any place to be explicit, it should at least be in a unit test. I would also keep mocking or patching to a minimum, preferably no mocking at all.

[river pilot]

how would you replace (for example) database access in business logic for a test?

[drifting sorrel]

I would isolate the parts that need a test, not test the part where the DB is accessed.

[river pilot]

doesn't the part that accesses the DB need a test? And how would you isolate those other parts?

[drifting sorrel]

If you replace the DB with a mock, what’s there to test?

[river pilot]

the code that collects the data from the DB. How would you isolate the parts you mentioned before?

[drifting sorrel]

Collects the data … the query? Wouldn’t test that. But if there’s data transformation, that part can be extracted to a separate place (function) and testable.

[river pilot]

hmm, an untested query?

i'm still trying to understand how you would isolate the parts you mentioned before.

[drifting sorrel]

Not sure if I understand what you mean by query, but if that queries a mock, it wouldn’t make sense to test.

[drifting sorrel]

Put parts in separate functions, that do calculations on input.

[river pilot]

tbh, i think that's called DI by people trying to sound fancy.

[drifting sorrel]

DI is about injecting. What i mean is splitting one function or method into several parts, where some of them can be unit tested.

[river pilot]

"injecting" just means "passing an argument" in many cases

[drifting sorrel]

I know that, but you can have a “wrapper” function in the production code - “get user” - that in turn calls one or more other functions. Those can be tested. That’s not DI.

[river pilot]

i'm interested to learn how you decouple components so they can work together in production, but be tested in isolation.

[drifting sorrel]

I usually take a functional approach, and separate the functions into two categories: actions and calculations.

Calculations are the testable ones: data in, data out.

Actions are the ones dealing with I/O and more difficult to test - needing mocks. So, keeping those ones as isolated as possible and focusing on testing the calculations.

[river pilot]

ok, but mocks. got it.

[drifting sorrel]

Also: treating data as data - basic data types and less custom objects.

With that I mean rather save(user) than user.save().

In the first one, user is only data: a dictionary or a data transfer object/dataclass. The second is combining data with behavior, and probably also state. Making things more complicated than necessary, and more difficult to test.

[river pilot]

i understand the pure functional calculation part. it was the mocks for io that i was looking for.

[drifting sorrel]

Ok! The ones dealing with I/O and needing a test I mostly use monkeypatch. Guess that is a mock, but more straightforward as I see it.

[river pilot]

yes, informally called a mock.

whether it's a mock or not depends on what you monkey-patch it with.

[drifting sorrel]

I usually patch with a function returning data. Guess that’s what is called a “fake” maybe?

[river pilot]

i think that would be a stub (canned answers)

fake would be simple but operational.

[maiden pawn]

sqlalchemy offers some typing features?

[hexed meadow]

My point wasn't introducing DI into tests

It was recognizing that DI frameworks (not manual DI) are doing something very similar to what unit test frameworks can do with fixtures

Or at least, similar to what pytest does

[drifting sorrel]

I agree!

[pearl cliff]

yeah, in 2.0 you can declare types like in pydantic, attrs, etc. and sqlalchemy will know what to do with them

https://docs.sqlalchemy.org/en/20/orm/mapping_styles.html#declarative-mapping

from sqlalchemy import Integer, String, ForeignKey
from sqlalchemy.orm import DeclarativeBase
from sqlalchemy.orm import Mapped
from sqlalchemy.orm import mapped_column


# declarative base class
class Base(DeclarativeBase):
    pass


# an example mapping using the base
class User(Base):
    __tablename__ = "user"

    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str]
    fullname: Mapped[str] = mapped_column(String(30))
    nickname: Mapped[Optional[str]]

[maiden pawn]

okay, it does work but your example was not correct fully

https://docs.sqlalchemy.org/en/20/orm/extensions/mypy.html

from sqlalchemy import Integer, String, ForeignKey
from sqlalchemy.orm import DeclarativeBase
from sqlalchemy.orm import Mapped
from sqlalchemy.orm import mapped_column
from typing import Optional
from sqlalchemy import Column, Integer, String, select
from sqlalchemy.orm import declarative_base
from sqlalchemy.orm.decl_api import DeclarativeMeta

# declarative base class
Base = declarative_base()

class Base2(metaclass=DeclarativeMeta):
    __abstract__ = True

# an example mapping using the base
class User2(Base2):
    __tablename__ = "user"

    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str]
    fullname: Mapped[str] = mapped_column(String(30))
    nickname: Mapped[Optional[str]]


class User(Base):
    __tablename__ = "user"

    id = Column(Integer, primary_key=True)
    name = Column(String)

if __name__=="__main__":
    pass
    # "some_user" is an instance of the User class, which
    # accepts "id" and "name" kwargs based on the mapping
    some_user = User(id=5, name="123")

    # it has an attribute called .name that's a string
    print(f"Username: {some_user.name}")

    # a select() construct makes use of SQL expressions derived from the
    # User class itself
    select_stmt = select(User).where(User.id.in_([3, 4, 5])).where(User.name.contains("s"))

    some_user2 = User2(name=123)

    # it has an attribute called .name that's a string
    print(f"Username: {some_user2.name}")

    # a select() construct makes use of SQL expressions derived from the
    # User class itself
    select_stmt2 = select(User2).where(User2.id.in_([3, 4, 5])).where(User2.name.contains("s"))

Defining in those two ways it works amazingly with integration to mypy (needing parameter in mypy.ini for that to work)

I RECEIVED TYPING ERRORS when i was trying to insert wrong ORM attributes or types!

That's... ****** amazing.

Static typing friendly ORM lib in Python.

Also, looks like some mypy typing integration was added to Django ORM too. i must try that.

Backend with types to ORM... that's... great. Even ORMs in static typed languages aren't that great tbh and not having this level of checking

[pearl cliff]

my example was copy-pasted from the sqlalchemy docs 😆

[hexed meadow]

trying to understand my options for converting some unittest style tests

the original code does

class TestFoo(unittest.TestCase):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.bar = ...
        self.baz = ...

    def test_one(self):
        ...

TestFoo is instantiated separately for each test

(that's how unittest works)

I'd rather keep the tests as methods, as there's a fair bit going on here

[pearl cliff]

this is probably the lowest-impact option, in that the test itself doesn't need to be rewritten at all:

class TestFoo:

    @pytest.fixture(autouse=True)
    def _init_bar(self) -> None:
        self.bar = 1

    @pytest.fixture(autouse=True)
    def _init_baz(self) -> None:
        self.baz = 2

    def test_one(self) -> None:
        assert self.bar == 1
        assert self.baz == 2

[hexed meadow]

hmm that's kind of an oof though

[pearl cliff]

yeah, but pytest doesn't like __init__ so that's all you got 🙂

you can of course put them into a single fixture if you want

[hexed meadow]

that's kind of a shame IMHO

in this particular case i feel like init is very natural

It's cool that pytest enables just writing fucntions and mixing and matching fixtures easily

but if you happen to be doing something that fits into a class why make it less ergonomic

[pearl cliff]

iirc it's because of something in the way tests are collected and run. the full range of things you might want to do in a unittest init method aren't supportable by pytest. so they take the easy route of not supporting it at all.

[hexed meadow]

🤨

[pearl cliff]

and you could do this as well if you wanted that init to run exactly once for all test methods:

class TestFoo:

    @pytest.fixture(scope="class", autouse=True)
    @classmethod
    def _init_bar(cls) -> None:
        cls.bar = 1

    @pytest.fixture(scope="class", autouse=True)
    @classmethod
    def _init_baz(cls) -> None:
        cls.baz = 2

    def test_one(self) -> None:
        assert self.bar == 1
        assert self.baz == 2

[hexed meadow]

like, in this case, AFAICS the "idiomatic" solution is to take this class, and remove all the tests as free functions

and then have them accept the class as a fixture

[pearl cliff]

there's nothing wrong with grouping pytest methods in classes. i do it all the time, specifically for class-local fixtures like this (as well as for logical grouping). pytest just refuses to collect any class with a user-defined __init__ method so you need to use an init fixture instead.

class TestFoo:

    @pytest.fixture(scope="class", autouse=True)
    @classmethod
    def _initialize(cls) -> None:
        cls.bar = 1
        cls.baz = 2

    def test_one(self) -> None:
        assert self.bar == 1
        assert self.baz == 2

this might be less offensive at least

or

class TestFoo:

    @pytest.fixture(autouse=True)
    def _initialize(self) -> None:
        self.bar = 1
        self.baz = 2

    def test_one(self) -> None:
        assert self.bar == 1
        assert self.baz == 2

[hexed meadow]

class FooFixture:
    ... # all the exact same code as before, minus the tests

@pytest.fixture
def foo_fixture():
    return FooFixture()

def test1(foo_fixture):
    ...

this hardly seems like an improvement

[pearl cliff]

well that's kind of different from what you described above

[hexed meadow]

why's that?

[pearl cliff]

because you're moving everything to a separate fixture data type instead of just assinging to self.

[hexed meadow]

i know, I'm just saying, it seems like this is the more idiomatic pytest solution, no?

[pearl cliff]

meh

it might be safer for something like pytest-xdist where you're parallelizing tests

otherwise the thing i showed certainly works

[hexed meadow]

the first thing?

[hexed meadow]

this?

[pearl cliff]

the no-self version might be safer with xdist. i don't know how how that works, and i don't know if there might be a data race assigning to self.bar with a function-scope fixture. class-scope seems safe but i don't actually know the semantics.

[hexed meadow]

this is - sorry - much much worse than the original code, because you have to define one method for each field in the class, not to mention that the logic that calculates the fields of the class is intertwined: now you need to try to factor out that logic into more functions, and it's just getting silly how much boilerplate this generates

[pearl cliff]

you can do it all in one fixture as in this example:

class TestFoo:

    @pytest.fixture(autouse=True)
    def _initialize(self) -> None:
        self.bar = 1
        self.baz = 2

    def test_one(self) -> None:
        assert self.bar == 1
        assert self.baz == 2

i would hope that any kind of parallelized pytest setup is smart enough to create separate instances of TestFoo for running test methods in parallel. but i don't know if that's true because i've never parallelized pytest (as much as i've wanted to try, just never sat down to work through how it would work)

i agree that defining a separate fixture for every assignment would be unequivocally worse than __init__. but the example above is functionally equivalent, especially if unittest creates a separate instance of TestFoo to run each test method

https://pytest-xdist.readthedocs.io/en/stable/

With this call, pytest will spawn a number of workers processes equal to the number of available CPUs, and distribute the tests randomly across them.
It looks like it uses parallel processes, so I can't imagine there'd be a data race. Should be safe.

i just pip installed pytest-xdist and that example above worked fine with pytest -n 2 ...

so my suggestion is to replace __init__ with an autouse fixture, and everything else can remain unchanged

[hexed meadow]

Does this run once per test, or once for all of the tests?

i'm pretty sure that pytest will do the reasonable thing wrt to parallelization

[pearl cliff]

_initialize runs once per test in that example. the default scope is "function". can't hurt to explicitly write the scope kwarg though.

[hexed meadow]

cool! I'll give that a shot. Why is autouse=True needed out of curiosity?
also, it feels like in the end you end up with something similar, except with a wierdly named method isntead of __init__

like, essentially __init__ could just ahve this behavior automatically it seems, I don't really see a downside

[pearl cliff]

autouse=True tells pytest to always run the fixture even when it's not requested by any test function. otherwise you'd have to write this:

class TestFoo:

    @pytest.fixture()
    def _initialize(self) -> None:
        self.bar = 1
        self.baz = 2

    def test_one(self, _initialize: None) -> None:
        assert self.bar == 1
        assert self.baz == 2

[hexed meadow]

hmm, I guess the issue is that self does not count as requesting the fixture

[pearl cliff]

or

class TestFoo:

    @pytest.fixture()
    def _initialize(self) -> None:
        self.bar = 1
        self.baz = 2

    @pytest.mark.usefixtures(["_initialize"])
    def test_one(self) -> None:
        assert self.bar == 1
        assert self.baz == 2

[hexed meadow]

I feel like it would be a lot more natural for pytest to just handle it that way automatically

like, if you have tests defined as member functions, that take self, obviously you need an instance

so, by default, you create an instance per test

[pearl cliff]

it doesn't really have a way to tell if self.bar is accessed without invasively modifying TestFoo. and it doesn't have a way to tell that self.bar is an "output" of the fixture, that would be hairy even if you're looking at the AST.

[hexed meadow]

i don't know why it really needs to tell

it should just do it?

[pearl cliff]

you mean, any instance method that's also a fixture should be autouse=True automatically?

as opposed to a staticmethod which would have autouse=False by default

[hexed meadow]

you need an instance to run a method, regardless

[pearl cliff]

that doesn't mean it should be autouse by default though

[hexed meadow]

?