unit-testing

[drifting sorrel]

If you explain what the problem is, we might be able help you out and share ideas on solutions.

[proud nebula]

Step one is to call the endpoint at all so you get some coverage

[lethal forum]

Thanks @drifting sorrel I got the solution from the Docs
@pytest.mark.anyio
async def testsample1_userstatus():
headers = {"Authorization": bearer_token_string}
async with AsyncClient(app=app, base_url="http://test/platform/inapp/v1/Users", headers=headers) as ac:
response = await ac.post(url="/userstatus", json=datasample1_userstatus)
assert response.status_code == 200
assert response.json()['data'] == responsesample_userstatus

This is the sample code for testing the routers I didn't split the code so its just a lame approach🙂

[slate rampart]

Im Working on
selenium project
with python,pytest,pytest-html

need to test SAAS webApp written in Reactjs and Node.js

i want to design framework can you people suggest me best.

[proper wind]

yeah in production

[modest portal]

e aassertion error : asset status 200
unlses you want =instes of = =
also i want to use redirect_function in code possible

[proud nebula]

You forgot a question

[abstract tendon]

With # pragma: no cover, how do I mark a branch without an else? i.e. ```py
if foo:
return

pragma: no cover

raise AssertionError()

This actually seems to work. ```py
raise AssertionError() # pragma: no cover

[proud nebula]

seems cleaner to write assert False

[abstract tendon]

ruff had auto-fixed that

[proud nebula]

wat

https://beta.ruff.rs/docs/rules/assert-false/

well I hate that

There's a reason people discuss removing -O all the time: it's enormously stupid.

[pearl cliff]

i don't agree, but i also think ruff is in the wrong for making this rule

fwiw i think -O is a bad name and it should be buried deeper in the options like --skip-assertions

so i say remove -O but don't remove the functionality

or make it implementation-specific and move it to -X skipassertions

personally i'd rather just disable that ruff rule

not all rules make sense in all projects

[bold solar]

Hey ya'll! Anyone can suggest websites that can help me start my QA career?

[viscid basalt]

All rules are optional 🙂

It's not even a ruff rule

https://github.com/PyCQA/flake8-bugbear - B011

[pearl cliff]

i realized that after, but i would disable that one too if i knew about it first 🙂

ive used bugbear before but typically end up refactoring out unreachable regions like this anyway so ive never hit this one

[viscid basalt]

Honestly I wouldn't generally use asserts in code

[pearl cliff]

i mostly use them with mypy or when there are specific pre-conditions for a function that aren't enforced anywhere in the surrounding code. basically only places where removing them wouldn't change runtime behavior

[viscid basalt]

If you're the only dev on the project - fine, just exclude that rule, otherwise I'd probably make some kind of exception class for technically unreachable code

Yeah, I have some situations like this too

[pearl cliff]

you might need statically-analyzable assertion of unreachability for tools like mypy

[viscid basalt]

I mostly encountered it when dealing with relationships in graphql, we use Foreign Keys in SQL, so object is guaraneted to be there, but our dataloader can technically return Object | None

[pearl cliff]

yup exactly, assert x is not None # Mypy type narrowing

[viscid basalt]

So I raise come kind of exception there:

async def relationship(self) -> Relationship:
    relationship = await load_relationship_somehow(self.id)
    if relationship is None:
        raise UnexpectedError()
    return relationship

But, you know

Db schema could change, I'd like to see that exception in sentry eventually

[pearl cliff]

also a valid way to do it

[viscid basalt]

Or just raise MypyTypeNarrowing(), seems to be pretty self documented 😃

[pearl cliff]

fair, but then i need the if and an extra level of indentation

[viscid basalt]

I personally don't consider it a problem, but yeah

[proud nebula]

I use them all the time. It's pretty great.

It can feel weird coming from C++ like I did, but once you get into it you can see it's really nice.

[viscid basalt]

I don't doubt it, but I generally don't need them or can use something else

[proud nebula]

We have many of them in iommi, where user error gets an assert and a nice error message telling you what went wrong. Developer error as assert is pretty nice.

And in prod it's nice for elses that should never be hit, or unfinished edge cases that might never be relevant, for example. Plus assertions of things you assume of course. The worst is when I see comments that say something is assumed, but no assert. That gives me the creeps.

[viscid basalt]

Well, I'd only use asserts if they would be raised during development

E.g.

def add_router(self, prefix: str):
    assert prefix.startswith("/"), "Message"

[proud nebula]

I use them in prod. Have done for like a decade. It's great imo :)

[viscid basalt]

I do as the linter guides

[pearl cliff]

for what, preconditions?

[proud nebula]

Assumptions. I don't like "preconditions" as a term personally. That implies to me that it's validation at the start of a function and not deep in the function or in a loop or whatever.

[verbal verge]

b = [23,44,87,100]
a = b[1:]

d = b[2:]
c = b
d[0] = 97
c[2] = 77
print(c)
print(b)

Here in last I expected b=[23, 44, 87, 100]
But it became [23, 44, 77, 100]

Can anybody explain this why it happened

[dreamy bough]

@verbal verge What is the name of this operator : in Python when you insert in a list?

[verbal verge]

It's taking from index 1 to end

[dreamy bough]

@verbal verge Yes. This called the slice operator in Python. When you insert in alist your giving the the starting-index and ending-index. So [start-index: end-index]

if you don't specify the ending-index Python will go to the end of the list.

`b = [23, 44, 87, 100]

print(b[1:]) # Outputs: [44, 87, 100]`

sorry about the big fontsize VS Code.

[verbal verge]

My doubt is not here
I have doubt that at last how the b[2] changed to 77 because it had 87

No operations are performed on b

[dreamy bough]

@verbal verge What kind of data type is list in Python ?

`b = [23, 44, 87, 100]
print("b: ", b) # b: [23, 44, 87, 100]
a = b[1:]
print("a: ", a) # a: [44, 87, 100]

d = b[2:]
print("d, ", d) # d, [87, 100]
c = b # c is being assigned to b ... I think both c and b are pointing to the same object in the private heap

so when you change the c[2] = 77 ... same happen to "b" cause they are point to the same array (list in this case ) object in memory

print("c, ", c) # c, [23, 44, 87, 100]
d[0] = 97
print("d[0] ", d[0]) # d[0] 97
c[2] = 77
print("c[2] ", c[2]) # c[2] 77
print("c ", c) # c [23, 44, 77, 100]
print("b ", b) # b [23, 44, 77, 100]` @verbal verge

Python lists are reference types. value types stores the actual value in memory. Reference types store pointers to an object in memory.
Both "b" and "c" are pointing to the same list. I hope this helps. It's easier to see this in programming languages like C# and Java.

[verbal verge]

@dreamy bough
Thank u very much
Your explanation cleared my doubt

[pearl cliff]

right. but i like to distinguish between assumptions that the user is expected to ensure are true, versus assumptions that i know are true as the programmer of the library/application

that is, assert is for internal stuff where failure would be a bug in my code. ValueError is for external-facing stuff where failure would be a bug in the user's code. imo users should never see an AssertionError.

[proud nebula]

A user will see a 500 page for any uncaught exception. So there isn't much of a difference.

[pearl cliff]

in my model, the "user" would be the person writing or debugging the web application code. i like to write my applications with clear public/private boundaries even when i'm the only developer, reserving assert for internal/private things only.

the web analogy would be that ValueError is a 4xx and AssertionError is a 5xx

[marble hatch]

tl;dr: if you have types which re-implement the same behavior but do not share a base class, how would you organize unit tests around them?

there's currently no base class shared among these types, and the implementations are usually highliy inlined as an optimization

specifically, I'm trying to think about how to organize tests for pyglet's color handling

However, this is a general question that applies elsewhere since Python doesn't have a concept of interfaces the same way Java does.

[pearl cliff]

you could define a typing.Protocol for them and organize your tests around testing your various implementations of that protocol

[marble hatch]

that's the approach I want to take, but I think at least one of the projects I want to do it for is wary of type annotations in general

the other idea i've had is some sort of annotation or test generator option, but i think that's dangerously magical

[marble hatch]

after reading your post and discussing it with others involved, I think the best option might be partial or wrapper funcs which return instances of the Protocol implementation as helpers for a pytest fixture

if you want to look at the closest thing to that in the pyglet codebase, it's https://github.com/pyglet/pyglet/blob/master/tests/unit/shapes/test_shapes.py

i'd probably move some of these tests a few levels up while re-using the same fixtures, and add entries for other things which implement the HasColor protocol or whatever its name would be.

ty for your advice.

[marble hatch]

follow-up question: is there a clean way to handle conditional monkey patching?

[marble hatch]

i'm probably going to go with the simplest approach due to having to support < 3.10: ifs and/or a tuple of module prefixes

the latter seems cleaner since it doesn't require isinstance checks

[pearl cliff]

what do you mean by "condition" here?

[pearl cliff]

maybe an abstract base class with register would work

https://docs.python.org/3/library/abc.html#abc.ABCMeta.register

[abstract tendon]

Does pytest support async generators as context manager fixtures?

Seems I need to use @pytest_asyncio.fixture()

[viscid basalt]

I'd use anyio instead
It could test with asyncio+trio if you need that, and generally would be already included if you use something like FastAPI

But both should support async fixtures

[abstract tendon]

Do I just need to set the asyncio_backend?

[viscid basalt]

Yes, it's both asyncio and trio by default

Provided via fixture params

https://gitlab.com/ThirVondukr/fastapi-cookiecutter/-/blob/main/tests/conftest.py?ref_type=heads#L18

Also add it to pytest_plugins

[lament heath]

Could you help me understand why you'd pick anyio instead of asyncio?

[viscid basalt]

Instead?

[lament heath]

Sorry. Wrong wording:

Why you'd pick anyio here?

[viscid basalt]

• I use anyio in some of my projects
• It already comes with fastapi
• I can test with both asyncio and trio backends if I need to

[pearl cliff]

is there any reason not to use anyio?

[viscid basalt]

True

[old venture]

hi everyone
i was wondering, can we use ai ml in automating API testing?
if yes how ?, i would like to do a small implementation of it

[proud nebula]

You can generate code with AI but it's like any other code: you have to have a competent developer check it. Maybe that takes more time.

[abstract tendon]

typing? asyncio.TaskGroup has better typing than anyio.create_task_group()

at least the last time I checked

though if you're not on 3.11 and you're using the taskgroup backport, it's not much better.

but gobot1234 has had a PR open for a few months now. graingert/taskgroup#1

[little panther]

GitHub

[taskgroup] #1 Typehint the public API

[marble hatch]

tl;dr pyglet has functions which return shader program objects in a number of modules. Those need to get monkeypatched during unit tests.

Since they have a consistent naming scheme, I've written a somewhat ugly set of helpers which use inspect to identify them and feed them to a monkeypatch_shaders fixture

I'll consider the ABC approach.

I was hoping to use Protocol types to describe things with batch, color, and other behaviors, but maybe you're right that ABC's register is the way to go

Protocols don't seem to have a good way to support registration out of the box.

[pearl cliff]

good to know, i hadn't actually used asyncio.TaskGroup yet

[pearl cliff]

i see. yeah i think the ABC registration thing makes sense here. or you can do something hacky like a global list of shader classes that you append to

[marble hatch]

ABC registration
My concerns with this are:

1. it mixes testing logic with implementation logic
2. If seems to be duplicate effort if I want to implement a Protocol type

something hacky like a global list of shader classes that you append to
that's the getter probing part

If you want to take a look, this is the part that does it https://github.com/pyglet/pyglet/pull/913/files#diff-54bd039a35025644488eb9fd2fa1265a0dc9851782bda560844b21000d0ed5f8R23-R161

If there are libraries you know of which provide similar monkeypatching helpers, I'd appreciate recommendations

I'm tempted to split this into its own project since it's useful, but I need to do the following first:

1. Make color consistent in pyglet
2. Understand use cases for auto-monkeypatching better

Another idea which straddles the line between good and awful: auto-replace shader getters at import time by extending import behavior

I'm very eager to hear criticism of these ideas

if they sound incomprehensible, I'd love to know that. It'd be useful feedback.

[stray basin]

We have a set of pytest tests that interact directly with a development DB which are relatively slow, during dev work it would be nice to save/cache the fixtures that we don't expect to have changed on the DB side for quicker iteration. These fixtures are static objects with some DB references (uuids, etc) and some static config.

To speed up dev I've just been hacking in a static return for some of these fixtures, e.g.:

# this fixture is used elsewhere; the write_op_x_ids are other fixtures that call out to the DB
@pytest.fixture()
def db_object(write_op_1_id, write_op_2_id, config):
  return DBObject(write_op_1_id, write_op_2_id, config)

# the hack
@pytest.fixture()
def db_object(): # remove all of the other fixtures so that we're not running their creates as well
  return DBObject('uuid', 'uuid', {})

Is there a good way to do something like this in pytest?

I don't think pytest-cache does what I'm looking for without a ton of fiddling

[marble hatch]

Is mocking the db not an option?

The tests you're describing seem more like integration than unit testing

[stray basin]

Yeah, that would probably be ideal, the test suite has sort of grown organically at this point and there's some momentum to overcome before we could change it for the better

[marble hatch]

>>> from unittest.mock import NonCallableMock
>>> class ExampleClass:
...     def __init__(self, a: int, b: int):
...             self.a = a
...             self.b =b
...     def do_thing(self) -> int:
...             return self.a + self.b
... 
>>> example_mock = NonCallableMock(spec=ExampleClass)
>>> example_mock
<NonCallableMock spec='ExampleClass' id='139979131454752'>
>>> isinstance(example_mock, ExampleClass)
True

If you need dunder methods, use the Magic version of that class

[marble hatch]

Since you described caching the instance, I think it will be less work to use mocks than it would be to build it from them from the ground up

If your team has an XML fixation due to Java, you might be able to find or build something Java's Spring for instantiating mocks from config files

[viscid basalt]

If you want to support both trio and asyncio anyio is (pretty much?) your only choice
Typing on create_task_group sucks, ngl 😅 though you can mitigate it by creating a typed wrapper for functools.partial
Also anyio memory objects streams are pretty cool 👀

[marble hatch]

I think there's actually a middle ground:

https://docs.python.org/3/library/unittest.mock.html#unittest.mock.Mock

look at the wraps argument

you can wrap an existing object, which may allow you to give fake results from the db during tests:

wraps: Item for the mock object to wrap. If wraps is not None then calling the Mock will pass the call through to the wrapped object (returning the real result). Attribute access on the mock will return a Mock object that wraps the corresponding attribute of the wrapped object (so attempting to access an attribute that doesn’t exist will raise an AttributeError).

i've never used that particular feature, so you may want to investigate it

I may be wrong

[viscid basalt]

What kind of db you're using?

[viscid basalt]

Honestly mocking db operations is a bad idea, they're in most cases an integral part of your application/business logic

Unless it's really basic crud operations

[marble hatch]

Honestly mocking db operations is a bad idea, they're in most cases an integral part of your application/business logic
For integration and end to end tests, absolutely

but the cached results sound like it's closer to unit testing

[viscid basalt]

I mean, there should be no reason to interact with a remote db during tests in my opinion 🤔

Ideally you'd run your tests in a transaction so DB is not affected at all, but adding fixtures with scope higher than function that interact with db could be tricky

In most cases it doesn't matter though, unless you write quite a lot of data to setup each of your tests 👀

[marble hatch]

agreed, sort of

unfortunately, it's understandable when you do it against a test environment

it makes it easier for people to collaboratively inspect the state of the system when things go wrong

the downside is that it encourages people to not make environments ephemeral

[viscid basalt]

It's not possible to collab using the same db during development

If developers need access to staging db - it's ok, but you shouldn't use it as your dev db

[slate night]

Is repl a good platform

[proud nebula]

you mean repl.it? and good for what?

[slate night]

Collaboration

[proud nebula]

that's increadibly vague

this is #unit-testing btw... so the question seems very strange

[slate night]

My bad wrong channel.

[pearl cliff]

it mixes testing logic with implementation logic
it's not just testing logic though. it's useful at runtime too, although maybe it's not something anyone needs right now.

If seems to be duplicate effort if I want to implement a Protocol type
yeah, this is something i've always wished for: a Protocol that can be automatically generated/derived from an ABC. for now i'd say pick one and not both.

i'm not really sure what you're trying to do with mocking here

as far as i understand, you just want to be able to loop over a list of all classes that implement a certain interface, and run some common test on all of them

actually reading over things, it seems like there isn't even a way to get a list of all registered subclasses for an abstract base class...

so maybe that's not even useful

from typing import Protocol, TypeVar, runtime_checkable

@runtime_checkable
class Getter(Protocol):
    ...

G = TypeVar("G", bound=type[Getter])

getters: list[type[Getter]] = []

def register_getter(cls: G) -> G:
    getters.append(cls)
    return cls

@register_getter
class Foo:
    pass

@register_getter
class Bar:
    pass

this seems like it could work, for example

from myapp.getters import Getter, getters

@pytest.mark.parametrize('cls', getters)
def test_implements_something(cls: type[Getter]) -> None:
    ...

[stray basin]

will take a look at mock, but I think this might not quite fit the use case... it sounds like some of our testing strategy could use a rethink, mocking more DB responses and keeping fewer integration tests

[stray basin]

snowflake, but I don't think that really makes a difference for this case

[viscid basalt]

And you need it to run your automated tests?

[stray basin]

we currently do yeah, a lot of our fixtures write to the dev db

[viscid basalt]

Can you do that with empty/new db?

[stray basin]

we could, but that wouldn't solve the problem that we're running into while developing new features; a lot of our fixtures hit the db to create data that we need in order to develop the new features

[viscid basalt]

Coud you try using local db? Also just how many fixtures?

[stray basin]

there's not really a local version of snowflake we can run, and the issue isn't so much in quantity of fixtures (in the hot path it might be like... 6?) but some of them take a while to compute

[viscid basalt]

6 isn't much

The issue comes from latency between dev machines and dev db

[stray basin]

think of it as 6ish "results" which are a composite of a lot more smaller things

and not quite, it's mostly compute time, latency between db and dev machines is a tiny part of it

[viscid basalt]

Why are you using snowflake btw? 🤔

[stray basin]

it's complicated but it fits our uses quite well

(without getting too much into what we do)

[viscid basalt]

Yeah but it seems to cause you some problems too 🙂

[stray basin]

no, I think the problems are mostly self-imposed; I asked a question about caching fixtures, but it really sounds like we should aim to change how we approach testing a bit

[viscid basalt]

Maybe if these fixtures are repeated and there's support for nested transactions you could use that?

I mean if the same fixtures are used in a lot of your tests

Honestly I didn't have a such problem since I mostly use postgres and not being able to host it locally isn't an issue

[stray basin]

it's complicated; we re-use what we can, but the data we need varies between test cases

[marble hatch]

This idea may also cause trouble, but it's worth considering:
since mocks specced from a type or instance report as of the same type, you might be able to make tests which run in either mock or "real" mode

I think part of my confusion is because the name of this channel is specific to unit testing rather than testing in general

[stray basin]

the channel description is a bit more general than just unit testing 😛

Everything related to testing your Python applications and libraries, and discussion of testing as a whole.

that kind of sounds like what I'd want to enable though

[restive bay]

Would it be possible to use sqlite memory so that instead of mocking and returing fixtures we just test it like the production dB?

[maiden pawn]

sure. it is possible to run db tests even if it is not sqlite engined

you aren't even obligated to use in memory sqlite for testing sqlite db ^_^ in memory db will be kind of helpful though

[maiden pawn]

now a fair warning, don't u dare to use sqlite engine db in order to test code meant for production db that uses another engine like (postgres/mysql/mariadb and etc)
it is a really bad idea

[ember maple]

In cases testing against real db is also done it sometimes is a timesaver,but always with the caveat that not testing against the real DB will eventually turn up really bad surprises ate the worst time

[maiden pawn]

recipe is easy one

1. run unit tests against database with same engine (running in local docker container, if it is postgres/maridb and etc) in CI pipeline
2. deploy to staging and observe it behaves correctly with some data replicated from prod ^_^ confirm everything is okay
3. deploy to prod 😉

[burnt ether]

Hi folks,

Need help framing unit test for method involving calls for asyncio

Python version is 3.7.13

Tried using AsyncMock not working with this python version

[abstract tendon]

Is there a good way to test authentication with authlib?

[ruby moss]

In my team, we are currently discussing our testing approach for a new API I have written. For this API, we collect some data from our database and check if specific entries are missing or available from the DB. I have written some tests that will create an in-memory db with some specific entries to mock. The tests will take the result of the API call, then check if the entries exist in the DB. I have made it such that this algorithm can be used to test every scenario but today I was told that you need to explicitly write the expected result in the test instead of having an algorithm calculate the result and compare with the output of the API. Is this best practice?

[maiden pawn]

it is best practice to assert only expected results, yes.
but if some part of result u don't need to verify, because it is irrelevant to you (for current test), then you just calculate what it will be.
u assert only things that matter for current test. (which is somewhat defined by test name or smth)

common practice goes next one

1. u have code that generates completely random records to db (factory-boy lib)
2. u specify to fixed values only things that matter for you during test
3. u assert only values that matters for you (the rest is random noise)

[ruby moss]

Let's say I have an API that can say if I have all the ingredients for a list of recipes in my database. The user asks for lasagna, then I look up the theoretical ingredients to that recipe, then I check if they exist in the database. I return a list of dictionaries containing the recipe name and a list of needed ingredients and whether they are in my db or not.

My suggestion for testing is that you can take the result of the API call, iterate through the dictionaries, iterate through the ingredients and then check if they exist in the mocked database. If all the ingredients lists are correct, then the API must be correct.

My coworker suggests that we query the API for a dish, then compare it with a prewritten result, like

result = ingredient_api(["lasagna"])
correct = [{"recipe_name": "lasagna", "list_of_ingredients": [{"name": "pasta", "in_db": True}, ...]}]
assert result == correct

[proud nebula]

I would also not mock stuff. Seems like you don't need to do any mocking here. Just create some rows in the db, call some endpoint, check results.

[marble hatch]

Since mock doesn't appear to have a good way to handle attributes on demand, is it acceptable to build your own thin versions of interfaces/protocols specifically for testing?

The context is that I'm still looking at the shader mock issue for pyglet.

[marble hatch]

today I was told that you need to explicitly write the expected result in the test instead of having an algorithm calculate the result and compare with the output of the API. Is this best practice?
writing the expected result instead of calculating does a number of important things:

1. keeps the test readable
2. keeps the test reliable

[marble hatch]

The second one is a big point in its favor, at least in my opinion

tl;dr if you make unit test generation tools, you effectively have a second codebase you might need to test

simple unit tests prevent that and keep the codebase accessible to new and less experienced team members.

[maiden abyss]

Has anyone used CodiumAI for generating UnitTests? I have been looking at it and wonder how efficient it is, and if it is even worth using, obviously its not going to be perfect but it provides a base, any opinions on this?

[weary quarry]

I have not looked at it so this is a knee-jerk opinion. I don't trust AI to generate my code. I can't see myself having any confidence that AI could determine how to test code it cannot understand.

Perhaps if it fit into a bucket of predicable patterns.

[maiden abyss]

True, thanks for the reponse. I was thinking somewhere on that same lines.

[marble hatch]

I somewhat agree with this.

There are two classes of concerns I see:

1. Trusting yet another service with access to code / your repo
2. Result quality

I think some of the quality concerns could be addressed if you used conventions or tools which help provide information about the data being passed, but...

at that point you may as well be using non-LLM tools.

[low oyster]

I'm just starting out with unit tests with my barebones application using pytest and I notice something weird:

Pytest tries to preload all import statements thereby causing my tests to immediately fail because the right env variable wasn't set. I've tried autouse=True and scope=session but the imports are still loaded before the actual test & fixtures are run.

I'll try putting in some min reproducible code but this is what it looks like roughly:

test_my_service.py:

from src.service import MyService
import pytest
... #other important imports

def test_my_service_init():
    ms = MyOtherService(dict(a=2))
    assert ms.value == 2

The test fails on from src.service import MyService because some env var wasn't initialized and I'm not even using that import in this test

[maiden pawn]

1. define sensible defaults for your env vars
2. or just input env vars values 🙂 as wev vars. .env file or at pytest.ini if possible

[low oyster]

yeah I kinda want to avoid the whole hassle of .env files if I can :/

[weary quarry]

Then it's sensible defaults for your env vars and dealing with import side-effects. Loading from the environ on import is something you need to consider. Is that the best time? Can you load on object instantiation or use instead?

[proud nebula]

Use what instead?

[maiden pawn]

best approach is having sensible default env values 😄

aim to have all defaults for unit testing / dev env out of a box after repo clone

modify for staging / prod

[red scaffold]

you guys use pytest?
i'm trying to use this faker fixture that supposed to be available out of the box according to this doc: https://faker.readthedocs.io/en/master/pytest-fixtures.html

but when i run this

def test_faker(faker):
    assert isinstance(faker.name(), str)

with pytest -vv, i get this error

fixture 'faker' not found

[weary quarry]

Faker is a plugin, you have to install it.

Out of the box it defines a global fixture, when installed.

[red scaffold]

ty

[red scaffold]

is xdist still the best plugin for concurrent testing in pytest or is there something better?

[flat phoenix]

Hello 👋
I am making a discord bot. I would love to add unittest to this bot and use github actions to run those tests before merging to main.
So I will need to plan everything about bots first (its almost planned, still dont have a proper written planning, its spread across different places), then write tests, then write code?

[maiden pawn]

Classical waterfall u named from old times

Be agile. Short planning, write tests/code to implement next step.
Again short planning, tests/code, repeat.
Make certain to select features important for first most minimal release (prune it from everything possible). Plan new features into next releases

[proud nebula]

to clarify previous answer: waterfall is bad. You learn by doing. Start doing now.

[muted fractal]

Never been very good at understanding how to layout (small) project.
code is in /code
tests are in /tests
where/how should pytest fixtures be defined, and how to import? I seem to by tying myself in knots figuring that out

[viscid basalt]

There's no definitive way, I usually store fixtures in tests/conftest.py
If you have a lot of them you can create a separate package with "plugins" and register them in your conftest.py

e.g.

tests/
   plugins/
      a.py
      b.py
   conftest.py

# conftest.py
pytest_plugins = [
    "tests.plugins.a",
    "tests.plugins.b",
]

[muted fractal]

I have the same pattern a bunch of times. Is this a decent case for a fixture?

    SAVED_JSON = {}

    def dummy_write_json_file(filename: str, data: dict, indent=4):
        SAVED_JSON[filename] = data

    monkeypatch.setattr("geojson.write_json_file", dummy_write_json_file)
    somelibrary.do_something(blah)
    info = SAVED_JSON["results/ABC/mytown.geojson"]
    assert info["type"] == "FeatureCollection"
    assert info["suburb"] == "MyTown"

[viscid basalt]

if you want to monkeypatch that geojson function in multiple places - yes

[flat phoenix]

okay thanks

[flat phoenix]

thanks 🙂

[elder sun]

Hard disagree, there is no binary solution to approaching software development stategies. Waterfall is simply an approach that will fit in a subset of situations depending on the type of software that a company is making. If something needs to be rolled out quickly, you're not going to use waterfall. There are situations however, where waterfall is the correct or - to say, a more optimized approach, given the circumstances.

[proud nebula]

Like? Even when doing something like a space probe, you will do agile as far as you can in a simulated environment. You can't be too agile after you blast the thing into space though:)

[elder sun]

Have you ever actually used waterfall, or any of these strategies? I've worked at companies where waterfall has been a perfectly viable way of developing software.

[proud nebula]

I have not used astrology either. I can still know it's bad. 🤣

Seriously though, I noticed you didn't answer the question.

[drifting sorrel]

One really bad thing with a waterfall-style approach is that you make a lot of decisions in beforehand and then execute on those. You’ll miss out on the feedback loops and will very likely have made a lot of wrong decisions, since made when you know very little about the thing to develop.

There’s several names for this: Big Design UpFront, RUP and Design-build-test-release.

Luckily, I haven’t come across this in many years, but have worked in organizations using RUP and the V-model 😱. Most software teams at least try to do something more “agile”, even if many only do a daily standup and think they’re agile. Avoid waterfalls 😁

[maiden pawn]

from unittest.mock import MagicMock

my favourite tool to plug any problem 😅

So many times it made my life easier inside and outside of tests

• needing to mock thing in unit tests? MagicMock!
• needing to make solution in case library is not importable? MagicMock! (And then we can keep same code further without making any if complexities)
• needing zero construct for smth object that was errored to be constructed? MagicMock! (And then we can keep same code further without making any if complexities)

Lazy answer for everything

[weary quarry]

If you mock it all with MagicMock, what are you testing? And do you actually use a test tool in implementation code?

[maiden pawn]

yeah, i abuse it for implementation code too

@contextmanager
def open_browser(awaited: Seconds = Seconds(10)) -> Generator[webdriver.Chrome, None, None]:
    chorme_driver_path = Path(__file__).parent.parent / "docker" / "chromedriver" / settings.DRIVER_VERSION
    chrome_options = headless_options if not settings.DEBUG else Options()
    service = Service(executable_path=chorme_driver_path)
    driver: webdriver.Chrome = MagicMock()
    try:
        driver = webdriver.Chrome(service=service, options=chrome_options)
        driver.implicitly_wait(awaited)
        logging.info("opened browser")
        yield driver
    except Exception as err:
        raise FailedOpenBrowser() from err
    finally:
        driver.quit()

for example i had unexpected exception in driver.quit() raised if driver is not defined yet.
MagicMock solved it to have correct code exiting.

i already captured exception in except line after all

=======
Or in django in wsgi.py file i needed to use

import wsgi

C library that is not available unless u run code via uwsgi.
I needed some solution for code to be still runable in order to run django shell and integration tests (with robot)

if "wsgi" is not in sys.argv:
i do wsgi = MagicMock() 🙂

[weary quarry]

That feels fragile.

[maiden pawn]

correction, i check if wsgi arg is not in sys.argv 🙂 then mock it

[maiden pawn]

https://youtu.be/YPN0qhSyWy8
i have mypy running in strict configuration ( https://careers.wolt.com/en/blog/tech/professional-grade-mypy-configuration ). static typing confusion is not really present in code
So it all just works

[weary quarry]

Sure, it works. Piping the output to /dev/null removes all the errors I get running my shell script too. ;)

[maiden pawn]

😛 in case of mocking driver, i already capture exception FailedOpenBrowser if i failed to open browser

MagicMock just removes me from second received exception in finally instruction (that var is not defined), that already duplicates a problem

[weary quarry]

But what are you yielding in that case? A MagicMock? How are you signaling to the caller that there was an issue in open_browser?

[maiden pawn]

if there is issue, they capture raised exception FailedOpenBrowser

everything is fine 😉

[weary quarry]

Type annotations in python are comments. Comments lie.

[maiden pawn]

mypy makes type annotations enforced 😛

even classes checked if they implement abstact things and interfaces (protocols)

for 90% they become not lying

[weary quarry]

Right, but my point is you can lie to mypy. That aside, I guess I don't see the use of MagicMock here on a personal choice, that's all. I'd have defined my driver as a valid option or None (optional) and on the finally I would only close it if it was not None.

[maiden pawn]

problem, it is raising exception on object creation though

um.. yeah, i can capture error from that too right away and to handle

yeah.. not fully justified, i agree

[weary quarry]

@contextmanager
def open_browser(awaited: Seconds = Seconds(10)) -> Generator[webdriver.Chrome, None, None]:
    chorme_driver_path = Path(__file__).parent.parent / "docker" / "chromedriver" / settings.DRIVER_VERSION
    chrome_options = headless_options if not settings.DEBUG else Options()
    service = Service(executable_path=chorme_driver_path)
    driver: webdriver.Chrome | None = None
    try:
        driver = webdriver.Chrome(service=service, options=chrome_options)
        driver.implicitly_wait(awaited)
        logging.info("opened browser")
        yield driver
    except Exception as err:
        raise FailedOpenBrowser() from err
    finally:
        if driver:
            driver.quit()

[maiden pawn]

Zero object saves me from not necessary if constructions and more complicated optional types 🙂
less nested ifs 🙂
if it is safer or not is another problem though

[weary quarry]

Just my experience; Test tools in implementation is a clever solution. Clever solutions are sleeping bugs.

But thank you for sharing your example. It's very interesting and gives me things to think about!

[maiden pawn]

fine

class ZeroDriver:
    def quit(self) -> None:
        pass

@contextmanager
def open_browser(awaited: Seconds = Seconds(10)) -> Generator[webdriver.Chrome, None, None]:
    chorme_driver_path = Path(__file__).parent.parent / "docker" / "chromedriver" / settings.DRIVER_VERSION
    chrome_options = headless_options if not settings.DEBUG else Options()
    service = Service(executable_path=chorme_driver_path)
    driver: webdriver.Chrome | ZeroDriver = ZeroDriver()
    try:
        driver = webdriver.Chrome(service=service, options=chrome_options)
        driver.implicitly_wait(awaited)
        logging.info("opened browser")
        yield driver
    except Exception as err:
        raise FailedOpenBrowser() from err
    finally:
        driver.quit()

more strict solution 😅 Proper zero object without tricks made

missing Protocol for proper type for happiness

too lazy for Protocol writing. Fine like this too.

in this way, yielded object will not propagate to surpress things by accident ever 🙂

hmm, i think mock library has smth to boilerplate thing like that too 🤔 perhaps i can just use regular Mock
nvm, fine as it is

[drifting sorrel]

I know many like mocking, but I’m not a huge fan of it and avoid it as much as possible in unit tests. In situations where it is difficult without it I go for the monkeypatch feature of pytest.

[weary quarry]

MagicMock or monkeypatch. Avoiding it is a good habit. It's only used in projects I work with to replace already tested injected dependencies. Even then, if the injected dependency can be used as-is then we skip mocking it. Only when the setup of the DI isn't worth the value does a mock get dropped in.

Turns out, that's pretty rarely.

[muted fractal]

Maybe the code is badly structured, but I'm adding tests to an oss project, have have to monkeypatch every damn thing to stop it reading/writing files, or talking to the network.... And my old brain doesn't naturally think in dependency injection...

[hexed cloak]

What's the project?

And what sort of network calls does it make, just http?

pytest tmp_path is good for testing file IO

[tranquil pecan]

i need information regarding how to start testing from the scractch please suggest book or tutorial for me as i am a beginner in this testing thing

[muted fractal]

https://github.com/LukePrior/nbn-upgrade-map

See /tests

[mossy ivy]

yoo wsp guys

[rancid tulip]

import ftplib
from pathlib import Path
import os


IP_ADDRESS = os.environ.get("EREADER_IP_ADDRESS")
EREADER_DB_PATH = Path.home().joinpath(".kobo-to-anki/KoboReader.sqlite")

def get_db() -> bool:
    """copy the KoboReader.sqlite database from the reader

    Returns
    -------
    bool
        boolean returned depends on success of copying the KoboReader.sqlite database
    """
    ereader = ftplib.FTP()
    try:
        ereader.connect(IP_ADDRESS)
        ereader.login("root","")
        with open(EREADER_DB_PATH, 'wb') as fp:
            print(ereader.retrbinary("RETR KoboReader.sqlite", fp.write))
        print(ereader.quit())
        return True
    except ftplib.all_errors as e:
        print(e)

Not sure on how to approach writing a unit test. Any pointers?

[proud nebula]

You sure you want a unit test? Seems like it doesn't make much sense. An end-to-end test seems more reasonable.

[rancid tulip]

This is what I thought, but had been told to create unit tests. Perhaps they meant this as an umbrella term. This is one code snippet of my project.

[proud nebula]

If it's a boss/mentor thing, you should ask them, not us :P

[rancid tulip]

How could I go about implementing a e2e test as creating additional tests is very new to me

[rancid tulip]

This is someone on the internet. Personal project 😛

[spark tartan]

hello, Ive been taking an intro to python class for my pentesting degree program at community college level, I just wanted to check on my part cause Im not getting much help from my professor here. I did some homework and finally got feedback that my data validation wasnt complete. I didnt get full credit for the assignment and was disappointed. I am pretty sure I tested for all data types and had screenshots for proof, regardless, not getting feedback from professor, so I want to ask here. What can I do better? I test for alpha, numeric, special characters - which are alpha apparently in my script, what am I missing? Would anyone have a guide on youtube theyd recommend, or a specific search query I should consider? afaik I am doing all I can, just need some tips or validation.

[maiden pawn]

Eh? testing edge cases.
kind of very specific subject

in general for application edge case testing ends at testing one positive and one negative scenario
and optionally extending to test more throughly with using Fakery lib to generate all possible data within allowed constraints (Optionally schemathesis can be used)
And optionally using mypy to have more proper static types to make sure proper types are going
And data validating in addition with pydantic + framework capabilities

So... essentially we check code just for all types of its logic branches.

If there is if condition to have some other logic of this code section? Then new test is required. Otherwise, not really

[spark tartan]

thanks

[maiden pawn]

TLDR: real life applications are tested in an time efficient way.
Only for stuff that is important (logic flow changes)

for all the rest errors, we invest into monitoring/logging/error tracking systems to react later

Overengineering in testing too many edge cases is not appreciated i think
Unless it is trully justified by the domain

[spark tartan]

• Proper entry and functioning of valid data
• Invalid data attempts on all inputs, include “fuzzy” data attempts
• Demonstration of multiple entries and accumulation.

I think I am doing every possible combination as far as allowed type of data for entry when prompted from user,

[maiden pawn]

Real life applications rely on other libraries/databases that already tested that functionality of types work correctly.
So we exclude in addition necessarity to test anything that is already tested by external parties 😄

We test our own applicationa and how it integrates with third party libs/systems
but we do not test third party libs and systems themselves

[spark tartan]

ok

[mossy ivy]

Hello

[spark tartan]

well I didnt show you but I do copy my code, along with screenshots for valid data entry to show the math works and that all areas of the program work. I them do the invalid data tests, covering all areas of the program. I just dont know what my prof is talking about. the assignment doest say, every possible combination, says every possible data type. There are only so many data types. Its a simple program, and theres only a dozen or less areas of control or input/ouput. I just dont know what more I can do.

[spark tartan]

hey

[maiden pawn]

words are words, show the code, then we can see what can be improved

[spark tartan]

well i cant attach the file

I have everything in a pdf with my personal info redacted

ok I can do it one sec

https://paste.pythondiscord.com/I2EA

well I cant attach the screenshots

so this program is short,

we are doing another one, a girl scout cookie order program, which is much longer and has twice the input prompts as well as conditionals. So The GSC order program has more testing.

Could this be why prof said, this program didnt test well enough? cause I did more for the other program?

I even tested the exit points

[maiden pawn]

i love refactoring the shit out of such things 😄
Question => Are you restricted in which python versions, standard libraries and third party libraries you are allowed to use?
And requirements for a program u have? (Task specifications?)

[spark tartan]

idk, but this is a basic intro course

I can use whatever IDE, im using codium cause its FOSS

I dont have any fancy tools installed, just the basic extension that allows me to use python.

I have python2 and python3 installed, we are using latest version.

I have no idea about libraries, prof says we can use idle, or pycharm, cs code, etc, and didnt mentions libraries or tools, or extensions.

He doesnt like that I come here on discord to get help

but he doesnt answer my questions so I go where I go

I have two screenshots for hte calorie program, and 5 forh te cookie order program.

[maiden pawn]

Is there any specific defintion of the required task?

[spark tartan]

the calorie program is modleed from his lecture videos, the gsc order prog is the one that requires the thinking

yea, test everything, fuzzy data testing, valid, invalid, all data types, at all points of the program, the y/n, for yes or 1, $, -0, etc . . . so on. I didnt test for every combination of character but only every data "type" be alpha, special character, numberical, I didnt do float . . . Im just wondering

did negatives, 0, -0

etc

we want only valid data entry

security this way

I guess we learnign this cause we are in a cyber secuirty program

strong data validation I mean

last time I failed this class so this time, I m getting it and its all clicking but prof sets me back, like Im not doing enough. so Im just trying to do my part and say okay, what am I not doing. 🙂

though he wont answer that

arf

I mean in comaprison to the two assignments handed in for that week, its all compiled into one word doc = pdf, so the second part has 5 pages of validation, the firt part only two, but what more can I do on the first part?

tehres only so many data types

Im on this cause Im about to start testing this weeks program

and this part is worth 100 points this week.

https://paste.pythondiscord.com/7ICQ this weeks assignment

for that calorie counter program, no cookie program this week

we are on functions, 2nd week

[maiden pawn]

i'll give it all check and suggest to restructure it once i get free from work shift

ergh... going to be a bit of spoon feeding, but lets say... try to follow example for later 😄

[spark tartan]

ok

can send me dm if you like or @ me here

[maiden pawn]

i'll try to go without third party libraries i assume. looks like it will be overkill

clarify your python3 exact version

[spark tartan]

latest version

(3.10.6-1~22.04).

[maiden pawn]

not really latest btw. but it is not important. i use same Major and Minor version

[spark tartan]

I just install codium from flatpack, installed pop os, updated system,

so I dont install anythign specific

im noob still, only thing I like is the dark high contrast of codium 😛

[maiden pawn]

wishing to relax from work.
Lets see...

infinity program with CLI interface reacting to user events
possible user scenario logics:

• adding item or not
• adding item carb/fat/protein

for sane writing of a program, i would be really wishing to use pexpect library tbh (as minimal requirement)
https://pexpect.readthedocs.io/en/stable/overview.html
Otherwise proper code to test it will be possible to write only in a more limited fashion

thinking u can handle pip install pexpect? 😄

[spark tartan]

yea

its apt install but one sec

[maiden pawn]

no, its pip.

its different

(ideally u need to use venv though)
like... python3 -m venv venv # to create venv
source venv/bin/activate # to activate venv
pip install pexpect # to install lib
python3 app.py # to run

[spark tartan]

pip install pexpect

sudo apt install python3-pip

[maiden pawn]

ergh, it will work

we can go without venv 😉

[spark tartan]

apt install python3.10-venv

trying to find app.py

[spark tartan]

got venv installed but the last command cant find app.py

[maiden pawn]

because we did not write app.py yet. in progress 😄

[spark tartan]

sounds good

just finishing up this weeks assignment, fuzzy testing

tidying up the errors with code scheme

[maiden pawn]

https://pastebin.com/gXBUyJbz

would you like to add another item?> (y/n) asdf
usage: babyraptorjesus_after.py [-h] {y,n}
str(err)="babyraptorjesus_after.py: error: argument answer: invalid choice: 'asdf' (choose from 'y', 'n')\n"
would you like to add another item?> (y/n) y
Enter item name?> asd asd
str(err)='must not contains spaces'
Enter item name?> banana
Enter grams of carbs?> asd
usage: babyraptorjesus_after.py [-h] answer
str(err)="babyraptorjesus_after.py: error: argument answer: invalid int value: 'asd'\n"
Enter grams of carbs?> 3
Enter grams of fats?> 6
Enter grams of proteins?> 5
would you like to add another item?> (y/n) y
Enter item name?> kiwi
Enter grams of carbs?> 3
Enter grams of fats?> 5
Enter grams of proteins?> 7
would you like to add another item?> (y/n) n
this meal has 2 for 171 calories

solved with style 😄

to have smooth retry, and defined rule checks

is it better readable? 😄 or i failed

easy to add a lot of validating rules (already validates a lot though)

[spark tartan]

I have it saved in my ide, and am looking at it. 🙂

there are a quite a few things in here I am new to,

cool, I have some stuff to learn

looks nice, advanced from my level of course

its nice to see a more advanced view at this problem set so I can see differetn ways of solving same problem

[maiden pawn]

i tried to make clean program flow of logic 🙂

class Session:
    def __init__(self):
        self.items: list[Item] = []

    def run(self):
        while True:
            answer = MyParser().add_argument(choices=["y","n"]).ask_question("would you like to add another item?> (y/n) ")
            if answer == "n":
                break

            self.items.append(Item())

        print(f"this meal has {len(self.items)} for {sum([item.calories for item in self.items])} calories")

much cleaner looking
as well ad item addition and etc

[spark tartan]

learning the structure and syntax

thanks,

hope it didnt take you long t odo this

[maiden pawn]

i did not write unit tests though, thought it is overkill (they should be ideally written too)

[spark tartan]

I would like to learn to do the same thing,

unit tests, ok

so what you just did is called, refactoring? like rewriting the code to be more efficient and simpler, etc?

[maiden pawn]

Refactoring => umm... tried to write it more readable / cleaner / simpler / better reusing logic / easier to extend with new logic

[spark tartan]

yep, good to know

cool

[maiden pawn]

u can see that in this program, u can easily add a lot of new features / items / rules

room for growth is way bigger, it will be still within human brain comprehension with new code additions

[spark tartan]

right

this is the first lecture for functions, so we only had a couple,

this weeks, https://paste.pythondiscord.com/45LQ I had to create 10 functions.

this is my rewrite of last weeks program

with this weeks requirements, etc

[maiden pawn]

well. i can complete solution by writing few unit tests just to be sure 🤔

then it will be 100% complete 🤔

[spark tartan]

if you want to, feel free!

but I hand in my code for the assignments

[maiden pawn]

well. screw it. lets stop it this point

https://docs.python.org/3/library/unittest.html
https://pexpect.readthedocs.io/en/stable/overview.html
just to be sure

the point of unit testing, to write scenarios to execute program

[spark tartan]

links saved

[maiden pawn]

and automatically checking that it fits expected logic

for positive and negative scenarios

[spark tartan]

I have no idea what this is

[maiden pawn]

and after you refactor program

that program is still working as expected

[spark tartan]

perfect

so this checks the code?

like an automated data validation test?

rather than human

[maiden pawn]

yeah. Read this book, it explains logic behind it

[spark tartan]

cool

[maiden pawn]

And this book shows on practice

[spark tartan]

will do 🙂

tyvm!

oh man best gift ever, today my bday got new books to read!

im sure i can find free pdf of these

[teal quartz]

Has anyone ever added custom markers in a junit report created by behave ?
In pytest, when having a custom marker registered in pytest.ini like test_id, enables you to pass a string constant to pytest tests that are then included into the junit xml file; that would look like this:
@pytest.mark.test_id("666") which would be put just before the function definition of the test executed by pytest.
I want to do the same in the junit report created by behave.

[forest palm]

I am new to python flask but while trying out app.testing, I am getting a value of false due to which my ReCaptcha is not getting validated. Any suggestions/solutions?

[maiden pawn]

your ReCaptcha?

how much is it yours ReCaptcha?

[spring sparrow]

we live in a society

[solemn nacelle]

Hi, this is a basic problem and I am actually not sure of what I did wrong. I am learning how to write tests for code using pytest. I wrote the function below in a file called calculator.py:

def main():
    x = int(input("What's x ? "))
    print("X squared is ", square(x))

def square(n):
    return n * n
if __name__ == "__main__":
    main()

And then I created a test file which I called test.py and wrote the following:

from calculator import square 

def test_positive():
    assert square(2) == 4
    assert square(3) == 9

def test_negative():
    assert square(-2) == 4
    assert square(-3) == 9

def test_zero():
    assert square(0) == 0

def test_str():
    with pytest.raises(TypeError):
        square("cat")

But when I run the test file, it doesn't run properly. It doesn't read the test file. What have I done wrong?

[weary quarry]

What command are you using to run the tests?

[solemn nacelle]

pytest

[maiden pawn]

U a missing pytest.ini to set more custom names for test file names.
I don't know default without it
tests.py?
test_smth?
smth_tests?

https://github.com/darklab8/darklab_darkbot/blob/master/OLD_CODE/pytest.ini
Found example
Write pytest.ini with discovery

[solemn nacelle]

Yeah, thanks a lot. I have been able to fix it with some help from someone here.

[vapid dome]

Hey folks, I've got a question. I have a number of fixtures, and I want to run the same test for each fixture. The trouble is, many are parameterized, and pytest keeps generating one test per combination of fixtures. Ideally, I'd like to pass in the fixtures in as @pytest.fixture(params=list_of_fixtures), except then the test doesn't create them as a fixture, but rather just another parameter, and I'm unclear how you'd tell it "hey, that's a fixture, treat is as such"

[weary quarry]

Can you show an example of what you are trying to do?

[hexed cloak]

How do people test rich widgets? I'm thinking snapshot testing is there a demo anywhere?

[maiden pawn]

clarify widget meaning and stack of technologies

[hexed cloak]

Something that uses rich.Live with a get_renderable and an update method?

[maiden pawn]

not really telling much. Beyond a guess that u probably use python backend framework for full stack purposes and smth happens in rendered frontend

u should be more specific what is happening

like what do u use for backend, for frontend is a good start (with example of code u wish to test)

[hexed cloak]

https://inspector.pypi.io/project/coiled/0.9.17.dev13/packages/7e/6c/0af7f551ebacd9d5755f9c86f6755d26304fe307c297d168f4eb514fd405/coiled-0.9.17.dev13-py3-none-any.whl/coiled/v2/widgets/rich.py

https://inspector.pypi.io/project/coiled/0.9.17.dev13/packages/7e/6c/0af7f551ebacd9d5755f9c86f6755d26304fe307c297d168f4eb514fd405/coiled-0.9.17.dev13-py3-none-any.whl/coiled/v2/widgets/rich.py#line.117

@maiden pawn I'm looking to render to a text file instead of a terminal

[maiden pawn]

looks like a matter of simple unittest/pytest function to test a single function to me. nothing really hard

u just write unit test, that invokes this function with different parameters and asserts results

feel free to parametrize to check multiple scenarios with reusing same code

[hexed cloak]

The result is something written to the terminal in a thread

[maiden pawn]

this function pretty much does not have any dependencies to need any other testing

the function in question is nasty in having too many if/else conditions

your test is supposed to be having no less than 20 or more parametrize variations 🙈 to test all this logic branches

or u should rewrite function to posses less if conditions

[hexed cloak]

Happy to take suggestions on that

Is there a function to take a rich.RenderableType and render it to a io.StringIO/BytesIO?

[maiden pawn]

consider using datetime.datetime formatter to output human readable datetime/timedelta?

[hexed cloak]

Oh is there a nicer version of format_seconds?

[maiden pawn]

write me example of expected input and output. throw some examples

[hexed cloak]

It needs to match the handwritten code in the JavaScript

[maiden pawn]

why and which one (write your JS code that expects certain input)

[hexed cloak]

"1 day 2 hours 3 seconds"

[maiden pawn]

can't u just return json at python, accept at Javascript JSON, convert to object, and taking from it days/hours/seconds normally to other code logic?

[hexed cloak]

Eh

The API gives seconds as a float

Are you suggesting to format it in the server rather than the JS web UI?

[maiden pawn]

what do u use for JS web gui for starters

vanilla JS?

[hexed cloak]

It's react see https://cloud.coiled.io/

[maiden pawn]

anyway, my suggestion is simple.
Remove all the if conditions of over complicated logic
Convert as early as possible your datetime or time delta into pure values days, hours, seconds as a pydantic BaseModel

Then make decision and either return with datetime.datetime formatted datetime/timedelta right away
Or optionally if u want to make nice output at front, then return isoformat or json of the made values in the BaseModel struct to front
and then format with some standard library your output (not with custom madness of if conditions 😄 )

whatever u do, just do with less if conditions and more reliance onto standard libraries that already solve it

and add obviously at least a single unit test to backend, and single test to frotnend that it works correctly

P.S. backend could generate openapi/swagger docs, and autogenerate typescript of backend endpoint for your react front 😉 https://github.com/drwpow/openapi-typescript
It will simplify/remove need in extra testing

[hexed cloak]

I just want to check it renders with some predetermined JSON

Oh I can just use
https://rich.readthedocs.io/en/stable/reference/console.html#rich.console.Console.export_text

[viscid basalt]

@river pilot Sorry for pinging you, but I saw that you're online right now.
Is that a valid behavior of coverage.py? 🤔

@strawberry.mutation
@inject
async def create_book(
    self,
    input_: Annotated[BookCreateInput, strawberry.argument(name="input")],
    command: Annotated[BookCreateCommand, Inject],
) -> BookCreatePayload:
    result = await command.execute(dto=input_.to_dto())
    if isinstance(result, Err):
        match result.err_value:
            case BookAlreadyExistsError():  # Line 30
                return BookCreatePayload(
                    result=None,
                    error=EntityAlreadyExistsErrorGQL(),
                )
    return BookCreatePayload(  # Line 36
        result=BookGQL.from_orm(result.ok_value),
        error=None,
    )

Name                                            Stmts   Miss Branch BrPart   Cover   Missing
--------------------------------------------------------------------------------------------
src\adapters\graphql\apps\books\_mutations.py      22      0     12      1  97.06%   30->36
--------------------------------------------------------------------------------------------
TOTAL                                             368      0     88      1  99.78%

[river pilot]

can you put the code in a pastebin with line numbers?

[viscid basalt]

Sure!

[river pilot]

(oh, you have them)

[viscid basalt]

😅

[river pilot]

perhaps the error is always a BookAlreadyExistsError?

[viscid basalt]

It is 🤔

[river pilot]

ok, then case never branches to return

[viscid basalt]

If the case itself contains return should coverage even report that?

[river pilot]

why shouldn't it?

why do you have a match case that always matches the first case?

[viscid basalt]

Hm, I'm not really familiar with how branch coverage should work, that's why I'm asking 😅
result itself is Union[Ok[Book], Err[BookAlreadyExistsError]]

[viscid basalt]

To have proper error handling with result library

If that's expected behavior I may just add pragma: no branch here

[river pilot]

that's one option. another is to assert the type of the error, and remove the match/case

[viscid basalt]

I don't quite understand what you mean by assert 🤔

just plain assert isinstance(...)?

[river pilot]

yes

one way to look at it is that you have a bug in your code: what happens if you get a different kind of error?

[viscid basalt]

But it may have different type if I change what that execute method returns

[river pilot]

ok, and you aren't handling that case.

[viscid basalt]

Mypy would yell at me

[river pilot]

coverage is already yelling at you 😄

[viscid basalt]

src\adapters\graphql\apps\books\_mutations.py:37:37: error: Item "Err[BookAlreadyExistsError | IndexError]" of
"Ok[Book] | Err[BookAlreadyExistsError | IndexError]" has no attribute "ok_value"  [union-attr]
                result=BookGQL.from_orm(result.ok_value),

[viscid basalt]

e.g. IndexError is not handled

[river pilot]

to answer your original question: this is what coverage is meant to do: there's a possible path through your code that isn't exercised.

[viscid basalt]

Yep, thanks for your time, just making sure I'm not missing something 👀

[proud nebula]

Mutations? What tool are you using?

(I'm the mutmut guy so mutation testing makes my ear point)

[viscid basalt]

Strawberry-graphql 😉

I'm still waiting for mutmut to support 3.11

[proud nebula]

Me too :(

[viscid basalt]

Use sqlalchemy

It's free 😃

[proud nebula]

They have a file called _mutations.py?

[viscid basalt]

It's my file, graphl uses "mutations" to change data:

@strawberry.type
class BookMutation:
    @strawberry.mutation
    @inject
    async def create_book(
        self,
        input_: Annotated[BookCreateInput, strawberry.argument(name="input")],
        command: Annotated[BookCreateCommand, Inject],
    ) -> BookCreatePayload:
        result = await command.execute(dto=input_.to_dto())

        if isinstance(result, Err):
            match result.err_value:
                case BookAlreadyExistsError():  # pragma: no cover
                    return BookCreatePayload(
                        result=None,
                        error=EntityAlreadyExistsErrorGQL(),
                    )

        return BookCreatePayload(
            result=BookGQL.from_orm(result.ok_value),
            error=None,
        )

[proud nebula]

Mutmut 3 doesn't need an sqlite db at all. And it's way way way faster.

[proud nebula]

Oooh. Yea ok.

[viscid basalt]

Very interesting 👀

I guess the bottleneck would be the test suite anyway

[proud nebula]

There's a branch for the WIP. It's just that I have a high tempo job at a startup now so not much time/motivation to work on it. It basically needs the UI plugged in and a bunch of dead code cleanup.

[proud nebula]

Mm3 has a fork model that can make it parallelized like crazy. Drops support for windows and many mutations but imo worth it.

[viscid basalt]

You should take your time

[viscid basalt]

I mainly use windows 🙃

[proud nebula]

Yea well... trying to revolutionize the home renovations market. Hugely important.

[proud nebula]

...I'm sorry

[viscid basalt]

lol

[proud nebula]

That windows-linux thing should be good enough though I guess?

[viscid basalt]

It would be a pain to use WSL instead of just running something in a terminal tbh

[proud nebula]

Meh. Maybe. But imagine it going literally 10x faster. That's what I'm looking at.

[viscid basalt]

@proud nebula You're pretty experinced with django, right? 🤔

[proud nebula]

Yea. You should use it. And iommi ;)

[viscid basalt]

https://gitlab.com/ThirVondukr/aioinject/-/tree/main/tests/ext?ref_type=heads
What would be the best way to test my library with a django app there?

With most ASGI frameworks I can just do something like

@pytest.fixture
def app(container: aioinject.Container) -> Litestar:
    return Litestar(
        plugins=[AioInjectPlugin(container=container)],
    )


@pytest.fixture
async def http_client(app: Litestar) -> AsyncIterator[httpx.AsyncClient]:
    async with httpx.AsyncClient(app=app, base_url="http://test") as client:
        yield client

[proud nebula]

Oh I have no idea about async in django. Sorry :)

[viscid basalt]

I actually wanted to test sync django, as async is in really rough shape atm

[scenic atlas]

I have a fixture, which returns a configuration dict. I want to parametrize my test on the values of a particular key. Eg:

@pytest.fixture(scope="session")
def config():
  return {"greet": ["Hola", "Hello", "Hi"]}  # just a smapple, actually generated from config files

#####
@pytest.mark.parametrize("greet", HOW_TO_GET_THE_LIST_HERE)
def test(greet):
    # checks if the greetings fit some criteria

I can perform a loop inside a normal test, but this approach as a few drawbacks such as:

• hiding the details when a test fails (for which input?)
• linear instead of parallel execution

[maiden pawn]

can i ask you not to use Dictionary as a common data carrier 😄 that should be the first order of business in my opinion.

use Pydantic BaseModel. it is really great

enforces types

and when u need at last moment submitting it as dict or json, it has func to transform it into it

[scenic atlas]

It is not a dict, I'm just making a simple-to-pose question

[maiden pawn]

@pytest.fixture(scope="session")
def config():
  return {"greet": ["Hola", "Hello", "Hi"]}  # just a smapple, actually generated from config files

@pytest.fixture()
def iterated_greet_key(config):
  for value in config["greet"]:
    yield value

#####
def test(iterated_greet_key):
    assert iterated_greet_key == smth

i think smth like this exists

[scenic atlas]

OO, lemme test this

[maiden pawn]

nope. Function has more than one Yield. error 🤔

[scenic atlas]

Yup

[maiden pawn]

probably it means using generating tests function perhaps

Or..

[maiden pawn]

import pytest

keys = ["Hola", "Hello", "Hi"]

@pytest.fixture(scope="session")
def config():
  return {"greet": keys}  # just a smapple, actually generated from config files


#####
@pytest.mark.parametrize("greet", keys)
def test(greet):
    assert greet == "Hello"

can't we just go like this? 😄

feel free to use Singleton to load those values at last moment

[scenic atlas]

Sadly, no. The config is actually generated from files, so we don't know the values in the list

I mean, yeah

The session fixture is a singleton as well

[maiden pawn]

let me correct u to code that will fit it, a moment

perhaps to try generating tests function? 🤔

[scenic atlas]

Any examples?

[maiden pawn]

cooking up example, a moment

[scenic atlas]

Ah, thanks. I haven't tried generating test functions ever

[maiden pawn]

import pytest

def create_config():
    return {"greet": ["Hola", "Hello", "Hi"]}

@pytest.fixture(scope="session")
def config():
  return create_config()  # just a smapple, actually generated from config files

def pytest_generate_tests(metafunc):
    if "param1" in metafunc.fixturenames:
        metafunc.parametrize("param1", create_config()["greet"])

#####
def test_compute(param1):
    assert param1 == "Hello"

[scenic atlas]

And test_compute will fail on Hola and Hi?

[maiden pawn]

Yup.

[maiden pawn]

E       AssertionError: assert 'Hi' == 'Hello'
E         - Hello
E         + Hi

test_abc.py:16: AssertionError
============================================================================== short test summary info ===============================================================================
FAILED test_abc.py::test_compute[Hola] - AssertionError: assert 'Hola' == 'Hello'
FAILED test_abc.py::test_compute[Hi] - AssertionError: assert 'Hi' == 'Hello'
============================================================================ 2 failed, 1 passed in 0.02s =

3 tests executed, 1 passed, 2 failed 🙂

[raw field]

Hi all. I need some help with Pytest indirect fixtures. My main issue is passing an indirect value to a fixture 2 levels deep. Namely:

@fixture
def test_db(request, tmp_path):
    db_name = request.param
    return f"sqlite:///{tmp_path}/{db_name}"


@pytest.fixture(autouse=True)
def with_test_settings(test_db):
    os.environ["database"] = test_db

@pytest.mark.parametrize(
    "test_db",
    ["my_test_db.sqlite3"],
    indirect=True
)
def test_my_db():
    # Do something and the `DATABASE` env variable should be set to `my_test_db.sqlite3`
    pass

I hope y'all follow what I'm trying to do. I am open to clarifying anything if need be. Thanks in advance!

[raw field]

Problem solved! It was something rather silly, I was running my test suite and was running a test that required test_db was failing that I had not yet parameterized. I didn't notice!

[cyan cliff]

Hello I am writing tests for my django application, I wonder what is the good practice? Because currently I am testing for absolutely everything. I have a main page with several tabs where the user clicks to be redirected to another page, I wrote a test script with selenium which tests absolutely all the tabs of the page but suddenly it takes a lot of time. .. Is this the right method to adopt?

[tall brook]

I always suggest this - https://www.obeythetestinggoat.com/

I don´t follow TDD but you can still learn and follow lots of good practices applied there

[lusty sparrow]

as a beginner to testing, is it better to start with either pytest or unittest?

[tall brook]

unittest provides more tools than pytest. unittest requires to understand less to achieve same result. Although I prefer pytest for its simplicity, how you can organize code with it and output format

[maiden pawn]

This book exactly tells what is best to test and what is not to test

golden book. literally golden

and yeah... writing selenium tests as main ones.. is not the best approach

they are needed sometimes... but still in average they are e2e tests which are too expensive to use

pytest is far better

[cyan cliff]

ok thank you

do you know if i can download it for free ?

[maiden pawn]

🙈

piracy is piracy. not going to recommend stuff

[cyan cliff]

bro because of 2008 i don't have any money left

[pearl cliff]

how do i choose between mutmut, cosmic ray, and mutatest? i know the mutmut author posts here occasionally, would appreciate any guidance on choosing among these. or is it just a "pick one and get started" situation?

[proud nebula]

I built mutmut because I couldn't even get cosmic ray to run. It's become a lot better since then though, mostly by copying mutmut ;)

Mutatests design sounds iffy based on the FAQ. Claims multi core but then also says "Since mutatest operates on the local pycache it is a serial execution process." Which is it?

I would try all of them. You want a very small start project to start with. Like 40 lines.

[pearl cliff]

I see, thanks. I have an existing mature project but I'm not confident that the tests are really testing what we expect them to test, so I was hoping to deploy mutation testing on a small section of business-critical code. Do you know if one of the mutation testing frameworks supports that?

I can of course start by trying it on a toy project

[serene basalt]

I have one branch that I need to run tests on locally on my computer but it will take 30 minutes for it to complete. I have another branch that I need to update documentation on.

Is there a way that I can start the test on the first branch and have it run and then I switch to the second branch and work on the documentation while the test is running?

I am on windows btw

[pearl cliff]

are you using git? if so, use git worktree

[pearl cliff]

fyi there seems to be a problem with the tri dependencies in mutmut. i see tri_declarative in .venv/lib/python3.10/site-packages/ but mutmut attempts to import tri.declarative, which causes it to fail

% .venv/bin/mutmut --help
Traceback (most recent call last):
  File "/.../.venv/bin/mutmut", line 4, in <module>
    from mutmut.__main__ import main
  File "/.../.venv/lib/python3.10/site-packages/mutmut/__init__.py", line 7, in <module>
    from tri.declarative import evaluate
ModuleNotFoundError: No module named 'tri'

% .venv/bin/pip list | grep mutmut
mutmut                     1.1.0

% .venv/bin/pip list | grep -F tri.
tri.declarative            5.7.0
tri.struct                 4.1.0

[proud nebula]

Why are you using mutmut 1.1.0? That's crazy old.

Released December 2018. Compared to the newest version 2.4.2 released December 2022.

[pearl cliff]

i see. it's whatever was fetched by pip-compile

maybe there's a conflict elsewhere in the dependency tree

i didn't do my usual due diligence of reviewing the changelog. figures this is what happens the one time i don't

i actually don't see a changelog. uncommon to see none at all nowadays

ahh, it's HISTORY.rst

sorry

yeah, seems like a dependency conflict. will need to debug that tomorrow

[proud nebula]

hmm, weird. Do let me know, because it's not like I have a lot of dependencies.

[pearl cliff]

it seems to be something in schemathesis

ah, it's actually junit xml

[
RequirementInformation(
  requirement=SpecifierRequirement('junit-xml<2.0,>=1.9'),
  parent=LinkCandidate('https://files.pythonhosted.org/packages/a6/21/6eb957cd581a5dacbd7bad008191caf0dfe22ecd8b714c53ea3b93e50b3e/schemathesis-3.17.4-py3-none-any.whl (from https://pypi.org/simple/schemathesis/) (requires-python:>=3.7,<4.0)')
),
RequirementInformation(
  requirement=SpecifierRequirement('junit-xml==1.8'),
  parent=LinkCandidate('https://files.pythonhosted.org/packages/92/7b/338ddf7802171d791ffe1ac73af172ce56060688cee47ae390ea3fb8b131/mutmut-2.4.3.tar.gz (from https://pypi.org/simple/mutmut/) (requires-python:>=3.7)')
)
]

why are people pinning this library so tightly...

[proud nebula]

Yea that's weird

[pearl cliff]

looking over the git history here https://github.com/kyrus/python-junit-xml/commits/master, it doesn't seem like the ==1.8 pin is needed, this library seems like it's very stable

[proud nebula]

wait, it's my fault lol

yea shit, my bad

[pearl cliff]

hah, i see you fixed it on master

time to cut a new pypi release? 😛

[proud nebula]

I accepted a PR for that it looks like, and didn't really think it through

meh. I don't want to. I have a mutmut 3 like 50% done that will be WAY WAY faster. Like potentially 100x faster. So not really keen on fiddling with the old one.

on the other hand.. no telling how many years that wil take. Bah

yea ok I'll cut a release

[pearl cliff]

yeah, i wouldn't mind trying mutmut now and not in 3 years 😉

i tried cosmic ray and wow this is (understandably) slow. i can see why the other author wanted parallelism

i did find some mutants though, so that was interesting

[proud nebula]

released a new version now

I really wish I had more time to work on mutmut. The new code is really really fast :(

[pearl cliff]

thanks! already found a bunch of surviving mutants

[proud nebula]

You've already got 100% coverage btw?

[pearl cliff]

should be close to it, i've actually been wrestling with coverage.py turning up false negatives for code paths that i know for a fact are being tested

i tried mutmut on one individual module that's not large but contains critical business logic, and quickly skimming over the mutations I would say they all seem like legitimate gaps in test coverage, but maybe not gaps that i care deeply about filling, eg a bound in the test being looser than a bound in the code, where the discrepancy was intended. but at least one of them was a legitimate gap in logical coverage even though that line was supposedly covered

i like that i can easily reapply a mutation to reproduce it

but yeah, parallel test running would be really useful here. i can see why that's a challenging design problem, maybe the easiest solution is to allow users to specify paths to clones or copies of the repository, and then the mutations can be parallelized over those clones

and maybe in the simplest scenario you could support automatically setting up a farm of identical venvs and mutating code in site-packages therein

[proud nebula]

Yea imo that's critical for practical use.

[proud nebula]

I have it already in the mutmut3-poc branch :) You can try it if you like. The UI isn't built yet though.

The trick is fork and mutation schemata. Blazingly fast.

[pearl cliff]

i see, that's cool. i don't think i can justify using alpha-level stuff on an actual project, but mutmut itself seems stable and i can easily justify using it

[serene basalt]

I'm developing a Python package and made an API object used as a pytest fixture for all my integration tests.

This API object previously had AWS S3 Cognito production details hardcoded in it.

class API:
    # hardcoded AWS S3 Cognito variables
    _IDENTITY_POOL_ID = "production identity pool ID"
    _COGNITO_LOGIN_PROVIDER = "production cognito login provider"
    _BUCKET_NAME = "production bucket name"

    def __init__():
        ...

    def my_other_method():
        ...

    def upload_stuff():
        """
        use the class AWS S3 Cognito variables with boto to upload to cloud storage 
        """
        ...

When running integration tests for my package, I want to be sure that I do not upload a bunch of test data to production cloud storage, and instead use the staging environment.

Previously, I used to override the API class AWS S3 variables in my fixture and it would work fine.

@pytest.fixture(scope="session", autouse=True)
def api():
    api = package.API(host=os.getenv("my_host"), api_token=os.getenv("my_token"))
    # Override AWS S3 cognito variables to staging values
    api._IDENTITY_POOL_ID = "staging identity pool ID"
    api._COGNITO_LOGIN_PROVIDER = "staging cognito login provider"
    api._BUCKET_NAME = "staging bucket name"

    yield api

I've now refactored to store these constants in api/config/aws_s3.py

# api/config/aws_s3.py
_IDENTITY_POOL_ID = "production identity pool ID"
_COGNITO_LOGIN_PROVIDER = "production cognito login provider"
_BUCKET_NAME = "production bucket name"

and pull them into the API class via import statements.

from api.config.aws_s3 import _IDENTITY_POOL_ID, _COGNITO_LOGIN_PROVIDER, _BUCKET_NAME

class API:
    ...

With the constants now in aws_s3.py, how can I modify this fixture to ensure I'm using the staging environment for tests, rather than production?

or am I refactoring/designing this thing badly?

[pearl cliff]

one simple approach to this is to put all of these things into a configuration file or environment variables, and leave the precise values out of your code. then you can have different configuration files for dev, test, and prod

[pearl cliff]

and one simple way to check that you are not accidentally running tests against the production system is to actually check that your bucket name, etc. do not match some pattern resembling your production bucket name

like literally check that the name matches, and if so raise an exception

of course that requires you to hardcode some knowledge of the production environment into your test environment, but it's more of a safety check than anything

[proud nebula]

I would try as much as possible to not do network calls in tests. That's asking for trouble.

[pearl cliff]

i'm working on an app where interactions with the database are a critical part of the business logic, so a lot of the test suite interacts with the database. there are plenty of unit tests as well, but i wouldn't be nearly as confident in the correctness of my app if it wasn't for the large amount of test cases that do nontrivial i/o

of course its just a local database running in docker compose but still

[boreal hill]

I have a dumb question: if I have miniconda and pytest installed, both without modifying PATH (as recommended by the installer?), how do I use pytest in VS Code?

[proud nebula]

path/to/python -m pytest?

[boreal hill]

would I have to open anaconda prompt beside my ide to do that? I need to select a specific env

[proud nebula]

With venv there's a path to the python used. Don't know how conda works.

[pearl cliff]

is there any practical difference between coverage run pytest and pytest --cov?

[pearl cliff]

i create my conda envs using -p so i can refer to them by their path just like a venv. for example

conda create -p .conda
.conda/bin/pytest

if you're using -n envs instead, you need conda run https://docs.conda.io/projects/conda/en/latest/commands/run.html

it's not a dumb question

but no, you don't need anaconda prompt as long as conda itself is in PATH (which might not be set up by default)

[pearl cliff]

also consider using micromamba which is much faster than conda and doesn't require a "base" environment (which is a common source of difficulty with conda)

[river pilot]

the pytest plugin also does combining and reporting, and will ensure pytest workers get the right settings.

[pearl cliff]

i see, thanks

[river pilot]

i'm not a huge fan of the plugin: the users don't know they are using coverage.py under the hood, so I monitor the plugin issues for things, and i think the plugin does too much: there's no reason for it to do reporting.

[pearl cliff]

understandable

is there anything it does that you can't do yourself with coverage run and such?

apart from reducing boilerplate

[river pilot]

i've lost track of what it does for spawned worker processes. I think they actually removed the multiprocessing help they used to do.

[pearl cliff]

that should work fine with plain coverage right? as long as you use the right settings

[river pilot]

yes, i use xdist with coverage without the plugin.

[serene basalt]

I think the way that I'm refactoring is just wrong and a lot of unneeded trouble for now. I can just leave that until the end

[spark locust]

sup yall do you unit test your ds/algo codings?

[spark locust]

coverage is really neat makes it easier to know what to test .

[river pilot]

thanks 😄

[proud nebula]

If you want to become more paranoid, you can try mutmut too :P

[pearl cliff]

do you recommend only using mutmut in "critical paths" of the code? it seems way too slow for a big codebase

or even a medium one, 1000s of LoC

[spark locust]

i am not familiar with mutmut is it better than coverage

[proud nebula]

Oh yes. It's not just slow to run, it's slow to work through and kill all mutants. (Auto complete suggests "kill all alligators" 🤣)

[proud nebula]

Totally different thing.

[proper wind]

any good books on testing someone can recommend?

[maiden pawn]

Unit testing by Khorikov

[spark locust]

during a technical interview . do yall go with test driven development , so you write the unit tests first before creating the coding, or vise versa?

[maiden pawn]

it depends on size of given task.
i will be voicing wishing to write unit tests first, or i will be insisting to write them first
but they could not agree to that

[proud nebula]

Depends on the problem. TDD is appropriate for some problems, but not all.

Being dogmatic just means you're branding yourself a noob ;)

[spark locust]

okay that makes sense thanks yall @boxed @maiden pawn

if i may give an example: A binary Search of a list to find a specific value : would you test first or start coding first ?

[proud nebula]

Test I think

Probably do some property based testing.

[maiden pawn]

for such task tests are expected to be provided or written first if they aren't available

tests are perfect here to define, what is expected to be done, to define requirements of the task

[proper wind]

Question... when running unit tests, if I have a class with a method that returns different types of objects (either a str or None), should I create a test method that checks for both? or seperate method for each return type?

For example:

class TestBird(unittest.TestCase):
    """Tests for the class Bird."""

    def test_get_random_noise(self):
        """Test that a random noise is returned."""
        birdy = Bird(3)
        self.assertIsInstance(birdy.get_random_noise(), str)

        birdy = Bird(1)
        self.assertIsNone(birdy.get_random_noise())

[maiden pawn]

separate test method is preferable default, because it allows u to write
test_get_none_because_explained_reason
u can document why it returns None in test name

allows makes more clear reason for test failure in case of trouble (because correct test name will be seen for failure)

[maiden pawn]

Also, be sure to know, that besides returning str or none, it is common to offer Null object in case nothing present.
Bird could be returning empty string for example

it will allow u to keep same type returned

that's atlernative approach to abusing None,that takes its meaningful place too

because it allows to remove all the extra if checks for None across your code logic

and just executing straight single path of logic instead

[proper wind]

How would I go about this?

[proper wind]

and thank you!

[maiden pawn]

from typing import Protocol, NewType
import sys

birdType = NewType("birdType", str)

class Birdy(Protocol):
    def get_random_noise(self): ...

class NullBird:
    def get_random_noise(self):
        return ""

class Falcon:
    def get_random_noise(self):
        return "SCREEEECH!"

def create_bird(type: birdType):
    if type == birdType("falcon"):
        return Falcon()

    return NullBird()

if __name__=="__main__":
    bird: Birdy = create_bird(type=sys.argv[1])
    print(f"{bird.get_random_noise()=}")

[proper wind]

Thank you. I appreciate it.

[proper wind]

Would any of you know why when running this unit test I'm only getting the output of one test on the command line/terminal?

    def test_get_random_quote(self):
        """Test that a random quote is returned."""

        archer_character = Character(3)
        self.assertIsInstance(archer_character.get_random_quote(), str)

        archer_character = Character(1)
        self.assertIsNone(archer_character.get_random_quote())

[proper wind]

ok I'm so dumb, I get it now, Python will return one test result for each test method, not for each assertion within a test method

[proper wind]

anyone know if it's worth writing a unit test for a class's str method?

[river pilot]

an old joke: "You don't have to write tests for all of your code, only the parts that you want to work."

[proper wind]

haha ok noted

[maiden pawn]

It depends on its complexity.
I would say u always need to have static typing with mypy/pyright

That ensures each place of your code that expects having __str__ defined, calls objects that have __str__ defined
That will be enough if you have __str__ simple enough to be always correct in its internal logic
static typing replaces needs for a miriad of tests, checking its pressence across the code 🙂

if it has complex logic and can break on run time internally if having different inputs, than it deserves having its own unit test in addition
(Optionally it can be super simple one written via doctest in docstrings of the function. Not obligated to have complex one with unittest/pytest
P.S. pytest calls doctest tests automatically included)

[river pilot]

one tweak to this answer: every object has __str__ defined, because object defines it. Mypy can't tell you that you've forgotten to define your own.

[maiden pawn]

https://media.tenor.com/T42cqp6YKEEAAAAC/damn-damn-damn-damn.gif

okay, then using __str__ is unreliable if u want ensure pressence of some custom str logic

define __str__ working through some another function and check pressence of this function

oh... u can also... make __str__ inherited from some base class / mixin that ensures usage of this logic 🤔

Yeah, we can check it then

[maiden pawn]

it works 🙂

from typing import Protocol

class StringedObject(Protocol):
    def __str__(self) -> str: ...

    def custom_descriptive_str(self) -> str: ...

class BaseExample:
    def __str__(self) -> str:
        return self.custom_descriptive_str() # type: ignore[no-any-return, attr-defined]

class Example1(BaseExample):
    def custom_descriptive_str(self) -> str:
        return "example1"
    
class Example2(BaseExample):
    pass

a: StringedObject = Example1()
b: StringedObject = Example2()

[cyan flicker]

Does anyone have guidance or an example on how to unit test whether or not a flag is set during a function? This flag is used throughout other parts of the program so I want to ensure it's never missed.

def _Update(cls):
    cls.variables['update_in_progress'] = True
    # Do the things
    cls.variables['update_in_progress'] = False

I've spent some time trying to come up with a solution but can't. I even created a thread that just while loop watches the class variable but that's obviously a race condition and hopefully more complex than it needs to be.

[river pilot]

instead of testing the flag, maybe test some result that requires the flag?

[river pilot]

cls.variables['something'] seems like an unusual construct. why class globals, and why a class global dict?

[cyan flicker]

@river pilot I ultimately just chose to create two methods for setting and clearing the flag and mocking away those which gave me the ability to add another flag like "update_in_progress_was_true".

[ember maple]

based on the limited information, this looks like a good use-case for a context manager and/or context decorator to extract the behaviour

see https://docs.python.org/3/library/contextlib.html#contextlib.ContextDecorator for a example (this ensures the functions decorates has the stuff set and then youcan test it in isolation

[maiden pawn]

Split your algorithm that sets this flag into smaller synchronous parts

First part of your function reaches stage setting flag true, second part continues work
(It can be done as part of made class with init function)

Then your unit test becomes easy peasy invoking those steps in sequence and asserting states before, between and after

[cyan flicker]

Interesting. This is somewhat of the approach I took. I believe not using the context manager is going to be more readable.

[cyan flicker]

I chose to use proxy variables since there's multiple pids that would be using the same info. Wasn't large enough to use a db, so I chose to use proxy variables and I think it doesn't its job.

[river pilot]

how do multiple processes get access to the class attributes?

[cyan flicker]

The class attributes are using basemanager proxy variables. https://docs.python.org/3/library/multiprocessing.html

[proud nebula]

I don't think you can use it as a class.

[cyan flicker]

Why's that? I've been using it that way for a couple years now in production. Even an instance variable would work but would be much more work for the manager pid since accessing managed variables is quite slow. ~3-5ms per request.

[proud nebula]

Oh. I thought you had trouble getting it to work at all 🤣

[cyan flicker]

nope. It was a class that I never got around to unit testing because of the complexity. But I thought I'd give it a stab so here I am 🙂

[tawdry saffron]

Hey everyone ! I have a problem with pytest and pytest-dependency : I can't run alone a test that have dependency.
Here's the thread : #1153233775136882718

[proud nebula]

Something seems wrong to me if you have dependencies between your tests like that.

[tawdry saffron]

why ?

I'm testing a webAPI. I can start an action, and actions have status and result files

if I can't start an action, testing its status doesn't make sense. if I can't get a status of an action, testing its result doesn't make sense

[proud nebula]

That sounds to me like you've bought into the Religion of Singe Asserts.

[tawdry saffron]

I'm not very used to unit test, whta would you recommand

[tawdry saffron]

?

[proud nebula]

Could you show two tests that have a dependency like this?

[lament heath]

Would you kindly expand on the religion part?

[tawdry saffron]

def test_start():
    pass


@pytest.mark.depends(["test_start"])
def test_status():
    pass


@pytest.mark.depends(["test_status"])
def test_result():
    pass


@pytest.mark.depends(["test_start"])
def test_bad_input():
    pass

[proud nebula]

Engineering is about tradeoffs. "One assert per test" is about Dogma.

[proud nebula]

No, I mean real code. Also that's 3 tests that depend on one. Just one that depends on one will do.

[lament heath]

Looks like something that could be done using fixtures

[tawdry saffron]

can you elaborate

[proud nebula]

Or just a more complex test

[proud nebula]

Do you know what a fixture is in pytest?

[tawdry saffron]

no, but I do use it to initialize my tests

[proud nebula]

You guys both should be more specific than just "can you elaborate?" 🤣

[tawdry saffron]

concretly, to reset the database before running the tests etc

[proud nebula]

That sounds wrong to me too. The DB should be in a clean state by default between every single test. Something smells weird here.

[tawdry saffron]

I mean

that's either something to do on startup or clean off

[proud nebula]

Ideally tests are run either in memory or in a transaction that is aborted at the end.

(but that's not always possible)

[tawdry saffron]

why would I do that

ok some parts of the api I'm testing requires some stuff in the database

I need to commit some data to properly test

[proud nebula]

You really should read up on pytest fixtures before asking more questions :P

[tawdry saffron]

alr thanks...

[lament heath]

Listen, pytest fixtures do exactly what you want. People are not trying to ask you away.

They don't do it in the way you expect but they really are the best solution.

If you believe that fixtures are not powerful enough for your use case — read more on them because they are always powerful enough.

[tawdry saffron]

I just haven't look at them enough, as it's not a concept I've saw in other languages/test frameworks. I'll def look more into it tho.

[scarlet aspen]

Is there any real difference between unittest anad pytest? Like does one have an advantage over another?

[maiden pawn]

pytest is supporting Everything unittest does, it can run unittest code
Plus it adds all its features

So..pytest is ultimately having more by design.. but

Unittest has one minor advantage. It is available without installing anything. Built in library. It is sometimes important too to have for certain written code

[river pilot]

pytest is much more powerful, and has addons that can provide more functionality

[scarlet aspen]

Alright. I already have to install other things for tkinter so whats one more thing.

Does anyone have a outline for the proper way to create a test db for pytest? RealPython has one for pytest but as a general sense.

[maiden pawn]

Depends on where using
If in Django, just connect pytest-django, it will do it for you

If customly... write pytest fixture with scope doing once for season to create temporal db and return connecting data in fixture to other tests
(Optionally without session scope, create temporal db for each test again)

[scarlet aspen]

Im wanting to make test data rather than using a premade database so i can check the edit functions.

But its for both

Desktop and web app

[maiden pawn]

Use factory boy library and its integration for your ORM to create rapidly for your tests data from zero thorough its fakery factories of models

[scarlet aspen]

Alright, im having a hard time wrapping my head around pytest for my file. Can someone help? Ill share the code in a minute

!paste

https://paste.pythondiscord.com/BCNQ

Oh I should probably share the table code. One second.

CREATE TABLE "vocs" (
"voc" TEXT UNIQUE,
"alt_name" TEXT,
"formula" TEXT,
"note" TEXT,
PRIMARY KEY("voc")
)

[maiden pawn]

Not seeing a question

Wrapping head is very subjective thing

[scarlet aspen]

What would a test for the load_vocs function look like. If I can see that within context of my code, I’m sure I can figure the rest out myself.

That functions going away anyway ask I’ve realized I should call a single record instead of load everything into a list. That works for 5 record, probably won’t pan out well for 500.

[scarlet aspen]

I can understand basic testing but when it comes to importing my file, I get lost.

[maiden pawn]

i can write you rough idea without exact working code 😄 a moment

[scarlet aspen]

It’s mostly getting the data and stuff across. I can figure out the assert data = record

[bitter wadi]

:ok_hand: applied timeout to @maiden pawn until <t:1695085668:f> (10 minutes) (reason: newlines spam - sent 103 newlines).

The <@&831776746206265384> have been alerted for review.

[sour lodge]

!unmute 370435997974134785

[bitter wadi]

:ok_hand: pardoned infraction timeout for @maiden pawn.

[maiden pawn]

sent to dm example of testing it

[maiden pawn]

thanks

[scarlet aspen]

Alright.

[maiden pawn]

https://paste.pythondiscord.com/JSMQ

screw it, sending like this 😄

[scarlet aspen]

Thank you. I’ll check in a minute, just got home from the store.

Theres a lot of syntax im not familiar with lol. Ill have to dissect this throught the night. Thanks for the help!

[sharp pumice]

Hi guys. I have been sharing my posts on certain channels. Many called it self promotional content. But this is not a promotional content. Just my article on A/B Testing. Like how all YouTubers share their knowledge, I try to share my knowledge via posting articles. Hoping it is helpful for everyone. https://www.linkedin.com/posts/isham-rashik-5a547711b_machinelearning-datascience-decisionmaking-activity-7109710963527839744-PWY9?utm_source=share&utm_medium=member_desktop

[lament heath]

Should someone ban it? It's promotional content and an annoying precedent

[maiden pawn]

!rule 6 , this stuff is forbidden, please delete.

[bitter wadi]

Rules

6. Do not post unapproved advertising.

[tawdry saffron]

Hi everyone

I've started using fixtures but I still have a remaining questions

let's say I have 3 functions of the same process to test : "start", "status", and "result"

let's say the result can accept some output format

I've written some tests to test the 3 functions, but now I'd like to tests every possible output formats

how should I go about it

@pytest.fixture
def start(format="3mf"):
  return SomeApi.some_start_request(format)


@pytest.fixture
def status(start):
  return SomeApi.some_status_request(start.id)


@pytest.fixture
def result(status):
  return SomeApi.some_result_request(start.id)


class TestSomeApi():
    def test_start(self, start):
        assert start.success and start.id
    
    def test_status(self, status):
        assert status == "done"

    def test_result(self, result):
        assert result.success and result.dest_file and is_file(result.dest_file)

^ illustration

[maiden pawn]

could u write some expected code what u wish achieve

failing a bit to comprehend your desired end result exactly

feel freely to write pseudocode of any type

[tawdry saffron]

sry

@pytest.fixture
def start(req):
  return SomeApi.some_start_request(req.param)


@pytest.fixture
def status(start):
  return SomeApi.some_status_request(start.id)


@pytest.fixture
def result(start, status):
  return SomeApi.some_result_request(start.id)


class TestSomeApi():
    def test_start(self, start):
        assert start.success and start.id
    
    def test_status(self, status):
        assert status == "done"
  
    @pytest.mark.parametrize('start', ['3mf'], indirect=True)
    def test_result(self, result):
        assert result.success and result.dest_file and is_file(result.dest_file)


    @pytest.mark.parametrize('start', [['Amf', '3mf', 'Obj']], indirect=True)
    def all_export():
        pass

[tawdry saffron]

so this is pseudocode, trying to use pytext parametrization

I have 3 fixtures, linked to each other. start, status and result

results generates a file, the file format can be changed

I'm trying to find a way to write a test taht will generate all files available

I mean for every format available

[maiden pawn]

i see. you wish to invoke as single fixture with different arguments 🤔with parametrize

[tawdry saffron]

I'm not even sure

[maiden pawn]

and sometimes chained function 🤔

[tawdry saffron]

since all the fixtures are linked in some way

i'm struggling to find the "good" way to write the test

I could like, call the fixtures directly

def test_exports(self, addfile, addpack, add_file_to_pack, pack_start):
        def test(addfile, addpack, add_file_to_pack, format):
            st = pack_start(addfile, addpack, add_file_to_pack, format)
            do = pack_status_done(addfile, addpack, st)
            re = pack_result(addfile, addpack, st, do)            
            assert re.success
            assert re.dest_file is not None
            assert isfile(re.dest_file)
        
        test(addfile, addpack, add_file_to_pack, "3mf")
        test(addfile, addpack, add_file_to_pack, "Stl")
        test(addfile, addpack, add_file_to_pack, "Obj")
        test(addfile, addpack, add_file_to_pack, "Amf")
        test(addfile, addpack, add_file_to_pack, "4da")

but that kind of defeat the purpose of fixture isn't it

[maiden pawn]

class TestSomeApi():
    @staticmethod
    def request(param: Param):
      return Request(param)

    @staticmethod
    def request_start(req: Request):
      return request SomeApi.some_start_request(param)

    @staticmethod
    def request_status(start: Request)
      return SomeApi.some_status_request(start.id)

    @staticmethod
    def request_result(start: Request)
      return SomeApi.some_result_request(start.id)
    

    def test_start(self):
        start = self.request_start(self.request())
        assert start.success and start.id
    
    def test_status(self):
        result = self.request_status(self.request_start(self.request()))
        assert status == "done"
  
    @pytest.mark.parametrize('param', ['3mf'], indirect=True)
    def test_result(self, param):
        result = self.request_result(self.request_start(self.request(param))
        assert result.success and result.dest_file and is_file(result.dest_file)

here u go 😄

[maiden pawn]

Fixture could be returning function if desired, which could be used to call it and get what u wish. but it will make it not very far different from just calling function then

[tawdry saffron]

thanks let me look into it

[maiden pawn]

the purpose of fixture to be a thing without parameters 🤔

but...

ohhh

[maiden pawn]

i remembered a funny way to approach it

[tawdry saffron]

failed: Fixture "pack_start" called directly.
oh well

[maiden pawn]

@pytest.mark.parametrize('param', ['3mf', 'abc', '123'], indirect=True)
@pytest.fixture
def request(param: Param):
  return Request(param)

@pytest.fixture
def request_start(request: Request):
  return request SomeApi.some_start_request(request)

@pytest.fixture
def request_status(request_start: Request)
  return SomeApi.some_status_request(request_start.id)

@pytest.fixture
def request_result(request_start: Request)
  return SomeApi.some_result_request(request_start.id)

class TestSomeApi():
    def test_start(self, request_start: Request):
        start = self.request_start(self.request())
        assert start.success and start.id
    
    def test_status(self, request_status: Request):
        result = self.request_status(self.request_start(self.request()))
        assert status == "done"
  
    def test_result(self, request_result: Request):
        result = self.request_result(self.request_start(self.request(param))
        assert result.success and result.dest_file and is_file(result.dest_file)

u could write smth like this

paramterizing at the level of fixture

u get the idea

[tawdry saffron]

fixture are still quite new to me, I'll try to look into this, looks interesting

thank you

[maiden pawn]

u a welcome. beware, it was just pseudocode. but as far as i remember such thing should be present

[tawdry saffron]

... I think I made it

it feels a bit hacky Idk if that was the best solution x)

I used your advices, but probably not as intented ?

@pytest.fixture
def format(request):
    return request.param

@pytest.fixture
def start(format):
    return api.start()

@pytest.fixture
def status(format, start):
    return api.status(start.id)

@pytest.fixture
def result(format, start, status):
    return api.result(start.id)

def default_format():
    return ['3mf']

def all_formats():
    return ['3mf', 'Amf', 'Stl']

class SomeText:
    @pytest.mark.parametrize('format', default_format(), indirect=True)
    def test_start(start):
         pass

    @pytest.mark.parametrize('format', default_format(), indirect=True)
    def test_status(status):
         pass

    @pytest.mark.parametrize('format', all_formats(), indirect=True)
    def test_result(status):
         pass

[solid venture]

Can some one understand why this test don't pass? ``` import unittest
from io import StringIO
import sys

class TestPrintOddNumbers(unittest.TestCase):
def test_print_odd_numbers_9_to_101(self):

    # Capture the printed output
    captured_output = StringIO()
    sys.stdout = captured_output

    # Call the function
    print_odd_numbers(9, 101)

    # Reset stdout
    sys.stdout = sys.__stdout__

    # Get the captured output as a string
    printed_text = captured_output.getvalue().strip()

    # Expected output (odd numbers from 9 to 101)
    expected_output = "\n".join(str(num) for num in range(9, 102, 2))

    # Compare the printed output with the expected output
    self.assertEqual(printed_text, expected_output)

if name == "main":
unittest.main() ```

[river pilot]

if you pastebin the unittest output, and the code you are testing, we might be able to help.

[ember maple]

also please use one of the stdlib context managers for stdio redirection (while in the example its not a problem, eventually a case wil lcome up where not having the context manager clean it up will bite error handling

(plus the cleanup is incorrect in case someone did a nested redirect or use other tools)

[neon kraken]

helpppppppppp....

[sleek plaza]

import unittest
from io import StringIO
import sys

def print_odd_numbers(start, end):
for num in range(start, end + 1):
if num % 2 != 0:
print(num)

class TestPrintOddNumbers(unittest.TestCase):
def test_print_odd_numbers_9_to_101(self):

    # Capture the printed output
    captured_output = StringIO()
    sys.stdout = captured_output

    # Call the function
    print_odd_numbers(9, 101)

    # Reset stdout
    sys.stdout = sys.__stdout__

    # Get the captured output as a string
    printed_text = captured_output.getvalue().strip()

    # Expected output (odd numbers from 9 to 101)
    expected_output = "\n".join(str(num) for num in range(9, 102, 2))

    # Compare the printed output with the expected output
    self.assertEqual(printed_text, expected_output)

if name == "main":
unittest.main()

In this modified code, the print_odd_numbers function is defined, which prints the odd numbers within the specified range. The test case test_print_odd_numbers_9_to_101 then verifies that the printed output matches the expected output, which is a list of odd numbers from 9 to 101

[serene basalt]

Hello everyone,

I am currently working on a project that makes extensive use of pytest for testing. Within my project, I have numerous doctests embedded within the docstrings of pretty much every methods. However, all of these doctests require a certain context to be established beforehand, namely, an active instance of our custom API.

Currently, I have been including a context manager within every single doctest example to ensure the necessary API instance is available. Here’s a simplified example to illustrate the setup:

class SomeClass:
    def __init__(self, name: str):
        """
        Examples
        --------
        ```python
> with API(host=None, api_token=None) as api:
>         ...     some_instance = SomeClass(name="example")
>         
        """


However, this makes the doctests quite verbose, harder to maintain, and somewhat obscures the actual examples I'm trying to provide. I am looking for a way to establish this context for all doctests within a module or class globally, so that I don't have to repeat the context manager setup in every single doctest.
 
 Is there an existing solution or best practice to manage the context for pytest doctests more efficiently? Or perhaps a way to establish a module-level or class-level setup that applies to all doctests within that scope?

[maiden pawn]

https://docs.pytest.org/en/7.3.x/how-to/doctest.html
U can invoke pytest fixture in doctest

I am very fascinated

[serene basalt]

but the conftest.py has to be within src/ right?
I have my conftest.py outside inside of tests/

so my project tree is like

src/
   my_file.py
stuff/
tests/
    conftest.py

[maiden pawn]

conftest should be at shared parent level. put in root or smth(or import at root)

[serene basalt]

okay I'll try putting it at root

[serene basalt]

moved conftest.py to root directory and everything seems to be working fine now

I had to add this to conftest.py

@pytest.fixture(autouse=True)
def inject_doctest_namespace(doctest_namespace, myt_api):
    doctest_namespace["api"] = my_api

[pearl cliff]

what if you add tests to pythonpath when running doctests? will that avoid needing to move conftest out of tests?

that said i've actually been interested in putting conftest at top level before

[serene basalt]

wouldn't that be harder then because every developer has to remember to run tests with terminal arguments?

am I understanding you correctly/

[serene basalt]

how come you have been interested in putting conftest.py at top level before?

I've thought of putting it at top level before too because it has all of my fixtuers and would make it easier for me to move all of my tests to each module instead of a big tests folder, but not sure how worth it that'll be for now

[pearl cliff]

i think you can tell pytest to add entries to path in ini options? that or i usually end up with a wrapper script for running tests anyway

[pearl cliff]

idk, i've just thought about it as a way to keep all "configs" together. but usually i do tests/conftest.py because it's convention and it's what people expect

i think pytest just looks for any module called conftest and imports the first one it finds, not sure though

the docs aren't clear about it

https://docs.pytest.org/en/7.3.x/explanation/pythonpath.html

[serene basalt]

I tried doing that before and after a bunch of time trying to figure it out I gave up and thought its not that important and moved on

now I just import all the fixtures into conftest.py via

from fixtures import *

[serene basalt]

yeah those docs killed me

I think the best thing to do is to ask them on their discussion channel and then try to update their documentation so that no one has that problem again in the future

https://github.com/pytest-dev/pytest/

[pearl cliff]

that'd be the best thing yeah

i'll throw it on my enormous list of things i'll never get around to 😬

[serene basalt]

same here, but if one person does it we'll all be better off and easier

[pearl cliff]

true!

[idle plank]

ok

[lament heath]

@proud nebula Trying out mutmut right now. As an idea, it sounds pretty amazing!

:3

[proud nebula]

If you have some piece of code that is enormously important to get right yea

[proud nebula]

that is pretty good so far

[lament heath]

I just try to cover 100% of stuff in my libraries and strictly using public interfaces with no mocking so I expected it to be at least okayish.

I guess it's not a question of enormous importance to me but more of a question of: How else can I improve the safety of my code. I try to type hint and test religiously so I'm kind of out of options.

Btw if you know of or have any other tools that help you make such improvements -- please, tell.

[proud nebula]

There's also property based testing. Personally I find it of limited practical use compared to mutation testing though.

And it's harder to do...

[lament heath]

Is it like hypothesis?

Okay, yeah, it is :D

Yeah, I was never a huge fan.

[proud nebula]

I've tried to use it a few times, but it's always very hard, and rare that my code has the right sort of situation for it.

[lament heath]

Yup. I've heard it workes well for API testing though.

⠴ 1061/1061 🎉 829 ⏰ 0 🤔 0 🙁 232 🔇 0

Sounds pretty good. Researching into it right now.

[proud nebula]

hm.. that doesn't sound right to me. "API" is just "function". But I assume you mean web API? Even that is just "function over http" which doesn't make anything more testable really.

To me property based testing makes sense for very algorithmic stuff. Like tree traversal or sorting

And that's just extremely rare that you do as a programmer I find.

[lament heath]

Yes, I am talking about a web API.

https://schemathesis.readthedocs.io/en/stable/

[proud nebula]

that's basically just fuzzing input as far as I can tell.. I don't like that

[lament heath]

It is, yes. I think, it makes a lot of sense for APIs where neither openapi is generated based on the code, nor the code is generated based on an openapi.

[proud nebula]

I don't see how it does... not any more than testing a typed function.

Another thing I dislike about PBT is that there's no end. You run it for an hour, or a day, or a week.. but are you done? 🤷‍♂️

MT you know if you're done.

In general it feels like MT is a useful tool everyone should know, but not use a lot. While PBT is a much more niche tool.

[lament heath]

I agree about MT.

I guess it just takes a few people doing a few talks and articles on them.

[proud nebula]

Probably. I have done some obviously, but I suck at marketing 🤷‍♂️

That's why mutmut is super niche and iommi is not very well known...

or any of my other projects for that matter heh

[river pilot]

the problem I had with mutation testing was the false positives

[proud nebula]

Yea. I skimmed your old article and there was one false positive due to a bug in mutmut and then one case of you not wanting to change the to make it mutation safe. And then lastly one False -> None mutant. The last one is more brutal I agree. Maybe it should be off by default? But we've found many bugs with this mutation in iommi so not so sure. Python can be weirdly sloppy about true/false.

MT is for very specific situations. Like TDD you shouldn't use it all the time. Coverage is at least 100x more useful than MT. If not 1000x.

But I find PBT must be a tenth as useful as MT.

[lament heath]

Could you share the article?

[proud nebula]

https://nedbatchelder.com/blog/201903/mutmut.html

[lament heath]

Thanks!

[river pilot]

@proud nebula i have definitely struggled to find a place for PBT also. I use it in the coverage test suite, but only on a very straightforward test scenario (compressing ints into a bitstream).

[proud nebula]

Ah. Yea that sounds exactly like what I would use PBT for.

[pearl cliff]

@proud nebula @river pilot i use PBT as an adjunct to "example-based" testing. as i started working with it more, i found that a lot of the example-based tests i wrote could be generalized relatively easily to property-based

that is, i suggest starting with concrete examples, and use that to guide you towards general properties

for example, you have a config file for your app with a field num_workers. maybe 0 is a sentinel for autodetection, and negative numbers are disallowed. so you want to enforce the property that num_workers < 0 leads to a config validation error.

[river pilot]

that's a good suggestion. I would love to know how to use it more.

[pearl cliff]

i think the examples that are often given (usually trivial nonsense examples like assert add(x, y) == x + y) aren't very helpful

i've found that using PBT has guided me in general to thinking about the properties/invariants that i am building into my application, and encourages me to make them explicit in the test suite

actually one interesting example is where i was implementing some geospatial routines that already existed in another library, so i was able to use that library as a comparison. the "property" in that case was that any valid input should return a result within 0.01% of the existing library's result (or something like that)

the problem there is that hypothesis sometimes really struggles with complicated inputs (e.g. constructing heterogeneous dictionaries like you might see in a REST API), and sometimes you have to just give up and rely on a handful of examples

in the geospatial example though, hypothesis has first-class support for numpy arrays

another related scenario is when you have two different implementations of the same algorithm, maybe one path for small inputs and a different path for bigger inputs

but that's relatively niche compared to the case of starting with an example of business logic and seeking to extract general properties/invariants from that example

[proud nebula]

I don't see how MT won't get you there faster and with less cognitive load.

[pearl cliff]

well, for one thing MT is practically slow to execute and comparatively hard to reason about. PBT is just drawing random values instead of you picking specific values, it's basically fuzzing on a smaller scale.

[proud nebula]

Isn't PBT pretty slow too? As it's potentially exploring an infinite space?

(At some point I will get mutmut 3 released and MT will be 10x to 100x faster than currently too)

[pearl cliff]

you set a max number of examples. if you want to fuzz the hell out of your app then yes, it's slow. but it's also much easier to parallelize, since you don't have this weird state-maintenance problem of literally having to copy the source code and/or bytecode and mess with it.

[proud nebula]

I don't have that in mutmut3 ;)

[pearl cliff]

fair enough!

[proud nebula]

But yea... it's just a prototype so far and I work at a high paced startup and have two kids. So...

[pearl cliff]

also MT is a really broad instrument. there is exactly one code path that's important to mutate (the < 0 condition), but there's no good way that i'm aware of to actually focus on mutating only that code path. and what happens if you're using pydantic? num_workers: Annotated[PositiveInt, Strict()] or something like that. how do you know to mutate the PositiveInt constraint?

maybe there's a computer science thesis in "coverage-guided mutation testing" if that doesn't exist already

[pearl cliff]

yeah tbh it sounds like a huge undertaking

i'm not trying to put down mutation testing to be clear, i'm only trying to answer the question of why PBT and MT aren't really equivalent in practice (given my limited messing around with MT)

the final thing i'll say about the comparison is that cognitive load is personal and subjective, but for me i think the cognitive load of property-based testing is relatively low once the actual properties have been identified. identifying useful properties is the harder part, but i think that's time well-spent anyway. it's a design-focusing exercise.

[river pilot]

in a nutshell, what makes it faster?

[chilly dove]

Guys, please I have a small problem with my test function. The test function calls a websocket server with the create_room event. Previously, this code (test function) worked correctly. It was until recently when the remote on GH was updated and it stopped working. I've not been able to figure the problem since then. Here's the test function code:

def test_create_room(socket_client: SocketIOTestClient, project: Project):
    data = {
        "owner": str(project.owner.id),
        "name": "Test room",
        "description": "Test description",
        "project": str(project.id),
    }

    socket_client.emit("create_room", data=json.dumps(data))

    response = list(
        filter(lambda dict: dict["name"] == "create_room", socket_client.get_received())
    )[0]
    response_data = json.loads(response["args"][0])["data"]

    # Check if `id` property is returned in response and that
    # the project id of the payload matches the response
    assert "id" in response_data
    assert data["project"] == response_data["project"]

    if project.import_type == ImportType.GITHUB:
        # Github projects should always be a directory
        assert project.is_directory == True

I'm testing with pytest btw. Notice the socket_client.emit() function call. The result of that is an empty array because socket_client.get_received() function is to get the response event from the server and that just returns an empty array which also makes the response list variable to be empty too thus failing to call json.loads().

[lament heath]

@proud nebula same question :)

[narrow beacon]

I am using Sqlalchemy to perform database operations. I am writing tests for the specific function created for database operations like insert, update, delete, search, select etc. How can i perform test without affecting my database in use.

[proud nebula]

Several things in combination.

1. Mutation schemata instead of on-disk mutation. This is that all mutants are generated up front and selection of mutant is done at runtime. In my prototype by setting an environment variable.
2. A sort of fake-coverage guide so that I know which tests affects which functions, cutting the set of tests by a lot.
3. Due to 1) I can run many runs in parallel so I can use multiple cores and saturate the CPU
4. To make this fast I use a fork model where I import as much as possible up front. This helps a surprising amount.
5. I also made some incidental changes to mutation generation that made that part way faster. That stuff was irrelevant in mutmut2 because everything else was so incredibly slow but made a big difference in mutmut 3.

[proud nebula]

Settings for tests should point to a different database.

[lament heath]

Do you have tasks that can be tackled by someone else or is this a one-man effort?

[proud nebula]

Honestly don't really know. Right now it seems totally broken for my test situation and I'm trying to figure out why. Something about django setup not being able to be called twice I think...

Found this gem:

        try:
            app_module = import_module(entry)
        except Exception:
            pass

in the django code and this seems to cause just endless suffering for so many people...

if you want to discuss helping out I would recommend the mutation testing discord and the mutmut channel there. ht tps://disco rd.g g/5B mcFBMuUN

I can't post the link because this server auto deletes discord invites heh

[lament heath]

Okie. Can't promise anything but I'm definitely interested in looking at it

[earnest skiff]

Hi, i'm trying to do test for aiohttp, but i'm unable to figure how to do it, i'm a noob doing test, and i saw that async test are "special" and i'm also unable to mock things for aiohttp, does any one have an example or docs to how to do it?, so i can learn.

Thanks in advance.

[hexed cloak]

You'll need an async test runner like anyio or pytest-asyncio

[hexed cloak]

vcrpy supports mocking aiohttp interactions

[hexed cloak]

pytest-recording is a nice way of setting up vcr

[hexed cloak]

What do you mean by "unable to mock things for aiohttp"? What did you try? And what happened?

[earnest skiff]

I'm trying to mock request to just test the function avoding the call to the web

[proud nebula]

Managed to get the mutmut3 POC working on the iommi test suite. Ran through the entire thing in ~42 minutes. ~1 mutation/s, killed ~28% of mutants. So... not great honestly. I would have expected a lot more kills.

[lament heath]

:(

[proud nebula]

Was a long time ago I ran a mutation tester on this code

We've let the coverage slip too I think. 98% seems lower than what I remember...

And the docs don't have 100%, which is probably even worse.

[lament heath]

Docs coverage?

[proud nebula]

Yea. We have all our docs as pytests and generate docs with iframes for output (instead of screenshots)

[lament heath]

Hm... I'll take a look. I wanna do the same thing. Sounds cool.

[proud nebula]

Yea, it was a bit of work. I had to write a lot of custom hacky code. But it's been worth it imo. Our docs are way better because of it and we found tons of errors too

[earnest skiff]

Thanks for the help, that it was i was looking pytest with vcr seems to be working as i wanted

[hexed cloak]

?

[earnest skiff]

About this 🙂

[hexed cloak]

@earnest skiff I'm not really sure what you're saying, I was asking about what you were trying before so I could help with that

[earnest skiff]

I was looking for this -> vcrpy supports mocking aiohttp interactions

[lament heath]

What is your opinion on pytest-postgresql and pytest-rabbitmq?

[pearl cliff]

is that the one that starts a postgres daemon as a pytest fixture?

[lament heath]

Yes.

[civic thunder]

Hi, if i use colocation for tests (the test file at same level of the file to test), where should I put the tests utils? like reusable mocks, or reusable matchers?

[proud nebula]

We do this in iommi. We still have a tests dir with utils, models+settings (this is a django project), and some tests that are not cleanly located next to the file under test.

We do foo__tests.py for the tests of foo.py, which makes it super clean to find the relevant tests for a module.

[prime willow]

hi all, need a quick help with mocking asyncpg calls, below is the code and pytest snippet and the mocking doesn't seem to work
async with self.pool.acquire() as conn:
async with conn.transaction():
async for record in conn.cursor(fetch_query):

mock_pool = mocker.patch("mymodule.processor.asyncpg.create_pool",new_callable=AsyncMock)
mocked = mock_pool.return_value.acquire.return_value
mocked.fetch.side_effect = [("row1", "data1"), ("row2", "data2")]

[river pilot]

have you made non-async mocks work? Usually it's because of needing to get the imports right.

[prime willow]

yes I did. I tried various approaches, most of them are raising the error below.
:test_something - AttributeError: __aenter__

[river pilot]

hmm, ok, i haven't used async like this before, sorry

[prime willow]

mock_pool = AsyncMock() with patch.object(mymodule.core.processor, 'asyncpg', mock_pool): mock_pool.configure_mock( **{ "create_pool.return_value.__aenter__.return_value": mock_pool, "acquire.return_value.__aenter__.return_value": mock_pool } ) mock_pool.fetch = [('a', 1), ('b', 2)] await processor.fetch_client_ids()

even the above doesn't work. I think I have wasted enough time, I'll just write a test class myself

[pearl cliff]

i've gravitated towards just running a local postgres instance for testing

you can set up and tear down a test database using a fixture, cloning it from a template app database

[slender pendant]

do you usually want to make your unit tests as small as possible, or can you aggregate all tests for a single function into a single test?
For example, I made this test:

def test_carousel():
    carousel_data = carousel()
    # Check that the settings are correct
    assert carousel_data.autoplay is True
    assert carousel_data.speed == 500
    assert carousel_data.slides_to_show == 5
    # Ensure proper data types for span children, as well as color
    for child in carousel_data.children:
        color = (
            isinstance(child.children[0].children, str)  # True
            and isinstance(child.children[1].children, str)  # True
            and child.children[1].style["color"]
        )
        assert color in {"green", "red"}

asked chatgpt like if it's good or not, and it said I should do it like this:

def test_carousel_settings():
    carousel_data = carousel()
    # Check that the settings are correct
    assert carousel_data.autoplay is True
    assert carousel_data.speed == 500
    assert carousel_data.slides_to_show == 5

def test_carousel_data_types_and_color():
    carousel_data = carousel()
    # Ensure proper data types for span children, as well as color
    for child in carousel_data.children:
        assert isinstance(child.children[0].children, str)
        assert isinstance(child.children[1].children, str)
        assert child.children[1].style["color"] in {"green", "red"}

[maiden pawn]

Different hot take. Just use Pydantic as data structures and then u can throw away all those tests.

because pydantic will validate for you correct types and structure of data 😅 including having correct Enum of colors

also just use in general strict mypy or pyright. Then u can throw away even more typing validating tests

leave tests for testing something more meaningful from functionality. Some heavy runtime logic

[robust siren]

hey i need to fixe something in my script for a telegram bot can someone help me

[limpid perch]

I have a test_one.py in folder /app/v1/tests and in it there is from app.v1.main import app which causes ModuleNotFoundError: No module named 'app' when I run pytest -s app/v1/tests/test_one.py from the /app folder

I am doing this through VSC on a windows machine (no virtual environment)

I have tried with and without a __init__.py in the tests folder and /app folder with no luck

[maiden pawn]

from typing import Literal, Union, NewType
WorkersMaxAmount = NewType("WorkersMaxAmount", int)
WorkersMaxUnlimited = Literal[None]
WorkersMaxChoice = Union[WorkersMaxAmount, WorkersMaxUnlimited]

a: WorkersMaxChoice = WorkersMaxUnlimited

Argument "workers" to "batch_action" has incompatible type "<typing special form>"; expected "Optional[WorkersMaxAmount]"  [arg-type]mypy
(type alias) WorkersMaxUnlimited: type[None]

Unable to use None as literal const. Any options? 😅 I could always go for normal constant i guess

WorkersMaxAmount = NewType("WorkersMaxAmount", int)
WorkersMaxUnlimited = Literal["WorkersMaxUnlimited"]
WorkersMaxChoice = Union[WorkersMaxAmount, WorkersMaxUnlimited]

a: WorkersMaxChoice = WorkersMaxUnlimited

Incompatible types in assignment (expression has type "<typing special form>", variable has type "Union[WorkersMaxAmount, Literal['WorkersMaxUnlimited']]")  [assignment]mypy
(type alias) WorkersMaxUnlimited: type[Literal['WorkersMaxUnlimited']]

not working too

i need type Alias may be? hmm it will change nothing

MaxAmount = NewType("MaxAmount", int)
MaxUnlimited = Literal[None]
MaxChoice = TypeVar("MaxChoice", MaxAmount, MaxUnlimited)

d: MaxChoice = MaxUnlimited

conquered! success.

[maiden pawn]

ops. broke again 🤔

[proud nebula]

Don't run it from the app folder. Up one level.

[limpid perch]

Finally got it working, from app folder 😉

[proud nebula]

By changing the import to v1.whatever?

Those are terrible names btw 🤣

program.app.main.v1.first.initial