internals-and-peps

[magic python]

Good point. Hmm. Maybe I should run mypy more as I go 🤔

[swift imp]

I dont know how I feel about static typing. For it being as old as it is, its still in a very limiting state. Especially if you arn't using 3.9 or higher

[magic python]

I wasn't sure at first, but having added it in a few places it does seem nice for documentation purposes etc

Tho sometimes I wonder if I should just need learning rust or something, I don't have any intuition for when I'd not use python as I only use python

[mortal ermine]

Most of my runtime type errors on projects that use mypy are because I'm doing gnarly things. mypy does go a long way. (disclosure: I'm a fan of the type hits.)

[magic python]

It's really encouraged me to use homogeneous list types

Or to even consider that as a thing, wouldn't have previously

[mortal ermine]

Ah that's interesting, do you find yourself using tuples more in those situations?

Where you still don't, that is.

[magic python]

In which, when there are heterogeneous lists?

[mortal ermine]

Naw, sorry I'm not sure how to ask what I'm wondering... how about, did it make situations that were homogeneous lists more distinct from ones where they weren't (eg. tuples)?

More disclosure, I've been away from Python for a while and just started fully digging into the modern stuff.

[magic python]

I mean, sometimes I might have had a list, dict etc with mixed types - it made that clearer

Using attrs and the validators has been good too

[mortal ermine]

Gotcha. Totally agree about attrs as well. dataclasses have been one of the new to me things I really like. I need to look closer into attrs, for a while I was under the impression it wasn't much of an improvement over dataclasses and I dunno why. I'll look into validators, too... sounds useful, thanks.

[acoustic crater]

how does gc.get_referrers work?

[swift imp]

attrs is a huge improvement. dataclasses is a neutered version, and attrs was around first and the inspiration of dataclasses.

[mortal ermine]

Good to know, thank you. I'll definitely give it a try.

[grave jolt]

attrs doesn't play nicely with pyright

[mortal ermine]

Does dataclasses?

[grave jolt]

yeah, dataclasses is in the stdlib

it's just that pyright doesn't have a plugin system, so third-party libraries that do some voodoo magic just don't get good completion/typechecking

[mortal ermine]

Gotcha, could somebody contribute support?

I know zilch about pyright.

[prime estuary]

All GC-compatible types have a 'traverse' function implemented, which has a function pointer as a parameter. The type then calls that pointer on every attribute/reference it contains (so dicts would call it on all keys and values, for example). This is how GC works, you start with the global modules and executing function scopes, then traverse that to find all objects that are still alive. Then anything that's not findable is inaccessible to any code, so it can be destroyed.

So gc.get_referrers() just runs a traverse cycle, but doesn't collect anything, just records the objects that say they have a reference to the target. gc.get_referrents() just calls the arg's traverse slot.

[mortal ermine]

Can't help but feel like there's a lot hiding behind "GC-compatible". edit: I love this kind of information, btw.

[grave jolt]

https://github.com/microsoft/pyright/issues/607

We will not be providing a proprietary plug-in model akin to the one provided by mypy. There are too many problems with such a model. It doesn't work across type checkers, maintenance of plugins is a problem, distribution and installation of plugins is a problem (since users need to know they exist and how to configure them), and there are potential security issues with plugins that are downloaded and run on users' machines.

[mortal ermine]

Super helpful, thanks.

[acoustic crater]

awesome! thank you!

[prime estuary]

Yep, the idea is that types need to support GC if they're able to produce "reference cycles" - IE there's a way that they can have a reference to themselves directly or indirectly.

[acoustic crater]

I was using it to go backwards through a singly linked lsit and was wondering why it was slower than reversing the list lol

[mortal ermine]

Hahaha

[prime estuary]

Okay yeah that's a very bad idea, it's intended for debugging only.

To do it it has to go through like all objects that exist...

[acoustic crater]

makes sense, I was just messing around

[mortal ermine]

Perfect.

[prime estuary]

The docs are here, if you're curious.
https://docs.python.org/3/c-api/gcsupport.htm

[acoustic crater]

dropped this ༼ つ ◕_◕ ༽つ l

ty tho

[mortal ermine]

I imagine #esoteric-python and #internals-and-peps collide a lot, but I'm new here.

[prime estuary]

#esoteric-python is less "serious", more fiddling around and doing weird or obfuscated things because you can. #internals-and-peps is more a discussion about the language itself as opposed to code - how things work, new PEPs and features, etc.

[mortal ermine]

I wasn't aware, thanks!

[visual shadow]

Oh nah. Esoteric just basically write code to make it as...bad esoteric as possible.

[prime estuary]

Such as 'code golfing' - solve a task in as few lines/characters/etc as possible.

[mortal ermine]

My context would be IOCCC.

What would we do without the people that push the boundaries.

[grave jolt]

the boundaries would push on us, of course

[mortal ermine]

Advanced discussion and wisdom.

[boreal umbra]

I thought that either the work of creating new nodes for the deque would add up, or the work of copying over the underlying array would add up. But I guess for obscenely large sequences, a list would end up being worse per O(n) > O(1).

[charred pilot]

have you tried any larger trials?

[visual shadow]

Wait... Deque should have constant time appends, shouldn't it?

[boreal umbra]

yes, but I assume it's still slightly more work than the best-case append to a list. But the worst-case append to a list is O(n) (if you trigger a resize)

[visual shadow]

Why would it be more work? (just curious)

[boreal umbra]

@swift niche Try asking in #discord-bots

[boreal umbra]

I assume you're creating a struct in C and linking it up to the chain, whereas with the underlying array, you're usually just moving some pointers around, yes?

I don't actually have a frame of reference for how long those operations take

[visual shadow]

Hm, I think you're assuming that a deque is doing something fundamentally different from a list and I don't think that's true

[charred pilot]

python's deque appends just adds another node to a linked list of constant sized arrays when it needs more space

[visual shadow]

I'll be honest, I'm not familiar with how deque is implemented in python, but I'm willing to bet it's functionally about the same

Is it a linked list?

[charred pilot]

yeah, a linked list of constant size arrays

[visual shadow]

Ah, so I guess iteration must be slower?

[charred pilot]

https://github.com/python/cpython/blob/main/Modules/_collectionsmodule.c#L35-L80

[visual shadow]

Appends wouldn't really change much

So okay. I see the overall issue

[boreal umbra]

So each node is an array? interesting

[charred pilot]

yeah, the rational is that it increases the ratio of elements to pointers, saving memory

since the arrays are constant size, you still keep the constant time appends and prepends

[boreal umbra]

makes sense actually

[visual shadow]

What is the size of this array btw?

So in this sense, I imagine lists might get slower to append in the long run somewhat no?

[charred pilot]

it's in that file i linked, 64

[visual shadow]

Ah ok

Ty

[elder blade]

Does the __hash__ implementation of a class need to be unique?

I have a value that is guaranteed unique, but I can remove some information to make it smaller (at the cost of it no longer being guaranteed to be unique, but the likelihood extremely low)

[peak spoke]

No

In [18]: hash(-1)
Out[18]: -2

In [19]: hash(-2)
Out[19]: -2

[charred pilot]

just that it doesn't change, right?

[peak spoke]

You just need to satisfy the equal objects have equal hashes condition, although having many collisions will decrease performance

[finite sparrow]

!e wait ```py
a = {-1: "-1", -2:"-2"}
print(a)

[fallen slate]

@finite sparrow :white_check_mark: Your eval job has completed with return code 0.

{-1: '-1', -2: '-2'}

[finite sparrow]

???? i thought dicts use hash?

[charred pilot]

yeah, but also eq

[finite sparrow]

ah

!e ```py
class A:
def init(self, val):
self.val = val
def hash(self):
return 1
def eq(self, other):
return True
def repr(self):
return f"<A val={self.val}>"

b = {A(1): 1, A(2): 2, A(3): 3}
print(b)

[fallen slate]

@finite sparrow :white_check_mark: Your eval job has completed with return code 0.

{<A val=1>: 3}

[finite sparrow]

makes sense lmao

[elder blade]

Yeah, that's the case for me so this is good.

[elder blade]

Isn't this because -1 is some special value for __hash__ 😅

[peak spoke]

It's probably used to signal an error in the C implementation of hash for ints so it can't return the number directly which is done for others that fit within the type hash uses

[spice pecan]

That was unexpected, but interesting

[uneven fable]

why would hashing an int ever fail

[charred pilot]

isn't the hash stored in a size_t? that's unsigned so it wouldn't be able to deal with the negative values

although, my C interpreting skills are not very good

[unkempt rock]

Who’s Joe

[peak spoke]

hashes are truncated to fit within a Py_ssize_t

[uneven fable]

why would it care if it's signed or not, it's just bits

[halcyon trail]

Usually when working with raw bits you use unsigned

[flat gazelle]

!e
why does

if 1:
    4
 4
```use unindent, but
https://github.com/python/cpython/blob/main/Parser/token.c#L15 uses dedent

[fallen slate]

Parser/token.c line 15

"DEDENT",```

@flat gazelle :x: Your eval job has completed with return code 1.

001 |   File "<string>", line 3
002 |     4
003 |      ^
004 | IndentationError: unindent does not match any outer indentation level

[sand python]

you're just asking why the terminology is different right? I found that the error messages were changed here https://bugs.python.org/issue400734 but no particular reason was given.

although IMO, unindent seems like the more natural term to use

[static bluff]

Regarding the singleton pattern

I have four objects which, functionally, should be singletons

The main Application object, the Window object, the Renderer object, and the Document object.

I know the singleton pattern is controversial and in many ways useless

I'm wondering what the prevailing wisdom is for an object that really shouldn't have more than one instance. I could throw an error when someone tries to create a second one or else re-return the first. Or I could do nothing

I suppose that in terms of function its useless to try to stop someone from making a second instance of an object thats supposed to be singular - its always possible to make a new instance if you really want to. So its a matter of improving usability and context is it not?

[raven ridge]

You should just make one instance of the object, and import that instance everywhere where you need it.

[sand rock]

guys any #math channel?

[raven ridge]

No, there isn't, though it's someone's discussed in the off topic channels.

[sand rock]

i do not find off topic either

[static bluff]

The issue is that I have no control over what my end user will do. It's not my object, but an object my API presents to its user. I guess what I'm asking is, should I npt care if someone screws up their program by creating a second one, or should I put safety-measures in place to dissuade it

[flat gazelle]

I would just put it in the docstring

or even make the class private

[flat gazelle]

!ot

[fallen slate]

Off-topic channels

There are three off-topic channels:
• #ot0-psvm’s-eternal-disapproval
• #ot1-perplexing-regexing
• #ot2-never-nester’s-nightmare

Their names change randomly every 24 hours, but you can always find them under the OFF-TOPIC/GENERAL category in the channel list.

Please read our off-topic etiquette before participating in conversations.

[static bluff]

In Python?

[flat gazelle]

yup

class _MyClassThatYouShouldNotInstantiate:
    ...
instance = _MyClassThatYouShouldNotInstantiate()

[static bluff]

Oh XD

[lament sinew]

If it's the case that having more then one instance will make your app misbehave then yeah, might be worth doing it.

(also the app is badly designed)

Otherwise I wouldn't bother.

[static bluff]

Sorry, you're saying that the API misbehaving is there is more than once instance is a sign of poor design?

[lament sinew]

Yup.

In general.

[static bluff]

I'm not 100% sure if it would misbehave, so much as there is no point

That said, thats a curious notion I hadn't considered

[sand rock]

thenks

[flat gazelle]

there are some classes where multiple instances are just semantically meaningless

I think just marking the class itself as not part of the public API is fine for most purposes.

[visual shadow]

i dont think API having one instance is a sign of bad design. it's fine.

[lament sinew]

I don't think you understood my comment.

[static bluff]

Its a wrapper around a wxPython App instance. It does a lot of work under the hood including (I would imagine) the management of threads. The Renderer object is an instance of cefPython's Browser object which itself handles numerous subprocesses. Now that I think of it in that light, I can see how multiple instances could gum up the gears; though I'd have to do some testing to really describe how/why.

Personally, I think an explicit error which states "you cannot have two of these because it will break the program" is reasonable, if such an architecture is not a sign of poor design to start with

So thats my question, really. You guys are the Jedis so I beseech your wisdom: is a program which is dependant on a singular (and functionally, global) object a sign of poor design?

[lament sinew]

Every software is one global object.

[flat gazelle]

global state exists, how you choose to represent it is up to you, I think a private class that exposes a public instance is good enough

[raven ridge]

Yeah. That's definitely how I'd do it.

[static bluff]

Music to my ears Godly

Thanks for the insight everyone

Also

😐 And I really like this pattern for aesthetic reasons - there's a pattern I'm employing which I'd like your input on. I'm loath to part with it but I wanna get this right

[lament sinew]

keep in mind, every design discussion is just proselytism

[static bluff]

class Public_API_and_Wrapper_Around_Widget_Literal(object):

  class Private_Direct_Subclass_of_Widget_Literal(wx.SomeWidget):
    def __init__(self, wrapper, settings):

      self.wrapper = wrapper
      
      settings = self.validateSettings()

      self.SetSomeSetting(settings.get('someSetting'))

    def validateSettings(self, settings):
      
      validated = {}

      if not instance(setting := settings.get('someSetting', SOME_DEFAULT), SOMETYPE):
        raise TypeError('wrong type')

      else:
        validated['someSetting'] = settings.get('someSetting', SOME_DEFAULT)

    def On_Some_Event_From_Render_System(self, event):
      """disregard the actual mechanics of the event system — the salient point is that
      events from the render system are routed from the widget literal to its wrapper"""

      self.wrapper.dispatchEvent(Event('some-event-type'))

  def __init__(self, settings):
    
    self.widgetLiteral = self.Private_Direct_Subclass_of_Widget_Literal(self, settings)

[steel rose]

anyone know a good vpn with port forwarding ?

[flat gazelle]

  class Private_Direct_Subclass_of_Widget_Literal(wx.SomeWidget):
```is not private

[static bluff]

This is my 'widget pattern' - a direct subclass of an actually-rendered widget, wrapped in my own public API. I like this pattern because it gives me fine grained control over how the actual rendered widget behaves, by way of subclass. But, it also gives me control over the public API of each widget - which would be filled with low level and (for the purposes of my tool) irrelevant methods and attributes

[flat gazelle]

I would suggest not nesting it in the class, since you then have to deal with name mangling or making it accessible to subclasses

[static bluff]

Excellent!

Thank you

[flat gazelle]

if you want a private module member, you do _name, but for classes, you have to do __name and access it like _Type__name

[lament sinew]

My problem with that is not really the nested classes (though i wouldn't recommend doing it anyways)

It's two objects keeping references to each other.

Or the wrappee keeping a reference to the wrapper.

[flat gazelle]

it's fine with singletons in terms of perf impacts, but I don't think you can do a whole lot about this, python doesn't have private inheritance like C++, so you can't exactly break the reference cycle without making the widget public api.

[static bluff]

Could you guys break this down into terms someone who hasnt yet got a comp sci degree can understand?

[flat gazelle]

essentially, you have two objects which hold references to each other

[static bluff]

(Starting this fall, whoo hook!)

Woohoo*

With you so far

[flat gazelle]

which creates some interesting considerations in terms of "not letting the memory usage get unreasonably high", but with singletons, that isn't a concern.
Nevertheless, it creates a very tight coupling between the two objects, which means that changing one means you have to change the other pretty much all the time, which is not ideal.
I would probably be ok with it in this specific case, since all the options to break this coupling tend to be quite verbose and often create excessive abstraction.

[static bluff]

Note, not all widgets are singletons

I'm personally fine with the tight coupling, this this may be my inexperience talking, but on a practical level the two classes are reflections of each other. How one works will always effect how the other works and the should really be considered in tandem.

As you say, I'm not really sure how you could uncouple them, let alone wether doing so is wise.

My most pressing concern is: the pattern on a macro scale is fine? A private override of the renderable, wrapped in a custom public api, that is

"Private override"

[flat gazelle]

you are effectively reaching for private inheritance of C++

and IG this is an adequate way to emulate it

[static bluff]

excellentttttt

My next big project is to teach myself language design (itll take years, and I'm more than fine with that)

And I'd like to try my hand at a python-like language (atleast where the syntax is concerned) with c like privacy and type enforcement

I'd like to have something waiting in the wings for when I do compiler design is year 4

[static bluff]

So to summarize: globals objects as a 'least worst' representation of global state - something that always does and will exist - is acceptable if considered appropriately. Throwing an explicit error to prevent multiple instances of an object, in cases where those additional instances would cause breakage in the program - is an acceptable option. My widget pattern is fine, though the tight coupling of the two classes is a curious case and should be considered thoroughly, and, weak references should be used to prevent memory leakage. The nested class is, however, frowned upon

Thanks my dudes

[gleaming rover]

man I remember this satire post about programming languages that described Objective-C as having "all the memory safety of C and the blazing speed of Smalltalk"

[static bluff]

I'm afraid that means nothing to me 😛

But, cool

[gleaming rover]

basically, C is super fast but memory safety in it is hell

so the joke is like...you would expect it to take the good parts of the languages that inspired it

actually I'll just link it https://james-iry.blogspot.com/2009/05/brief-incomplete-and-mostly-wrong.html

[static bluff]

XD ohhhhhh

@gleaming rover anything to add to the other stuff wee discussed? I'm interested in all the input I can get 🙂

[charred pilot]

nim is pretty good

[grave jolt]

Nim? 🤔

oh, I didn't see the last message

[boreal umbra]

@stuck valley @spark cosmos Your messages were not on-topic for this channel.

[spark cosmos]

where i find that TAP, ?

[boreal umbra]

TAP?

[spark cosmos]

Related channel!

[manic sundial]

I have a question about https://github.com/brainix/pottery. It provides a nice Pythonic API by wrapping Redis constructs with Python Redis-backed data structures (Dict, Deque, etc.). I am using it in a Fastapi microservice project, which is obviously async. My question: is it worth wrapping functions consuming pottery Redis-backed construct with loop.run_in_executor(None, ...)?

I am asking this because it uses redis under the hood which probably already has a form of concurrency in threadpools, so I am unclear about the cost I am paying by calling from Fastapi (async) into redis (sync but with threads)

[elder blade]

Here it says: https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types

If one of those methods does not support the operation with the supplied arguments, it should return NotImplemented.

Does that mean that if other is of an unsupported type I should raise NotImplemented? In my opinion TypeError or ValueError would make more sense but I see the argument for NotImplemented.

[peak spoke]

TypeError will be raised if it can't be handled by the other type

[elder blade]

Ah thank you

[peak spoke]

at least if it doesn't raise its own error

[peak spoke]

Oh, also mind that you're returning NotImplemented, not raising it

[elder blade]

Yeah I noticed that 😅

[static bluff]

I need to build an importer that looks an functions exactly like the built in import handler, except it also searches for .html files when is successfully locates a package

I can't remember where to find that loader so I can subclass from it. Does anyone know off the top of their heads?

[paper echo]

importlib, i assume

[small quest]

SourceFileLoader

[olive marsh]

What is tp slots in Cpython?

[magic nova]

tp slots in CPython are how types written in C implement things like constructors, allocators, what happens during addition etc. There's a list of them here: https://docs.python.org/3/c-api/typeobj.html

[olive marsh]

Yes, I am reading that only. Is there a reason it is being named slots.

[magic nova]

Nothing to do with the language slots feature if that's what you're confused about. I think it was named that because all those slot methods are just a sequence of fields in a struct and you leave them NULL if they're unimplemented

So in a way you're slotting in what you implement

[olive marsh]

Cool, if you want to implement sequence and iterator protocol then implement those in your custom type and you will get it. Thanks man.

[prime estuary]

tp_ also stands for 'type' - in C there isn't really a concept of namespaces, so there's often a convention of adding prefixes like that.

[magic python]

is logging in python confusing or is it just me? I've not logged in anything else, but i find logging quite hard to understand and sometimes it behaves unexpectedly (to me). The docs are also quite hard to read

might just be me not knowing wtf i'm doing though

[sacred tinsel]

I also didnt find the logging module entirely intuitive at first, but I also wasnt very familiar with other languages' solutions

Theres a good diagram in the docs afaik, it should help understand how messages propagate and what filters, handlers, loggers etc do

I'd also suggest looking at how open source projects use it (eg the Python bot would be a good reference I think)

The setup there is not complicated but it should demonstrate the basics well

[magic python]

Fair - not being familiar with other langs i suspect that logging might just be more complicated than i expect, all the handlers and stuff are pretty confusing atm... probably just a case of finding time to read the whole doc tho 😓

i have yet to find a clear explanation of logging across multiple modules across multiple directories though

[paper echo]

what behavior was unexpected?

[magic python]

i was downloading data from gcp and all the calls to that were output to the log lol as DEBUG level

i didn't expect that to happen unless i'd explicitly written somewhere - but this is my lack of understanding of how logging works (to be clear)

i could have basicConfig in a module and import that as a side effect, but autoflake would strip the import lol

[sacred tinsel]

apply it in your init or main

or have a logging utils module that offers a setup function(s)

thats usually my approach

fwiw I do not use basicConfig

[magic python]

main here is main file or main in the module?

bc i have multiple mains in multiple modules

[sacred tinsel]

should each main use a different logging setup? if so, let each main configure logging as appropriate; else define 1 setup func somewhere and import & call it from your mains

[paper echo]

the tldr on logging is that you can define a "logger" object and attach handlers to it.

the loggers are defined globally and are hierarchical, in that the logger a.b is a "child" of the logger a, and all loggers are children of the "root logger". setting a log level and handlers on a parent logger also applies to child loggers.

the common newbie trap is to just work with the root logger, which is what basicConfig does. in your case, you probably want to use your own logger with a unique name, and attach handlers to only that logger.

a common pattern is something like this in every module in your application:

import logging

logger = logging.getLogger(__name__)

then you have a 1:1 relationship between loggers and modules

[magic python]

idk what you mean tho - a module like:

# setup_log.py
import logging
def setup(*, name): 
    logging.basicConfig(
    format=( "%(levelname)s"),
    level=logging.INFO,
    filename="./log.txt",
) 
  logger = logging.getLogger(name = name)
  return logger

[magic python]

right - but then where's the config done ?

[paper echo]

in __main__ for your application

[magic python]

I dont have that typically

[paper echo]

sure, just wherever the application is run

never inside a "library"

[magic python]

there are multiple modules that are called from multiple scripts, there is no centralised "main"

that calls everything else

[paper echo]

can you be more specific

[magic python]

i mean - there is no __main__

it's not a thing i can put basicConfig in

i'm working with a clusterfuck of modules in a mono repo not some tidy library : ' )

[sacred tinsel]

each of your scripts must have a conceptual "main", something that orchestrates everything, this is where you generally call the logging init function

probably as soon as possible

[paper echo]

in a single "application" you might want to do something like this in myapp/logging.py:

import logging

log_format = "%(levelname)s, ms: %(relativeCreated)d, line: %(lineno)d, module: %(module)s, msg: %(message)s"
log_date_format = "%m/%d/%Y %I:%M:%S %p"

default_level = logging.INFO
default_filename = "./log.txt"

logger = logging.getLogger("myapp")

def setup_logging():
    file_handler = logging.FileHandler(default_filename)
    formatter = logging.Formatter(log_format, datefmt=log_date_format)
    file_handler.setFormatter(formatter)
    logger.addHandler(file_handler)

then in myapp/__main__.py you might do

from myapp.logging import setup_logging
from myapp.run import run

setup_logging()
run()

[paper echo]

but you don't type clusterfuck on the cli and stuff just happens

what actually is the setup you're working with?

[magic python]

no i type dvc repro ...

[paper echo]

ok, so you have a bunch of standalone .py files and you want each one to log to some sensible and/or configuration-defined location?

[magic python]

yes - there's some logical grouping of them in terms of directory structure

there's not a python package structure though i guess

i might also have a/b/c/x.py log somewhere else than a/b/d/y.py i guess, as a/b/c is not part of a/b/d or related to it

[magic python]

I guess with this approach one could just do

from  .. import setup_logging
setup_logging.setup()

or whatever though - there's no requirement for it to be in __main__? Or is there 🤔

[paper echo]

ok, that helps.

you can still use this pattern in scripts/a/b/c/x.py:

import logging

logger = logging.getLogger(f'myproject.{__name__}')

def main():
    logger.info('Starting run!')
    ...

if __name__ == '__main__':
    from utils.logging import setup_logger
    setup_logger(logger)
    main()

and something like this in utils/logging.py:

import logging

log_format = "%(levelname)s, ms: %(relativeCreated)d, line: %(lineno)d, module: %(module)s, msg: %(message)s"
log_date_format = "%m/%d/%Y %I:%M:%S %p"

default_level = logging.INFO
default_filename = "./log.txt"

def setup_logger(logger, level=default_level):
    file_handler = logging.FileHandler(default_filename)
    formatter = logging.Formatter(log_format, datefmt=log_date_format)
    file_handler.setFormatter(formatter)
    logger.addHandler(file_handler)
    logger.setLevel(level)

restricting this to "main" (whatever "main" actually is) helps prevent the situation where you accidentally enable logging where it shouldn't be enabled

on the other hand, maybe you want to always enable all myproject.* loggers in every run, so then you'd do this in scripts/a/b/c/x.py:

import logging

logger = logging.getLogger(f'myproject.{__name__}')

def main():
    logger.info('Starting run!')
    ...

if __name__ == '__main__':
    from utils.logging import setup_logger
    setup_logger(logging.getLogger('myproject'))
    main()

or maybe you want enable DEBUG logging on some loggers and INFO on others

i really wish basicConfig let you choose a logger other than the root logger

it would make it so much easier to deal with logging

all it needs is a logger= kwarg...

[magic python]

i found some dictionary config that might have had something along those lines, i can't remember though

[paper echo]

meh

i wonder if this has been discussed on python-ideas ever

i'm sure it's been proposed before... right?

maybe there's a good reason it was rejected

[magic python]

I'm going to try the approach above - but presumably i'd have this for each a/b/c, a/b/d/ and so on, i'd have some a/b/<whatever>/log_setup.py which was used for those modules

[fallen slate]

Hey @olive marsh!

It looks like you tried to attach file type(s) that we do not allow (.epub). We currently allow the following file types: .gif, .jpg, .jpeg, .mov, .mp4, .mpg, .png, .mp3, .wav, .ogg, .webm, .webp, .flac, .m4a.

Feel free to ask in #community-meta if you think this is a mistake.

[paper echo]

make it easier on yourself and put the log setup code in the same place for every script

in my data science projects i tend to keep a top-level lib directory with various one-off modules and packages that i reuse across the project

[magic python]

and then over ride if necessary?

[paper echo]

depends on how widely it has to vary

[magic python]

no idea atm just setting it up

[paper echo]

you can start by just copying and pasting the log setup boilerplate in to your "main" section

[magic python]

for every module?

[paper echo]

(and yes i recommend the if __name__ == '__main__' pattern even if you don't strictly need it, if only for visual clarity)

[magic python]

I use that

[paper echo]

copy and paste it 3 times, then start abstracting the parts that are obviously duplicated 🙂

[magic python]

well i'll copy paste it 6 times right off the bat

i don't expect it to vary between those, as they're all in the same dir, which is what i was confused about setup

i'm going to say this is not a straightforward module at this point 🙃

[paper echo]

does this help at least elucidate how the logging system works?

[olive marsh]

Does anybody wants to read this book with me https://leanpub.com/insidethepythonvirtualmachine ?

[magic python]

if i can get it working yes - if not then i guess no lol

i get the idea that - for any given run time - there's a single logger process and things attach to that... but how they do is messy atm

it still prints out loads of stuff from gcp and slows shit down as well

meh, i quit logging lol

[paper echo]

can you show some of your code?

what do you mean "from gcp" - from inside some google gcp sdk?

it's possible that they wrote their logging code badly

[magic python]

Sorry i've deleted logging and restored print lol

it works, i can redirect to a file, i'm not getting the last two hours wasted on that module back at this point : ' )

[magic python]

maybe , no idea - when i have things which are downloading from gcp i get a lot of debugging output written that i didn't ask for - and it makes everything slow as hell

[paper echo]

i was more interested in how you "set up" the logging

if you did logging.basicConfig(level=logging.DEBUG) then i'm not surprised you got a bunch of debugging output from deep inside every library you used

like i said, that applies to the root logger

and all loggers are child loggers of the root logger

so the level you set on the root logger applies to all loggers

[magic python]

tried a bunch of ways lol, fair, i accept i'm not getting logging at this point

[paper echo]

i think you're just busy mid-project

there's not much to get, but i think you will do better if you can sit down for an afternoon and focus on it

there is the "logging cookbook" but i don't know how useful it really is

i would love to find a good python doc that spends a lot of time covering the basics

[magic python]

eliot logging looked interesting

[paper echo]

very very carefully and thoroughly checking for understanding with exercises, etc

eh, it's a totally different system. i still suggest learning logging

eliot is definitely interesting but the fact that it doesn't seem to play nice with regular logging was a turn off

[rugged latch]

Hi everyone

I hope you are doing fine

[magic python]

regular logging seems shit tho, idk why i should spend an afternoon dedicated to what's essentially printing a message to a file

[paper echo]

because it's not shit, it's complicated because it's flexible

print means "do i/o all over your application"

that's not good in a lot of situations

[magic python]

What proportion of that flexibility is required and what proportion gets in the way?

Seems the balance is off

[paper echo]

it's a sensible balance for "app dev"

[magic python]

*required for most purposes

[paper echo]

maybe a bit heavyweight for scripting

[magic python]

Well maybe I'll look at loguru or something, seemed to just have colours, but if it's easier then I'll take it

[paper echo]

idk, in your case it sounds like you don't need it

print and redirect stdout to a file

also if basicConfig had a logger= kwarg you wouldn't have this problem

[magic python]

I don't know at what point I'd require logging Vs printing

[rugged latch]

@paper echo excuse me I am just beginner in python

[magic python]

Obviously you can change levels etc

[rugged latch]

Still in for loop stage

Actually I need guidance

[paper echo]

@rugged latch if you have a question, you might want to ask for help in a help channel. see #❓｜how-to-get-help

[paper echo]

if you want to log messages from inside a library, but give the end-user control over when to actually generate log messages and where to send them

e.g. turn on/off debug messages in said library

or if you're writing a web server and you want to do different things with different kinds of log messages

[rugged latch]

@paper echo thanks

[halcyon trail]

if you have packages that are logging too much

you can easily prevent that by just setting alogger for tha tpackage with no handlers and propagate = false

I had to do that with I think paramiko

and that's basically exactly the reason you use logging, because you can do that

if you use print statements everywhere, then you don't have control of these things without changing the source code

logging gives you tremendous code over what's happening in other code without changing it

the logging solution in python is honestly pretty sweet, I've worked with various approach to logging and I think this is about as good as it gets, and I think the docs are excellent.
IME all that most python files need is

logger = logging.getLogger(__name__)

that's all, one line setup. if your file can actually be invoked as a script, i.e. if it can be main, then you need to decide how to handle your logger setup

[shadow spade]

any django web developer here?

[grave jolt]

If you have a question about Django, you can ask in #web-development. Instead of asking whether there's someone who thinks they're an expert, just ask your question there.

[shadow spade]

thanks. 🙂

[paper echo]

is this enough to safely prevent a circular import between models.user and models.subscriber?

models/user.py:

import attr
from models.subscriber import EmailSubscriber

@attr.s()
class User:
    username: str = attr.ib()
    def make_email_subscriber(self, email: str) -> EmailSubscriber:
        return EmailSubscriber(email, user=self)

models/subscriber.py:

from __future__ import annotations
from typing import Optional, TYPE_CHECKING
if TYPE_CHECKING:
    from models.user import User
import attr

@attr.s()
class EmailSubscriber:
    email: str = attr.ib()
    user: Optional[User] = attr.ib(default=None)

maybe make_email_subscriber should be a top-level function in models/subscriber.py but you still get a circular import w/ the type annotations

[halcyon trail]

not sure what you have omitted but if this is really the heart of it then it seems pretty simple and preferable to just break the circular dependency completely

make_email_subscriber doesn't really touch the insides of User at all

user.make_email_subscriber(email) is the same as EmailSubscriber(email, user)

[paper echo]

@halcyon trail yeah, assume there's additional data processing that needs to happen between the "user" and the "email subscriber"

basically just enough that you don't want to leave it as boilerplate, in my case

[peak spoke]

I think that should work, doing the import directly with the non-from syntax may also prevent it from erroring

[paper echo]

i think i tried that and got the "uninitialized module" error

i really don't like the if TYPE_CHECKING thing but i guess that's the only thing to do

oof, i just found a runtime circular dependency between these classes

[halcyon trail]

yeah, circular dependencies...

[paper echo]

each User occasionally needs to look up or create an EmailSubscriber, while each EmailSubscriber occasionally needs to look up a User

[halcyon trail]

idk I see questions about circular design a staggering amount here, on the C++ discord, etc, relative to how often they are appropriate in real life (almost never)

So I think about 95% of all cases where people ask about a circular design, it's not appropriate. I just don't know which 95%.

[paper echo]

in this case it's a very very "natural" thing to do

i'm trying to figure out a pithy example to demonstrate why it's so easy/tempting in python

[halcyon trail]

yeah, I think that's the problem, people learn OO from cats and dogs and squares and circles and try to design things based on what feels natural, instead of software engineering principles

[paper echo]

it's semantically natural, yeah

other than "it needs to not crash" i don't know if there's a software engineering principle to be invoked here, but that's why i ask questions

[halcyon trail]

well, since I don't know your overall domain/design I have to kind of clutch at straws

[paper echo]

yeah let me try to work an example

[halcyon trail]

but AFAICS, user seems like a member of an EmailSubscriber.

The question is, why does the user need to know about the EmailSubscriber class at all? User seems like a lower-level thing.

Some higher level class ought to be responsible for doing the lookups

[paper echo]

the situation is:

• people can sign up for mailing lists without being users
• any given user might or might not be signed up for a particular mailing list

it's easy and natural to want to create a new email subscription based on a user

and that's something the application has to do

[halcyon trail]

Yes, but just because you want to be able to create a new email subscription based on a user

doesn't mean that the user has to know about that operation

those are two different things

[paper echo]

indeed, that seems like the logical place to work on refactoring. the problem however is that the application currently has a "hook" to automatically keep all email subscribers in sync w/ a user, when the user is saved

and the way the application is designed, this hook is a method of a User

[halcyon trail]

Yeah, I mean I'm not sure who wrote the code and how long ago

[paper echo]

i have enough control over this code that i can refactor it, but i'm not sure how

[halcyon trail]

but whatever design decisions people make, obviously further design decisions get made on top of that and they get "baked in"

I could probably give suggestions but I'd need to know more of the overall structure

that said maybe it's just not worth it

like, if the code works, okay, just because I wouldn't write it with a circular dependency, doesn't mean it's worth rewriting

(or refactoring, or whatever)

in fact that's like 80% of real life coding 😛

[paper echo]

hah yep

class Document:
    def save(self):
        self.do_database_stuff()
        self.post_save_hook()

    def post_save_hook(self):
        pass


class User(Document):
    email_address: str

    def post_save_hook(self):
        self.update_email_subscriptions()

    def update_email_subscriptions(self):
        subscriptions = EmailSubscription.find_by_email(self.email_address)
        ...


class EmailSubscription(Document):
    user_id: str

    def get_user(self) -> User | None:
        return User.get_by_id(self.user_id)

[halcyon trail]

I'm adding a feature to our code right now. The guy who used to be the lead developer on the team had a class design where there was this "low level executor" class whose job was to write the API of the trading infrastructure we use, and handle the job of placing trades. Then he had a "high level executor" class that reflected our actual strategy, approach, etc to execution.
This is all reasonable enough although the execution (pun intended was messy).
However, when another guy on m y team wanted a different kind of "high level" execution, and found that the low level executor didn't have a flexible enough API, he wrote another low level executor.

So, by writing another class, he completely defeated the purpose of the design of having a high level, and low level, executor. And at this point it would really just be easier probably if they were one class. But we have two high level, and two low level executors and I just don't have the energy, patience, time, etc to fix it 🤷‍♂️

[paper echo]

this is a very rough sketch; both EmailSubscription.get_user and User.update_email_subscriptions are used in real code. i do have the freedom to (gently) refactor the code that uses these methods, but i don't see a sensible way to do it that isn't convoluted.

[paper echo]

oof yeah that's no good

bad abstraction vs no abstraction

[halcyon trail]

yeah

[paper echo]

no reason to write the low level executor if you don't need a low level executor

just write the high level one and extract common stuff if needed

also sans-io is your friend here

i made that dev's mistake many times before i realized how to not make it

what's bothering me here is that i don't see any individual obvious bad design decision

each individual decision here makes sense, but the result is something that "shouldn't happen"

the truth is that in the domain itself there is a circular dependency between users and email subscriptions

[halcyon trail]

I mean the init for EmailSubscription can simply update the document's subscriptions

[paper echo]

what do you mean by that

[halcyon trail]

well, idk, maybe you should show a complete example where post_save_hook actually gets called

[paper echo]

oh it's only called by Document.save

but we call .save on User instances all over the place

and the decision to update the email subscription on user save was not a throughtless decision, we considered various alternatives and decided that this was the most sensible thing to do

what is less valuable is the EmailSubscription.get_user method, although there is at least one instance where we really do need to look up the user from the email subscription

[halcyon trail]

It's really hard for me to say without seeing more of it. Like I said, you could just update user's subscriptions whenever a new EmailSubscription is created

[paper echo]

what ORMs might call a "backreference" i guess

[halcyon trail]

it depends on the different ways in which a user's subscriptions can be changed

it seems like here, everything is being mediated via globals. Some global registry of subscriptions gets updated, and then when you "save", the subscription list gets updated from the globals.

[paper echo]

no these are mongodb documents

sorry, head too deep in this

[halcyon trail]

yes, and your mongo database state is a global

[paper echo]

sure, whenever the global "user" list is updated, some work needs to be done to keep the global "subscription" list in sync

[halcyon trail]

it just depends I guess if you are concerned with updates happening elsewhere on other threads, processes, etc, or just local stuff

but then this is getting into really big picture design

[paper echo]

yeah synchronizing/collating/reconciling concurrent updates is out of scope for this situation (fortunately)

[halcyon trail]

if it's out of scope then that means that ultimately all of the updates to subscriptions is happening based on local stuff

if that's true, then I'd rather update it based on local variables and arguments that get passed around rather than changing something in a global registry and then looking it up in that global registry

If the only possible way to add a new email subscription is to ultimately call EmailSubscription(email_address, user), then EmailSubscription.__init__ can update the subscriptions member on its user argument

[paper echo]

oh, i see

no... some email subscriptions don't have an associated user

but sure, you could have some kind of cascading behavior like that

[halcyon trail]

if there's no user, then the second argument is none, and then it doesn't update the subscriptions

[paper echo]

what happens if the user later signs up for an account 🙂

or if they change their email address

[halcyon trail]

Then there's another function or class for that, and that specific function or class is responsible for maintaining those invariants

[paper echo]

that's fair and an interesting idea

[halcyon trail]

if someone is signing up for a User account, then the User instance is being created right there in that function

when that User is created, you would look up the existing subscriptiosn for that email, and set subscriptions correctly

[paper echo]

sure, that's what we had originally

then we did that in 5 different places and it was decided to enforce that it always happens on save

now what you could do is set up a kind of "3rd party" invariant checker/enforcer thing

[halcyon trail]

I would be fine with having the save thing verify that the subscriptions match (assuming again that concurrent changes aren't a thing)

the problem is that by using save, like I said, you've channeled lots of important logic to go through mutation of globals

making it harder to read the code, harder to unit test, etc (all the usual downsides of globals)

[paper echo]

at some point you have to mutate the database, im not sure thats the issue

[halcyon trail]

i'm not saying you don't have to mutate the database. I'm saying that if you want to change foo, then change foo directly by passing it in as an argument. You're changing the database and then making foo call some function to ensure it gets updated from the database.

like, to me what you're doing is taking

def assign_foo(x, f):
    # update database for this assignment
    x.foo = f

And now you're writing

def assign_foo(x, f):
    # update database for this assignment
    # don't update x.foo; instead the user of assign_foo will call `x.update_from_db`

[paper echo]

are you suggesting something like this? it kind of helps because it moves things out of the "subscriber" and "user" namespaces, but then you also have to define your invariants in a separate location

_post_save_hooks = []

def register_post_save_hook(func):
    _post_save_hooks.append(func)

def save(obj):
    save_to_database(obj)
    for hook in _post_save_hooks:
        hook(obj)

[halcyon trail]

i didn't think so, no 🙂 But probably it's just too hard for me to give suggestions that make any sense, I don't understand enough of your overall setup

[paper echo]

think: ORM

except with mongodb instead of mysql/sqlite/etc

i'm actually not sure how ORMs handle this, because i imagine they'd have this problem now and then

CREATE TABLE user (
  id INTEGER PRIMARY KEY, email TEXT
);
CREATE TABLE subscription (
  id INTEGER PRIMARY KEY, email TEXT, user_id INTEGER,
  FOREIGN KEY (user_id) REFERENCES (user)
);

that's basically what we have, pretty straightforward stuff

i guess this is my point: the easy and obvious solution is a direct consequence of something that happens in "real life", but is not representable in code

there are entirely different ways to structure applications, and i think that's what you're suggesting

but even in those cases, you still end up with circular dependencies between namespaces, even if you entirely discard OO notions

[halcyon trail]

why do you think you'll always up with circular dependencies between namespaces?

[paper echo]

not always of course, but surely it makes sense to define a lookup_user_in_db function in a "users" module and a lookup_subscriber_in_db function in a "subscribers" module

at some point, code that deals with users has to call lookup_subscriber_in_db, and code that deals with subscribers has to call lookup_user_in_db

[halcyon trail]

at the end of the day, it's always a choice about how to represent things. users don't have to know about email addresses, and even vice versa, they could both just store "primitive" information and you can have a separate class for trackling relationships between them

[paper echo]

hm, that is one option

[boreal umbra]

@unkempt rock well for one thing, you can add arbitrary attributes to functions, and they get stored in the __dict__

You can use that functionality if you want some kind of state persistence between function calls

Though it would be annoying to refactor

I'm not really sure

[prime estuary]

It exists because there needs to be somewhere to store the variables a function closes over.

[fallen slate]

:incoming_envelope: :ok_hand: applied mute to @unkempt rock until <t:1627947123:f> (9 minutes and 59 seconds) (reason: duplicates rule: sent 4 duplicated messages in 10s).

[lusty scroll]

what about just having both the User and the EmailSubscription class in the same file?

[paper echo]

not possible or feasible in this case, but yeah that is one way to fix it

[unkempt rock]

hi, I am trying to imitate the python dictionary but upon reaching 800,000 values ​​in the following code, seems to see cpu overhead (idk) but it takes a long time to look up the value in the list, I don't know if this looks like a b-tree:

class Map:

    def __init__(self):
        self._storage = []

    def get(self, key):
        size = len(self._storage)
        minbuffer = 0
        maxbuffer = 1000

        while True:
            for _key, _value in islice(self._storage, minbuffer, maxbuffer):
                if _key == key:
                    return _value
            minbuffer = maxbuffer
            maxbuffer *= 2
            if minbuffer >= size:
                return None
        return None

    def put(self, key, value):
        self._storage.append((key, value))

    def delete(self, key):
        size = len(self.__storage)
        minbuffer = 0
        maxbuffer = 1000
        indexnum = 0
        while True:
            for _key, _value in islice(self._storage, minbuffer, maxbuffer):
                indexnum += 1
                if _key == key:
                    del self._storage[indexnum]
                    return True
            minbuffer = maxbuffer
            maxbuffer *= 2
            if minbuffer >= size:
                return None
        return None  

@hearty drift

[hearty drift]

at first glance this literally just looks like a list and linear search

not sure what the buffer logic you have there is doing

@unkempt rock can you confirm

[unkempt rock]

For some reason in other codes islice it was faster for me

[hearty drift]

doubt it

could be simplified to

for _key, _value in self._storage:
  if _key == key:
    return _value
return None

Also, yeah, this is neither a B-tree or a hashmap, this is literally just an array and linear search

[charred pilot]

plus when you delete you're just deleting values from the list, which is O(n)

python's dict uses a hashmap

[unkempt rock]

It's like you say, linear search.

[hearty drift]

yes, so it runs in linear time

[charred pilot]

the time it takes to search is O(n), where n is the number of elements

[hearty drift]

explains why it's slow

[charred pilot]

accessing a value in a dict is O(1), it's always the same (theoretically)

[unkempt rock]

This block?

[unkempt rock]

I did not understand much about hashmap, if I know that it uses hash but how is it possible to return the saved value over a hash?

[charred pilot]

are you following a resource about data structures?

[paper echo]

@unkempt rock this maybe belongs in #algos-and-data-structs

or at least you might get more relevant help there

[visual shadow]

Basically, this is not a dict. That's why it's slow. This class doesn't capture the essence of what makes a dict work, which is hashing. So this class will never match the performance characteristics of a dictionary. You might want to read up on how dictionaries work

[verbal escarp]

except if the thing is sorted

and you can bisect

or use another feature to guess the position

then you can easily approach the performance of a b-tree and even might have a better performance than a hashmap

[verbal escarp]

which is also why that statement isn't true for all inputs

hey @paper echo we took your suggestion about using pip in a subprocess and worked around it, interested?

it's still highly experimental and doesn't work on all platforms

[flat gazelle]

the class doesn't implement a btree map, it implements what's more or less a lisp alist

but yes, you can make a dict without hashing

and even then, a dict backed by a bisectible vector would work poorly with insertions

[verbal escarp]

a sorted list can do very well if you can guess positions

[flat gazelle]

yes, but that is not what that class does

[verbal escarp]

true

[flat gazelle]

and well, inserting into a sorted list is O(N) for at least some inputs

so... is that really the best option for a mutable mapping

[verbal escarp]

depending on dict implementation and collision handling, you can have similar problems

the python dicts are extremely well designed though, a lot of work has gone into those

https://www.youtube.com/watch?v=npw4s1QTmPg - just to appreciate how much work actually went into them

[flat gazelle]

yeah, that's a great talk

[paper echo]

Curious definitely

[verbal escarp]

def load_mod(package, version):
  venv_root = \
    Path.home() / ".justuse-python" / "venv" / package / version
  if not venv_root.exists():
    venv_root.mkdir(parents=True)
  venv_bin = venv_root / "bin"
  python_exe = Path(sys.executable).stem
  if not venv_bin.exists():
    venv_output = subprocess.check_output(
      [
        python_exe, "-m", "venv",
        venv_root
      ], shell=False, encoding="UTF-8"
    )
    log.debug("venv created: venv_output=%r", venv_output)
  current_path = os.environ.get("PATH")
  pip_args = [
    "env",
    f"PATH={venv_bin}{os.path.pathsep}{current_path}",
    python_exe,
    "-m", "pip",
    "--no-python-version-warning",
    "--disable-pip-version-check",
    "--no-color",
    "install",
    "--progress-bar", "ascii",
    "--prefer-binary",
    "--no-build-isolation",
    "--no-use-pep517",
    "--no-compile",
    "--no-warn-script-location",
    "--no-warn-conflicts",
    f"{package}=={version}",
  ]
  pkg_path = None
  log.info("Installing %s, version=%s", package, version)
  while not pkg_path:
    output = subprocess.check_output(
      pip_args, shell=False, encoding="UTF-8"
    )
    log.debug("pip subprocess output=%r", output)
    match = re.search(
      f": {package}=={version} in (?P<path>(?:(?! \\().)+)",
      output
    )
    pkg_path = match.group("path") if match else None
  assert Path(pkg_path).is_dir()
  orig_cwd = Path.cwd()
  try:
    sys.path.insert(0, pkg_path)
    os.chdir(str(pkg_path))
    return importlib.import_module(package)
  finally:
    sys.path.remove(pkg_path)
    os.chdir(str(orig_cwd))

[visual shadow]

Yes. Though I say thats a strawman argument. Time complexity is generally accepted to apply for general cases, not hand tailored inputs. It would make for rather tiring conversations if it had to be clarified each and every time. In any case, I don't wish to clarify edge cases when I refer to performance characters of dicts, since its just noise for those new to it and redundant information for those who already know.

[verbal escarp]

calculating a hash for very large inputs can take much longer than just comparing

[visual shadow]

Sure. But again, forest or the trees something something

[verbal escarp]

nodnod

[paper echo]

oh interesting, you just went with import_module and sys.path

note: i would put sys.path.insert(0, pkg_path) outside the try

why sys.path and chdir?

why --no-use-pep517 --no-build-isolation --no-compile?

also interesting that you used env with PATH

i would have expected this:

pip_args = [
    str(venv_bin/'python'),
    "-m", "pip",
    "--no-python-version-warning",
    "--disable-pip-version-check",
    "--no-color",
    "install",
    "--progress-bar", "ascii",
    "--prefer-binary",
    "--no-warn-script-location",
    "--no-warn-conflicts",
    f"{package}=={version}",
  ]

(it'd be really nice if subprocess.* accepted Paths as args and str-ified them for you; or does it?)

[verbal escarp]

@broken furnace questions for you to answer 🙂

[paper echo]

I suppose setting the path could be safer for really ill behaved set up scripts

[broken furnace]

well, with just chdir(), tests failed, so it seems like it needed to be added to sys.path for some reason

[paper echo]

But those kinds of scripts would be breaking peoples dockerfiles and such anyway, I don't think it's worth accommodating them and they are rare

[paper echo]

I would imagine that you only need sys.path and not chdir

[broken furnace]

probably so, there's some extra fat for sure

is there a way to get the pkg_path without running pip twice?

[paper echo]

Good question, can it be parsed from the installation output

Really frustrating that there's no stable python API and to my knowledge no stable machine-readable output format/schema

I know, small team underfunded with too much to do

[lusty scroll]

it's probably in the verbose output; maybe I'll see if it cam be parsed out of there

[lusty scroll]

if pip is installed in the venv, it says:
Using pip 20.3.4 from <venv dir>/lib/python3.9/site-packages/pip (python 3.9)
that's the closest thing I see

this gives the same path, along with several others:

>>> site.getsitepackages(site.PREFIXES)
['<dir>/lib/python3.9/site-packages', '<dir>/local/lib/python3.9/dist-packages', '<dir>/lib/python3/dist-packages', '<dir>/lib/python3.9/dist-packages']

[unkempt rock]

I think the dictionary is built in C so equal performance was not expected but at least it doesn't take 10 seconds and overload the CPU to look up a value.

[charred pilot]

the language isn't the issue here. you're using an entirely different algorithm

[gleaming rover]

the point is

the time complexity is different

you have not built a dictionary at all

also, I would suggest #algos-and-data-structs instead

[unkempt rock]

https://github.com/python/cpython/blob/main/Objects/dictobject.c

[unkempt rock]

👍

[gleaming rover]

yes, we know the CPython dict is implemented in C. but again, I would say one of the fundamental characteristics of a dict is sublinear time insert and lookup in the general case. what you built doesn't have that.

[visual shadow]

I don't mean in terms of the implementation speed. I mean the algorithm or "logic" of a dict is absent here

[split needle]

Hi all, this code changes hair color in a photo. I am having trouble, just making the color black by changing the RBG: https://github.com/zllrunning/face-makeup.PyTorch/blob/master/makeup.py

Can anyone look at this small file real quick, and tell me what I might be doing wrong?

I tried just changing RGB to all 0

[visual shadow]

This is not a help channel. Please see #❓｜how-to-get-help

[unkempt rock]

@unkempt rock Please don't randomly post links here. This is an on-topic channel.

@unkempt rock My brother did to bother me.

[verbal escarp]

oh, you're not fond of filenames as APIs? 😉

[static hinge]

python actually does compile down to bytecode.

[verbal escarp]

hu?

that's pretty much unrelated..

[flat gazelle]

@echo steppehello, we do not allow recruitment

[echo steppe]

Ok

[gilded ice]

Yo anyone here use heroku?

[spark magnet]

You can ask in #python-discussion , but maybe change your pronoun joke first?

[unkempt rock]

Need some advice, perhaps this is a more advanced question.

I've written a package MyApiLib that exposes resources on a REST API as a Pythonic objects. It uses the requests library to make the low-level requests to the upstream API server, and the rest is all Python.

I'm making a frontend in Flask which uses MyApiLib to communicate with this server. How do I manage multiple request libraries for multiple users concurrently using this package's library?

Currently I tried using contextvar and setting a global instance of the request.Session, but that seems to assume a single instance of the object. Contextvars also don't seem persist within flask request contexts (or get overwritten or something. I'm not sure I get it). Regardless, I'm not sure a contextvar is really what I need, given that I want to support multiple user sessions having multiple requests to this API server via MyApiLib. Thoughts?

[gusty folio]

Think this fits more in #software-architecture but have you considered to make the requests async?

[elder blade]

You could create a request.Session on each request and place it on app, session or g (not global).

[sturdy timber]

Is making a method a classmethod because it just need's to call a staticmethod on it's class (and would otherwise be a staticmethod itself), a weird thing to do? I'm not sure why it would be weird, it's just not something i've seen before. Alternatives I can think of would be to move the staticmethod out of the class, or into the function itself, not sure what would be preferred, or if it doesn't matter.

[open trout]

Can you show where you encountered it?

Sounds pretty interesting. My gut feeling says that it's a valid design choice.

class Foo:
  @staticmethod
  def bar():
    print('bar')
  
  @classmethod
  def baz(cls):
    cls.bar()

so like this?

[sturdy timber]

    def generate_poem(self):
        rhymes = {}
        for line_index in range(4):
            end = rhymes.get(line_index)
            line = self.chain.generate_sentence(end)
            if line_index in RHYME_SCHEME:
                rhymes[RHYME_SCHEME[line_index]] = self.get_rhymes(line[-1])
            yield line

    @classmethod
    def get_rhymes(cls, word):
        word = cls.clean_for_rhyme(word)
        res = requests.get(f"https://api.datamuse.com/words?rel_rhy={word}")
        for word in res.json():
            if word["word"] in self.chain.chaindict:
                return word["word"]
        return None

    @staticmethod
    def clean_for_rhyme(word):
        word = "".join(i for i in word.lower() if i in ascii_lowercase + "'").strip("'")
        if word.endswith("'d"):
            word = word[:-2] + "ed"
        word = word.rsplit("'", maxsplit=1)[0]
        return word

yeah, this is the context pretty much.

The clean_for_rhyme is specific to get_rhymes, so I don't think it would really be used anywhere else

[open trout]

I'll give thumbs up for this here, I see no reason for why it would be a bad idea

[sturdy timber]

Oh wait a second lol, self.chain.chaindict, get_rhymes relies on the instance anyway🤦‍♂️

[open trout]

lel

[undone hare]

I think it makes sense, it is quite inheritance friendly this way

[sturdy timber]

Ah yeah, hadn't thought of it in terms of inheritance

[unkempt rock]

Would I have to manage mapping sessions with request.Sessions, or does Flask do this automatically?

[elder blade]

I am not the best with Flask, but there should be a session object you can import from flask I think.

[paper echo]

you could attach the session to the app as an "extension". see
https://flask.palletsprojects.com/en/2.0.x/api/?highlight=session#flask.Flask.extensions

def create_app():
    app = flask.Flask()
    app.extensions['myapp_http_client'] = requests.Session()
    return app

(this belongs in #web-development or #software-architecture )

[elder blade]

These two code blocks are the same right?

class A:
    pass

class B:
    pass


class C(A, B):
    pass

class B:
    pass

class A(B):
    pass

class C(A):
    pass

[flat gazelle]

in regards to class C, yes

[elder blade]

Yeah thank you

[wide shuttle]

They are not necessarily the same, although they are similar

Consider B implementing __init_subclass__: the first one will run that only once; the second will run it twice

If your inheritance tree gets more complex, the linearisation of the __mro__ may not be the same

[elder blade]

I see, thank you.

[boreal umbra]

I know I'm late to the party on this one. It used to be that in PyCharm, if you have a method that doesn't use self, it suggests that you make it a static method. Now it suggests that you move it out of the class and does it for you. However, even in a static method, I'm pretty sure you can refer to methods of the class by the name of the class, but it will always be that exact class even if you're accessing it from a subclass.

class A:
    @staticmethod
    def foo(bar):
        return A.some_other_static_method(bar)

(Which is why factory methods, or whatever those are called, should be class methods.)

[halcyon trail]

i think in all cases, whether it's a factory function or not, you ought to consider whether you can/want to support inheritance

for factories I would usually default to classmethods, but not always, but in the other example, maybe you don't want to support derived classes overriding the behavior of your class/static method

basically, not every class has to be inheritance-friendly, IMHO

[grave jolt]

well, there's typing.final for that 🙂

!d typing.final

[fallen slate]

typing.final

@typing.final```
A decorator to indicate to type checkers that the decorated method cannot be overridden, and the decorated class cannot be subclassed. For example...

[grave jolt]

In Kotlin it's the other way around -- you have to explicitly make a class or method open. I think that makes a lot of sense

[elder blade]

Python is duck typed, so I think it makes more sense in this context that you explicitly define something as final

As well as with Python not having private and public anything, it goes with the rest of Python.

[spark magnet]

python isn't good at saying, "you can't"

[flat gazelle]

it is, however, good at saying "you shouldn't", and that's generally sufficient

[magic python]

is there any reason to use a function as a decorator rather than using a class and creating a __call__ method ? I've only recently started messing around with them, and when there are three nested function it seems kind of messy? I'm not sure it's pure choice, or whether there are some scenarios one would prefer to use a class as a decorator than a function

[halcyon trail]

@grave jolt yeah, I mean a lot of things in Kotlin are just based on recommendations that grew out of java

so '"Effective Java" will actually tell you to slap on final by default

and a pretty broad swathe of the Java community (at least, out of the people who aren't stuck in over-OO of a couple decades ago) agree

so Kotlin logically just makes final the default, you have to add open

once you have interfaces and final-by-default, it really decreases the OO headaches by a huge amount

[flat gazelle]

it mostly doesn't matter, functions are preferred generally just because they are less code, but whatever you find nicer goes

[magic python]

looking at the pandas source they seem to use both, not sure why

maybe i just need to read more

(here: https://github.com/pandas-dev/pandas/blob/master/pandas/util/_decorators.py)

[paper echo]

class decorators can make it easier to handle complicated decorator logic, decorators that have some kind of internal state, etc.

it's my go-to for decorators with arguments

the differences might be just "who contributed the code"

in the case of Substitution, it's a class precisely because it has internal state and an instance method; you can't do that with nested functions

in the case of Appender, it seems arbitrary

[fallow apex]

hello

how to do concurrent programming in python ??

[half wharf]

what's concurrent programming ?

@fallow apex

[fallow apex]

do in python things that golang does

[raven ridge]

Check the pinned messages in the #async-and-concurrency channel

[fallow apex]

ty

[half wharf]

👍

[unkempt rock]

Does anybody know a book or resource for software design in python? I think i might be overbloating stuff a bit by following what seem to be too strict java oop and tdd conventions

[amber ledge]

dude im looking for something similar

[unkempt rock]

the abstract bloat is real 😅

[lofty agate]

cosmic python?

[unkempt rock]

thx, i will give it a look, have you read it? can you give me a short idea of what i can expect, if its not too much to ask?

[lofty agate]

it's a whole book. So check out the ToC

[unkempt rock]

yeah im reading the preface

[finite sparrow]

python isn't good at saying, "you can't"

you can, function based decorators can act almost like classes because they are objects

it is, however, good at saying "you shouldn't"

but you really shouldn't

[deep bramble]

is dict_keys type acting as a set in a sense that it has O(1) lookup? Or is it more like a list with O(n) lookups?
I know that doing x in dict will take O(1) time, but will x in dict.keys() also take O(1)?

[charred pilot]

https://docs.python.org/3/library/stdtypes.html#dictionary-view-objects

Keys views are set-like since their entries are unique and hashable.
i would assume this means it has O(1) lookup

[deep bramble]

yeah I know they'll be unique and hashable, so I also figured it's probably O(1) however is this actually documented somewhere? The docs you sent just document that you can do x in dict.keys() but they merge this with dict.values() and even with dict.items() and dict.values() certainly won't be unique so that must have O(n), I'm therefore just wondering whether the keys do actually have O(1) lookups or not

[charred pilot]

i don't think time complexities are documented anywhere

[elder blade]

You want to check if a key is in a dict?

[deep bramble]

well yeah, but I'm just wondering about the dict_keys datatype, I know generally I would just do x in dict rather than x in dict.keys()

[elder blade]

!e ```py
print('abc' in {'abc': 123, 'xyz': 987})

[fallen slate]

@elder blade :white_check_mark: Your eval job has completed with return code 0.

True

[deep bramble]

that's not what I'm asking about... as I said in the reply, I know we can just do x in dict and I know x in dict.keys() works too, what I don't know is whether x in dict.keys() lookup will act like a set and work in O(1) or like a list and work in O(n)

[elder blade]

dict_keys and the other objects are kind of special. If you do dict.keys() and then change the dict, that will show up in the dict_keys object.

I guess you can call it a proxy (if I understand the definition correctly).

[deep bramble]

well yeah, but that doesn't tell me whether it has O(1) lookups or not

[elder blade]

Ah, yeah I think they both do the same thing. My guess is that __contains__ of dict_keys calls dict's __contains__.

[charred pilot]

you could just check the source

[unkempt rock]

In [1]: first = dict.fromkeys(range(1000)).keys()

In [2]: second = dict.fromkeys(range(1000000)).keys()

In [3]: timeit 999 in first
66 ns ± 2.72 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [4]: timeit 999999 in second
64.2 ns ± 0.587 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
``` sure seems to be O(1)

[deep bramble]

oh, I didn't realize I could just check the speeds like this, smart, thanks!

[paper echo]

this is ipython https://ipython.org/, %timeit is a special ipython command

[spice pecan]

You can use the timeitmodule for a standard solution, albeit it's a bit less convenient

!e ```py
from timeit import timeit

print(timeit("999 in d", setup="d = dict.fromkeys(range(1000)).keys()"))

[fallen slate]

@spice pecan :white_check_mark: Your eval job has completed with return code 0.

0.08245383203029633

[spice pecan]

there are different ways to use it that have varying degrees of convenience, but yeah, IPython's %timeit is way easier to use for quick tests

[grave jolt]

yay, immutables now has proper typings 👍

[paper echo]

ipython's %timeit also automatically figures out the number of iterations. i really wish they had a standalone library for it

[verbal escarp]

i don't like it only gives you mean and std, min would be more useful in some cases

[paper echo]

it'd be even nicer if it gave you the full list of timings

and had .mean, .std_dev, etc. methods

[peak spoke]

You can get the TimeitResult obj from it

In [68]: x = %timeit -o time.sleep(0.2)
205 ms ± 1.93 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [69]: x
Out[69]: <TimeitResult : 205 ms ± 1.93 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)>

In [70]: x.average
Out[70]: 0.2051083285810559

In [71]: x.all_runs
Out[71]:
[0.20412989996839315,
 0.2080345000140369,
 0.20717900001909584,
 0.20368340000277385,
 0.2022181000211276,
 0.20619320002151653,
 0.20432020002044737]

[paper echo]

TIL you can do x = %timeit

it'd still be nice if i could use this outside of ipython, but this pretty helpful as an intermediate

[verbal escarp]

new to me as well, curious

[halcyon trail]

nice, TIL as well

[verbal escarp]

@paper echo omg, i'm laughing so hard right now, it works

from types import ModuleType
import threading
from importlib import import_module

mod = import_module("numpy")

class ProxyModule(ModuleType):
    def __init__(self, mod):
        self.__implementation = mod
        self.__condition = threading.RLock()
    
    @property
    def random(self):
        print("totally random")
        return 4

    def __getattribute__(self, name):
        if name in ("_ProxyModule__implementation", "_ProxyModule__condition", "", "random"):
            return object.__getattribute__(self, name)
        with self.__condition:
            return getattr(self.__implementation, name)

    def __setattr__(self, name, value):
        if name in (
            "_ProxyModule__implementation",
            "_ProxyModule__condition",
        ):
            object.__setattr__(self, name, value)
            return
        with self.__condition:
            setattr(self.__implementation, name, value)
            
proxy = ProxyModule(mod)

module properties, here we come

[grave jolt]

module-level __getattr__ with RTX on

[verbal escarp]

rtx?

[grave jolt]

dead joke about enabling Nvidia RTX and making something more beautiful/advanced https://www.nvidia.com/en-gb/geforce/rtx/

[verbal escarp]

ahhh xD

well, i was considering putting dunder methods in ProxyModule, but i don't think people would appreciate the elegance of mod @= decorator xD

[native flame]

oh my god

[verbal escarp]

numpy + math!

module properties on the other hand might come in handy

[spice pecan]

import numpy

numpy.random = lambda: 4

I am fun at parties

[verbal escarp]

that's not the same

[nova iris]

from py3 docs: https://docs.python.org/3/reference/datamodel.html#object.__getattribute__

This method should return the (computed) attribute value or raise an AttributeError exception.

does anyone know what computed means in this context (this occurs in other dunder method docs as well in this section)?

[peak spoke]

Just that it can do some work in the method instead of returning a direct attribute from somewhere I believe

[nova iris]

ah, okay. thanks!

[crisp zinc]

yours changes it globally?

i'd have to test it

[verbal escarp]

i was saying that having it as a property is not the same as replacing it with a function, properties are way more trickier

you wouldn't do sys.path()

but with path being a property of sys, one could serve different versions of sys.path depending on who's calling it

[verbal escarp]

that's actually a trickier question than i thought

normally you could replace a loaded module by switching it in sys.modules, but where does sys live?

buildins?

oh, it just works

it's one and the same

confusing

i guess that answers your question

you can do import sys; sys.x = 4; import b in module a, and import sys; print(sys.x) in module b

from types import ModuleType
import threading

class ProxyModule(ModuleType):
    def __init__(self, mod):
        self.__implementation = mod
        self.__condition = threading.RLock()
    
    @property
    def path(self):
        print("accessing sys.path..")
        return self.__implementation.path

    def __getattribute__(self, name):
        if name in ("_ProxyModule__implementation", "_ProxyModule__condition", "", "path"):
            return object.__getattribute__(self, name)
        with self.__condition:
            return getattr(self.__implementation, name)

    def __setattr__(self, name, value):
        if name in (
            "_ProxyModule__implementation",
            "_ProxyModule__condition",
        ):
            object.__setattr__(self, name, value)
            return
        with self.__condition:
            setattr(self.__implementation, name, value)

import sys
sys.modules["sys"] = ProxyModule(sys)
sys.x = 4

import asdf

with import sys; print(sys.x, type(sys)) in asdf.py - prints 4, ProxyModule

works just like that

mindbending

[spice pecan]

I assume the interpreter just keeps a reference to the original module

[visual shadow]

Modules are cached and a second import just uses the cache, yes

So yes, it's why you can freely monkey patch modules as needed

[slim torrent]

anyone got any codes which can make create a game loop?

[fallen slate]

@meager imp Please don't try to ping @everyone or @here. Your message has been removed. If you believe this was a mistake, please let staff know!

[charred pilot]

he's tried it like 10 times now, in multiple chats

[past iron]

Trying to understand this piece of code
def genPrimes(l=[]):
return l.clear() or (l.append(p) or p for p,_ in enumerate(iter(int,1),2) if all(p%i for i in l))
Can anyone help?

[stuck valley]

iter(int,1) is an infinite iterator

[boreal umbra]

@past iron what happens if you run it?

[past iron]

error

[boreal umbra]

@past iron what error?

[past iron]

syntax error

[sand goblet]

!e ```py
def genPrimes(l=[]):
return l.clear() or (l.append(p) or p for p,_ in enumerate(iter(int,1),2) if all(p%i for i in l))

x = genPrimes()

for _ in range(10):
print(next(x))```

[fallen slate]

@sand goblet :white_check_mark: Your eval job has completed with return code 0.

001 | 2
002 | 3
003 | 5
004 | 7
005 | 11
006 | 13
007 | 17
008 | 19
009 | 23
010 | 29

[sand goblet]

It works here

@past iron

[past iron]

thanks so much

[sand goblet]

I didn’t change anything

[past iron]

but you solved it

[unkempt rock]

ooo that looks cool

[sand goblet]

It the same program

[unkempt rock]

I just now figured out what python is

lmao

[sand goblet]

I didn’t change it at all

[unkempt rock]

it seems cool

[haughty sand]

No, he taught you how to use a generator

[sand goblet]

It’s his friend’s code, and he posted it and I just ran what he posted

[haughty sand]

yes.., you taught him how to use it..?

[sand goblet]

No I just ran what he posted to show that it worked

He was saying it had a syntax error

[sand goblet]

Maybe you just didn’t save the py file or something

[past iron]

yah i guess

[sand goblet]

So was he showing off his one-lining?

Because it doesn’t need to be written like that

[past iron]

haa.. i dont know

[sand goblet]

import itertools

def genPrimes2():
    l = []
    for p in itertools.count(2):
        if all(p % i for i in l):
            l.append(p)
            yield p```

He could have done this too

!e ```py
import itertools

def genPrimes2():
l = []
for p in itertools.count(2):
if all(p % i for i in l):
l.append(p)
yield p

x = genPrimes2()

for _ in range(10):
print(next(x))```

[fallen slate]

@sand goblet :white_check_mark: Your eval job has completed with return code 0.

001 | 2
002 | 3
003 | 5
004 | 7
005 | 11
006 | 13
007 | 17
008 | 19
009 | 23
010 | 29

[gusty marsh]

Is it possible to set a function's docstring manually like?

def function():
  """default docstring"""
  __doc__ = "overwritten docstring"

!e

a = "Apple"
def function():
  f"""default docstring, injecting {a}"""
  ...

print(function.__doc__)

[fallen slate]

@gusty marsh :white_check_mark: Your eval job has completed with return code 0.

None

[gusty marsh]

I thought this could do that, but nope

[boreal umbra]

@gusty marsh the attributes of the function object isn't the same as the symbol table for the function's scope

Unlike a class, where that's pretty much exactly how it works for the class scope.

[gusty marsh]

I see, so there is no way out right?

[boreal umbra]

I believe the docstring is read only, though I'm not sure

Probably is so that you can't set it to something other than a string, since built-ins depend on that being the case.

[gusty marsh]

I could possibly edit after the function is created, like

>>> def c(): "formatted docstring works %s"
... 
>>> c.__doc__
"formatted docstring works %s"
>>> c.__doc__ %= 'after'
>>> c.__doc__
"formatted docstring works after"

[boreal umbra]

Does that work?

[gusty marsh]

Yeah, but I don't really like that implementation

well, I found something helpful

https://stackoverflow.com/a/9745145/14863850

def custom_docstring(*args):
  def wrapper(func):
    func.__doc__ = func.__doc__.format(*args)
    return func
  return wrapper
``` I could do something like this then

lemme try

yep!

[sand goblet]

!e ```py
from timeit import timeit

setup = """
def f():
return all(ind == n for ind, n in enumerate(range(10000)))

def g():
result = True
for ind, n in enumerate(range(10000)):
if ind != n:
result = False
break
return result
"""

print(timeit(setup=setup, stmt="f()", number=1000)) # longer
print(timeit(setup=setup, stmt="g()", number=1000)) # shorter```

[fallen slate]

@sand goblet :white_check_mark: Your eval job has completed with return code 0.

001 | 2.1543522216379642
002 | 1.4388898788020015

[sand goblet]

People only use any and all with generator expressions to save lines, not because it’s faster?

[verbal escarp]

faster than what alternative?

[flat gazelle]

very few decisions in python codebases are made for performance

and any/all are significantly simpler than writing it out as a for loop

[valid rose]

mainly to reduce lines with the added benefit of being usually faster

[verbal escarp]

eh. those timings are misleading

timings = []

def test1():
return all(ind == n for ind, n in enumerate(range(10000)))

for _ in range(1000):
before = monotonic()
test1()
timings.append(monotonic() - before)

mean(timings), stdev(timings), min(timings), sum(timings)

(0.0011077024637488647,
0.000590969766133669,
0.0007980040027177893,
1.1077024637488648)

timings = []

def f():
result = True
for ind, n in enumerate(range(10000)):
if ind != n:
result = False
break
return result

for _ in range(1000):
before = monotonic()
f()
timings.append(monotonic() - before)

mean(timings), stdev(timings), min(timings), sum(timings)

(0.0008553606179993949,
0.0003762659363172025,
0.000588546994549688,
0.8553606179993949)

urgs.

hard to copy&paste from jupyter 😦

[peak spoke]

all/any will be faster if it doesn't mean you're iterating twice, but readability comes first

[valid rose]

if for some reason i were to use c macros inside python, is there any legitimate use or should i be banished for treason against python 🤔

[verbal escarp]

whatever makes most sense for your situation 😉

[valid rose]

i am asking for situations that makes sense, because i've got an hammer and i want nails to hit

[verbal escarp]

maybe have a look at numba?

[valid rose]

numba?

i think you misunderstood this, im asking about C preprocessor macros

[fallen slate]

Hey @rocky willow!

It looks like you tried to attach file type(s) that we do not allow (.exe). We currently allow the following file types: .gif, .jpg, .jpeg, .mov, .mp4, .mpg, .png, .mp3, .wav, .ogg, .webm, .webp, .flac, .m4a.

Feel free to ask in #community-meta if you think this is a mistake.

[elder blade]

See #c-extensions, Python has a complete ABI.

[valid rose]

im not doing anything c extension related haha 😅

im abusing c preprocessor macros in python

[elder blade]

How does that work? Do you compile CPython yourself?

[spice pecan]

Could you elaborate? Are you running the preprocessor on python code?

[valid rose]

no, you can use the gcc preprocessor any file

i run the .py file through the preprocessor and then execute it

[spice pecan]

I'd say that falls under the cursed category

[valid rose]

it is :)

[elder blade]

Oh really? That's cool

[valid rose]

for example

look at this cursed code

this is python

[spice pecan]

I don't see anything inherently bad about it

It's just very unexpected and your IDE won't like it, but other than that uh

Why not I guess

[valid rose]

i was just looking for some clever thing to do with it, tho using macros is a terrible terrible idea

[spice pecan]

I just don't see much incentive to use that either

[valid rose]

im just fooling around

im not going to use this for real things

[elder blade]

There's actually a project I've thought about where you can run your code through a preprocessor during installation (setup.py) that will remove stuff such as logger calls

[valid rose]

hmm

[elder blade]

Again, that's little to no gain, but 🤷‍♂️

[valid rose]

you can do it

#define DEBUG

#ifdef DEBUG
code...
#endif

[valid rose]

true 😅

[spice pecan]

I can't really come up with an example other than stuff like this tbh

[valid rose]

haha thats to be expected

macros arent needed in python

[spice pecan]

Not C-style macros at least

[valid rose]

although i had a lot of fun doing this

[hasty swift]

you shouldn't really want to remove logger calls, you can disable loggers for a reason, but the logs should remain so that whoever ends up running the software can debug it when things go wrong

[torpid hatch]

Help my pleas
how to create a porogram on python

[verbal escarp]

that's not a very "advanced" discussion, is it 😉

[elder blade]

That is true, but I wanted to have other functionality. Like a bool you could import that would change which code runs.

That said, there is an issue with f-strings being a bit selfish. They evaluate immediately and so even if you don't use the log, the string computation has been made.

Have you guys seen the template string PEP? What do you think about it?

[verbal escarp]

which one?

[elder blade]

https://www.python.org/dev/peps/pep-0502/

This one has been rejected because it directly competed with f-strings, but there's another one

[verbal escarp]

not sure what exactly you're asking about

any reason why you're bringing up this PEP?

[elder blade]

I was talking about f-strings evaluating immediately, t-strings (template strings) would evaluate with a function call

This would mitigate the issue of even if you don't use the debug prints, they can slow down performance critical code.

Another application was SQL code, with the usage of t-strings the library itself could control how it was formatted into it's final form.

[spice pecan]

I mean

[elder blade]

I can't find the PEP

[spice pecan]

Shouldn't this work? lambda: f"string here"

[elder blade]

https://mail.python.org/archives/list/[e-mail]/message/LYAC7JC5253QISKDLRMUCN27GZVIUWZC/

Here it is I think

[peak spoke]

if the cost of the interpolation matters then I imagine the overhead of calling the lambda when logging is enabled would also be considerable

[spice pecan]

fair enough, but I don't see how a template string evaluated with a function call would be much different

[peak spoke]

If it is a concern distributing a debug build along with a normal release feels like the best idea

and you can use if __debug__ or the preprocessor mentioned above

[elder blade]

Yeah exactly

[hasty swift]

mjea, that's why there are logging best practices like using "%s" placeholders etc. instead of f-strings directly

[verbal escarp]

there's also .format?

[flat gazelle]

that's what loguru uses for placeholders, yeah

[peak spoke]

you do trade readability for it when using the placeholders logging supports

[quasi hound]

Is there a way to completely disable error traceback when I run a python script

I don't wanna see any of it in the terminal

Maybe I could do something like this?

open_terminal()
try:
   run_everything()
except:
   close_terminal_and_exit()

[flat gazelle]

you could rebind sys.excepthook to something that will not print the traceback, but try except is also fine

[quasi hound]

Thanks

Yeah I was thinking isn't there a place where errors usually go

stderr

Which is like stdin or out

Could probably put them into a file somewhere

[flat gazelle]

yes, you could also redirect stderr, though that would hide more than just exceptions

sys.excepthook will happen specifically for exceptions

[quasi hound]

K ty

[boreal umbra]

@unkempt rock This is not a help channel. See #❓｜how-to-get-help

A feature I've wanted for some time is function composition with __matmul__ as the default behavior for all callables that don't implement that method. For example:

user_num = (int @ input)("Please enter a number: ")

I doubt they'd ever go ahead with it, but I'm thinking of pursuing (and subsequently giving up on) a reference implementation, though I'm not quite sure where to look in the cpython source code. object is the only class that all callables inherit from, and it isn't callable itself, so I imagine this would involve changes in a lot of places.

[quasi hound]

Would look better if you replaced it with ->

(input -> int -> print)("enter number: ")

Since that's already a thing

[boreal umbra]

@ already represents function composition in the context of decorators. Also, what you're proposing isn't a thing; it would require the grammar to be changed.

[quasi hound]

I know it's not an operator

But already exists in the language

[boreal umbra]

!e

import numpy as np

a = np.random.random((4, 3))
b = np.random.random((3, 2))
print(a @ b)

[fallen slate]

@boreal umbra :white_check_mark: Your eval job has completed with return code 0.

001 | [[0.59711628 1.20779292]
002 |  [0.6526827  1.50215726]
003 |  [0.65091109 1.65541153]
004 |  [0.56062185 1.2990863 ]]

[quasi hound]

I know what it does

[boreal umbra]

Anyway, I'm more interested in what parts of the source code might need to be changed, not in debating the aesthetics.

[visual shadow]

i suppose you'd just need to define matmul on all these builtins

[boreal umbra]

lovely 😛

[visual shadow]

perhaps as a middle ground, you could define a function that did this. i imagine that would be perfectly doable

fuse(int @ input)("Please enter a number: ")

and whether you give it comma separated args or still want to be fancy and use @, it's your call

[peak spoke]

something on type that uses __call__ of the types should work

[visual shadow]

actually that might be downright easy (with comma separated args that is)

[boreal umbra]

this would require a grammar change, yes?

[visual shadow]

with the @ it still would require a change or at least a weird patch somewhere, yes.

this makes me want to try something foolish out...

[boreal umbra]

I don't see how fuse(int @ input) is better than fuse(int, input). But int @ input is still my wishlist item.

[visual shadow]

TypeError: can't set attributes of built-in/extension type 'type'

ah right ofcourse...

[boreal umbra]

class compose:

    def __init__(self, *args: tuple[Callable, ...]):
        self.funcs = args

    def __call__(self, *args, **kwargs):
        funcs = reversed(self.funcs)
        current = next(funcs)(*args, **kwargs)
        for func in funcs:
            current = func(current)
        return current  # this statement was here the whole time

I think that would work.

[visual shadow]

from forbiddenfruit import curse

def __matmul__(self, other):
    return lambda item: self(other(item))
    
curse(type, "__matmul__", __matmul__)

result = (int @ float)("42.5")
print(result)

that's always fun 😛 even if this isn't really all that well thought out.

oh..why am i assigning and printing. pfft

from forbiddenfruit import curse

def __matmul__(self, other):
    return lambda item: self(other(item))
    
curse(type, "__matmul__", __matmul__)

(print @ int @ float)("42.5")

[boreal umbra]

(print @ int @ float)("42.5")

[visual shadow]

so yeah..you could sneak that in a codebase somewhere, and watch the chaos that ensues

[undone hare]

So function composition? That's kinda sick

[static bluff]

So I wanna ask you guy's advice on something

from pyscript import application, window, renderer, document, console

My toolkit is a graphics library that walks and talks like an javascript runtime, in the important ways. In Javascript the window, document, and console objects are built in and hence, accessible from anywhere

The closest I can get to this behaviour is importing them into every module you want to do your UI stuff in.

[spice pecan]

You can inject them into builtins

That way if there are no globals/locals with the same names, they should be accessible from any module in the same process

But most if not all IDEs won't like that

[static bluff]

But this means I have to dance a delicate dance. These objects need to get created at some point, and, they take user defined settings. My options are to either create the objects implicitly and then accept user-defined configuration settings in a secondary 'application.initialize(<settings>)' step, or have the user create them themselves in their entry point script. The second option seems more pythonic, but it adds the extra complication of demanding any additional user modules to be imported after the objects are created. It exposes more of the internals than I'd like

I hadn't thought about the builtins approach. Its by far the least pythonic of all approaches - though function does trump purity

[spice pecan]

It's definitely not something I myself would do, but it does provide that exact functionality

But yeah, linters won't like that one

[static bluff]

from pyscript import application, window, renderer, document

import logging

application.initialize(
    applicationSettings={
        'application-name': 'My Awesome Application',
        'author': 'Skywalker on Discord',
        'version': (0, 0, 0),
    },
    consoleSettings={
        # 'logs-directory': os.path.expanduser('~/Desktop/My Application Logs'),
        'logs-directory': None, # if none, Pyscript will use the default location
        'print-severity': logging.WARNING,
        'log-severity': logging.INFO,
    },
    windowSettings={
        'title': 'My Awesome Application',
        'background-color': 0x000000,
        'aspect-ratio': 1.0,
        'width': 500,
        'height': 500,
    },
    serverSettings={},
    cefSettings={},
    cefSwitches={},
)

if __name__ == '__main__':
    application.start()

This is what the first option would look like. I havn't written out the second option but I'm sure you can imagine what it would look it

[spice pecan]

I think this version makes more sense

Though the second gives more power to the end user I'd say and definitely has some merit

[static bluff]

True. And in later versions it may be required because if I want to support more than one window then the user would be creating them themselves by definition

[spice pecan]

Depending on the implementation it could also encourage them to deal with those objects however they'd like, lifting that responsibility from you

If they want to import those objects into every module, sure, if they want to inject them into built-ins - they can do that as well, if they don't want direct bindings and access those through the module - sure

[static bluff]

My main concern is simplicity

Not for me, but for the user

[spice pecan]

You could have a default option, kind of like sympy's init_session

[static bluff]

I don't normally agree with this tenant of the zen, but this is a case where there should one and only one way of doing things - whatever that way may be

I saw you typing for a second @grave jolt , did you want to weigh in?

[spice pecan]

Supporting both approaches would be a valid way to treat that IMO

[static bluff]

So, a quickstart function, with support for the manual for advanced usage

[spice pecan]

Initialize with some default values and simple calls and inject into built-ins for those who don't want to be bothered, with support for manual creation and overrides

[spice pecan]

Yeah, pretty much

I'd have the main package contain the quickstart variation, and have the rest distributed in submodules/bundled into a single submodule so both kinds of users can have their way

[static bluff]

This does circle back, however, to the import/globalization issue. How do I get the objects into subsequent modules?

[spice pecan]

Or the other way around, pyscript contains all you need and importing pyscript.quickstart will perform autoinitialization with built-ins injection

[spice pecan]

Depends on what you want the end user experience to be

For internal purposes I'd keep them grouped via some object in your modules that you import that doesn't pollute built-ins

[flat gazelle]

I would go the cljs way and just provide an object of sorts that will just have them as attributes

[spice pecan]

With some variation of a use function that would replace/add a new object to that mix

[flat gazelle]

(well, cljs gives you a namespace, so IG you could use a module)

[spice pecan]

Basically that

[static bluff]

I've got a dataclass called 'runtime' that contains global state, like the OS and the program's bundled status

[spice pecan]

That could keep references to the window, document, etc. objects as well

For internal use

[static bluff]

Yup. And type hinting the attributes with a starting value of None will keep the linters happy

[flat gazelle]

honestly, with highly dynamic objects like that, I would just use the : Any typehint

[spice pecan]

If they go with the advanced approach, they'd call pyscript.use(Window(...)) (for example) manually, which would add a reference to runtime, and if they import the quickstart package, you do that yourself with some suitable defaults

IMO that gives you the best of both worlds

[static bluff]

I'm not certain what you mean by this 'use' practice. I could just have the class's __new__ method register it

Or application.createWindow, more likely. Certain implementation details would require that anyway

[flat gazelle]

you could also look into how brython does things

[spice pecan]

You could, and that's up to you, use is just a placeholder for registering in general

[static bluff]

Goooooood callllll

[spice pecan]

How exactly you accomplish that depends on how your package is structured, so I'm not one to judge here. __new__ or create_CLASSNAME would be very much suitable, but if there could be some sense in creating, but not yet registering an instance, a more manual mechanism could also prove useful

.use would be familiar to some JS devs, those with a Vue background for example

Though you could argue that it's there because certain features that Python has are lacking there

[static bluff]

So in summary: supporting both a quickstart approach and manual construction of the object's is preferred. In both cases, do not create the objects implicitly. The user must be responsible for importing the application/renderer (singleton) objects and the specific window/document instances at play in a given module, on their own

But, I should prevent some sort of streamlined interface for doing so

[spice pecan]

That sounds solid

[static bluff]

I suppose the simplest approach would be to import pyscript at the start of every script, and just reference pyscript.application, pyscript.renderer, pyscript.windows.get(...)

Might change renderer to chromium, since both the application and 'renderer' objects are actually renderers. I do want to be 'correct' in my naming

Or I could just treat them both as a single class 😮 As it stands, one is responsible for starting and manually 'pumping' the other

Hmmmmmmmmmmm

Do you mind if I pick your brain on something else?

[grave jolt]

nah, just trying to make a bad joke, as always

[static bluff]

XD

[static bluff]

Regarding memory abuse

I know memory is cheap these days, but I don't want to waste it either

So, each element in my program has a 'style' attribute, which is pretty much just a dictionary style attributes and their values

In total there are, I dunno, maybe three hundred

I'd LIKE to type hint each one, but that means tying 300 null/potentially-not-null values to every element. There could be hundreds or even thousands of elements

The alternative is just to just store the values that that present. Thoughts?

[flat gazelle]

that is never going to have a measurable impact

even if they were 1kib each (which they aren't), it's 300KB, which is a rounding error on anything desktop

[static bluff]

XD Okay, excellent

I had a feeling it would be something along those lines

import pyscript

application = pyscript.Application(
    applicationSettings={
        'application-name': 'My Awesome Application',
        'author': 'Skywalker on Discord',
        'version': (0, 0, 0),
        'logs-directory': None, # use default logs-directory location
        'browser-support-directory': None,  # use default logs-directory location
    },
    serverSettings={
        'server': None, # custom server can be supplied, or None to use the built-in
        'address': '0.0.0.0',
        'port': 5000,
    },
    consoleSettings={
        'log-severity': pyscript.logging.INFO,
        'print-severity': pyscript.logging.WARNING,
    },
    diagnosticsSettings={
        'log-severity': pyscript.logging.DEBUG,
        'print-severity': pyscript.logging.NOWRITE,
    },
    menubarSettings={
        'main': (
            ('Some Command', ('command', 'shift', 'a')),
        ),
        'file': (
            ('Some Command', ('command', 'option', 'b')),
            ('Some Command', ('command', 'alt', 'c')),
            ('Some Submenu', (
                ('Some Command', ('command', 'option', 'd'))
                ('Some Command', ('command', 'option', 'e'))
                ),
            ),
        ),
    },
    toolbarSettings={
        # functions similar to the menu, above
    },
)

window = pyscript.Window(settings={
    'window-name': 'My Window One', # used for identification purposes only
    'title': 'My Awesome Application',
    'height': 500,
    'width': 500,
    'aspect-ratio': 1.0,
    'background-color': 0x000000,
})

Me, working away

This all seems fine to me, except for the menu bar/toolbar stuff. The other option is to do the menu bar stuff objectively once the application is created — its all widgets in the end

[verbal escarp]

i think what you're suggesting is called piping and is done in bash with | for ages, there's also a piping package that makes this possible in python, but it could be better. also, don't assume you can monkeypatch things in python (builtins etc), since they're either __slotted__ or not written in python

https://pypi.org/project/pipe/

it works, but it has some quirks that i find unelegant enough to make me not want to use it

you'd need to deal with exactly the same issues if you used @ instead

i actually had a real usecase for piping, which is why i actually gave pipe a serious chance

[static bluff]

This is similar to (probably built from)the infix hack, which isn't too hard to do.Might be worth a look. Conventional wisdom says such things shouldn't be used in production code- but I'm not seeing anything particularly unsavoury. Its just using the syntax to your advantage. https://code.activestate.com/recipes/384122-infix-operators/

[verbal escarp]

it was a dark, cold, winter night when i did some experiments with data in jupyter.. the function calls grew more and more until i had something like semilogy(np.abs(np.fft.fft(np.abs(np.fft.(np...(( ..(( and then i forgot how many closing brackets i needed

[static bluff]

The horror

[verbal escarp]

when i remembered dear raymonds creed "there has to be a better way!"

after taking a long, hard look at the pipe package, i realised that it couldn't help me and i was very sad

.oO(damn, my storytelling was better :p )

[verbal escarp]

well, as i said, you can't count on objects to be compatible with the infix calls, also my biggest beef with pipe is that you still need to feed data to the pipe, which is done by a regular call () at the end of the pipe

so the order is all messed up

it's not A -> B -> C -> D but B -> C -> D -> and oh, btw, A -> B

[static bluff]

I've done a lot of experiments trying to modify Python's syntax and I've learned the only way to do it properly is to literally modify the syntax.

And that's a bad idea. Others won't know what's going on. Use the syntax, abuse the syntax even, but don't change it

[verbal escarp]

yeah, wise words

[static bluff]

Even the descriptor pattern is a bit offputting because without proper documentation, how is someone supposed to know that thing-x in the definition space of the class is a descriptor?

[verbal escarp]

i wished we had a |> syntax of sorts to properly support pipes

[static bluff]

For me, I wish we had an attempt operator - for simple one line try except clauses

result = (method, KeyError, IndexError) !? (arg1, arg2, arg3='some-value')

[verbal escarp]

that would be cool.. like the proposed ?x syntax for Optional

[static bluff]

Or, perhaps a bit more useful

value = ((getattr, AttributeError) !? (instance, attribute)) | SOME_DEFAULT

[verbal escarp]

well, ! and ? are still open tokens for syntax

although, i'd imagine try/except could made into a one-liner just as if/else

[static bluff]

The javascript syntax isn't bad

try {...} catch {...}

[verbal escarp]

why am i dropping words all over the place 😐

[verbal escarp]

how is that different from try/except?

[static bluff]

try {value = getattr(thing, attr)} catch {value = default}

Its not, except for the braces and catch instead of except

[verbal escarp]

forget braces 😉

[static bluff]

But braced-codeblocks are not a trivial thing to write into the grammar, even for someone who knows that their doing

Even single-line braced-blocks

[boreal umbra]

Well, you know what they say about a rose by any other name, but "function composition" is a thing.

[verbal escarp]

sure, and i wished it were that easy in python

[boreal umbra]

so that's why we need to band together and take our demands to the steering council.

[flat gazelle]

it is my opinion that python is too far gone to try and make long expressions viable, and instead should focus on somehow making the highly statement based programming style feel at least vaguely modern.

[verbal escarp]

hmm.. wait a sec

[boreal umbra]

what if more mutator methods returned self?

[verbal escarp]

what if we make a wrapper that actually delegates | to ()

[flat gazelle]

you still have to parenthesise things to do ```py
a
.b()
.c()

[static bluff]

What do you mean by 'modern'?

I agree about the too far gone part. I think they should start again from scratch - using what they've learned to build a better (and more modern) language

With runtime type enforcement, damnit!

Typing hinting is great but its a half measure

[verbal escarp]

or.. equally feasible and not at all magical - just have a Pipe(...) that takes any number of callables and first argument being the input

and then have Pipe call the arguments in that order

[flat gazelle]

there are languages building on the experiences from python in development and out there already

e.g. nim

[verbal escarp]

Pipe(data, func1, func2, func3, print)

not as elegant as |, but easy and much better than ((())))

[flat gazelle]

python devs should not drop python, some other group will eventually make a language that is better enough than python to switch, and I eagerly await that day.

[flat gazelle]

python's highly statement oriented programming style comes from some variant of algol. If you look at any modern language, you will notice they use expresions much more that statements, sometimes forgoing statements as in lines without a result entirely (e.g. raku). Python is kind of stuck with this style, but I think there is more that can be done to make it nicer to use than it is now

[static bluff]

All of this is fine. But who to better design a new language than the Python devs? I can't really think of a better trained or curated team

[verbal escarp]

and if it works well enough as a Pipe() class/function for composition, syntax like |> could be added easily enough

[flat gazelle]

I have rewritten the clojure arrow macros into python using tuple literals before.

it's just a matter of whatever readability you gain you can also gain by creating intermediate variables, and that has the added advantage of being obvious to people not privy to how exactly your pipe/thread function works

[gleaming rover]

sometimes people talk about wanting a managed Rust (keeping the syntax and most of the semantics, but being more chill on lifetimes etc.)

I feel like a statically typed Python with some syntax improvements would be nice too

[verbal escarp]

you mean julia?