#internals-and-peps

1 messages · Page 88 of 1

spark magnet
#

What would you like sorting to use?

boreal umbra
spark magnet
#

i guess that would have been another way to go.

#

you could default __key__ to returning self, and then existing classes would work as they do.

boreal umbra
#

am I right in thinking that if (__lt__ or __gt__) and __eq__ are implemented, you can infer logically sound implementations for __ge__, __ne__, and __le__?

spark magnet
#

if your comparisons "make sense", sure

boreal umbra
spark magnet
spice pecan
#

cmp_to_key is a very smart solution, yeah

feral cedar
#

functools really has everything huh

raven ridge
#

I never looked into how cmp_to_key works - obvious in retrospect, but I wasn't coming up with it myself when thinking about it. The key object it returns is actually a class instance that calls the cmp function in its __lt__ etc.

#

If there's no __eq__ and no __hash__ defined, user defined classes are hashed by identity (essentially they have a default __hash__ that is hash(id(self))). It's only user defined classes that have an __eq__ and that don't have a __hash__ that aren't hashable

#

Most instances of user defined classes are hashable, in my experience, because most don't define __eq__, and many of the ones that do also define __hash__.

swift imp
#

Attrs is superior in every way. I started out not knowing of attrs and used data classes, ended up implementing the entire validator functionality of attrs for data classes. Attrs is a superset of data classes.

boreal umbra
raven ridge
#

or to put that differently, dataclasses is the subset of attrs that enough people agreed upon to allow it to become part of the standard library.

boreal umbra
raven ridge
#

yep.

#

and the dataclasses module is based on it.

boreal umbra
#

and it directly informed the design of -- wow

#

speaking of

#

dataclass instances are still wrappers around a dict like most custom objects, yes?

raven ridge
#

yes, AFAIK they never use __slots__

boreal umbra
#

how significant is the memory footprint reduction if you use slots, do you think?

#

Let's assume, say, five attributes.

raven ridge
#

!e ```py
import sys
print(sys.getsizeof(dict(a=1, b=2, c=3, d=4, e=5)))
print(sys.getsizeof((1, 2, 3, 4, 5)))

fallen slateBOT
#

@raven ridge :white_check_mark: Your eval job has completed with return code 0.

001 | 232
002 | 80
raven ridge
#

give or take.

spark magnet
#

that seems like an odd way to compare

boreal umbra
#

the tuple in the last line doesn't have named attributes

spark magnet
#

the names will be in the class, not the instance, i think

boreal umbra
#

ah right

#

the class would have a dict mapping attribute names to indices, yes?

raven ridge
#

yeah.

#

but yeah, perhaps that is an odd way to compare, and I should have just made a slots class and a non-slots class and checked the size difference. Let's try that...

#

!e ```py
import sys
class C:
def init(self):
self.a, self.b, self.c, self.d, self.e = (1,2,3,4,5)

print(sys.getsizeof(C()))

class C:
slots = ("a", "b", "c", "d", "e")
def init(self):
self.a, self.b, self.c, self.d, self.e = (1,2,3,4,5)

print(sys.getsizeof(C()))

fallen slateBOT
#

@raven ridge :white_check_mark: Your eval job has completed with return code 0.

001 | 48
002 | 72
boreal umbra
#

😮

raven ridge
#

hm, that seems wrong.

#

what'd I get wrong? 😄

mint forge
#

What even is the point of __slots__?
Like where do we use it?

raven ridge
#

to prevent the creation of an instance dictionary for storing instance variables, and its associated overhead.

#

it's used very rarely, as an optimization in code that will create a huge number of instances of objects of the same type.

mint forge
#

oooo

raven ridge
#

it's essentially a builtin implementation of the flyweight pattern into the language itself, allowing some data to be shared across all instances of the class that would otherwise need to be stored on a per-instance basis.

mint forge
raven ridge
#

yes.

#

!e ```py
class C:
pass
print(vars(C()))
class C:
slots = ()
print(vars(C()))

fallen slateBOT
#

@raven ridge :x: Your eval job has completed with return code 1.

001 | {}
002 | Traceback (most recent call last):
003 |   File "<string>", line 6, in <module>
004 | TypeError: vars() argument must have __dict__ attribute
mint forge
#

oh ok, so __slots__ replaces __dict__ basically

raven ridge
#

more or less. Instead of each instance storing a dictionary mapping attribute name to value, the class itself stores a sequence of attribute names, and the instance stores a sequence of attribute values. You look up the name against the class's slots to find the index for that attribute, and then look the value up in the instance's sequence of values.

#

inheritance makes it more complicated than that, but that's the basics, anyway.

mint forge
#

i see, which one is faster?

raven ridge
#

slots, usually, unless you have a ton of attributes.

boreal umbra
#

My guess is that __dict__ is slightly faster, but not in a way that scales

boreal umbra
#

but __slots__ also entails accessing something from a tuple? and that these are both constant time?

raven ridge
#

I don't believe so - I believe that for __slots__ classes there's no hash table involved; it looks up interned strings in a sequence (tuple or something)

#

!e ```py
class C1:
slots = "a",
def init(self, a):
self.a = a

class C2:
def init(self, a):
self.a = a

import timeit
print(timeit.timeit("c.a = 5", setup="c = C1(10)", globals=globals()))
print(timeit.timeit("c.a = 5", setup="c = C2(10)", globals=globals()))

fallen slateBOT
#

@raven ridge :white_check_mark: Your eval job has completed with return code 0.

001 | 0.08612881787121296
002 | 0.10498212021775544
raven ridge
#

there's the proof it's faster - by about 15%

#

!e ```py
class C1:
slots = "a",
def init(self, a):
self.a = a

class C2:
def init(self, a):
self.a = a

import timeit
print(timeit.timeit("x = c.a", setup="c = C1(10)", globals=globals()))
print(timeit.timeit("x = c.a", setup="c = C2(10)", globals=globals()))

fallen slateBOT
#

@raven ridge :white_check_mark: Your eval job has completed with return code 0.

001 | 0.05832714191637933
002 | 0.07088103494606912
raven ridge
#

^ and it's faster for getting attribute values as well, not just for setting.

paper echo
#

Also you can't assign new attributes to classes with slots

rain hawk
#

What was the site name where share large codes (I just forgot its name) ?

true ibex
#

!paste

fallen slateBOT
#

Pasting large amounts of code

If your code is too long to fit in a codeblock in discord, you can paste your code here:
https://paste.pydis.com/

After pasting your code, save it by clicking the floppy disk icon in the top right, or by typing ctrl + S. After doing that, the URL should change. Copy the URL and post it here so others can see it.

true ibex
#

@rain hawk is this what you were looking for?

rain hawk
#

Nah.. however you remind me of that .. Hastebin

#

thanks

swift imp
#

Wait so can I have instance and class attributes with __slots__ and only the ones in __slots__ can stored as instance attributes while rest are class attributes or is this something else completely

mystic cargo
#

this seems like it belongs in one of the off topic channels

quaint zenith
#

People over there are usually not that serious about answering such question but alright.

mystic cargo
#

perhaps, but this channel is reserved for discussion about python itself 😄

boreal umbra
#

you may consider joining the math discord.

paper echo
#

!e ```python
class A:
val = True
slots = ('x', 'y')
def init(self, x, y):
self.x, self.y = x, y

print(A.val)

a = A(-1, 1)
print(a.val)

fallen slateBOT
#

@paper echo :white_check_mark: Your eval job has completed with return code 0.

001 | True
002 | True
paper echo
#

@swift imp apparently yes, class attributes are not subject to the __slots__ restriction

#

i suppose this is because the class object itself still uses dict lookup, while the instance does not. but that also means there is still some complicated attribute dispatch happening, where if the attribute name is not present in slots it still looks it up in the class

#

so the "failure" path with slots is not likely to be faster than without slots

#

but that's not usually an important case

radiant fulcrum
#

iirc the __slots__ restrictions are only applied to a class when a instance is created not when they're parsed, hence why you can still define attributes with __slots__ = (), but the methods then become read only after being instantiated at that point

cobalt hamlet
#

anyone want a challenge?
Create a substitution table for all 8 bit values. so that when you do, this for every pair of bytes

count = 0
for a in itertools.permutations(range(256),2):
    if substitution_table[a[0]] & substitution_table[a[1]] == substitution_table[(a[0]&a[1])]:
        count += 1
```you get the smallest number possible i got 736
boreal umbra
paper echo
#

interesting

#

i dont hate it, although ? might get conceptually messy if we also end up with None-safe operators

#

x, ?, z = foo?.coordinate

#

two different meanings

#

i prefer .

#

x, ., z = foo?.coordinate

boreal umbra
#

@paper echo the benefit is that that object won't have an existing reference that could affect its garbage collection schedule. But I'm not sure when that would really matter.

#

they could also imbue _ with that special property for non-repl settings. That might as well be backwards compatible.

swift imp
#

Is there even a pep for None aware operators?

peak spoke
#

There's a deferred pep

paper echo
#

Im still bummed they deferred it but are charging ahead with the half baked pattern matching

#

Unless the pattern matching pep got better

boreal umbra
#

it's unfortunate that a majority of core devs want patma but a minority are satisfied with the current spec.

#

Last I checked, at least

#

Do I summon aeros...

raven ridge
mint forge
#

I mean we have regex so why not just improve that rather than adding pattern matching to the language?

raven ridge
#

because it solves an entirely different problem. regex is limited to matching strings, and cannot reasonably be extended to match arbitrary objects without inventing an entirely new syntax and winding up with something that no longer particularly resembles regex.

boreal umbra
paper echo
#

Ill read those thanks @raven ridge

raven ridge
#

start with 636 - that's the quickest read and best overview.

boreal umbra
#

the idea is that patma is danker syntax for chained if statements, yes?

raven ridge
#

not particularly like regexes at all.

brave badger
boreal umbra
raven ridge
#

or try / except (TypeError, ValueError)

brave badger
#

True

boreal umbra
#

how does patma integrate exception handling?

raven ridge
#

it doesn't - if a matcher raises an exception, it breaks out of the pattern matching.

brave badger
#

iirc, the syntactic macros PEP provides a compelling alternative to pattern matching which ultimately just compiles down into ifs and elifs

raven ridge
#

I really like the look of the syntactic macros PEP from a user facing point of view, but a core dev friend made the argument that it's terribly underspecified how it would actually work, and very difficult to implement efficiently as it's currently proposed

boreal umbra
raven ridge
#

they pretend to like me at least 😛

boreal umbra
raven ridge
#

My hope is that it would get rid of some incredibly "creative" hacks in major libraries that are used today. PLY came up here recently, and is a great example - it abuses the docstring of functions as the regex for tokenization, and I have to imagine that if syntactic macros existed it wouldn't have done it that way.

boreal umbra
#

The one case that comes to mind where I wish there was library-specific syntax is df[df['x'] > y] expressions with pandas.

raven ridge
boreal umbra
#

because I don't want to be sad.

raven ridge
#

might make you angry? ¯_(ツ)_/¯

boreal umbra
#

I don't want that either.

raven ridge
#

nah - it's clever, and it looks nice enough, but it's cleverer than code should have to be.

#

it's clearly implementing a DSL in Python, and it has to do some creative hacks to get something that's both valid Python and reasonably nice as its own language.

#

For example, this rule matches numbers and converts the string into a Python integer.

def t_NUMBER(t):
    r'\d+'
    t.value = int(t.value)
    return t

So - it looks up the module's globals(), finds all the ones that start with t_, decides those must be its token rules. Then it checks whether each of those globals is a string or a callable. If it's a string, then that's the regex to use for matching that token. If it's a callable, then it contains some processing to do on the token when parsing one, and its docstring is the regex to use for matching that token.

boreal umbra
#

so it's very... meta...

raven ridge
#

it's clever. It's a nice DSL that happens to be valid Python code. It's less absurd than the C++ parsing DSL in Boost. But if syntactic macros existed, they would provide a way to declare tokens with less implicit hacks going on.

azure oar
#

too many things out there parse docstrings and do things with them. I don't... hate it? But given my rather nice experience with pyparsing I'm not sure why I'd use it

brave badger
#

It's more of a niche use-case if anything, not too absurd, but certainly a bit too clever

raven ridge
#

I have no particular opinions on how PLY stacks up against other parsers - it's just a nice example of something that did something weird which wouldn't have been necessary if syntactic macros existed.

#

I'm sure it's nowhere near the weirdest thing out there, just something that came up recently and was on my mind.

sacred yew
#

cant you just replicate that with decorators

azure oar
#

I'm curious how PEP-638 will do

raven ridge
#

sure, or with attributes on the function. It's done the way it is because someone thought the syntax was more readable than the alternatives, not because no alternatives exist.

azure oar
#

yeah, i mean, you can do some pretty cool macro-like metaprogramming with python, but it definitely has its limits in how far you can go

deft pagoda
#

uses some newer python meta programming to get even coolor syntax

raven ridge
#

Looks like SLY is intended to be the replacement for PLY once it's stable?

deft pagoda
#

probably

unkempt rock
#

where can I get python's round() function source?

print(round(0.5))
print(round(1.5))

0
2

This is something like banker's rounding right?

Isn't it here?
https://github.com/python/cpython/blob/master/Python/pymath.c#L72
But I see this and I can only see "usual" rounding here..

GitHub
python/cpython

The Python programming language. Contribute to python/cpython development by creating an account on GitHub.

charred wagon
#

That's the source for the math module

#

round() is different - it's a built-in function

#

My understanding of this code is that the type defines its own round method

gleaming rover
#

if you want the implementation for float itself

charred wagon
#

And this built in just calls that

gleaming rover
#

^

gleaming rover
#

which you don't use directly

gleaming rover
#

in particular, you can see the "round to even" behaviour here:

        if (fabs(x-rounded) == 0.5)
            /* halfway case: round to even */
            rounded = 2.0*round(x/2.0);
#

...where round is the abovementioned C function.

unkempt rock
#

yea this is what I was looking for, thanks!

spark magnet
#

i'm still surprised they changed the behavior of round() and didn't provide an option for how it should work

gleaming rover
spark magnet
#

python 3

urban quiver
#

python3 round() implementation of banking round surprised me too since JS round() do round half down which caused problem of conversing

prime brook
#

hi, new here ,(see last para for actual question) ill get straight to the point if thats okay , (btw idk where to ask this i think this should be the best channel out of all tht i ve checked but i could be wrong)

I am a beginner at python, currently doing a course in AI , i know the basic syntax and stuff, that's mostly it. recently I have learnt numpy (via the same course) , we will learn pandas and some more stuff relevant to data science in the coming time then at the end learn how to use all of it for data science. Okay so , I feel like i ve developed a nice grasp on the numpy syntax ( i m a beginner at coding ^_^) , I have finished my numpy assignments and i d like to think i did well enough , and I have some spare time (like a few days to a week) .
SO finally to my question that i came here to ask: What can I do with numpy? also , doesnt have to be related to data science even or could be, doesnt matter, like what can i do or make using the numpy knowledge that i have. I want to apply that into making smthing useful, not too complicated cos ofc i m not at that lvl yet. But I have no idea what to do or even what it can do now that i know the basic stuff . (was that too long, sorry)

(also do tell me where this question ideally ought to go so i can be more careful in the future)

elder sky
#

Hi there I was wondering if someone can explain to me some algorithm code for the travelling salesman problem. I’m looking to implement the A* algorithm and am not sure where to start. I have the distance matrix of the cities but don’t know how to implement to heuristic function in Python. Thanks

astral gazelle
#

This channel is for discussion of python itself, you'll have better luck claiming a channel and asking there #❓｜how-to-get-help

raven ridge
boreal umbra
boreal umbra
boreal umbra
#

If there exists a reference implementation of a PEP, will a link to that implementation always be in the PEP?

true ridge
#

Like, usually yes. But not always. There is no such strict rule

unkempt lynx
#

Yeah true

swift imp
#

When you guys have a function/method that is overloaded by multiple types for a single arg how do you type annotate it without the documentation from help looking like shit? I have a few unions which cause the signatures to go way best 80 chars and the help just looks like pure crap

gleaming rover
swift imp
#

It makes help look better?

gleaming rover
#

honestly I don't know but I believe it should...could be wrong.

swift imp
#

I wish I wasnt so tired and my laptop near me or id check lol

#

15 hr code binge got me burnt

spark magnet
#

@swift imp would it help to define the union as its own named thing?

#

one option is to not overload so much

swift imp
#

I think help does the replacement.

#

Like even if I assign the union to something, help prints out the full thing

spark magnet
#

@swift imp can you link us to the code?

swift imp
#

No but I can give an example.


from dataclasses import dataclass
import pandas as pd

@dataclass
class Foo:
    bar: Union[pd.Series, pd.DataFrame]
#

It's a less extreme one

spark magnet
#

is there not a type that encapsulates series and dataframe?

swift imp
#

There is within pandas I believe but I'm pretty sure using help will cause a substitution

#

So like I use the encapsulation in the definition but the. help replaces it

#

If I'm making any sense...

spark magnet
#

i don't use typing enough, or help at all, so I don't know.

swift imp
#

You really never use help?

#

I use that more than the online docs for libraries usually

spark magnet
#

hardly ever, and never on my own code

swift imp
#

Interesting. Of course a few people have told me that here before

balmy patrol
#

I still flake8 all my crap in the build so I'm forced to write it though 😄

spark magnet
#

i write docstrings all the time

severe pine
#

speaking of pandas, ( 🙂 ), anyone have any idea about the question in #help-honey

paper echo
#

@spark magnet there isn't a generic "pandas thing" class that's part of the stable public API

boreal umbra
#

I've been working on a project where a couple of classes inherit from a do-nothing class in case someone wants to do some hacky injection thing that I'd probably hate.

tranquil atlas
#

1 of these days ill belong here

mint forge
#

!pep 3155
Hey so I just found out about __qualname__ but I can't really think of a useful case for it.
The pep shows examples such as

>>> class C:
...   def f(): pass
...   class D:
...     def g(): pass

but I don't think anyone would be doing this.

Where would we use __qualname__?

fallen slateBOT
#
**PEP 3155 - Qualified name for classes and functions**

Link

Status

Final

Python-Version

3.3

Created

2011-10-29

Type

Standards Track

formal python
#

Anyone knows how would I be able to make standalone exes/binaries without packaging the whole cpython/pypy interpreter?
I know that I can use cython to produce .c files, but iirc, they require the cpython interpreter?

#

ping me ^^, also tell me if it's not the right channel to ask this

grave jolt
#

@formal python You can use Nuitka, it can compile Python to C. It still links against libpython, there's kind of no way around it

halcyon delta
boreal umbra
#

it seems to have worked for me?

flat gazelle
#

Call the function

boreal umbra
#

...

#

yeahhhh

#

Yep, recursion error on Windows.

#

I wonder why that is.

#

I wouldn't actually expect it to work on Linux either.

flat gazelle
#

It probably deletes itself in __del__ though that alone shouldn't cause this

boreal umbra
#

You're switching to a directory that will get deleted

#

wouldn't it delete itself in __exit__ since it's in a with block?

flat gazelle
#

Yeah, nvm. It can't be del since it also errors outside of a function.

halcyon delta
#

looking at the stacktrace it seems that it's due to rmtree failing because of permissions:

Traceback (most recent call last):
File "C:\Users\Gabriele\scoop\apps\python\current\lib\shutil.py", line 617, in _rmtree_unsafe
os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\Gabriele\AppData\Local\Temp\tmpks1f12o0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\Gabriele\scoop\apps\python\current\lib\tempfile.py", line 801, in onerror
_os.unlink(path)
PermissionError: [WinError 5] Access is denied: 'C:\Users\Gabriele\AppData\Local\Temp\tmpks1f12o0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\Gabriele\scoop\apps\python\current\lib\shutil.py", line 617, in _rmtree_unsafe
os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\Gabriele\AppData\Local\Temp\tmpks1f12o0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\Gabriele\scoop\apps\python\current\lib\tempfile.py", line 801, in onerror
_os.unlink(path)
PermissionError: [WinError 5] Access is denied: 'C:\Users\Gabriele\AppData\Local\Temp\tmpks1f12o0'

#

typical Windows

flat gazelle
#

Yeah, it can't del the folder it's in

#

That is sensible

#

The issue is why does the impl loop

storm fiber
#

I’m need advice on a decision about choosing one of two courses.

The first is an “advanced” python-course focused on AI implementation of statistical data.
The second is the programming language R.
I can’t take both, I don’t know why.

Which course should I choose? 🤷‍♂️

unkempt rock
storm fiber
#

Sounds about right

boreal umbra
# storm fiber I’m need advice on a decision about choosing one of two courses. The first is ...

well, a lot of people here are very biased towards Python, so I'll tell you why I prefer Python: You can probably do whatever you'd like in R, but you'd be in an ecosystem oriented specifically towards things that you can do with data. Python has a much larger ecosystem, so if you wanted to have some AI/ML in a Discord bot, you absolutely can. It also appears that the data science community is gravitating towards Python and that this is only going to continue--I've taken three data science-oriented courses for my degree (CS senior), and all were in Python. R never came up.

A potential benefit of R: Numpy (arrays/matrices) and Pandas (tabular data in general) are two of the fundamental libraries for doing data science in Python, and they both share an interesting syntax for iterative operations that isn't found in Python's standard library. They achieve that syntax by leveraging Python's data model in a way that I typically find enjoyable to use, though it can also sometimes be a bit awkward to read, and for that reason I think there's a strong argument to be made that Python could have benefited from having a special, native grammar for "rectangular data". I believe that R does, though I'm not sure what it's like.

#

To be specific: if x and y are both numpy arrays of the same shape, then x + y performs iterative, element-wise addition without having to write out a loop, and I like that. But if you wanted to, for example, increment every non-zero value in an array by two, it would be something like x[x != 0] += 2, and I don't like how it looks when you select from an array in terms of itself. These operations would look the same in Pandas.

round kiln
#

If you're end goal is to go into data science professionally, python is more common, at least in my experience (~7 years working professionally).

#

You do still find R (and it has great visualization libraries and some other stuff not as easily available in python).

#

R is also pretty easy to pick up & learn on your own (even if the syntax is a little awkward sometimes).

storm fiber
#

@boreal umbra Thank you all for the information!
We've been using scikit-learn, numPy and pandas for a few assignment and I feel like I don't understand all of their functions, there are a lot of functions lemon_eyes . I guess that comes with different projects and this is one of the reasons i'm thinking of taking the python-course, but on the other hand R seems like a great tool for data science. I don't have that much experience with big projects yet but aren't python struggling in that regard?

deft pagoda
#

i've used numpy for a while now, I think the library is too big for one person

#

just use what you need and keep the docs close

boreal umbra
#

@storm fiber yeah, those three libraries are meant to provide you with a comprehensive toolset for certain kinds of problems. It's not expected that you will "learn how to use scikit learn" or "learn numpy". It's also not expected that any one person will ever need everything that's in those libraries.

#

What do you mean by "I don't have that much experience with big projects yet but aren't python struggling in that regard?"

storm fiber
#

One of my teachers briefly mentioned that some other programming languages are more suitable for running big data science projects where python would work but less effective.

boreal umbra
# storm fiber One of my teachers briefly mentioned that some other programming languages are m...

Python is known for having a slower runtime than other popular languages, though I don't know enough to fully speak to why that is. Part of it is that Python gives you a lot of flexibility, so the interpreter has to do more work to afford you that. Or something. That being said, in the data science scene, Python is more of a "glue language". In all the libraries we've discussed so far, a lot of the heavy lifting is performed by compiled C code.

swift imp
#

Python is by definition a glue language iirc

#

No one does real heavy lifting with python its self, python is for the interface, giving a nice api to the ugly ass c/c++

boreal umbra
#

Though the fact that you can get something approaching a best of both worlds between readability and performance has probably contributed to Python's success.

raven ridge
#

Python is slow, in large part, because no one has ever bothered to put in the effort to make the core CPython interpreter fast. Extensibility, maintainability, portability, and readability have been prioritized over optimization. And the relative ease with which you can build Python extension modules in other languages makes that choice justifiable. There's already an escape hatch for things that need more performance than CPython can give.

peak spoke
#

imo it is a part of the problem, not having to use C for speed would be great. I personally only did it a few times thorough cpython but a lot of dependencies of projects have to use them and taking the build process from there would be beneficial for both the maintainers (along with maintaining a very different code base) and users

raven ridge
#

Cython lets you write code that is basically Python with some extra annotations and get C extension modules with C performance. The most annoying part of the whole process is building wheels for new versions of your library.

boreal umbra
peak spoke
#

Anyone is free to do so, but I don't think there are that many people with necessary expertise and the time to do it, and with large scale changes a big amount of time would be spent on discussions

radiant fulcrum
#

It's hard to get Into, I'm sure alot of people would like to support it but don't want to go through the pain of it in C

boreal umbra
#

If I maintained a certain component of cpython, and an excellent optimization would require me to re-familiarize myself with what had been my own code, I don't think I'd be enthusiastic about that.

raven ridge
#

Ah, I think most people would be enthusiastic about it, honestly.

boreal umbra
#

maybe I'm just too resistant to change 🙂

radiant fulcrum
#

Tbh if exelent optimisations came around I think it would be a good idea to go through with it regardless although annoying, if it doesn't cause you to drop it entirely that is.

raven ridge
#

But yes, I agree that it's just a matter of finding people with both the interest and expertise to optimize things. Optimizations will generally be accepted, as long as they don't come at too great a maintainability or flexibility cost.

#

And there are those among us who very much enjoy C 😄

radiant fulcrum
#

What NodeJs has over python, written in the slightly more user friendly cpp

raven ridge
#

The only major advantage of C++ over C that I see is destructors. That'd let you get rid of some gotos and explicit decrefs.

swift imp
#

Or once a pep is rejected it's rejected?

paper echo
#

Sphinx uses this

#

Even though they aren't official

swift imp
#

I know people don't care about help but I do and I want the feature to be compatible with that lol

paper echo
#

Right, understandable

swift imp
#

You can do it with descriptors but it's just overhead

paper echo
#

The issues identified in the pep still aren't solved in py 3.9

fringe canopy
#

Speaking of speed and python, no one has mentioned the pypy project yet?

swift imp
# 
class Descriptor:

   def __init__(self, doc_str):
       self.__doc__ = doc_str

   def __set_name__(self, owner, attr):
       self.attr = attr

   def __get__(self, instance, inst_type):
       return getattr(instance, f"_{self.attr}")

   def __set__(self, instance, value):
        setattr(instance, f"_{self.attr}", value)

class Foo:
   bar = Descriptor("Example Docstring")

foo = Foo()

help(foo)
#

Makes help work on instances and the class but it's still not as nice as property style docstring

raven ridge
#

Hm. typing.Annotated might be able to accomplish this?

swift imp
#

I wonder how property does it. If you added a property with docstring to my example whether you did help(foo) or help(foo.prop) or help(Foo.prop) it brings up the correct docstring

#

The descriptors doesn't work out too nicely though

paper echo
#

What about it @fringe canopy ?

fringe canopy
# paper echo What about it <@164064286522605568> ?

It was talk about cpython execution speed and if that was inherit to the language, and why there wasn't that much optimization done to the cpython implementation (which is not correct, but is usually not huge speed increases). Pypy kinda answers both questions

swift imp
#

I heard the python c code base is pretty attrocious

#

Something like a ridiculous large amount of code to do simple stuff

raven ridge
#

hm. not really. not more than a C programmer would expect, I think.

swift imp
#

No no it's pretty large. There was a video posted on it before but I can't find it. Some very simple operation took like close to a 1000 lines in C

raven ridge
#

I write code that uses the CPython API nearly every day. The amount of code it takes to do something usually somewhere around 5x would it would take to do the same thing in Python, which is pretty typical of the relative difference in verbosity/expressiveness between C and Python

#

CPython isn't particularly verbose or arcane for a C library.

swift imp
#

This is going to bug me, it was really interesting because it showed just how many checks the c code goes through due to the high dynamicism that python offers

#

I think it was literally adding 2 numbers

raven ridge
#

well, are you talking about the implementation of +, or are you talking about what C code that wants to add two Python objects together does? Because that's two very different beasts

swift imp
#

I mean source python

#

I said python c code base

raven ridge
#

yes, but which source? Source that wants to add two objects? Or source that implements how to add two objects?

swift imp
#

The logic that python goes through to add two objects

raven ridge
#

If you've got two objects, a and b, and you want to add them together and store the result in c, and you're writing Python, you would write: py c = a + b If you're doing the same thing in a CPython extension module, you would do: ```c
PyObject *c = PyNumber_Add(a, b);
if (!c) {
return NULL;
}

swift imp
#

Checking for add and then radd and so on and so fourth

#

No dude I'm not talking about writing cpython

#

I'm talking about the literal C code that python is written in

raven ridge
#

yes, that's CPython.

#

you're talking about what CPython does to implement PyNumber_Add, and I'm saying that that's only done in one place, and everywhere else can just call the PyNumber_Add library function.

#

if PyNumber_Add weren't implemented in C, you'd have to write a whole lot of Python code to emulate it, too. Probably somewhere around 20% as much code as the C version takes.

swift imp
#

This is about to go real off topic

#

Didn't I say that like 3 times in less precise terms bc Ive never written cpython

raven ridge
#

¯_(ツ)_/¯

#

maybe

swift imp
#

I wish I could find that video

raven ridge
#

if so, I didn't follow what you were trying to say. You're right that Python's dynamism means that the interpreter has to do a lot of work and runtime checks in order to execute operations that seem very simple in Python code. But that's got nothing to do with C - the interpreter would have to do the same amount of work and the same amount of runtime checks, regardless of what language it was written in

#

what I was trying to get across is that, if you were to write all those same checks and do all that same work in Python, it would be roughly 20% as much code as the corresponding C code is, and the extra C code is mostly extra repetitive boilerplate

swift imp
raven ridge
#

oh. not really. There are languages that are just as dynamic as Python but that have much more optimized interpreters. JavaScript, for instance.

#

And the big difference is that JavaScript is the only language that can realistically be used in the browser (or at least, was for a long time), and there was no escape hatch that allowed dropping down to a faster but less expressive language, and so a whole bunch of money and time and research went into optimized JavaScript interpreters

#

no one has ever put in that effort for Python, in part because there's less need, and in part because there's less corporate backing.

swift imp
#

What's special about js that only it can be ran in browsers

#

Maybe Microsoft will dump money into optimizing python KEK

raven ridge
#

accidental history, mostly. Some browsers started supporting JS, and then those browsers became popular, and popular websites began depending on it, which meant that new browsers needed to support JS in order to compete

#

it's not that it's the only language that could be used in a browser, it's just that in practice it's the only language that is. Or at least, the one that's used in 99% of cases.

#

WebAssembly being the other option that now exists.

swift imp
#

Never heard of it

raven ridge
deft pagoda
#

dabeaz wrote a python stack machine that interprets webassembly

raven ridge
#

some people are way more productive than me 😄

deep steeple
#

is there a transcriber from python to js?

spice pecan
#

Several

#

Though they tend to be more of a toy-ish thing than a production-ready solution

cosmic eagle
#

Hello there, i'd like to be using a live video from a capture card with python, checked online and apparently it doesn't work with all models.
What i'd need to do is get the screen streamed from another computer to the one running python and then crop/process/do things to that video live with python.
Anyone who got experience with that, could guide me to what capture card i have to get and so on?

deft pagoda
#

this is the wrong room

cosmic eagle
#

opened 2 the past 24 hours didn't get anything back from anyone. so i thought i might get some back here @deft pagoda. sounds like an advanced concept to me as google can't get me any answers after a bunch of hours searching.

deft pagoda
#

please read the topic description

#

this is a discussion channel about python

cosmic eagle
#

Anyways, happy new year, same old as 2020

deep bramble
#

I was playing with classmethod and discovered something weird, I noticed that doing classmethod.__call__ will return the call method of it, used for the decorator, but when the function is decorated, the instance of classmethod doesn't have this __call__ anymore.

I than tried to do a = classmethod(Foo.instance_method) and call a(), and indeed, a seemingly isn't callable, what's weird is that when I then do Foo.a = classmethod(Foo.instance_method) and call Foo.a() it works just as it should.

I can't seem to wrap my head around how was something like this implemented, it's very weird

📎 unknown.png
#

does somebody here know why is this happening and what's actually going on?

flat gazelle
#

What does Foo.cm tell you? @deep bramble

deep bramble
peak spoke
#

They work with the descriptor protocol, which won't get invoked when you use a normal name outside of a type's attribute access

deep bramble
#

very interesting, I've seen descriptors before, but only briefly, what exactly is the bound method here, and how is it different from a normal function?

peak spoke
#

When you access a method from an instance (or a class for classmethods) the __get__ binds it to that object which will implicitly pass the first arg

deep bramble
#

oh, yeah I see, how would I replicate something like that? I assume just something like this?

def pass_first_param(func):
  @wraps(func)
  def wrapper(*args, **kwargs):
    # Let's just assume `first_arg` was defined
    return func(first_arg, *args, **kwargs)
  return wrapper
``` what's interesting to me that the type is saying `bound method` rather than `function`, which wouldn't happen in my case
peak spoke
#

The get returns instances of MethodType, which handles the arg along with other things

deep bramble
#

where is that MethodType defined? would it be possible to manually define my own method?

peak spoke
#

You can access it from the types module. You can instantiate it normally, it accepts the callable as the first arg and the object to bind to for the second arg

deep bramble
#

that's very cool

#

what I was initially doing was making a class decorator to apply some other decorator to all of my functions, but it was failing the callable() check because of this, would it be possible to somehow decorate a classmethod?

def debugmethods(cls):
  for key, val in vars(cls).items():
    if callable(val):  # will fail for staticmethods/classmethods
      setattr(cls, key, debug(val))
  return cls
peak spoke
#

The objects returned by the decorators store the original function under __func__ so for something simple you could just handle it separately through that and then convert them back to what they were before when setting the attribute

deep bramble
keen arch
#

Hi

tidal marten
#

Hello, can __del__ be used to properly release objects being held by the class?

#

I'm trying to wrap the PyTessBaseAPI (for tesseract) in a class and it requires closing the object

# 
class WrapTest:
    def __init__(self):
        self._api = PyTessBaseAPI(...some parameters...)
    def __del__(self):
        self._api.End()
cloud crypt
#

maybe you should create a context manager that will close the API afterwards

sage scroll
#

i suppose to get 200

mint forge
#

@sage scroll this is not the channel for that

pure oak
sage scroll
#

oh okay leeme try

sage scroll
cloud crypt
# 
class Wrap:
    def __init__(self):
        self._api = PyTessBaseAPI(...)

    def __enter__(self) -> PyTessBaseAPI:
        return self._api

    def __exit__(self, type: Type[BaseException], value: Exception, traceback: TracebackType) -> None:
        self._api.End()``` or something
# 
with Wrap() as api:
    ...  # actual code here```
tidal marten
cloud crypt
#

oh, I see

#

well then, maybe you should register the shutdown with atexit module?

#

you can not rely on __del__, that is for sure

#

!docs atexit

fallen slateBOT
#
`atexit`

The atexit module defines functions to register and unregister cleanup functions. Functions thus registered are automatically executed upon normal interpreter termination. atexit runs these functions in the reverse order in which they were registered; if you register A, B, and C, at interpreter termination time they will be run in the order C, B, A.

Note: The functions registered via this module are not called when the program is killed by a signal not handled by Python, when a Python fatal internal error is detected, or when os._exit() is called.

Changed in version 3.7: When used with C-API subinterpreters, registered functions are local to the interpreter they were registered in.

tidal marten
#

The object is long-lived but can be invalidated in the middle of the lifetime of the program, I don't want to accumulate unused resource?

#

Or should I just let it accumulate and let the OS collect everything once it's exited?

cloud crypt
#

if it can be invalidated, you can unregister the cleanup

tidal marten
#

Last time I ran the script with __del__, it throws an error about self._api not existing. But now it just works, huh...

tidal marten
# 
class Invalidate(abc.ABC):
    @abc.abstractmethod
    def invalidate():
        pass
cloud crypt
#

and yeah, writing abstract Invalidate class sounds fine

tidal marten
#

I'll see what I can do. If the script is not running long enough or the object allocation is negligible, I might just let it accumulate...

swift imp
#

The new pep 642 is so confusing. The longer it takes to get into agreement on pattern matching the more confusing it gets and more I'm against it.

radiant fulcrum
#

tbh im starting to agree with that aswell

#

The only sort of match and pattern matching so far that i really like is Rust's system, Python's seems a little to pedantic ig you could say?

livid pollen
#

I think it makes much more sense in languages with really strong static typing (Rust's enums-with-values-inside are glorious)... destructuring is great, but I just don't see the need for it in Python

paper echo
#

It's especially offensive in the absence of a switch statement

#

I want guido back

radiant fulcrum
#

i do agree tbh

#

they just like idk

flat gazelle
#

python does need destructuring

#

a lot of code is literally

if is_a_structure:
    destructure
```but I don't think patma is the best solution
radiant fulcrum
#

all the examples they really show in the pep and are discussing basically just seem like a way to make code just less readable and more of a pain to write

fresh cargo
#

I feel like the relationship between patma and if is_a_structure does resemble that between switch-case and regular if, although the former seems to be slightly more useful. Can't say I like the current proposal tho

sturdy timber
#

I've lost track of where it's at now, is 642 the most recent/up to date proposal?

raven ridge
#

642 doesn't seem like a replacement for 634, but an alternative.

indigo notch
#

hey

atomic egret
#

What's the reason behind ever using try-finally block?
finally is something that's supposed to execute no matter if there were any errors or not, but that brings a question, why would I ever choose finally instead of just following up with my code?

try:
   f = open("test.txt")
   ...
finally:
   f.close()
``` vs ```py
try:
   f = open("test.txt")
   ...
f.close()

I just don't understand the reasoning behind every using something like that

brave badger
#

@atomic egret Outside the context of exception handling, try-finally is there to make sure that something is run unconditionally; something which can be applied in context managers

atomic egret
#

yeah, but I can just put it outside of my try and it will run unconditionally too

#

why would I choose to use finally?

#

could you show me some example where finally is absolutely necessary and just putting the code outside of the try block wouldn't work? because I just can't imagine any scenario like this

brave badger
#

It will not, actually

>>> from contextlib import contextmanager
>>> @contextmanager
... def do_something():
...   try:
...     yield None
...   finally:
...     print("Always Run")
...
>>> @contextmanager
... def do_something_else():
...   yield None
...   print("Never Run")
...
>>> with do_something():
...   raise Exception()
...
Always Run
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
Exception
>>> with do_something_else():
...   raise Exception()
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
Exception
atomic egret
#

yes but if you did ```py
def do_something_else():
try:
yield None
print("Alwats run")

brave badger
#

Nope, try would always come with either an except/finally

spark magnet
#

@atomic egret if your try block raises an uncaught exception, the line after the try won't run.

#

@atomic egret try this:

try:
    1/0
finally:
    print("finally")
print("after")
atomic egret
#

why didn't the try block catch it?

spark magnet
#

@atomic egret becuase there's no except clauses that matches the exception

atomic egret
#

I thought doing py try: ... was the same as ```py
try:
...
except:
...

spark magnet
#

nope

peak spoke
#

Most context managers can be remade with a simple finally (or are implemented through it)

#

You don't see it as much because of that, but there are still plenty uses

spark magnet
#

@atomic egret the try block i showed won't catch any exceptions

atomic egret
#

so why would someone ever use pure try-finally without except?

#

oh, nvm, I get it

spark magnet
#

@atomic egret because you want something to run no matter what happens

atomic egret
#

yeah

#

that's cool, thanks for explaining it!

steel surge
#

HI guys I would like to check what is the standard practice of unit test do we make an API call to mock server or to create a static json file and make requests to the static file itself to get the response and request?

pulsar lava
#

is there a reason anyone would mutate a string for a variable using slices and concatenations to make a new string instead of just rewriting the string?

boreal umbra
#

If you're asking why anyone would need to use the "step" part of a string slice, well, I've never actually used it lemon_pika

pulsar lava
#

this would be an example

#

I'm not sure why you wouldnt just overwrite the name variable

raven ridge
#

because you still want to reference the original string by the original name

boreal umbra
raven ridge
#

or because you want to be very explicit that it's now a different string, and should therefore have a different name.

pulsar lava
#

hmm okay thank you guys, I was just interested in where this would be applied

boreal umbra
#

I've never re-defined a string variable in terms of a string literal, though. It's always when I'm reading files where the start and end index of a string is all I have go to by

#

@raven ridge knows all about my library for that 🙂

#

(or at least, more than most people.)

raven ridge
#

sometimes people have good reasons for why they do things, sometimes not - my best guess about the name versus newName example above is that they were trying to be very clear that it does not modify a string (because strings are immutable), which is a thing you might expect to see in code that's trying to teach you something

#

later on, when people assume you already know the language, they might reuse the same variable name for two different strings, because they can count on the reader to know that they're two different strings.

tidal marten
#

Hey, any idea if opencv/numpy would benefit from multithreading in Python?

boreal umbra
tidal marten
#

I'm not sure if any of them are thread-safe

raven ridge
tidal marten
#

I'm not sure if small burst of matrix operations would benefit from multithreading to be honest...

#

What I do is many small burst operations using opencv and numpy

raven ridge
tidal marten
#

I think the bulk of the performance improvement comes from the video I/O. I only do bitmap manipulation. I have tried multiprocessing and it works great though...

raven ridge
#

well, in general "about the language" terms, a Python extension module can release the GIL, but once it has released the GIL it can't create Python objects or manipulate Python data structures or call Python functions until it reacquires the GIL. So whether or not any given C extension does release the GIL is a question of whether or not it can do enough of its work with the GIL released for releasing the GIL to be useful.

#

it should always release the GIL before performing blocking IO - failure to do that is a bug in the extension. But when it's doing any CPU bound work, it's a question of whether it is even able to do that work without the GIL.

tidal marten
#

I don't think my code is thread-safe. It just segfaulted when I use threads, lol...

grave jolt
#

I fell for this with google's own API wrapper

#

tried to run it in a threadpool executor, got some nice double-frees

raven ridge
#

that's worth reporting as a bug. My opinion is that it should never be possible to invoke undefined behavior by using the public interface of an extension module.

tidal marten
#

Using Thread directly actually runs slower than if I just run the operations sequentially

#

Well, I guess no multithreading for me

tidal marten
raven ridge
#

regardless of what API it is, that's a bug in the Python API - if it's not threadsafe, it should be using locks to guarantee that things run sequentially rather than in parallel.

tidal marten
#

Yeps, wrapped the Tesseract API in threading.Lock and there's no segfault

raven ridge
#

yeah - definitely sounds like a bug that should be reported to the owners of the Tesseract API, if it hasn't been already.

tidal marten
#

Hmm, yeah, I'll see if I can at least get useful information. I'm using tesserocr btw.

#

I'm trying to compare between spawning threads and using threadpoolexecutor. Thread pool runs slower.

raven ridge
#

for the same number of worker threads?

tidal marten
#

Yeps

#

Each operation takes at most 420ms (cold run) and at least 370ms (after initial run). Thread pool takes like 1800+ms to finish 5 operations. Manually spawned thread takes like 1100ms.

#

Oh wait, I put in the wrong parameter. It was 5 workers with 10 operations for the thread pool.

#

Running 5 workers with 5 operations takes about the same amount of time

raven ridge
#

That's what I'd have expected.

tidal marten
#

So multithreading does work. I just need to make sure that my components are thread-safe.

grave jolt
radiant scroll
#

I was messing with python's import machinery and found out about sys.meta_path which holds the finders to resolve imports into the ModuleSpec objects.
What's weird is that when I cleared the cache (sys.modules = {}) and set this meta_path to empty list, somehow it was still able to look up some modules, like os or sys. I was following the official python documentation: https://docs.python.org/3/reference/import.html from which, as I understood it, was clear that the finders in meta_path should be the only thing that's being done when resolving imports (apart from initial cache check with the sys.modules), but if that's true, how come I was still able to import os, without having any finders in the meta_path?

raven ridge
languid dagger
# radiant scroll

Not sure exactly why but all the modules that were loaded in sys.modules when the interpreter was started still imports but any others such as itertools will fail

undone hare
#

I am guessing it has to do with the importlib bootstrapper

radiant scroll
#

is it possible to somehow avoid this behavior?

undone hare
#

I'm not sure, the bootstrapper is a critical part of the interpreter

#

Are you trying to limit the available libraries?

radiant scroll
#

I was just exploring what can be done, I'm not currently using it for anything specific, but I would be interested in knowing exactly what's going on here because it's very odd and following the official docs doesn't provide anything about why would this happen

raven ridge
#

!e ```py
import sys
sys.modules = {}
sys.meta_path = []
import os

fallen slateBOT
#

@raven ridge :warning: Your eval job has completed with return code 0.

[No output]
raven ridge
#

!e ```py
import sys
sys.modules.clear()
sys.meta_path = []
import os

fallen slateBOT
#

@raven ridge :x: Your eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 4, in <module>
003 | ModuleNotFoundError: No module named 'os'
undone hare
#

huh

radiant scroll
#

that's odd

languid dagger
#

Oh it was the references

undone hare
#

IIRC the docs say something about that

languid dagger
#

sys.modules must be passed somewhere else

raven ridge
#

Something has retained a reference to the old sys.modules, and you've made a new one without clearing the old.

radiant scroll
#

hmm, that's interesting I see, thanks

tidal marten
#

How are you supposed to do imports in module with nested modules?

mint forge
#

I think a question for software design, but i m not sure

tidal marten
#

Last time I tried using relative imports, I was not able to use isinstance because somehow a class is defined with two different module path. But now it needs relative imports.

prisma furnace
#

Is there anything wrong how I'm loading type objects from pyarrow with this dict comp?

import pyarrow as pa
import yaml

schema = yaml.load(open(schema_yaml))
load_columns = schema['columns']
dropped_columns = []
my_types = {k: getattr(pa, v['pa_type'])()
    for k, v in load_columns.items()
    if k not in dropped_columns}
rose anchor
#

hmm

#

i will look about it

#

i have to get the package

boreal umbra
#

Are there dunder methods that linters rely on?

#

I know that each scope has a __annotations__ dict

boreal umbra
#

New topic: Some people have raised the point that having None as the sentinel value for mutable default arguments is bad because then None can't really be a valid argument if it's actually what you need. But what about NotImplemented?

#

It's kinda hacky, but it's a singleton that everyone has. I can't think of a time you'd pass it and I can't think of a time that would mess with the mechanisms behind infix dunder methods.

pliant tusk
#

I tend to use singleton = object()

peak spoke
#

If you do need None to be a meaningful value then that is a better choice imo instead of repurposing a completely unrelated object

boreal umbra
#

I've made sentinel object()s in the past

livid pollen
#

I've always thought that ... would be a good sentinel

silk coral
#

Is there a list of all built-in Python types that can't be referenced by name? (Eg. tracebacks, modules, list iterators, etc.) The types module has a few of these, but seems to be missing some.
(Idk where to put this so I'm just gonna leave it here)

pliant tusk
#

its possible to map them all out

boreal umbra
silk coral
#

Cool

silk coral
pliant tusk
#

@silk coral gimme a sec ill write it

silk coral
#

there's a lot afaik

pliant tusk
#

yea there are

peak spoke
#

getting all names from the types and collections.abc modules ought to get you most of it

silk coral
#

I forgot about collections.abc

#

I think there might still be some missing but that should cover a lot of it, yeah

spark magnet
#

@silk coral lots of the names are boring, like Ellipsis

pliant tusk
#

@peak spoke that doesnt include subclasses

silk coral
#

what about dict_reversevalueiterator?

#

!e print(type(reversed({}.values())))

fallen slateBOT
#

You are not allowed to use that command here. Please use the #bot-commands channel instead.

silk coral
#

o

swift imp
#

There's an NotImplemented object that's different from NotImplementedError?

spark magnet
silk coral
boreal umbra
#

it tells the interpreter to try the reverse operation using methods from the other object

#

which I know sounds confusing

silk coral
swift imp
#

What's an infix

silk coral
#

( #bot-commands )

boreal umbra
swift imp
#

Oh so a binary op

#

Oh ok ok

spark magnet
swift imp
#

That how the interpreter knows to try say radd

silk coral
pliant tusk
# 
def get_all(cls=object):
    # check if cls is builtin, then:
    yield cls
    for c in type(cls).__subclasses__(cls):
            yield from get_all(c)
peak spoke
#

I don't believe those are exposed anywhere without getting the type yourself (and are useless to handle), but every builtin etc. that needs an iterator or some other object to create will have its own type with the behaviour

spark magnet
#

@silk coral that finds you an object with that type, bbut the name isn't available.

pliant tusk
#

this will get all registered subclasses of object, and their subclasses recursively

silk coral
spark magnet
#

there isn't one master list like that

silk coral
#

oh ok

pliant tusk
# 
>>> c=lambda n,l=lambda l,o=object:sum([l(l,c)for c in type.__subclasses__(o)],[o]):{f'{c}'.split("'")[1]:c for c in l(l)}.get(n)
>>> c('dict_reversekeyiterator')
<class 'dict_reversekeyiterator'>
>>> ``` @silk coral this function `c` will let you grab any currently initialized class (that the interpreter knows about) by name (if its in a package you'll need to specify the package as well, ie: `c('importlib.abc.FileLoader')`
silk coral
#

wait huh

pliant tusk
#

some c modules (and the interpreter sometimes as well) will create a class, but not register it, as it should not be initialized by the user

silk coral
#

this syntax is confusing me 😅

pliant tusk
#

oh yea its a janky lambda i wrote last year

#

lemme clean it up

silk coral
#

ty

pliant tusk
# 
def get_cls(name, base=object):
  if repr(base).split("'")[1] == name: 
    return base
  for cls in type(base).__subclasses__(base):
    if ret := get_cls(name, cls):
      return ret
``` @silk coral
#

it splits the repr of the class to get module.name for non-builtins, but name for direct builtins '.'.join([base.__module__, base.__name__]) would result in stuff like builtins.dict being required

cloud crypt
#

oh damn, fair

#

was about to ask it

raven ridge
raven ridge
cloud crypt
#

dynamic linters hmmm

tidal marten
#

PyCharm can't even check for wrong function names and stuffs, smh

#

Like half of my bugs are just wrong function calls because I changed some stuffs...

brave badger
#

Project-wide refactors do exist though

tidal marten
#

I was changing the class that I want to instantiate, but somehow it doesn't detect that the previous function calls are invalid. This was in the same file.

#

Sometimes I forgot to use refactor as well, so I'm stuck finding all the previous instances of that function call.

radiant fulcrum
tidal marten
#

Probably the former. I didn't touch anything tho...

#

Or maybe I ran out of RAM, lol

#

Also just found that deleting a Thread object does not kill the thread, so this code is dangerous:

from threading import Thread
from time import sleep

def infinite_thread():
  while True:
    sleep(3)
    print('Hello there!')


v = Thread(target=infinite_thread)
#v.start()  # Uncomment
del v
#

Does anyone know how to kill an orphaned thread?

grave jolt
#

Maybe you need a daemon thread?

wicked crown
#

Iirc the way to kill off threads is to interrupt them and have behaviour defined for how to stop the thread upon an interrupt. I think you use the threading.event to do this

tidal marten
#

Ideally all the resources are initialized when the thread is initialized and kept until the program is terminated.

#

I don't think I can do that with thread pool?

radiant fulcrum
#

seems pretty unnecessary considering the fact that threads are pretty cheap to make and would be more hassle than to just make a new one

tidal marten
#

I was thinking of having multiple threads that do something like Tensorflow, so each thread would have its own model for prediction.

#

Just for experimentation though. But I think using multiprocessing would be better.

grave jolt
#

Maybe you need a thread pool?

tidal marten
#

Thread pool would delete the tensorflow model each time the operations end, right?

#

I was thinking about having a fixed number of threads on infinite loop and communicate with them through channels.

#

I have some idea on how to do it in Golang, but not in Python currently...

paper echo
#

@tidal marten you can do something like this

N_THREADS = 16
threads = [Thread(target=foo) for _ in range(N_THREADS)]
#

honestly python is really bad with multithreading and multiprocessing

#

however there are things like queue.Queue which kinda act like a channel and are thread-safe

#

there are also locks, semaphores, etc

#

but you still have to worry about the GIL

#

which might not be an issue in tensorflow because it calls out to C code

deft pagoda
#

i feel like one could do actors with some queues sockets and multiprocessing

distant quartz
#

Does that new multiprocessing call happen on a new machine in the cluster (if using something like Kubernetes), or does Python try to cram it onto the same machine it was called on?

boreal umbra
paper echo
#

@deft pagoda fwiw actor is more of a programming model than anything. BEAM for example has its own scheduler & all threads are "green threads" that might or might not be run on their own OS threads, at the whim of the scheduler

#

Ruby 3.0's fiber implementation did something similar, where they rewrote all the i/o primitives to be "async optional", so you don't need async/await keywords nor do you need a whole new stack of i/o primitives, you can just either run it in a Fiber (where the i/o routine might or might not hand off control), or without (where the i/o routine will be fully blocking)

#

actually wait.. im wrong on that, they have special async i/o methods

#

but they don't have this language "color problem" that python has

deft pagoda
#

yep, but you can still implement actors in python with queues sockets and multiprocessing

paper echo
#

yes. but the point is that you're not implementing "actors" as much as you're implementing your own OS type of scheduler thing

deft pagoda
#

those are actors

paper echo
#

i'd argue that it's one way to implement actors

deft pagoda
#

why isn't there an actor library somewhere

#

is there?

paper echo
#

seems like there are a few

deft pagoda
#

they're all so old 😦

distant quartz
#

Hey all - got a quick question that may be slightly broader than just python. I'm running a function that uses multiprocessing on my cluster. Say that I'm already using my cluster and it has 4 CPUs left, but I run that function with 8 Processes
Does that new multiprocessing call happen on a new machine in the cluster (if using something like Kubernetes), or does Python try to cram it onto the same machine it was called on?

radiant fulcrum
#

Ive been working on implementing a actor framework that fits ontop of the exist async / await key words

#

fun but pain

#

i dont think it'll be able to keep the async / await keywords unless i can override the coroutine class

paper echo
#

@radiant fulcrum you can use await but you might have to replace async with your own class that implements __await__

radiant fulcrum
#

yeah which is le issue

paper echo
#

you an also use async and then write your own constructor that does what asyncio.create_task does

radiant fulcrum
#

i dont actually know if you can even use await without it being part of the default coroutine class

paper echo
#

!e ```python
import asyncio

class MyAwaitable():
def await(self):
return 'asdfjkl'

async def amain():
awaitable = MyAwaitable()
print(await awaitable)

asyncio.run(amain())

fallen slateBOT
#

@paper echo :x: Your eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 11, in <module>
003 |   File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
004 |     return loop.run_until_complete(main)
005 |   File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
006 |     return future.result()
007 |   File "<string>", line 9, in amain
008 | TypeError: __await__() returned non-iterator of type 'str'
paper echo
#

interesting

#

!e ```python
import asyncio

class MyAwaitable():
def await(self):
yield 'asdfjkl'

async def amain():
awaitable = MyAwaitable()
print(await awaitable)

asyncio.run(amain())

fallen slateBOT
#

@paper echo :x: Your eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 11, in <module>
003 |   File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
004 |     return loop.run_until_complete(main)
005 |   File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
006 |     return future.result()
007 |   File "<string>", line 9, in amain
008 |   File "<string>", line 5, in __await__
009 | RuntimeError: Task got bad yield: 'asdfjkl'
paper echo
#

gotta go dig into the docs i guess

#

not sure what the data contract is supposed to be here

radiant fulcrum
#

await needs to return a iterator

paper echo
#

is a generator not an iterator?

#

!e ```python
import asyncio

class MyAwaitable():
def await(self):
return iter(['asdfjkl'])

async def amain():
awaitable = MyAwaitable()
print(await awaitable)

asyncio.run(amain())

fallen slateBOT
#

@paper echo :x: Your eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 11, in <module>
003 |   File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
004 |     return loop.run_until_complete(main)
005 |   File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
006 |     return future.result()
007 |   File "<string>", line 9, in amain
008 | RuntimeError: Task got bad yield: 'asdfjkl'
radiant fulcrum
# 
import asyncio

class MyAwaitable():
    def __await__(self):
        self
    
    def __iter__(self):
        self
    
    def __next__(self):
        return "stuff"

async def amain():
    awaitable = MyAwaitable()
    print(await awaitable)

asyncio.run(amain())```
paper echo
#

!e ```python
import asyncio

class MyAwaitable():
def await(self):
self

def __iter__(self):
    self

def __next__(self):
    return "stuff"

async def amain():
awaitable = MyAwaitable()
print(await awaitable)

asyncio.run(amain())

fallen slateBOT
#

@paper echo :x: Your eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 17, in <module>
003 |   File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
004 |     return loop.run_until_complete(main)
005 |   File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
006 |     return future.result()
007 |   File "<string>", line 15, in amain
008 | TypeError: __await__() returned non-iterator of type 'NoneType'
paper echo
#

it seems that the asyncio event loop doesn't like what is actually being yielded

radiant fulcrum
#

🤔

paper echo
radiant fulcrum
paper echo
#

i think ensure_future is related here

radiant fulcrum
#

should of probably looked at my own notes

#

ensure_future makes sure that any future is wrapped in a coroutine class

#

as create_task enforces it being a coroutine class

paper echo
#

right, but i remember reading the ensure_futures source and it let you use general __await__-implementing objects

radiant fulcrum
#

yeah cuz it wraps the custom class

paper echo
#

right

#

either way, yield foo should do +/- the same thing w/ respect to returning an iterator (or at least a generator which afaik is a specialization of an iterator)

radiant fulcrum
#

yeah

paper echo
#

(or maybe a generator is iterable and not an iterator as such but i cant imagine that would cause a problem)

radiant fulcrum
#

pretty sure its because asyncio.run() will enforce the coroutine class requirement

#

i think

paper echo
#

im curious what the "bad yield" is about

#

!e ```python
import asyncio

class MyAwaitable():
def await(self):
yield 'asdfjkl'

async def amain():
awaitable = asyncio.ensure_future(MyAwaitable())
print(await awaitable)

asyncio.run(amain())

fallen slateBOT
#

@paper echo :x: Your eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 11, in <module>
003 |   File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
004 |     return loop.run_until_complete(main)
005 |   File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
006 |     return future.result()
007 |   File "<string>", line 9, in amain
008 |   File "/usr/local/lib/python3.9/asyncio/tasks.py", line 687, in _wrap_awaitable
009 |     return (yield from awaitable.__await__())
010 |   File "<string>", line 5, in __await__
011 | RuntimeError: Task got bad yield: 'asdfjkl'
paper echo
#

gotta look at the tasks.py source i guess

radiant fulcrum
#

does it return the correct StopIteration(value) response?

paper echo
#

it's calling yield from awaitable.__await__() so that isn't the problem

radiant fulcrum
#

bearing in mind asyncio gets the result from the value of the error

paper echo
#

not sure what it's expecting, but it doesn't like the string being returned. need to look at the source to see what triggers that RuntimeError

radiant fulcrum
# 
class MyCoroutineCopy:
    def __init__(self, arg1):
        self.arg1 = arg1

    def __await__(self):
        return self

    def __iter__(self):
        return self

    def __next__(self):
        raise StopIteration(self.arg1)

async def amain():
    awaitable = MyCoroutineCopy("hello")
    print(await awaitable)

asyncio.run(amain())
#

that should work

paper echo
#

!e ```python
import asyncio

class MyCoroutineCopy:
def init(self, arg1):
self.arg1 = arg1

def __await__(self):
    return self

def __iter__(self):
    return self

def __next__(self):
    raise StopIteration(self.arg1)

async def amain():
awaitable = MyCoroutineCopy("hello")
print(await awaitable)

asyncio.run(amain())

fallen slateBOT
#

@paper echo :white_check_mark: Your eval job has completed with return code 0.

hello
paper echo
#

ah

#

interesting

radiant fulcrum
#

looks like it doesnt return the correct raise

paper echo
#

so it's expecting nothing to be yielded

#

and instead expects to try/except a StopIteration

#

which it then pulls the value from

#

......why

radiant fulcrum
#

well it expects either a return

#

or a StopIteration error

#

I imagine its because of the C api

paper echo
#

a return?

radiant fulcrum
#

return essentially says im not ready yet

#

raise StopIteration(arg) says that it's returning and is complete

#

it's the equivalent of yield + return for a normal function Or should be atleast

paper echo
#

i don't think you can use a return in a generator at all

#

can you?

#

or is that the value that will be sent along with StopIteration

radiant fulcrum
#

thats how teh __next__ method wants

paper echo
#

oh, yeah. i mean with yield syntax

radiant fulcrum
#

its diffrent with for loops

#

return in __next__ is like yield

paper echo
#

__next__ returns the next value of the iterator or raises StopIteration which triggers yield from and for to stop iterating

#

that much i know

radiant fulcrum
#

yeah

#

its a bit of a backwards method lmao

paper echo
#

nah i get that

#

i was just confused about what you were describing

radiant fulcrum
#

it makes a degree of sense

#

considering you dont have yeild or that in the low level api

paper echo
#

def foo(): yield 'a' returns a generator which does all the stuff above

#

!e ```python
def mygen():
yield 1
return 2

g = mygen()
while True:
try:
print(repr(next(g)))
except StopIteration as e:
print(repr(e))
break

fallen slateBOT
#

@paper echo :white_check_mark: Your eval job has completed with return code 0.

001 | 1
002 | StopIteration(2)
paper echo
#

so it looks like yield is always expected to be None

paper echo
#

ah it looks like they used StopIteration so that you could use yield from in the old style coroutines

raven ridge
#

!e I hit something similar the other day trying to implement my own __await__ - turns out you can do this:

import asyncio

class MyCoroutineCopy:
    def __init__(self, arg1):
        self.arg1 = arg1

    def __await__(self):
        return self._my_value().__await__()

    async def _my_value(self):
        return self.arg1

async def amain():
    awaitable = MyCoroutineCopy("hello")
    print(await awaitable)

asyncio.run(amain()) 
fallen slateBOT
#

@raven ridge :white_check_mark: Your eval job has completed with return code 0.

hello
raven ridge
#

Which is at least slightly less arcane

radiant fulcrum
unkempt rock
#

hello anyone here?

#

I have an advanced request

#

just trying to build a team*

#

hahaha

radiant fulcrum
#

going back to the topic of actors

# 
import cobalt
import time

from typing import Any


class Actor:
    def __init__(self):
        self._actor = cobalt.CobaltActor(self.on_message)

    def on_message(self, message: Any):
        pass

    def send(self, message: Any):
        self._actor.send(message)


class Actor1(Actor):
    def on_message(self, message: Any):
        print(f"hello world! {time.time()}")
        time.sleep(5)


class Actor2(Actor):
    def on_message(self, message: Any):
        print(f"hello world! {time.time()}")


def test():
    act1 = Actor1()
    act2 = Actor2()
    act1.send("hello world! 1")
    time.sleep(1)
    act2.send("hello world! 2")

    time.sleep(12)


cobalt.run(test)
#

Work stealing scheduler is freaking hot

# 
Running Cobalt reactor!
Actor is running
Actor is running
hello world! 1609882165.2952862
hello world! 1609882166.296234```
karmic cypress
#

not sure which channel fits this, but does anyone know a decent free netflix api?

charred wagon
acoustic estuary
#

Anyone here familiar with working with cpython directly? I am running the unit tests and so far taking over 2 hours and not completed, is this normal?

2:18:50 load avg: 0.52 running: test_ssl (2 hour 9 min), test_selectors (2 hour 10 min)
2:19:21 load avg: 0.52 running: test_ssl (2 hour 10 min), test_selectors (2 hour 11 min)
2:19:51 load avg: 0.52 running: test_ssl (2 hour 10 min), test_selectors (2 hour 11 min)
2:20:21 load avg: 0.52 running: test_ssl (2 hour 11 min), test_selectors (2 hour 12 min)
2:20:51 load avg: 0.52 running: test_ssl (2 hour 11 min), test_selectors (2 hour 12 min)
2:21:21 load avg: 0.52 running: test_ssl (2 hour 12 min), test_selectors (2 hour 13 min)
2:21:51 load avg: 0.52 running: test_ssl (2 hour 12 min), test_selectors (2 hour 13 min)```
karmic cypress
#

ty

desert peak
#

I'd probably ditch Optional for | None

#

first 39 makes it easier to use generics, now 310 makes it easier to use unions? neat

sacred yew
#

!pep 563

fallen slateBOT
#
**PEP 563 - Postponed Evaluation of Annotations**

Link

Status

Accepted

Python-Version

3.7

Created

8-Sep-2017

Type

Standards Track

sacred yew
#

oh yeah now there aren't any performance issues

astral gazelle
boreal umbra
#

I don't necessarily understand what people dislike about lambda other than that there are often other alternatives. But what about using from?

list_.sort(key=x[0] from x)

how bout it?

cloud crypt
#

lambda is wordy as well

#

also operator.itemgetter(0) is something I'd use here still

boreal umbra
#

I think that that use of from is nice and pythonic-looking. I imagine the new parser can figure it out more easily.

#

(it wasn't my idea btw)

cloud crypt
# 
import operator

get_first = operator.itemgetter(0)

...

points.sort(key=get_first)``` I'd use something like that
#

and well, I lowkey like from, but not sure

boreal umbra
#

that's explicit, I suppose

#

and that's good

#

though it's also an extra statement

cloud crypt
#

can we get python to be faster and then work on more beautiful additions ;)

raven ridge
# boreal umbra I don't necessarily understand what people dislike about `lambda` other than tha...

I think lamdba dislike comes in three different forms:

  1. Some think it's not powerful enough (limited to a single expression, no statements, etc)
  2. Some think it hurts readability (doing def would force you to name the thing, which would help readers understand it, and prevents the need to use awkward structures to keep things in one expression)
  3. Some think the syntax for defining lambdas isn't Pythonic enough.

Of the 3 concerns, I expect 3 is the least serious. 1 and 2 are at odds, anything that makes the proponents of 1 happy will make the 2s unhappy, and vice versa.

boreal umbra
astral gazelle
#

lambda limited to a single statement is definitely not a good thing, look at js and its arrow functions, you could do anything in the world inside them

spiral willow
#

2. Some think it hurts readability (doing def would force you to name the thing, which would help readers understand it, and prevents the need to use awkward structures to keep things in one expression) that's my take, it's also limiting, if future requirements mean lambda won't cut it, it's problematic to expand out of it

cloud crypt
# 
what about my [secret weapon](https://github.com/nekitdev/braces.py), haha?```
boreal umbra
cloud crypt
#

I have made braces for python

boreal umbra
#

I can't.

cloud crypt
#

^~^

#

might want to rework it at some point, though

boreal umbra
#

There are two possible cases:

  1. Your code blocks are short enough that you can see where they started and know what is in what block, and curly braces afford you nothing
  2. Your code block is so long that you can't see where it started, and curly braces afford you nothing
#

whitespace syntax is the most obvious win for Python and you'll never be able to convince me otherwise.

spiral willow
#

eh, I do work with Powershell as well as Python, {} has it place

boreal umbra
#

when you're writing Powershell stuff, is that a repl-like situation?

spiral willow
#

but also {{{{}}}} gets annoying as well

#

We use Powershell at work for everything management and REST API End to End Testing

raven ridge
#

The best argument for braces is that they'd make it possible to write one liners in a shell.

boreal umbra
#

that was easy

raven ridge
#

And shell commands don't need to be only one line, so that's mostly a stylistic complaint, anyway.

spiral willow
#

it's easy for it not to happen

raven ridge
#

They just don't want indenting in their shell commands.

spiral willow
#

like python if x is blah: if y is blah: if z is blah: #please stop

#

that can get annoying so we can sometimes don't tab EVERYTHING for easy of reading

spiral willow
raven ridge
#

I rarely write Python in the shell, but that's because of a huge number of different things that make it annoying to use in short commands. Indenting is one of them, but not the only one, and not even really the biggest.

spiral willow
#

{} in Pwsh is mainly because C#

#

the biggest thing I'm glad I have in Powershell when I need it is static typing

#

Anyways, Python

#

I do dislike Lambdas just because how limiting they are and it's not terribly difficult to do def blah(input): return blah

peak spoke
#

I believe that allowing more than an expression in them would bring more cases that wouldn't be as readable, although there are some situations where it could be handy

spiral willow
#

I feel like more expression would result in more cursing by next developer who has to modify your code

#

but that's my watching developers spin around code they thought could written to be easier even if it's more verbose

wind bridge
#

what is "+=" "-=" nd "=="

#

"!=" .....??

radiant fulcrum
#

== is equal to e.g. a Boolean comparison
!= is not equal to e.g. doing not (x == y)

+= is the same as doing x = x + n
-= is the same as doing x = x - n

wind bridge
#

oh

wind bridge
modern night
#

but in essence x += n does basically the same thing as x = x + n most the time

radiant fulcrum
#

was trying to not make it too complicated lul

#

but yeah

rose anchor
desert peak
#

I saw something recently I didn't understand

#

a library was re-exporting its members using

from X import X as X
#

what is the point of re-naming the export to the same thing it already was?

#

it might've been

from pkg.mod import X as X
#

is the as X pointless?

#

the library was doing it for all exports

flat gazelle
#

probably some convention, it is indeed superfluous

unkempt rock
#

hi i joined this server

#

i have an error

desert peak
#

read the header.

mystic cargo
honest narwhal
#

Is there a way to track how much memory a function is using? Also how do I set a timeout on a process started by concurrent futures ProcessPoolExecutor, if a function runs for more than the given amount of seconds, stop the process completely or if the function uses a given amount of memory

@limit_memory(bits=16_000_000)
def function():
...    while True:
...      pass
>>> function()
MemoryError: function exceeded memory limit```
like limit the amount of memory a function can have?

or maybe something like 
```py
@limit_memory(bits=number)
@limit_runtime(seconds=10)
>>> def function():
...    while True:
...      pass
>>> function()
TimeoutError: function took longer than 10 seconds to run```, I'm making my own decorators. All I need to know is how to track the memory, this code shows exactly how it should look like

I'm currently working on the limit_runtime part that limit the amount of time a function can run, but I'm stuck on giving the `concurrent.future.ProcessPoolExecutor` a timeout thing
boreal umbra
#

@honest narwhal I'm not sure that you can know how much memory is specific to a given function.

#

@honest narwhal I'm not sure that you can know how much memory is specific to a given function.

honest narwhal
#

so I can't limit the amount of memory the function can use?

radiant fulcrum
#

That's not something you can control

honest narwhal
#

okay and what about the second thing

#

timeout for the process

boreal umbra
honest narwhal
#

yeah I wanna prevent that

boreal umbra
honest narwhal
#

I don't know

desert peak
#

while True

honest narwhal
#

well I mean it's not always like that

desert peak
#

this sounds like the halting problem, however

honest narwhal
#

Yeah I know

#

it's not supposed to know if it halts or not

boreal umbra
#

If this is ultimately intended to help you debug another problem that you're having, there is probably a better solution.

honest narwhal
#

I just like need to make it so it has a time limit

#

okay

boreal umbra
#

Try opening a help channel and describe what the actual function (not an example function) is intended to do, with some examples of when it returns and when it appears to start running forever. See #❓｜how-to-get-help

honest narwhal
#

Alright

#

well I don't have an actual function

#

I have the decorator function made

#

but not an actual function to use it on because the decorator doesn't work

true ridge
#

interesting problem, though seems like a bit overengineering

boreal umbra
true ridge
#

I guess the best way would be fixing the function instead of trying to figure out how to prevent it from consuming infinite amount of resources

boreal umbra
#

if you have a function that is unexpectedly running forever, having a system for killing that function if it runs for too long won't actually solve the fact that it has unexpected behavior.

#

but I might be misunderstanding what your goals are.

true ridge
#

the best you can do for calculating how much of memory a function uses is, traversing everything in the frame recursively including all the children. If you compile python with a debug build, there are also some different stuff that you can take advantage of, but basically these things should never happen in real life.

#

(from what I see) as everyone else said, it is much better to either fix your function or create a limited environment (like a python subprocess)and let the OS to handle this stuff (something like a ulimit -v in place)

raven ridge
#

ProcessPoolExecutor was mentioned earlier... You can limit know much memory a process uses, or kill a process after a certain amount of time from another process

prisma grove
#

can anyone point out the exact differences between python on windows and linux,?

raven ridge
#

You could even do it from another thread in the child process

cloud cedar
#

how do you install pygame and add it to atom text editor???

radiant fulcrum
#

Well after a boredom spree i got https://github.com/ChillFish8/tactix working properly, somehow fixed it deadlocking the GIL.
Surprisingly affective and pretty nice to have essentially a multi-threaded asyncio, although this is pretty under-developed

GitHub
ChillFish8/tactix

A work-stealing actor framework for Python built on Tokio.rs - ChillFish8/tactix

desert peak
#

multi-threaded asyncio
I think you missed the point of an event loop

radiant fulcrum
#

no no i get the point of the event loop

#

The behaviour per-thread is still like asyncio, in the sense that you can have one thread and many thousands of tasks on a single thread without blocking

#

the system is just able to switch out tasks between threads and their related schedulers

radiant fulcrum
desert peak
radiant fulcrum
#

I am aware

#

That's just the logic of Tokio.rs and therefore the runtime logic of the actor framework

desert peak
#

oh, you're emulating rust's eco

radiant fulcrum
#

the runtime is actually directly built on tokio

#

you dont have the CPU power though as you would though because of the GIL

#

but it does allow you to essentially automatically load balance and handle blocking tasks while running many concurrently

#

to python they're seen as a sort of weird thread ig is the wording?

paper echo
#

pardon my ignorance but what does "work stealing" mean

#

ah i see

#

so this is still cooperative multitasking

#

but the scheduling model is that a thread of execution pick up and work on any task from any coroutine at any time

#

is that accurate ish?

radiant fulcrum
#

the scheduler is what is essentially implementing the tasks and control their execution, basically what asyncio is

#

and then the work stealing is essentially allowing many schedulers to co-operatively work together

#

In the real world it is incredibly nice

paper echo
#

right, so the multitasking model still requires you to be running "coroutines" which can yield control to the scheduler, and which can be resumed later by the scheduler

radiant fulcrum
#

sadly python limits the process power so the impl in python isnt any faster, just cool to see work

#

yeah you still need coroutines

#

or actors

#

either or, as either allow you to in some way produce a state machine

paper echo
#

right. so the "work stealing" aspect is a model for distributing cooperative multitasking across multiple OS threads

radiant fulcrum
#

yeah Python has a OS thread handle i think

#

to rust they're actual threads though, python just wont ever see their power lul

paper echo
#

well yeah threading.Thread is literally an OS thread, just blocked by the GIL

radiant fulcrum
#

tactix can use coro's and call them just with coro() when they're wrapped cuz behind the scene it's turning the coro into a actor state machine

paper echo
#

hm

#

ill read your code, this seems very interesting

radiant fulcrum
#

pretty fun to mess with, although deadlocks were, a pain to say the least

paper echo
#

fwiw im trying to move away from python for day to day hobby programming

#

there are so many more interesting and (imo) comfortable ways to write software that python doesn't really let you do

#

its become a pretty nice "day job" language, still moreso than javascript imo

#

but for my own personal stuff i feel like id rather use a naturally actor-based language rather than hacking it into python, if only while im still getting used to using this stuff to solve real problems

radiant fulcrum
#

yeah

#

actors in python are pretty pointless

#

atleast imo

#

What makes them so powerful in things like Rust is the fact that they're lockless and then can be used across threads

#

which basically maximises program power

#

but for python there isnt much point because its still going to have the same processing power regardless and a standard event loop for python is alot cheaper to run

paper echo
#

right

#

thats part of whats cool about it. i want to think less about telling the computer what to do

#

and more about describing what i want it to do, and letting the computer figure out how to do it

#

at least at a low level in terms of how to best make use of threads, processes, etc

radiant fulcrum
#

yeah

#

why i love about tokio in rust, or the rust async eco system in general

#

i dont like actix so much though as it more encourages, many seperate processes all running single threaded

#

like you would have Python do

deft pagoda
#

ever tried ponylang?

#

that's a fun actor-based one with python-like syntax --- except for all the

    end
  end
end
paper echo
#

Ive read about it

deft pagoda
#

the typing is mind-warping though

#

cause refcaps

paper echo
#

What's that

deft pagoda
#

reference capabilities, they let you know who can read/write an alias

grave jolt
#

like &T and &mut T in Rust?

deft pagoda
#

i'm not sure, maybe; but i think pony refcaps are also unique in some way

harsh thistle
#

anyone can help me here?

deft pagoda
harsh thistle
#

i cant able to install dotenv module so anyone can help me

deft pagoda
#

this isn't a help channel, but a discussion channel

harsh thistle
#

can u join my DM

deft pagoda
#

you can get help here: #❓｜how-to-get-help , which explains how to open your own help channel; if your question isn't answered you can always open another one

harsh thistle
#

ok

deft pagoda
#

helpers typically don't help in DMs, both because we'd like to keep questions/answers public so that many people can be helped by or help with and also because this is a very populated server and we can get overloaded with dms

grave jolt
#

Can someone share a more or less accessible and well-known example of inheritance in Python's standard library? Besides:

  • pathlib
  • logging
  • exceptions
harsh thistle
deft pagoda
#

uhh, the collections.abc module?

#

feels like a pretty idiomatic use of inheritance

grave jolt
#

hm... well, I was thinking of something more easy to grasp or well-known for beginners

deft pagoda
#

there are classes in the stdlib i can think of that are meant to be subclassed too

grave jolt
#

Maybe I could just make up an example, but I hate tutorials that use animals/shapes

#

Animals -- because it's useless, and shapes -- because it causes the circle-ellipse problem

deft pagoda
#

maybe use gui mixins / behaviors examples?

grave jolt
#

Maybe

            RWFile
              |
              |
  +-----------+------------+
  |                        |
  V                        V
ReadableFile        WriteableFile
  |                        |
  +-----------+------------+
              |
              V
            File
#

but that's multiple inheritance, I'd rather pick single...

raven ridge
#

collections.OrderedDict or open() are my two go-to examples depending on what I'm trying to illustrate

grave jolt
#

I really need an example of a subtyping, not necessarily inheritance, but I don't think many beginners are familiar with type hints, especially with stuff like Union or Optional

#

Maybe I could explain subtyping without inheritance, though?

#

Like, list, tuple and dict all have a length

#

but then, I'd need to explain how types in dynamically typed languages are a bit broader than the literal concrete type

deft pagoda
#

i dunno, i think the collections.abc stuff is really simple for the most part

grave jolt
#

yeah, but it's not immediately obvious that list is an instance of Collection, yet it doesn't inherit from it

#

or something like that

deft pagoda
# 

class Sized(metaclass=ABCMeta):
    @abstractmethod
    def __len__(self):
        return 0

class Container(metaclass=ABCMeta):

    @abstractmethod
    def __contains__(self, x):
        return False

class Collection(Sized, Iterable, Container):

can simplify it to this

grave jolt
#

yes, it's probably a good example, but it has foreign stuff like

  • abstractmethod
  • metaclass
  • classes to which objects belong without being instances of
deft pagoda
#

i think you can use it to make a more boiled down simple example though

grave jolt
#

yeah, maybe I can just make up something

#

it doesn't even need to be language-specific 🤔
but something more or less real-world

raven ridge
#

I think the different types of file-likes in io (which open can return) are good examples of this.

glad shoal
# grave jolt I really need an example of a _subtyping_, not necessarily _inheritance_, but I ...

In the object-oriented framework, inheritance is usually presented as a feature that goes hand in hand with subtyping when one organizes abstract datatypes in a hierarchy of classes. However, the two are orthogonal ideas.

Subtyping refers to compatibility of interfaces. A type B is a subtype of A if every function that can be invoked on an object of type A can also be invoked on an object of type B.

Inheritance refers to reuse of implementations. A type B inherits from another type A if some functions for B are written in terms of functions of A.

However, subtyping and inheritance need not go hand in hand. Consider the data structure deque, a double-ended queue. A deque supports insertion and deletion at both ends, so it has four functions insert-front, delete-front, insert-rear and delete-rear. If we use just insert-rear and delete-front we get a normal queue. On the other hand, if we use just insert-front and delete-front, we get a stack. In other words, we can implement queues and stacks in terms of deques, so as datatypes, Stack and Queue inherit from Deque. On the other hand, neither Stack nor Queue are subtypes of Deque since they do not support all the functions provided by Deque. In fact, in this case, Deque is a subtype of both Stack and Queue!
https://stackoverflow.com/a/36266550/1305461

raven ridge
grave jolt
#

Alright, I think I'll pick some silly toy example, but not as silly as with animals

#

something like Building <- Library

grave jolt
grave jolt
raven ridge
#

why?

#

they happen to be ABCs, but you don't need to mention that - the docs barely do.

#

and the docs only mention it in the context of adding your own number types.

grave jolt
#

Oh, right, Decimal and Fraction actually inherit from Rational

#

oh wait

#

Fraction inherits from Rational, but Decimal doesn't. ?????

raven ridge
#

Decimal doesn't have numerator and denominator, and Rational requires those

grave jolt
#

right...

raven ridge
#

though there's no reason why it couldn't have numerator and denominator... Perhaps it ought to...

grave jolt
#

@raven ridge It has as_integer_ratio, but no numerator and denominator

raven ridge
#

right.

grave jolt
#

I think I found a real example in urwid

  Widget
    ^
    |
BoxWidget
    ^
    |
SolidFill

But, damn it, BoxWidget is deprecated... has been for a really long time; instead, a different way, without inheritance, is used

#

Other examples from urwid I could find probably would'be been better off with composition?.. like how Edit inherits from Box

#

Maybe I should look in Tkinter

#

I haven't worked with it very much, but many beginners did, I suppose, so it'd be immediately accessible

raven ridge
#

how about dict vs collections.Counter?

#

maybe I'm still not really grasping what specific subtyping aspect you're trying to capture

grave jolt
#

maybe I should've mentioned, yeah

#

I'm trying to create a resource explaining variance

#

Maybe I don't actually need inheritance. Maybe I could use lists and tuples, for instance. Or some other collection.

#

Yeah, lists are a great example of invariance.

raven ridge
#

and you want examples for covariance and contravariance?

grave jolt
#

Maybe I won't use formal notation, I'll just explain in words

grave jolt
raven ridge
grave jolt
#

It's a good explanation, but again, it uses some typing stuff and type annotations.

#

I think I'll go with a toy example for covariance

raven ridge
#

yeah - its examples for contravariance and invariance are more independent

#

I think it doesn't bother to explain covariance in as much detail since that's just a normal subtyping behavior

grave jolt
#

For some reason I really struggle with the idea of variance when I first encountered it... a few months ago?

#

now that I look back, it's pretty basic

raven ridge
#

I have to double check the terms every time someone uses them, heh - but the concept isn't so tough.

grave jolt
#

don't know if you checked the staff channel (which is a mess rn), but I also found this fun correspondence

📎 unknown.png
#

it lives in the same world as other type-proposition examples, I guess

raven ridge
#

so, covariance - what's wrong with dict <- OrderedDict? No special typing markup, no ABCs. An OrderedDict can be used everywhere where a dict is expected, and in fact is a subclass of dict, but has some extra functionality as well, so passing a regular dict where an OrderedDict is expected won't work

grave jolt
#

There's an example of bool being a subclass of int, but I would never subject learners to that unfortunate relation 😅

#

and also a trivial example of <any type, e.g. int> and object, but it's too trivial

raven ridge
#

right - OrderedDict is better than that, it both inherits some meaningful behavior from the base class, and overrides some methods, and adds some new methods

grave jolt
#

Yeah, I think that's a good example

#

I suppose I'll use that, and I'll use some toy example which might be less technical, like Building <- Library

raven ridge
#

datetime.date <- datetime.datetime?

#

ooh!

#

what about: tuple <- collections.NamedTuple, @grave jolt

grave jolt
#

yeah, I thought about that too 🤔

#

that's a nice example

#

or maybe even -- a 2-sized tuple and any tuple of size 2 and bigger

#

nah, namedtuple is good

glad shoal
grave jolt
#

It's a subtype of Mapping, but not a subclass

raven ridge
#

right. And dict is a subtype of Mapping as well

glad shoal
#

Ah, I see it here. Just went into the source as well

#

Good to know, thanks

grave jolt
glad shoal
#

Have you guys done research on the cpython implementation? Been meaning to watch a series of lectures on the topic, but haven't gotten around to it

#

Specifically the C implementations- core stuff.

raven ridge
glad shoal
#

Just realized you guys on a different topic- didn't mean to interrupt. 🙂

grave jolt
#

Yeah, I guess I could use that in the types-aren't-just-what-type()-says

grave jolt
#

And, well, I think I found my examples

glad shoal
#

you don't know how often I get that in friendly (real life friends) discord chats ^^

grave jolt
glad shoal
grave jolt
#

Yeah, I've read it. I've just had a bit of that here, no denial

glad shoal
#

Not enough context on my end to comment- but if you say so, I'll take your word for it

raven ridge
#

I write a decent amount of code that interfaces with the CPython C API, where the source can often explain things that the docs don't...

glad shoal
#

I've been stuck in a problem for the last week that I have yet to come to figure out if its even possible in Python due to the cpython limitations

glad shoal
raven ridge
#

what CPython limitation are you hitting?

glad shoal
#

Specifically telling if a function was awaited. I mocked a solution with inspect, but it's far from satisfactory.

from toolbox.experimental.asyncdispatch import asyncdispatch
import asyncio

@asyncdispatch
def func():
    return "sync"

@func.register
async def _():
    return "async"

async def main():
    print(func())          # >>> sync
    print(await func())    # >>> async

asyncio.run(main())
grave jolt
#

Well, it's not really about CPython, is it?

glad shoal
#

Been looking into collections.abc.Awaitable, but haven't deep-dived.

grave jolt
#

an async def-ed function is just a function that returns a coroutine

#

What about

x = func()
await x
```? It should be exactly the same as
```py
await func()
glad shoal
#

I suppose so- but I was hoping cpython source might give me a hint to implement a more generalized decorator that might work in different Pythons

grave jolt
#

Why would you want that? You can just make two different functions

glad shoal
grave jolt
#

I meant func being an async def-ed function, i.e. a coroutine function