internals-and-peps

[unkempt rock]

but thats it

bob()

^ that constucts it?

[acoustic crater]

but u are missing hte bob argument

'so it'd be an error

foo(bob) constructs foo

[unkempt rock]

Ok

what about

def foo()
pass

No foo further

Is foo constructed?

i say no?

[acoustic crater]

that constructs an instance of the function class

and defining a class constructs an instance of a metaclass, usually type

[unkempt rock]

o0, so just by coding it, not doing anything with it, you're creating a construct

[acoustic crater]

everything in python acts like an instantiated object

[unkempt rock]

^

[acoustic crater]

so even defnitions themselfes are objects

which sets it apart from a lot of langs

[unkempt rock]

Its why I always refer to python stuff as parent/child/sibling. I don't know enough of the proper verbology to talk on this level

Yep, eveything is an obj

[acoustic crater]

I didn't really get it viscerally till I did a project that involved digging around in the C

of CPython

[unkempt rock]

Even now, coding python since 2013.... I have yet to implement super because I just haven't found a good gosh darn reason why

I'd love to have a reason to learn C.....

Time is the killer for me.

[acoustic crater]

that means you're solving tasks that don't need it then

probably

[unkempt rock]

^

But super just gobbles up the parent class

never saw a need to do so, etc.

[acoustic crater]

nah

[unkempt rock]

f containing a.b seems needless

[acoustic crater]

super figures out what method to inherit by goign up the method resolution order

[unkempt rock]

f.b

vs

^ I should attend whatever class taught you what you're speaking

Sounds like itd be worth the time

i fscking love python, can do it fluidly, but the minutae... kills me.

[acoustic crater]

google fu haha

[raven ridge]

super() can delegate to a sibling class. Not just a parent class.

[unkempt rock]

^ But I've never seen a reason to do so

Sure I can, but why?

[acoustic crater]

I just had to experiment a lot to understand this stuff

[unkempt rock]

^ same, but I'm practical

Why bother...

[raven ridge]

mixins are the usual example of why.

[unkempt rock]

Hrm>

[acoustic crater]

and why not just use super() if it's gonna work, it means there's less to type if you change stuff at least

but a lot of problems don't even require classes

[unkempt rock]

I suppose its the way I self-taught classes to myself

I won't say the word I use to describe it

But I love to pass around objects in Python

classes to other classes, etc.

[acoustic crater]

that's composition

rather than inheritance

[unkempt rock]

Coolest thing ever I learned was I could pass self from a given class, to an instantiation of another class.

self.bob = foo(self)

etc.

[raven ridge]

I rarely use multiple inheritance, and if I do it's almost always for mixin classes. But if you want to use mixins, you need to understand super and the MRO.

[acoustic crater]

lol that's inheritance but wonky

_>

sorta

[unkempt rock]

Me spoony?

[raven ridge]

And mixins are a nice way to factor some code to enable reuse.

[acoustic crater]

ya

[unkempt rock]

I know its goofy how I do it 🙂

But fsck... It works.

[acoustic crater]

that's directly coupling two classes in a way that they contain eachother

[unkempt rock]

and thats always been my model. Inspect and Test, that which you Expect to a reasonable degreee.

[acoustic crater]

if nobody else has to read your code ever, do whatever you want haha

[peak spoke]

you will have to read your code

[unkempt rock]

It could be why me ole github has little traffic 🙂 My methodologies are self-taught

But in self-teaching, I came across super, read up on it, and just kinda shake my head as to why bother

self it out and go

[acoustic crater]

someone who helped me learn a lot about this stuff is a guy who recently started making youtube videos

[unkempt rock]

Never had a concrete hardcore explanation/reason presented that said to me in Edison terms, Yes do it

[acoustic crater]

https://youtube.com/playlist?list=PLC0nd42SBTaNuP4iB4L6SJlMaHE71FG6N

he does a good job at showing what inheritance, coupling, cohesion, and other oop principles are in python

[unkempt rock]

I bookmarked it, too lat/early in the day to comprehend lowlevel stuff, but thank you for the link

[acoustic crater]

np

[unkempt rock]

So it seems like this out of all the avail chans is the sane one to be in

[acoustic crater]

lately I watch a lot of talks from python conferences

[unkempt rock]

The others are like help vampire channels

How do I echo?

etc.

[acoustic crater]

as long as you're talking about how python works

[unkempt rock]

and its funny.... I've been looking for a discord svr to talk on at 5am est, etc..

This one is like a shotgun to the face

right-click, "read all", 2 secs later, 10 chans are populated...

Had to slow my roll

[acoustic crater]

lol yeah just keep it to how python works so it stays that way

[unkempt rock]

Fair nuf... I just hope you're ready for a n00b like me 🙂

Cause I got mad questions I've always wanted answered, but rarely am I around other people who code Python

So whats your opinion on slots?

I implemented it for some code recently, a backport as it were

It certainly feels spunkier

[acoustic crater]

use them last if they will optimize your code

[unkempt rock]

But it could be a placebo too.

They only optimize it because you told me too drill sgt

[acoustic crater]

no reason to add them while still building your project

[unkempt rock]

Did it really though? I'm too lazy to unit test, etc.

I backported

and the backport blew me away...

it certainly and really really seems like it made an impact.

I can now seemingly keep up with C level code for TCP injection

and if you don't play with scapy you wouldn't know what I mean as a baseline

I had a bottleneck somewhere in the code, probably scapy itself. But for giggles after I read the pep or whatever, i decided to implement.

Now it keeps up again in 2021.

[acoustic crater]

lol mayb cuz of some weirdness introduced cuz you instantiate classes with self as initialized attributes XD

[unkempt rock]

Granted, it never kept up prior to, but it far and fast exceeds now based upon prior experiences with speed testing for the injection

Naw, it's scapy Spoony

If you don't know the lib, please do learn it.

It's a blessing of a socket wrapper

[acoustic crater]

I will either get into networking stuff or async next

[unkempt rock]

As an example

[acoustic crater]

idk

probably both as I find use cases

[unkempt rock]

from scapy.all import *
p = sniff(iface = 'wlan0mon', prn = lambda x: x.summary, filter = 'tcp port 4200', count = 1)
print(p)

[acoustic crater]

well actually I just learned unit testing nad my next thing to learn is github and PyPI

[unkempt rock]

I just did in three lines what takes quite a bit in raw sockets

[acoustic crater]

but yeah that's what python is for

[unkempt rock]

^^^ this ^^^

Even though we got all the Python haters out there.

[acoustic crater]

https://www.youtube.com/watch?v=wf-BqAjZb8M

that's basically what this talk is about

[unkempt rock]

Pep 8... haha

[acoustic crater]

hettinger is my new hero, what a legend

[unkempt rock]

Anyways, I should really be asleep. Glad I found this channel and I'm sure I'll chirp back up at some pt.

[acoustic crater]

lol nn

[unkempt rock]

^ cheers

[visual shadow]

One tip that can come in handy for questions like these... Often we're using python as if we're the consumers, the users of Python to achieve some means. Try to think of what would happen if you were the one making a library for other python users to use as they see fit. Usually you'll see all these questions that start with "but I'll never* need this" answered if you start thinking about having to write code or tools for others. (this is ofcourse a generalization but I hope it's a useful one anyways). So yeah, super is super useful if you want to let someone else inherit from and modify a class you make without breaking. Same for many other specific portions of the language.

So, tl;Dr yes, you don't need to use all features of a language while doing some task, because those features cater to specific needs.

[acoustic crater]

but I think self might be objectively the last parameter you should be using to instantiate another class as an instance attribute in __init__ haha

[paper echo]

it's also worth considering that python is meant to be "general purpose" and it has grown to be very powerful. it's not a minimalist language with a tidy set of primitives like lua or scheme. the memory model is pretty elegant (IMO), but otherwise it's a sprawling collection of hooks and overrides and customizations. you would probably never design a language like this today, but python as it exists today was not really "designed" as much as it "evolved". individual features were designed, yes, but it's not like when python 1 was written guido was already thinking about the semantics of __getattribute__ versus __getattr__

[ashen monolith]

Languages are either continually evolving or dead; I'd say there is no language worth using that has been designed from the ground up and never changed thereafter.

[static bluff]

This is true, but python takes it to a whole new level. It is in my opinion the spiritual successor to C in many way ways- a language that was designed from the ground up as an interface with your computer and every one of its capabilities.

To salt's point about no one designing a language like this today, I disagree. If I were to ask myself to build a general purpose language with easy syntax and an object centric philosophy, itd look a lot like python. The only reason you wouldnt build a language like Python today is that python already exists and you'd need the same 30 years python took took to get to where it is to build one as effective as python.

If and when it's time to abstract yet another level above the python of today, I feel it would either be built from python or else heavily inspired by it

I'm something of a mad scientist, so I've spent the better part of the last few years trying to hack python (finally starting to make some headway). My only criticism is that I wish it were a tiny bit less rigid, which is to say, I wish more of the internals were exposed. That's just me.

Also, its import system, while highly ingenious in some ways, has a few silly bits to it

@unkempt rock I'm just reading through your posts. Your coding philosophy is a lot like mine

[gleaming rover]

it would look like it from a user's point of view

...but would the internals be the same?

[static bluff]

I'd imagine for the most part. Python's major claims to fame are (in my opinion); easy syntax, an advanced mro algorithm, sophisticated attribute lookup algorithms, metaclassing, an objective nature, dunder methods corresponding to top level syntax, and iterators

Now, an mro is an an mro and while the algorithm itself might turn out different the result would be largely the same. Once you have an mro that facilitates attribute lookup on a linearized resolution order, the same sophisticated lookup system would probably just fall into place

An objective nature is, if its your goal, simply what you'd hold in your mind as you were designing and all of the rest of your languages implementation would revolve around that

Top level syntax methods and iterators were strokes of genius in my mind. I might never have thought of it. But if you knew its what you wanted then again, the implementation might differ somewhat but the result would be the same. A six of one, half dozen of the other type situation

The only real place in all of python that I dont think would be self evident when trying to build a general purpose abstracted language is metaclasses. I could see that turning out majorly different depending on, you know, lots of stuff

[static bluff]

Lexing is fascinating

I'm a bit afraid of the parser, when I finally get to it

But this is such an intricate process- and people have come up with so many interesting little tricks

[unkempt rock]

@acoustic crater So I did a grep to try and find some examples where I do the self passing, here you go
https://github.com/stryngs/piCopilot/blob/master/SRC/DEB_picopilot-idrop/opt/piCopilot-idrop/lib/os_control.py#L35

[fallen slate]

SRC/DEB_picopilot-idrop/opt/piCopilot-idrop/lib/os_control.py line 35

Hopper(self)```

[unkempt rock]

With respect to the discussion about super from earlier this morning, I'd love to know a better way to kick off Hopper, with Hopper knowing about the internals of Control. I ask not for "help", but the curiosity of maybe this bridges the gap in my lack of "need for super()" calls and why perhaps I should start? using super().

I have more examples if that isn't clear enough what I'm doing.

Hey @unkempt rock, this channel is meant for the discussion about Python itself - you're probably looking for a help channel (#❓｜how-to-get-help) or #software-architecture

No Qwerty, I'm speaking to this mornings discussion about Python itself =), scroll back

My code works just fine, but I'm very very very different? So Spoony Bard and I were discussing that. Rather than leave it at where it was, I figured I'd dive a bit into it

My question revolves around super() vs the crazy way I implement my own super()

and I very much respect the purpose of the channel -- Quite fascinated I hadn't found this earlier. I'm always looking for conversations around the guts of Python, well... Ask and ye shall receive it seems.

I even had to re-read instance method vs class method based on Spoony's thoughts about 3.10 enhancements. Probably have to re-read it again just to memorize it.

There, I rephrased what I was saying to make it more apparent for the channel. Sorry if I wasn't clear on what I was asking @unkempt rock

[raven ridge]

That code seems very strange to me. Why make Hopper a class at all, instead of just a function, or better yet an instance method of the Control class?

[unkempt rock]

I tend to like Classes a bit much as it were.....

But it seems like a great, hey dummy super() is for this example, thus why I dropped it as a q

[raven ridge]

Right, but there's a problem with your premise. We can't tell you why you should use super() here, because you shouldn't, because Hopper shouldn't be a class at all.

[unkempt rock]

Hmm. Okay fair enough

I'll look around a bit later and see if I can find a Class with my methodologies that is a bit larger in scope than the 8 or so lines Hopper is. very curious about super as it were. I see it enough on github that it makes me think I should use it, sure does make following code hard when something is popular but you yourself haven't used it you know?

[winged bane]

hi guys

i.m doing a mini project useing python script i hjave a doub how can send the output pdf file to whatsapp useing python script

[visual shadow]

This is not a help channel. Please see the channel topic and refer to #❓｜how-to-get-help

[winged bane]

Hy guys.. I'm doing a mini project on python I have a doubt who can I send pdf file to whatsApp useing python script

[acoustic crater]

all super does is figure out where to inherit a method from and call the method as if the class is instantiated, it's exactly the same as using the class the method is inherited from itself and passing in self it's just dynamic instead of hard-coded

[tough mist]

On the topic of classes; this is really not "advanced" but I'd like to hear what the current opinion is on the role of [what a python programmer implements as] constants and if there are any standards or practices in the choice of their data-type/data-structure.

[acoustic crater]

https://www.python.org/dev/peps/pep-0008/#constants

[raven ridge]

it's exactly the same as using the class the method is inherited from itself and passing in self it's just dynamic instead of hard-coded
It needs to be dynamic, because you can't necessarily know what class it's inherited from to hardcode it.

[acoustic crater]

yeah in many cases

so it's best pratice to use super() if only because it is one less line of code to change if you change the inheritance

but also allows for dynamically building classes and for defining methods outside of classes

[raven ridge]

!e ```py
class A:
def init(self):
print("Initializing A")
super().init()

class B:
def init(self):
print("Initializing B")
super().init()

class AB(A, B):
pass

AB()

[fallen slate]

@raven ridge :white_check_mark: Your eval job has completed with return code 0.

001 | Initializing A
002 | Initializing B

[raven ridge]

You need super() if you want multiple inheritance to work reasonably.

[acoustic crater]

!e

class A:
    def __init__(self):
        print("Initializing A")
        B.__init__(self)

class B:
    def __init__(self):
        print("Initializing B")

class AB(A, B):
    pass

AB()

[fallen slate]

@acoustic crater :white_check_mark: Your eval job has completed with return code 0.

001 | Initializing A
002 | Initializing B

[acoustic crater]

if you know how mro works you can technically hardcode it but yes it won't adapt to changes

[tough mist]

Well, I do like the Eighth Pep but I am surprised the whole thing can be summed up in one sentence.

[raven ridge]

nah, now you can't make an A instance anymore.

[acoustic crater]

I've tried to find more guidance before and really it's up to you beyond the words of PEP8 is what I've found

[raven ridge]

you've made it so that A() calls B.__init__, but it shouldn't, because B isn't in the MRO of A.

[acoustic crater]

yeah so for use cases other than this hard coded one it doesn't have the same behavior

would need to define __init__ in AB for that behavior without super

[raven ridge]

right.

[raven ridge]

constants are usually just implemented as global variables, named in UPPER_SNAKE_CASE, and assigned at the top of the module.

[raven ridge]

FWIW, this doesn't work in cases where there is a common base class to multiple of your base classes. If you have:

class Base:
    def __init__(self):
        print("Base initialized")

class A(Base): pass
class B(Base): pass

class AB(A, B):
    def __init__(self):
        A.__init__(self)
        B.__init__(self)

AB()

then "Base initialized" will be printed twice.

whereas if you used super, it would use the linearized MRO, and only initialize it once.

[acoustic crater]

good points yeah

That inheritance structure makes me anxious haha

[feral cedar]

(diamond)

[tough mist]

puns

[civic onyx]

wrong channel, sorry.

[boreal umbra]

in my programming languages class, the instructor tried to pose multiple inheritance as introducing a dilemma where you can get unpredictable behavior. I'm not really sure why.

[raven ridge]

hm. multiple inheritance prevents local reasoning. When you use super() in a class, you don't know what other class you might be delegating too

[boreal umbra]

true...............

[raven ridge]

there's nothing unpredictable if you know the MRO, but you can't know the MRO by looking at any single class and its base classes, only the final/most derived one

[boreal umbra]

that's what I wrote for the questions about it, and I got credit 🤷🏻‍♂️

[raven ridge]

that's the charitable interpretation of what he meant, at least. 🙂

[boreal umbra]

the three languages we looked at in the course were lisp, python, and java. The slant they took with lisp was "pure functions are cool" and the slant they took with python was "it's bad but a lot of people like it and the PyPI is huge". The python assignment was deliberately made to be too difficult, but came with the caveat that you could use any PyPI package.

[raven ridge]

heh. What was the slant for Java?

[boreal umbra]

"you already know this because that's the language we made you use in every non-data science class up to this point. but look, threads!"

[raven ridge]

"a lot of people like it" is describing network effects, which is a genuine pro for a language - it's easier to learn and use when other people use it and talk about it. PyPI having a huge ecosystem is also a legitimate pro.

[rich cradle]

Did they give a reason why they thought Python was bad?

[grave jolt]

duh, it's not lisp

[boreal umbra]

for some reason they really hate the python docs (and so does that person on python-ideas), and they don't like dynamic typing.

at least, not in the context of CS education. But then they also said that they don't care if non-CS students don't learn about types. In my opinion, everyone who writes any code should understand types. If someone asks you what your code is doing and you can't articulate what your variables are, you can't get help.

[raven ridge]

I don't get the Python docs hate. I think it's quite well documented, other than that it tends to be loosey goosey about types

[rich cradle]

Dynamic typing can be great for a lot of things, and the Python docs are pretty good. So idk why that is a bad thing.

[boreal umbra]

I like dynamic typing. I also like type annotations.

except typing.Literal

[swift imp]

I don't understand why literals aren't just interpreted

Like why do I have to wrap it in Literal

[boreal umbra]

they organized their thoughts on Python into "the good", "the neutral", "the bad", and "the ugly", and for some reason "the docs are bad" was "the ugly". Of all the criticisms of Python to be the worst possible thing one could say about it, I feel like the docs shouldn't be it.

[swift imp]

I think the look of the docs are ugly but the content is solid

[raven ridge]

yeah, that's a weird choice for "the ugly" - it's got both a good tutorial and good reference documentation, split between the language itself than the standard library.

[boreal umbra]

by "the ugly" they meant "the worst thing about Python".

[rich cradle]

Have they heard of type annotations?

[boreal umbra]

those were beyond the scope of the class. they probably know about them, though

[raven ridge]

I'm not sure what you mean here - the idea behind Literal is that it lets you say that this function takes either "text" or "binary" as arguments, but not other arbitrary strings. Are you suggesting that you'd like to see that be annotated as "text"|"binary" rather than Literal["text"]|Literal["binary"] ?

[swift imp]

Yes

[raven ridge]

ah. Well, that can't be done because that syntax already means something else.

at least, not without deprecation and PEPs and arguing and a backwards incompatible change period, etc...

granted, the syntax already means something that it's no longer required for - now that we have from __future__ import annotations by default. So maybe we could eventually recycle that syntax for something else.

[boreal umbra]

On an unrelated note, does the stdlib have a way to create data types with a fixed number of attributes, each instance using a fixed amount of memory to store those references, but the references can be changed?

so like a namedtuple, but you can switch out the references.

[halcyon trail]

dataclass with slots?

[boreal umbra]

seems like an inelegant way to get that behavior

[spark magnet]

@boreal umbra __slots__ sounds like exactly what you are asking for

[sour dragon]

Does anybody have any good resources regarding Pythonic / idiomatic ways to consume APIs that don't already have good Python clients?

[boreal umbra]

but I want to be able to do

class Thing(typing.mutablenamedtuple):
    a: int
    b: int

and not have to write __slots__

also I don't want it to be named mutablenamedtuple, since that makes no sense.

[raven ridge]

the only way to get the inability to add new attributes to an object defined in Python code is to use __slots__

[boreal umbra]

so one would have to make a class decorator to apply __slots__ implicitly?

[halcyon trail]

https://stackoverflow.com/questions/50180735/how-can-dataclasses-be-made-to-work-better-with-slots

one of the answers there gives a library which can do this

actually

attrs supports this

so just use attrs

[spark magnet]

why do you want to prevent adding new attributes, and why "fixed memory" whatever that means.

[halcyon trail]

strictly speaking I actually recommend using attrs over @dataclass anyway

I started with @dataclass and kind of regret it

not enough to make the effort to switch

but I do

[boreal umbra]

if one anticipates needing to create tons of instances

[sour dragon]

What do you mean?

[halcyon trail]

attrs

the library that dataclass is based on

[raven ridge]

I don't think you can define __slots__ on an existing class - you could set it from a metaclass, but a decorator couldn't change it for an existing class, only recreate the class with __slots__ added - I think.

[halcyon trail]

a decorator can set the metaclass though

a decorator is recreating the class anyway

[boreal umbra]

don't normal class instances each have their own dict, and don't dicts take more memory than tuples?

[raven ridge]

It can, but it doesn't have to.

[spark magnet]

they do

[sour dragon]

Where are you running into performance issues?

[raven ridge]

but it's still fixed.

[halcyon trail]

Well, depends how exactly you look at it, but sure, ok

[boreal umbra]

I'm not, as this is hypothetical.

[raven ridge]

the overhead for a dict is higher than for a tuple, but it's still a fixed amount of overhead for a given number of elements.

[halcyon trail]

but the point is that adecorator can achieve this

so it's not really accurate to say that a decorator can't change it for an existing class, except in some very narrow technical sense

[sour dragon]

Hypothetical Python memory management will drive you insane

the way that the GC works is not even in the slightest bit intuitive

[halcyon trail]

agree

[raven ridge]

I don't believe __slots__ can be changed for a class once it has been created.

[spark magnet]

Can you say more about that?

[sour dragon]

https://docs.python.org/3/library/gc.html

It's not Rust or C, the rules are probably too complicated to keep in your head as you program

[spark magnet]

i know what it is, thanks. I meant about your difficulties with it.

[boreal umbra]

I worked on a project where an earlier contributor put a bunch of things in tuples, and I switched it to named tuples to make the code more communicative. Fortunately, the tuple elements never needed to change.

[sour dragon]

Maybe I'm just not intelligent enough to do so

[halcyon trail]

@raven ridge attrs' decorator takes a slots=True argument

so I think it does, unless again you are talking in some narrow sense

[sour dragon]

but remembering the edge cases of the Python GC is too much for me

[raven ridge]

right, but attrs' decorator returns a different class than the one it's called on - right?

[spark magnet]

why do you need to know the edge cases?

[halcyon trail]

yes, that's the narrow technical sense

[sour dragon]

I don't, I just don't use Python if I'm getting to the point where thinking about such things seems like a good idea

[raven ridge]

ok. I don't think that's a narrow detail, but sure - I agree a decorator can make a different class with the same name and attributes that does have __slots__

in fact, that's what I originally said, heh

[halcyon trail]

decorators are basically always making different functions/classes

anyhow, we agree in the substance

[spark magnet]

Perhaps its all premature worrying

[sour dragon]

the most I do in terms of Python memory management is not keep data too big to keep in memory in memory needlessly (ie. list comprehensions for massive datasets in lieu of a generator expression when a generator expression can work) and maybe tweak some gc.set_threshold() values around

Yeah, I agree. I just don't concern myself with it unless it's a demonstrable problem

[spark magnet]

good

[raven ridge]

there are tons of decorators that don't - like flask's @app.route(), for instance. And class decorators are more likely to modify the existing class than function decorators are to modify the existing function, since there's more interesting stuff that you can do with an existing class than an existing function.

[sour dragon]

idiomatic code in Python is much prefferred over more performant code and often times there is no conflict

iterating directly over an iterable is usually more efficient than iterating over range(len) for example, which I didn't find very intuitive

[halcyon trail]

a huge fraction do though and in most cases people just talk about how the decorator "affects" the function/class, they aren't strict int alking about the new class/function and the old, because they never see the old in the typical case

at any rate, agree to disagree. the important point is that decorators can solve the original problem

[raven ridge]

possibly. As long as you don't need the original class. 🙂

[spark magnet]

I prefer iterating over the collection just because it's more direct code. I'm not sure where there's an efficiency difference.

[halcyon trail]

99% of decorator use you never see the original class

if you needed the original class, you wouldn't use the decorator syntax

as to the memory issues though, having gone through cases where a python program was actually using a large fraction of the available memory, and trying to do things to lower it back down by simply "avoiding copies" or things like that

and it was basically hopeless

If you ever reach that point in python, AFAICS, you need to change the approach to drastically bring down memory usage. You can't count on the GC to behave sanely no matter what you do, you can try calling del and calling functions to force garbage collection, it's very hard to get the memory back to the OS when you'd want it (or even, still with the python interpreter, but ensuring it gets used in the next allocation instead of more memory being asked for)

[raven ridge]

if you're running out of memory, either your working set is too large (meaning you need to make your objects smaller, or keep fewer objects in memory at a time), or you're keeping objects longer than they're needed (perhaps because of reference cycles, meaning that the objects hang around until the cycle collecting GC picks them up)

depending on which of those two situations you're in, the solution may be different.

[halcyon trail]

there were lots of examples in my code of things with no reference cycles not getting collected

[spark magnet]

Did you find out why that happened? it sounds not-possible.

[halcyon trail]

i'mnot saying it was never collected, it just wasn't collected fast enough to get memory usage to an acceptable level

You could assign something to one variable, and then del the variable, and then do gc.collect

etc

and it would not work

[spark magnet]

I see

[raven ridge]

that would mean that there were reference cycles. If there weren't any reference cycles involved, then the object would be destroyed as soon as the reassignment happens

at least, assuming we're talking about CPython.

[halcyon trail]

are you saying this based on theory or have youa ctually debugged this problem in practice?

[raven ridge]

gc.collect() only collects objects that are trapped in reference cycles. That's its purpose.

https://docs.python.org/3/library/gc.html
Since the collector supplements the reference counting already used in Python, you can disable the collector if you are sure your program does not create reference cycles.

[halcyon trail]

there's no way for me to know that the internals of some object I'm dealing with don' thave internal reference cycles, so even if for example I delete the last reference to a dataframe, I don't really know for sure it will get collected immediately. which is why I used gc.collect as well.

[raven ridge]

or, rather: gc.collect() runs the cycle-collecting garbage collector immediately, rather than waiting until its next scheduled run. The cycle-collecting garbage collector only collects objects trapped in reference cycles, whose reference counts will never drop to 0.

[raven ridge]

Yep, that's true - though if investigation pointed to internal reference cycles in library objects you're using, there may be ways to break the cycles before you lose your last reference to the object, so that you don't need to depend on the GC.

But that would need to be looked at on a case by case basis, of course.

[halcyon trail]

sure. but in any case this was irrelevant since it didn't collected (immediately) at all. gc.collect cannot hurt.

[spark magnet]

it's easy to believe that dataframes involve cycles

[halcyon trail]

indeed, they are massive beasts

i started with just doing del on my single local varaible, then thought of this, and was also calling gc.collect

the one problem I can think of that's much easier to debug in C++ than python 🙂

[raven ridge]

yeah - there aren't currently any great memory debuggers for Python.

We've been working on one at work, and I'm cautiously optimistic it'll get open sourced... I think everyone who needs to buy in on that plan has bought in

[halcyon trail]

Coming full circle

import attr

>>> @attr.s(slots=True)
... class Coordinates(object):
...     x = attr.ib()
...     y = attr.ib()

actually, sorry:

@attr.s(slots=True, auto_attribs=True)
class Coordinates:
    x: int
    y: int

I've actually been burned by dataclass shortcomings relative to attr before in totally unrelated ways. E.g. dataclass does not have a kw_only option which causes the generated init function to use keyword only arguments

This actually has some implications, without kw_only, because init is being generated directly from annotations in order, all defaulted attributes have to follow all non-defaulted ones

which is a huge headache once you start having inheritance

So I really would recommend attr even if you didn't want slots

[raven ridge]

there was a discussion about that on python-ideas - I believe the consensus was that 3.10 would have a change to dataclasses allowing keyword-only

[halcyon trail]

ah that would be amazing

[raven ridge]

hm, no - https://docs.python.org/3.10/whatsnew/3.10.html#dataclasses

[halcyon trail]

@dataclass
class Base:
    x: int
    y: int = 0

@dataclass
class Derived(Base):
    z: int

this not working was really irritating

[raven ridge]

looks like setting __slots__ for dataclasses made it into 3.10, but the keyword-only arguments didn't make the cut.

well, I think it'll be in 3.11, then.

[halcyon trail]

yeah. at this point though I'd just suggest starting with attrs. It's bound to have something you want at some point

and it has equally good mypy/IDE support

[raven ridge]

the biggest advantage of dataclasses is, and always has been, that it's in the stdlib and so it's one less dependency for your project. Other than that, attrs existed first, and has always been more powerful

[raven ridge]

keyword-only arguments for dataclasses has landed on master - https://github.com/python/cpython/pull/25608

ah, actually, that change is on the 3.10 branch - it did make the cut, just barely.

looks like it just hasn't been added to the changelog yet

but it's on https://github.com/python/cpython/blob/3.10/Doc/library/dataclasses.rst if you ctrl-f for kw_only

[halcyon trail]

ah that's nice, glad that that will sort itself out without my having to do any work

if I could go back it and do it again though I'd still attr. Depends what you'r edoing but most people are going to have dependencies anyway. 0 or 1 dependencies is a big deal, N vs N+1 meh

the other kick in the teeth which applies to both attr and dataclass though is that if you try to define a trivial decorator of your own, that simply calls to attr/dataclass while changing the default

mypy/IDE don't know what to do with it

[sand goblet]

x = []
y = []
x.append(y)
y.append(x)
del x
del y```
Why does this have a problem where it’s not able to decrement the reference counts to 0?
How I would think it would work is this:
When it deletes x, it decrements the reference counts of everything within the list referenced by x, then it decrements the reference count of the x list itself. At that point, x and y each have a reference count of 1.
When it deletes y, it does the same thing with the list referenced by y, so x and y would have a reference count of 0.
Why isn’t that how it works? I don’t get it.
Same thing with this:```py
x = []
x.append(x)
del x```
I don’t get why that kind of thing is a problem. Why wouldn’t deleting x just end up decrementing the reference count by 2 there?

[tawny current]

Can we evaluate here?

Yes

!e ```py
class Undeletable(list):
del=lambda s:print('deletion success')
def init(s,l=()): super().init(l)
def test():
a=Undeletable()
a+=[a]
test()
print(1)

[fallen slate]

@tawny current :white_check_mark: Your eval job has completed with return code 0.

001 | 1
002 | deletion success

[tawny current]

Ah

I see

[raven ridge]

You can evaluate here, but this is a discussion channel - it would be preferable to use #bot-commands

[feral cedar]

that's collected by the cycle counting gc right?

[tawny current]

Im just trying to replicate the latest assertion

So, 1 prints before garbage collection yeets the self referencing object

[sand goblet]

I’m reading this: https://scoutapm.com/blog/python-garbage-collection
And when seeing the examples and thinking about them for a second, I realized I didn’t understand why it’s a problem

[raven ridge]

Why isn’t that how it works?
Because that would make a = b a linear time operation, instead of constant time. It would need to descend recursively through each object accessible via b and increment the reference count of that subobject by 1

[sand goblet]

I thought it wouldn’t since it’s the same object, so it would only increment the reference count of whatever object b was directly referencing

[raven ridge]

if b is a list of a million elements, with your schema a = b or b = None would require changing the reference count of a million and one objects. As Python's actually implemented, both of those require changing the reference count of at most 2 objects (the old object referenced by the name, and the new one)

[sand goblet]

What I’m thinking of is that when it sees a list or something’s reference count reaches 0, it goes through that object and decrements the reference counts of all the objects it contains

But if you add a reference to the list, or you decrement it’s reference but it’s not at 0, it just increments or decrements the reference and doesn’t do anything else

[raven ridge]

it does do that, but neither of the list's reference counts ever reach 0.

working through your example one line at a time: py x = [] y = [] x and y are each lists with a reference count of 1. py x.append(y) Now x has a reference count of 1, y has a reference count of 2 (it's y and also x[0]) py y.append(x) Now x has a reference count of 2 (it's x and also y[0]), so both objects have a reference count of 2. py del x makes the first list's reference count drop from 2 to 1, and py del y makes the second list's reference count drop from 2 to 1, and now neither x nor y exists anymore, but the two lists both do, and they both have a reference count of 1, because each is referred to by the other.

[sand goblet]

Oh yeah. I knew I was overlooking something obvious like that. Now it makes perfect sense

Ok, it’s really clear and obvious now, thanks guys

[raven ridge]

it may be easier to picture with only one object, also.

x = []
x.append(x)
del x
``` is also a reference cycle - the list winds up with a reference count of 1, because it owns a reference to itself.

[sand goblet]

Yeah, and it only goes through the list and decrements the references within it when it has a reference count of 0

But it doesn’t reach that point

[raven ridge]

Exactly

[glossy pike]

Can someone help me solve this

Or this

[grave jolt]

@glossy pike This isn't a help channel, this is a discussion channel. You should see #❓｜how-to-get-help and claim a help channel. You should explain what difficulties you have, and what you've tried so far.
Keep in mind that if it's an ongoing exam, we can't help you (see #rules, rule 5).

[tawny current]

It isn't even advanced python

Smh

[charred wagon]

It might seem advanced for someone of their current skill level, which is why it's so hard to come up with a good name for this channel.

[gritty sequoia]

Gotta love how useful the second explanation is. "The answer is 14." Well, thanks, for that. LOL.

[acoustic crater]

maybe #metapython?

[raven ridge]

That might lead people to thinking the channel is about Python metaprogramming, as opposed to discussing Python.

[grave jolt]

We should make a separate channel for discussing the name of this channel.

Should we name it #meta-advanced-discussion?

Hey, that suggestion is server-meta and channel-meta

[sand goblet]

https://devguide.python.org/garbage_collector/#collecting-the-oldest-generation
What is long_lived_pending / long_lived_total?
Is it this?
gen3_not_yet_checked / current_living_gen3
Or is it something different?
And also what does the 3rd value of gc.set_threshold do if gen3 only gets collected when long_lived_pending / long_lived_total goes over 0.25? Is it that python suggests that maybe gen3 should be collected after every 10 (by default) times gen2 gets collected, but it only actually collects it if also long_lived_pending / long_lived_total is over 0.25?

[swift imp]

That belongs in #community-meta

[grave jolt]

yep, and your message as well

[static bluff]

So, what are you guys' thoughts on multiline lambdas?

[swift imp]

They're unnecessarily hard to read

[grave jolt]

yeah, sometimes they're write-only...

imagine embedding Perl into your language

and I think they wouldn't play nicely with indented syntax

[static bluff]

True, unless (I think) you take the arrow notation approach

Once you wrap them in braces the compiler doesn't care about the indents inside

[grave jolt]

CoffeeScript somehow made it work, though.

[static bluff]

note to self

[grave jolt]

You mean, change indentation to braces?

[static bluff]

No no

God no

[grave jolt]

well, basically, how would you delimit the beginning and the end of a multiline lambda?

[static bluff]

But once you wrap something in some kind of parentheses ( '[', '{', '(' ) python stops caring about the indentation

So

[grave jolt]

right, but if a lambda has multiple lines, it can have nested indentation

and it has to separate the lines somehow

[static bluff]

anon = ( *args, **kwargs ) => { ... }

Different syntax, but it's still the same animal

[grave jolt]

okay, how do you make this

def foo(bar, baz):
    bar.boom()
    if bar or baz:
        for i in bar:
            baz(i, bar)
    bonk(bar)

into a multiline lambda?

[static bluff]

Ohhhhhh I see what you mean

[grave jolt]

also, {} already delimit a set or dict literal

[static bluff]

The parser can make the distinction

[spice pecan]

What if you just want a lambda that returns a dict/set?

[grave jolt]

ah yes, the horrible behaviour of JS
x => { x } is a no-op function returning void
x => ({ x }) is a function returning the { x } object literal

[static bluff]

But I see what you mean about the nested blocks.

[grave jolt]

with indented syntax, multiline lambdas will get even worse than in whitespace-ignorant (like JS), I think

[flat gazelle]

I mean

print(call(lambda bar, baz:
  bar.boom()
  if bar or baz:
      for i in bar:
          baz(i, bar)
    bonk(bar)))
``` is parsable

[grave jolt]

by computer

yes

by human

no

looks like an anonymous class or something

[static bluff]

What about treating everything inside the braces, once recognized as being a block, as an independent token stream and ast?

[flat gazelle]

the issue isn't parsing multiline lambdas, the issue is making a syntax that isn't terrible for humans to read

[static bluff]

Pop it out from the normal flow, parse it, throw errors as needed, reinject

[grave jolt]

you can always just make a standalone function

why does Python need multiline lambdas?

Where would they be useful?

[flat gazelle]

callbacks

[spice pecan]

def callback():
    # lines go here
    pass

[grave jolt]

well, maybe it'd be useful in some UI frameworks...

[flat gazelle]

until you need to add them in a loop and forget about closure rules

[static bluff]

Well for one, writing the function in-place exposes the anon to the scope of the enclosing function. That gets lost if you define the function as, say, a separate method of a class

[grave jolt]

how would multiline lambdas change that?

[static bluff]

And as lakmatiol says- callbacks. As python embeds itself into more and more event driven systems- UI programming for example, the need for a more concise callback syntax becomes greater

[spice pecan]

Lambdas still have the same issue, so unless there will be a change, it's not a big difference

[flat gazelle]

I meant with regards to reassigning the callback name, but that doesn't really happen

[spice pecan]

Oh, got you

[grave jolt]

writing the function in-place exposes the anon to the scope of the enclosing function
You can do the same with a named function

[flat gazelle]

honestly, decorators are more than adequate for enough cases

[grave jolt]

yeah, usually event handlers are added with decorators

[static bluff]

Fair enough

[flat gazelle]

and I mean, java didn't have lambda until java 8 and was the primary language way before then

[static bluff]

They didn't have cars until the 1900s, and people still got around

XD

[grave jolt]

well, Java had anonymous classes, right?

[static bluff]

Anon classes?

[grave jolt]

and they could access the enclosed names

[static bluff]

Now this I gotta hear more about

[flat gazelle]

yes, but you used regular inheritance, not that abomination

anon classes are a way to create a subclass at instation time

essentially

a = Foo():
    def overriden_method(self):
        return 1
``` but python wisely doesn't have such a feature

[static bluff]

Quick side note ( I'm working while we talk) can binary, octal, and hex numbers be fractional in python?

i. e. have a decimal part?

[flat gazelle]

no

[static bluff]

Rockin

So lemme ask you guys

If there is anything you could change about python, what would it be?

[flat gazelle]

consistent naming across all builtin types

[grave jolt]

and names

like, no logging.createLogger or str.removeprefix

[flat gazelle]

which begs the question of whether map is a function or a type constructor

[grave jolt]

easy question, remove map altogether

functional nerds can build their own library!

[static bluff]

I think its sorta both, like type

[grave jolt]

Make the type annotation system more like typescript (well-defined rules and semantics, powerful inference, accounts for existing idioms).

[flat gazelle]

also, remove filter and map or move them to functools next to reduce, move id into sys, remove the is operator and replace it with a function in sys, implement some form of except try

[spice pecan]

I'd love built-in piping and function composition

[flat gazelle]

@ as function composition would be nice

[spice pecan]

Yup

[static bluff]

Dude I am so there

[grave jolt]

@ looks cryptic, and doesn't have direction

Maybe >>?

[spice pecan]

>>/<< would be nice for piping

[flat gazelle]

though lack of partial application makes that a bit less pleasant

[spice pecan]

I like the idea of matmul, but it's true that it doesn't have direction, that's a bit confusing

[flat gazelle]

a function to add 5 to a number in python is really verbose

[grave jolt]

X + 5

[flat gazelle]

it works like the mathematical ○ does

[static bluff]

And a built in oxford-comma/or method would be nice

[grave jolt]

yes, and the mathematical ○ is confusing, I look it up every time

you just got used to it

just like mathematics often use a single-letter variable for every single thing

[red solar]

Does python have a rule in the small print somewhere saying you can’t inherit from multiple exception types in the same class?

[flat gazelle]

you can inherit from multiple exceptions

[spice pecan]

Seems to be working as expected, yeah

[acoustic crater]

!e

y = lambda x:(exec('''
if x > 2:
    globals()["g"]=4
else:
    globals()["g"]=6''') or g)

print(y(2))
print(y(3))

multiline lambda

[fallen slate]

@acoustic crater :white_check_mark: Your eval job has completed with return code 0.

001 | 6
002 | 4

[red solar]

Lol

Hmm ok lemme test the exceptions rq

[static bluff]

See, this is what I was talking about when I said you might be able to pull of multi-line lambdas by removing it from the flow of the what's outside the lambda and parsing it seperately

[acoustic crater]

and it;s possible so now ppl can stop complaining hehe

u can even do if True:/n if True:/n for however many levels of indent you need

and if u dont wanna use globals use a dict in the scope you need to define stuff in

[red solar]

!e

class OtherException(Exception):
    def __init__(self):
        self.reason = "hello"

class DerivedException(NameError, OtherException):
    pass

d = DerivedException("hmmmm")
print(d.reason)

[fallen slate]

@red solar :x: Your eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 9, in <module>
003 | AttributeError: 'DerivedException' object has no attribute 'reason'

[red solar]

shouldn't this work fine?

[acoustic crater]

u inherit init from NameError

[red solar]

or does DerivedException need to call super.init()?

[acoustic crater]

I think

you need to switch NameError with OtherException in hierarchy

!e

class OtherException(Exception):
    def __init__(self, *args):
        self.reason = "hello"

class DerivedException(OtherException, NameError):
    pass

d = DerivedException("hmmmm")
print(d.reason)

[fallen slate]

@acoustic crater :white_check_mark: Your eval job has completed with return code 0.

hello

[red solar]

huh

@raven ridge what was the issue with my exceptions then?

[acoustic crater]

NameError init is what doesnt call super

[red solar]

should it call super?

(i mean you're right, it doesn't, it just initializes BaseException)

[static bluff]

Quick question

Converting something from python to cython almost universally leads to performance gains, but this is esspecially true for loop heavy code, correct?

[flat gazelle]

it is true for code that doesn't need python types

[static bluff]

Okay cool

[flat gazelle]

so number crunching that can put up with bounded ints

[main cradle]

I am new in this python comunity , can anyone tell where to start from

[raven ridge]

I don't agree with this statement. Converting Python to Cython can result in performance losses, too, if you type the wrong things

For instance, the Python function py def add(x, y): return x + y should be strictly faster than the Cython function py def add(x, int y): return x + y because the Cython function does all of the same dynamic dispatching, plus an extra isinstance check.

Cython is only faster than Python when the Cython function is able to skip work than the Python function does, but if you're careless, you can make it need to do extra work, instead.

[static bluff]

List manipulation (appending, slicing, popping) is slower than attribute lookup, right?

[raven ridge]

Slicing a large list is slower than attribute lookup, appending is faster, popping from the end is faster, popping from the start is slower

[static bluff]

Hmmmmm

What about a deque?

If I need to pop from the start

I have an iterator that uses a queue, adding one to the end and then yielding the firstmost

[raven ridge]

Deque is the fastest data structure for that.

[static bluff]

I could use a deque, but I think I could also just give my iterator 'previous, current, next' attributes, shifting them on each call to next

[raven ridge]

So the number of queued items is limited to 3?

[static bluff]

Yes

I have thoughts on implementing a similar mechanism is seven in the future but for this instance, three

[raven ridge]

Then all 3 will be very fast. List might be the fastest, you'd need to profile and see.

[static bluff]

Righto. I don't normally think about performance unless I have to- python is nearly always fast enough.

But right now, I have to 😛

How do I iterate groups in a regex match object?

[flat gazelle]

for group in re.match('(.)'*9, 'abcdefghijkl').groups():
    print(group)

though this kind of question is better suited to help channels

[static bluff]

Sorry, right, my bad

[paper echo]

Im familiar with these, but are they cpython implementation details or is the list implementation part of the language spec?

I assume the former

[feral cedar]

i don't think there are constraints on time complexity for these operations, so it would be impl specific

[hot valve]

hello

can someone shed some light on how python implements exceptions?

[static bluff]

I might be able to shed a bit of light

Why, what's your question?

[hot valve]

@static bluff , I want to know how exceptions are implemented in python

I don't have a specific question in mind! I wan't to understand the implementation and performance costs

[raven ridge]

The language spec never requires particular time complexity, as far as I know, but what I said applies to every Python implementation that I know of except for MicroPython / CircuitPython

[static bluff]

So I'm trying to get indentation working in my lexer

Would it be fair to assume that:
a) indentation never matters inside some kind of parentheses
b) if a ':' operator is found to be followed by a newline or comment, an indented block is expect
c) if a ':' operator is found to NOT be followed by a newline or comment, an indented block is NOT expected
d) any time a logical line leads with less whitespace than the expected indentation, a dedent token is created
e) any time a line leads with more than the expected indentation, an 'outer indentation does not match' error should be raised
f) any time a line leads with some mix of spaces and tabs, an 'inconsistent tabs and spaces' error should be raised

[static bluff]

What you're talking about called Search Engine Optimization or SEO

Its a field unto itself- there are professionals whose entire job is to reverse engineer google's (and others') algorithms. It isn't an entirely impossible task and there are indeed ways to game the system

Sorry, could you rephrase that? I'm not sure what you mean

Oh, I see

Your own search engine you mean

https://en.wikipedia.org/wiki/PageRank#See_also

There's a start

And that will probably link you to further reading

[feral cedar]

of course, the person who made the algorithm wouldn't have its Wikipedia page open while writing

[grave jolt]

what if it's a time traveling algorithm?

[torn sorrel]

my best day

[unkempt rock]

Can I have some help in #help-chocolate, thank you very much

[spark magnet]

@unkempt rock don't advertise help channels please, especially not in multiple channels at once

[unkempt rock]

ok

[static bluff]

So do you guys have any (advanced) tips for optimization?

[tropic fulcrum]

There is no really applicable-everywhere answer to that question. Any advanced optimization tips will be dependent on what it is you are optimizing. And if you haven't done any profiling yet, you shouldn't be optimizing anything (because you will almost surely be micro-optimizing non-bottlenecks).

[static bluff]

Fair

[unkempt rock]

Code reviews help for sure. Look at your code twice, thrice, etc.. Find those human bottlenecks when you wrote it.

[tropic fulcrum]

Human intuition is really not that good at predicting bottlenecks though.

[unkempt rock]

I disagree

I suppose it depends on the level you want, what your own personal definition is, etc.

timeit works, but sometimes you the human can find things faster just by native intuition of the boolean logic workflow you know?

pen + paper == mightier than the python tongue sometimes

[undone hare]

Hey @patent tiger, we won’t help with ongoing exams as said in your 5th rule

[patent tiger]

Ok

[static bluff]

@tropic fulcrum you aren't the person who wrote pythonparser are you?

The m-labs version?

[brave badger]

@static bluff Perhaps: https://github.com/pyparsing/pyparsing/

Really nice parsing library by the way

[bitter shoal]

How does python addresses collision in its dictionary implementation?
Is it a combination of some probing techniques or does it uses chaining? Or both?
Or something else?

[deft pagoda]

i think it just uses == comparison for all the things with the same hash

[bitter shoal]

I just found out that it uses pseudo-randomized probing to address a collision

And if 2 values have same hash == , they can still exist in the dictionary.
I dont understand how a lookup will work in this case? And
How can 2 different values with the same hash even exist if we are using open addressing and NOT chaining?

[deft pagoda]

i don't know how it's implemented, but i think you could wrap two or more collisions in some sentinel class and iterate through that

[bitter shoal]

But won't that give worst-case lookup O(n) for n items assuming we have m slots such that m > n
I thought for m> n , if we are doing open addressing (assuming no primary or secondary clustering), we would indeed get O(1)

[bitter shoal]

Wont that be similar to chaining?

[deft pagoda]

i don't know

[bitter shoal]

I read that python does not use chaining, it uses probing instead.

[visual shadow]

What's probing and chaining?

[deft pagoda]

i think there are O(n) worst-case lookups though

[bitter shoal]

https://hg.python.org/cpython/file/52f68c95e025/Objects/dictobject.c#l296

[deft pagoda]

i think accessing is only amortized O(1)?

not completely sure here

[visual shadow]

Yes accessing isn't guaranteed to be O(1) always, it can't because of collisions.

But I'm not familiar with the terms probing and chaining, what do they mean?

[bitter shoal]

Yes, in general hash tables would have O(n) worst case and amortize O(1)

But
https://cs.stackexchange.com/questions/249/when-is-hash-table-lookup-o1

[gleaming rover]

probing is like

how you resolve collisions

like your strategy to move between successive slots until you find an unoccupied one

basically

chaining is what Python does (I THINK)

[bitter shoal]

Well you can think chaining as @visual shadow :
Suppose during a collision, you are simply creating a linked list, at that same hash value and all elements with similar hash value would go to the end of linked list

[gleaming rover]

like instead of one result per "slot" you have a linked list of slots and you just add to the end

yeah

that

I'm not very familiar with this kind of thing so feel free to correct me if I'm wrong

[bitter shoal]

No no , python does probing

https://hg.python.org/cpython/file/52f68c95e025/Objects/dictobject.c#l296

[deft pagoda]

server unavailable

[gleaming rover]

oh really

[bitter shoal]

Yes, it happened with me too, I refreshed a few times . Wait

[gleaming rover]

I'll take your word for it

[prime estuary]

Here's a massive wall of text explaining the chosen algorithm:
https://github.com/python/cpython/blob/main/Objects/dictobject.c#L137

[fallen slate]

Objects/dictobject.c line 137

/*```

[undone hare]

Nice, a comment

[bitter shoal]

/*
The basic lookup function used by all operations.
This is based on Algorithm D from Knuth Vol. 3, Sec. 6.4.
Open addressing is preferred over chaining since the link overhead for
chaining would be substantial (100% with typical malloc overhead).

The initial probe index is computed as hash mod the table size. Subsequent
probe indices are computed as explained earlier.

All arithmetic on hash should ignore overflow.

[bitter shoal]

for those server shows "unavailable"

[prime estuary]

The Mercurial repository is old, CPython switched to git a while ago.
https://github.com/python/cpython/blob/main/Objects/dictobject.c#L770-L793

[bitter shoal]

Oh, this is better.

[visual shadow]

Okay, so chaining makes linked lists. Probing makes a.. New hash that points to a different bucket?

Or perhaps more precisely, a hash + post processing to point to a diff bucket?

[prime estuary]

Yeah. The simplest version: move to the next index, and keep going until you find a free one.

[deft pagoda]

yep

"next index" being a relatively simple modular function that has the same period as the size of the table

[visual shadow]

OK. So how does the lookup for a collisioned item proceed? Does it store the specific "post processing" on the first item of a collision?

[bitter shoal]

Yes, there are different kinds of probing though, it might happen that you are not getting new slots at all and only getting same values. Its not realy "moving" onto the next slot

[visual shadow]

I suppose the part that threw me off is that they said they're not going sequentially, but randomly

[prime estuary]

When it looks up, it finds the original key doesn't match (does a Py == comparison), then does the same hop.

[deft pagoda]

their probing hits all the slots in the table

[bitter shoal]

In linear probing, primary clustering can happen, you are not moving onto the next slot, but you are computing the next slot and getting an already occupied one. And it might happen that during probing you are not getting a new slot at all.

That's where double hashing comes in

[prime estuary]

It's sequential, but with a more complicated sequence (IE multiply, add a constant).

[visual shadow]

Oh ok. So probing doesn't need to be stored because it's deterministic for a given dict?

[prime estuary]

Yeah. When deleting values, the slot is set to a special deleted key value to ensure it can still hop to the right spot.

[visual shadow]

Ah beautiful. You answered just what I was about to ask before I even knew I needed to ask it myself

Thanks!

It's almost scary how elegant this is.

[prime estuary]

If the dict gets too full or empty, it triggers a resize, rebuilding the table so they don't collide anymore.

[bitter shoal]

Yes at every 2/3rd fill, so collisions are less

[visual shadow]

Oh i didn't realise it resized down also.

I saw the comment about 2/3

[feral cedar]

does it resize down?

what it you're sitting right at 2/3 full then you just add and remove keys, that would give terrible performance

[prime estuary]

Looks like it doesn't, I don't think.

[bitter shoal]

It makes less sense to size down though.

[flat gazelle]

In [2]: a = dict.fromkeys(range(10000))

In [3]: a.__sizeof__()
Out[3]: 294984

In [4]: for k in range(10000):
   ...:     del a[k]
   ...:

In [5]: a.__sizeof__()
Out[5]: 294984

In [6]: a
Out[6]: {}

yeah, doesn't seem to scale down, unlike a list

[prime estuary]

One clever optimisation they do do however: for instance attribute dictionaries, they use a "split" dictionary. The dict is split into the hash table/keys, and a separate values array. So all instances of a class can share the hash table, in the regular case where they all always the same attributes set in __init__. If you do delete or add new ones, then the oddball instance converts back to a regular dict.

[bitter shoal]

So that would only be optimal for lookups across various instances and not for deleting or adding.

[peak spoke]

Does a clear size it down?

[flat gazelle]

yes

[prime estuary]

Yes, a clear makes it actually use a shared empty table.

https://github.com/python/cpython/blob/main/Objects/dictobject.c#L1792-L1796

[fallen slate]

Objects/dictobject.c lines 1792 to 1796

dictkeys_incref(Py_EMPTY_KEYS);
mp->ma_keys = Py_EMPTY_KEYS;
mp->ma_values = empty_values;
mp->ma_used = 0;
mp->ma_version_tag = DICT_NEXT_VERSION();```

[bitter shoal]

into play.  This is done by initializing a (unsigned) vrbl "perturb" to the
full hash code, and changing the recurrence to:

    perturb >>= PERTURB_SHIFT;
    j = (5*j) + 1 + perturb;
    use j % 2**i as the next table index;

Can someone please explain what is perturb, and why will this make it depend on every bit of the hash code. ?
https://github.com/python/cpython/blob/45862f9f5ef5d3c9da37f35e4fe4b18618530cfa/Objects/dictobject.c#L189

[spice pecan]

IIRC the sizing down happens on the next insertion

[prime estuary]

If you scroll up, it's defined as "5".

[flat gazelle]

In [15]: a = dict.fromkeys(range(10000))

In [16]: for k in range(10000):
    ...:     del a[k]
    ...:

In [17]: a[0] = 4

In [18]: a.__sizeof__()
Out[18]: 294984
``` doesn't seem to be the case.

[prime estuary]

Since we're using the least significant bits, the perturb value causes it to shift down 5 and pull in the other values each time.

As it describes in the text, Tim Peters apparently did a bunch of testing, and found that was a good value.

[spice pecan]

Apparently not, yeah

[bitter shoal]

Ok, that makes sense now. Thanks a lot for helping.

[visual shadow]

Just catching up on convo so it doesn't resize down then yeah?

So only on too full

(well, except clear looks like)

[tiny cave]

is there any code that "bypasses" text?

[feral cedar]

what does that mean?

[tiny cave]

it puts ˇ´ on it

Lïkê ťhįš

[visual shadow]

Just make a dictionary, mapping normal ascii to those noisy unicode characters.

[tiny cave]

what

how

[visual shadow]

I guess the fundamental point to understand is this: you aren't "adding" funny symbols on top of letters in terms of computer land.. Those are literally distinct and different unicode chars.

Once you understand that, you'll understand that it's not a "adding symbol" problem, rather its simply a problem of "replacing" letters.

[tiny cave]

my head exploded, i do python only for 2 sayd

days

[visual shadow]

Ah yeah, computers do some things differently than how we humans think of it. I would say if you want specific help on this, try our help channel system #❓｜how-to-get-help

This room isn't really appropriate for giving help, more so for discussion about the language itself.

[lost mica]

I have made a cool snippets but it isn't very useful : https://0bin.net/paste/32cu+FlI#ZbEzcK0q892xXLYR86zIetVWGYHFoxQWLxaseJAb1GN

do you see some use cases ?

[tropic fulcrum]

No, I wrote and maintain pyparsing (https://github.com/pyparsing/pyparsing), and a couple of others (like this pyparsing spinoff, an embeddable parser/evaluator for arithmetic expressions without the security issues of eval https://github.com/pyparsing/plusminus; and this small-footprint ORM-ish library for easy import/export of CSV's and some simple db functions including join and pivot https://github.com/ptmcg/littletable).

[stone field]

@tiny cave you can get a very close estimation by normalizing the string into it's decomposition representation, then removing all non-ascii characters:

>>> unicodedata.normalize('NFKD', 'Lïkê ťhįš').encode('ascii', 'ignore').decode()
'Like this'```

[viscid mesa]

Have anyone read Fluent Python? Any good?

Also, what is the best resource to learn "advanced" Python?

[feral cedar]

yep, I recommend it after getting to know python a bit

[viscid mesa]

Been working with it for a couple of years.. feeling ready to go to the next level

[feral cedar]

however, this isn't really on topic for this channel, try #python-discussion

[viscid mesa]

Awesome. Thx.

[craggy horizon]

is anyone familiar in buildimng a gli gaming site

[rich cradle]

You might want to ask in #python-discussion or a Topical Channel, this channel is for discussion of the semantics of Python itself.

[craggy horizon]

ight

[static bluff]

I am having

So. Much. Fun building this lexer

Performance continues to be an issue, but I think I'm starting to 'get' what all of the other lexers I've looked at are doing

[static bluff]

You guys working on anything fun?

[tiny cave]

thank

[devout hollow]

I'm building a Markup language that compiles into LaTeX

The compiler will be written in Python

[grave jolt]

I made a lisp-like markup language that compiles into HTML:
https://decorator-factory.github.io/python-fnl/

interesting stuff

[devout hollow]

Looks pretty cool

[static bluff]

Cool!

So, about lexing/parsing and performance

My reading has shown that it's often best to rely on a lexer generator, as they are heavily optimized. I'm also under the understanding that there are faster alternatives to regex based lexers but that those alternatives are a bit harder to understand

I just wanted to get you guys' thoughts on the question as a whole- performance in the context of lexing and parsing that is. Specifically, I want to know if transplanting the code from python to cython would make any improvements

Ultimately, I should probably just bite the bullet and write it in C. I'm not quite ready to learn a whole new language yet though. I'd rather figure out what in the hell I'm doing first, then make the transition

[grave jolt]

eh

is performance really that important?

How much time does it take to lex, parse a 1000-line source file?

[scarlet marlin]

hei there, is this a good place to post a problem that I have and someone might help with?

[static bluff]

Ideally, no time at all

[grave jolt]

well, you can't do that in 0 seconds

[static bluff]

_>

XD

[grave jolt]

Hi, check out #❓｜how-to-get-help

[static bluff]

Really though, what you're saying drives exactly to my point. Even a lexer written in a lighting fast language should be carefully configured for speed. Imagine if they weren't and instead of each of your modules taking 10ms to compile (a four step process by the way) it took 100ms

That's 100 modules per second versus 10 modules per second

[grave jolt]

but you can make the interesting parts of the language first, and keep a simple and readable lexer & parser

by the point you'll care about parsing speed, you'll have changed the syntax, the lexer, the parser and everything else

right?

[static bluff]

Well let me give you an example

One thing I'm learning (this is new territory for me) is that one big fat regex is faster than many small ones. Additionally, compiling that regex takes a little time, and so doing it once in a metalexer class, with all subsequent lexers not having to recompile, saves on some overhead

[flat gazelle]

even a slow lexer should be really fast, this is a lexer written by nedbat for JS, and it doesn't really focus on performance and is pure python + regex. It takes 1ms on my device to lex all of jquery
https://github.com/django/django/blob/main/django/utils/jslex.py#L101-L178

[static bluff]

Ahhh yes, JSLex

I guess my question is one I'm asking more as someone who is lost in the weeds

You guys know how this works, I don't- hence the questions

[flat gazelle]

the slowest part of compiling e.g. C++ is linking, the compilation step itself is very rarely not realtime, especially with debug configs

parsers and lexers are almost never the bottleneck

[static bluff]

And I know, I should be doing my research, which I am, but getting it straight from the horses mouth is an important part of the learning process also

[flat gazelle]

asking questions is good

[static bluff]

I won't ask you guys for a seminar but, what do I absolutely need to know

Side note: I don't think I've ever enjoyed a coding project this much 😄 😄 😄

[flat gazelle]

having a working language that takes 30ms to lex a source file is more valuable than having just a lexer that takes 1ms

you can always optimize later

[static bluff]

Good to know. My first few attempts were taking about 150ms to lex even a tiny snippet, but I've learned a lot since then

[flat gazelle]

besides, the solution to slow compile times is less faster compilers, but more compiling only what changed and caching builds. 10 minutes vs 2 minutes to compile a large project is never real time, but if it is 10 minutes and .2 seconds per change, vs 2 minutes every time, it is clear which is better for the developer

[static bluff]

🤯

If nothing else, this experience has taught me the importance of comments

[compact salmon]

hi all, I have a question that's about the internals of python. I posted on #help-pie but I realize maybe I need to check out here (if it's ok, otherwise I apologize) because it's an advanced question I'd say. Thanks in advance 🙂

[static bluff]

Hey GG, your chan is long since dormant, but, did you get an answer to your question?

What are you guys' preferred nomenclature for metaclasses? I just hate naming my metaclasses 'metawhatever'

I asked in the general chat too, but I get the feeling you guys will have more nuanced thoughts on the matter

[halcyon trail]

I usually just do MetaWhatever

the one time a year I write a metaclass

[static bluff]

:3 metaclasses are fun (IMO)

[halcyon trail]

sure, it's just almost never called for

[flat gazelle]

even the standard library does metawhatever

there is no real concept that corresponds to metaclasses, so you just have to use Meta

[tropic fulcrum]

@static bluff - have you written out your BNF? I encourage people to this early on in their parser endeavors, no matter what parsing framework they are using. You can do some up-front ambiguity checking, and once you have it how you like it, it gives you a roadmap for your parser development.

[acoustic crater]

since __init_subclass__ is a thing now, is there really ever a time metaclasses are required any more?

[undone hare]

Couldn’t you say that metaclasses are only used in libraries then? I doubt it is as used in real word apps than in libraries. Sure, you still have some utilily classes using some fun meta stuff, but it is more rare, at least in my opinion.

[flat gazelle]

every metaclass I have seen was just to create a specific modified class, not modelling a real concept. so yeah, I would agree with that

[halcyon trail]

init subclass is just one use case of metaclasses, right

There are still use cases of metaclasses, but I do agree that it seems like usage of them should be extremely rare

When you look at inheritance + decorators + hooks like init_subclass, you can achieve a crazy amount already in python, and even using those tools, there's a lot of complexity, and you need to do things carefully so that they are robust

[grave jolt]

There are still use cases of metaclasses,
Overloading operators on classes

[halcyon trail]

Can't you do that via inheritance?

[grave jolt]

how?

[halcyon trail]

define the dunder methods?

[grave jolt]

That would define the operators for instances of those classes

[halcyon trail]

oh, lol

see, that's such a rare and crazy thing to want to do that I didn't even take your meaning

[grave jolt]

!e

class Meta(type):
    def __add__(whatever, man):
        return "quack"

class Foo(metaclass=Meta):
    pass

class Bar(metaclass=Meta):
    pass

print(Foo + Bar)

[fallen slate]

@grave jolt :white_check_mark: Your eval job has completed with return code 0.

quack

[halcyon trail]

And if your coworker tries to commit code like this, and tries to argue that "this is a DSL that makes doing such and such more elegant", you drive them to a remote location and leave them there

[visual shadow]

classes define behaviour of objects. metaclasses define behaviour of classes.

[halcyon trail]

other than that one sentence blurb, you honestly need to know nothing about metaclasses for a very very very long time writing python

unless you happen to have a framework that forces you to use them directly, not sure how common that is

I believe usually it's presented as inheritance instead

and the meta-classing would be an implementation detail

[mild flax]

Method Resolution Order

[halcyon trail]

MRO is something that comes up basically in complex inheritance hierarchies

[mild flax]

when subclassing, python might have to look in multiple places to resolve an attribute reference

[halcyon trail]

basically, it becomes significant when you have multiple inheritance

[native flame]

when you call object.method(), the MRO decides which class's method its going to use

[halcyon trail]

Again, this isn't usually something you need to worry about too much

python code where you really need to know/look up the MRO rules would probably be in the 99.9th percentile of complexity of python code (IMHO)

🤷‍♂️ 🙂

Not sure what kinds of things you're already familiar with, but knowing say generators, context managers, decorators, well, is probably more mileage

[raven ridge]

https://rhettinger.wordpress.com/2011/05/26/super-considered-super/ explains MRO well.

[pale flame]

Say you have a list of tuples or a list of lists and you wanted to iterate through the first element of each tuple or list of the main list:

list_of_lists = [[1,2,3],[4,5,6],[7,8,9]]

Why does python not have some kind of syntax such as

for element[0] in list_of_lists:
  print element

I feel like theres some handy syntax to have for this - not for any particular reason though. It's probably a bad idea, i'd like to know why.

[undone hare]

I think it isn’t clear what you mean here

But it can work with for element, *_ in list_of_lists

[pale flame]

I have a list of lists. I would like to only iterate through the first element of each list inside of list of lists

[undone hare]

It is just the unpacking syntax which is different

Yep

[pale flame]

Huh this works

Ive never seen the use of *_ before

[undone hare]

!e Are you familiar with syntax like

my_list = [1, 2, 3]
a, b, c = my_list
print(f"{a=} {b=} {c=}")```?

[fallen slate]

@undone hare :white_check_mark: Your eval job has completed with return code 0.

a=1 b=2 c=3

[pale flame]

nnnnnot familiar with the a= stuff no

[grave jolt]

!e
Just the unpacking part:

my_list = [1, 2, 3]
a, b, c = my_list
print(a, b, c)

[fallen slate]

@grave jolt :white_check_mark: Your eval job has completed with return code 0.

1 2 3

[pale flame]

I see what it is doing. That's interesting

[undone hare]

And by prefixing one with an asterisk, you try to take as many elements as possible

[pale flame]

What are the advantages of that? Quick assignment of variables from list?

[acoustic crater]

it's not rly the most clear but if I have a tuple I want to be a list for example I'll usually do this [*my_tuple] instead of this list(my_tuple)

list(*my_tuple) would turn the tuple into separate arguments so it wouldn't work but a list literal like [1, 2, 3] has comma separated values so unpacking into it works

[feral cedar]

it's iterable unpacking

!pep 3132

[fallen slate]

**PEP 3132 - Extended Iterable Unpacking**

Link

Status

Final

Python-Version

3.0

Created

30-Apr-2007

Type

Standards Track

[halcyon trail]

The main advantage/usage IME is to quickly grab say the first and last elements from a list

or the first two

etc

or if a function returns a tuple and you only care about some of the arguments

it's not life changing, just occasionally slightly nicer

[pale flame]

Thanks guys. This helps clear things up

[acoustic crater]

np

[undone hare]

It is quite useful in my opinion when you want to return many variables from a function, just pack them into a tuple when returning and unpack them into variables right away

[halcyon trail]

you don't need the * for that though

although when you say "many" that seems a bit scary, I'm picturing a function returning a tuple with like 10 items 🙂

[narrow kettle]

i like it personally

[grave jolt]

If you want to get the first and last elements of a list, you do

first, *_, last = your_list

this isn't allowed in javascript, which hurts my little heart

[grave jolt]

it can be any iterable, actually

a, b, c = map(int, input().split)

[halcyon trail]

I know it can be any iterable, the person I was responding to said tuples though

I guess basically I like * (rather minorly) more in the context of lists, where having a variable number of elements is more of a thing

with tuples, I'd probably rather unpack the whole thing explicitly since you usually only have 2-4 elements, if you have a tuple with so many things that * is useful maybe it shouldn't be a tuple

[visual shadow]

there's also new_tuple = (*list1, *list2) and things along those lines for unpacking and merging stuff into new containers.

[halcyon trail]

for lists you'd usually just use +

** is nice for dicts though, although I seem to recall that an operator overload was suggested for dicts

I actually really wish + worked on iterators

itertools.chain is so annoying comparatively

Ah, apparently this is annoying in python because of course iterators are a protocol, not a type, so there's no really way to do this and have it work for all iterators

[grave jolt]

what if you need to merge a list, tuple, a set and a range?

!e

a = [1, 2, 3]
b = (4, 5)
c = {6}
d = range(7, 11)
e = [*a, *b, *c, *d]
print(e)

[fallen slate]

@grave jolt :white_check_mark: Your eval job has completed with return code 0.

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

[halcyon trail]

then you probably went wrong a while ago 🙂

[acoustic crater]

| or/union is used for dicts in case that's what you mean

I think it's weird that it's a non-commutative use of | but I guess it's less weird than using +

(right operand's values of same keys overwrite left operand's)

[halcyon trail]

I dunno the commutative argument has been IMHO pretty week for a long time now

• has been standard for string concatenation in many languages for ages

You're right, it is |

https://www.python.org/dev/peps/pep-0584/

i guess this was accepted to 3.9

[grave jolt]

naaaaaah

actually, type checkers generally like [*, *] better than addition of lists of the same type

[halcyon trail]

>>> {0} | {False}
{0}
>>> {False} | {0}
{False}

😱

[grave jolt]

well, sets always had |

as far as I remember at least

[halcyon trail]

they did

but the behavior above is horrific

why would type checkers have any issue with addition of lists of the same type?

I've never had an issue with it type checking

[acoustic crater]

!e print(issubclass(type(True), int))

[fallen slate]

@acoustic crater :white_check_mark: Your eval job has completed with return code 0.

True

[acoustic crater]

that's why a set can't have a bool and its integer representation, they're the same to a set

or actually I don't really know I should look

[feral cedar]

well

!e print(0==False)

[fallen slate]

@feral cedar :white_check_mark: Your eval job has completed with return code 0.

True

[paper echo]

!e print(isinstance(True, int))

[fallen slate]

@paper echo :white_check_mark: Your eval job has completed with return code 0.

True

[grave jolt]

With different types, sorry

[halcyon trail]

yeah, still awful though

[grave jolt]

bool < int is cursed indeed

[halcyon trail]

for different types, why would you expect it to work in either case?

Are you saying that [x**, y**] does a better job inferring a supertype than x + y?

oh

Or do you mean like concatenating a list and a tuple?

[grave jolt]

yup
(also depends on the type checker, as always)

[grave jolt]

in that case it will just be a runtime error

[halcyon trail]

yeah that's why i was confused

do you have a concrete example or should I just use inheritance

class Foo:
    pass

class Bar1(Foo):
    pass

class Bar2(Foo):
    pass

x = [Bar1()]
y = [Bar2()]

z1 = x + y # doesn't type check?
z2 = [*x, *y] # does type check?

[acoustic crater]

only 1 asterisk to unpack sequences

2 for mapping

[halcyon trail]

my bad

it shows that I use two more often hah

interesting, you're right

mypy complains about z1 but not z2

[acoustic crater]

cpython casts both things to PyObject as w and v then checks if (v == w) and yeah both True and int(1) are gonna be pointers to 1 as a PyObject I think

[halcyon trail]

it seems like mypy doesn't handle the covariance correctly here basically

[paper echo]

i wonder if this is also true in pyright, pyre, pytype, and/or pycharm

[acoustic crater]

pycharm complains

just a little though

[paper echo]

the problem is that the type of the z list could be list[Bar1 | Bar2] which is totally valid

in fact id be annoyed if this didnt pass type checking

[paper echo]

yeah thats just wrong imo

these both should fail due to contravariance:

z1: list[Foo] = x + y
z2: list[Foo] = [*x, *y]

and these both should pass with the inferred type of list[Bar1 | Bar2]

z1 = x + y
z2 = [*x, *y]

(in my opinion)

[acoustic crater]

yeah it's silly

[halcyon trail]

why should they fail?

[acoustic crater]

in general I wish pycharm let you customize it more

rly should switch to vscode

[halcyon trail]

that's not how it works in languages that have contravariance/covariance

[acoustic crater]

pycharm also hogs hella memory and freezes my other apps

[paper echo]

what should it be @halcyon trail ?

[acoustic crater]

and lags a lot

[paper echo]

are lists supposed to be covariant?

i think this is all stated in a pep somewhere but i cant remember which

https://www.python.org/dev/peps/pep-0484/#covariance-and-contravariance

[acoustic crater]

!e

from ctypes import py_object
print(id(py_object(True)), id(py_object(1)))

also yeah same pointer

[halcyon trail]

sorry, you're right, I forget that list is mutable. x+y can pass with type Sequence[Foo] though

[paper echo]

so i think they are invariant because mutable

[fallen slate]

@acoustic crater :white_check_mark: Your eval job has completed with return code 0.

140165620876736 140165620876736

[halcyon trail]

which is more typically how it would be handled in languages that heavily feature inheritance

e.g. in Kotlin this is how + would work

[paper echo]

are kotlin lists immutable?

[halcyon trail]

List[Foo] is not immutable but it's a read-only API

so it is covariant (or is it contravariant, always get the details mixed up)

basically, List[Foo] is parent to List[Bar1]

(since Foo is parent to Bar1)

[paper echo]

yeah i think that allows it to be covariance

https://mypy-play.net/?mypy=latest&python=3.9&gist=dafcaa9570bf1a59326188473c11ebe1

interestingly mypy gets the type inference wrong too (again imo)

[halcyon trail]

It is the same as python:

The read-only collection classes in typing are all declared covariant in their type variable (e.g. Mapping and Sequence)

Even though Mapping and Sequence are not immutable

I'm not even really comfortable with the terminology "read only collection classes", it's very misleading

[paper echo]

yeah, they are "effectively" immutable

[halcyon trail]

No

they are read only interfaces

of things that could be mutable

[paper echo]

right, which is as good as it gets in python. otherwise everything is mutable.

[halcyon trail]

tuple is immutable 🙂

So are FrozenSets

etc

strings

[paper echo]

well a tuple is a Sequence

[halcyon trail]

yes

Sequence can be satisfied by both immutable and mutable classes

that's the whole point

but Sequence isn't a "read only collection class", it's a read-only interface/API to a concrete type that may or may not be read-only/immutable

[paper echo]

right. a Sequence is a read-only interface to things that might or might not be immutable. but the idea is that if something implements Sequence but not MutableSequence it should be considered immutable from the perspective of the type checker. likewise, if something is locally annotated Sequence then it isn't being mutated by anything locally (in a perfect world where the type checker is complete) and can be treated by the type checker as immutable.

[halcyon trail]

it's not considered immutable from the perspective of the type checker, I'm not even sure what that actually means

it just says that through something whose static type is Sequence, you cannot do any mutations

[paper echo]

right, so the type checker can treat it as covariant because you aren't (or shouldn't be) mutating it

[halcyon trail]

Right, it is safe to treat it as covariant

if the underlying container is mutable, then the underlying container is invariant, which is a stronger guarantee, so you are safe

[paper echo]

right, but the type checker doesnt know or care about the underlying container.

[halcyon trail]

Indeed.

[paper echo]

thats all i meant by "treated as immutable"

[halcyon trail]

But yeah, that's why z1 = x + y should work, and probably should have type Sequence[Foo] or something similar

Yeah, that's not really correct usage of immutable, but I understand it's more convenient to use it that way then to spell out what' sactually happening

[paper echo]

right

re z1, in this case x and y are list[Bar1] and list[Bar2], so that's why i said it should infer list[Bar1 | Bar2]

[halcyon trail]

Sequence doesn't even give you "local immutability" from two separate angles, so it's not much of an actual benefit in verifying that things don't change. It's only an aid in saying you aren't allowed to change them.

Sure, in a language that puts more emphasis on sum types than inheritance, that would be reasonable

it's just that python isn't such a language

[paper echo]

isnt that more a question of type system completeness than anything

[halcyon trail]

So I think Sequence[Foo] makes more sense personally. but you can go either way.

Not sure what you mean by "completeness"

[acoustic crater]

x: List = [Bar1()]
y: List = [Bar2()]

seems like the way to go if you want List[Any]

or just do List[Any]

[halcyon trail]

You don't want List[Any] though, you want List[Foo]

[paper echo]

e.g. in kotlin, can you bypass the read-only sequence API in a local section of code where something is annotated as a read-only sequence?

[halcyon trail]

you could annotate x and y as List[Foo] of course, but that's not ideal either

you can downcast it

but I'm not talking about that

[acoustic crater]

in this use case sure but Any would allow concatenation of lists with anything inside

[paper echo]

as in, is it possible to mutate a sequence that's annotated as read-only, and fool the type checker into thinking it is not in fact mutated?

[halcyon trail]

First of all, someone else could have a reference to the underlying mutable container under a mutable type

and simply mutate it

second of all, these guarantees are not transitive so in any case a Sequence[Foo()] isn't immutable unless the Foo's are immutable as well

So in both cases it' strivial to mutate such a thing

[paper echo]

yeah transitivity is a whooole different issue

[halcyon trail]

Yeah, but without it you aren't immutable

[paper echo]

you can have an immutable container of references to mutable things 😛

[halcyon trail]

but even the aliasing issue

it's not really immutable at that point, but again, I get what you'r esaying

[paper echo]

yeah i see the aliasing issue

is it not?

a tuple can contain lists

[halcyon trail]

no

[paper echo]

the tuple is immutable, no?

[halcyon trail]

it's not true immutability, no

shallow immutability I guess would be a term?

[paper echo]

sure, i think ive seen that term used actually

regarding z1, i dont see why the type checker should infer it's a list of Foo and not a list of Bar1 | Bar2

[acoustic crater]

I don't understand whyt he type checker doesn't allow concatenation of lists containing different types

when they aren't used later for anything

does it just assume they need to stay the same type and be used later because it's harder to actually determine in a language like python?

[paper echo]

i assume because + is defined like this

A = TypeVar('A')
def __add__(x: list[A], y: list[A]) -> list[A]:
    ...

and not like this

A = TypeVar('A')
B = TypeVar('B')
def __add__(x: list[A], y: list[B]) -> list[A | B]:
    ...

[halcyon trail]

@paper echo that's how it works in Kotlin fwiw, and probably Java too

[paper echo]

kotlin would infer list[Foo]?

[acoustic crater]

oh makes sense

[paper echo]

there are probably type stubs for this somewhere in the cpython source. can check to confirm

or maybe its hard coded into mypy

[acoustic crater]

it's in mypy

it's umm

typeshed

https://github.com/python/typeshed someone told me the other day

[halcyon trail]

So List[Foo] is parent to List[Bar1] right

[paper echo]

if lists are contravariant then list[Foo] is a child of list[Bar1]... i think

[halcyon trail]

No

[acoustic crater]

https://github.com/python/typeshed/blob/e14085014f71c316bfdd5d6183b2b01b8665e978/stdlib/builtins.pyi#L774

[fallen slate]

stdlib/builtins.pyi line 774

def __add__(self, x: List[_T]) -> List[_T]: ...```

[halcyon trail]

Right, so it's a bit problematic to handle this correct in python

the problem is that you can't actually define functions that act on interfaces, I suppose. So you have to define add on list, and then it assumes that the type of self is list as well, which is invariant

in Kotlin, add is defined as an extension function that acts on the List interface. That means that when you have a MutableList<Bar1>, Kotlin doesn't care about the mutable part anyway, it only sees the List<Bar1.

since List<Foo> is parent to List<Bar1>, when it's looking at the + operator, it will also consider List<Foo>'s version of operator+

List<Foo>'s version of operator+ takes, of course, another List<Foo>, and MutableList<Bar2> is a List<Foo>