internals-and-peps

[verbal escarp]

i'm not sure if performance would be a big issue in this case because scientific tooling mostly uses numpy for calculation-heavy stuff, so most of the time wouldn't even be spent inside python

the python layers would mostly be abstractions for communication between parts of the program, basically

[lusty scroll]

it won't... that wouldn't be possible

[red solar]

I think I would still have a clear boundary between only allowing imports compatible with the current Python version, and allowing all imports which might need a subinterpreter

Oh

So subprocess it is then?

[verbal escarp]

possible upside: we could use specialized interpreters like cython maybe?

still not sure about communication between the different interpreters, though

[red solar]

I’m more worried about how you’re going to emulate stuff like inheritance or imported decorators

[lusty scroll]

is this to "use" a package that's incompatible with the python that's running?

[verbal escarp]

yes

[lusty scroll]

I think you would basically need a proxy to the module and all of its classes

[verbal escarp]

which we have..

wait. all of its classes?

you mean a recursive proxy?

[lusty scroll]

and any data that wasn't pickleable would need proxied as well (like call args)

[verbal escarp]

oh man

couldn't we just take @red solar's atomic buffer thingy and use that as proxy? 😄

[red solar]

Lol wut? How would that help? 😂

[verbal escarp]

just generating crazy ideas 😄

[lusty scroll]

I think it's possible - GDB has a 'libpython' that basically does it for any python you debug, but it's mainly meant for generating deep repr's of objects

I don't think this would be possible in most languages without code generation, but I think it would in python

[verbal escarp]

time to write a new issue "research: find a way to communicate with incompatible python interpreters as subprocesses" label: "good first issue"

maybe someone like @paper echo takes the bait 😉

hope

[red solar]

@verbal escarp are you also hoping to support py2, or just py3?

[verbal escarp]

oh god no

although it probably could

not sure if that's a good thing :p

[red solar]

I mean imo a really big use case is using python2 libraries that haven’t been updated for python3

[lusty scroll]

wrapt has ObjectProxy

[verbal escarp]

yeah.. well. ideally, you could have a python 3.12 main program and simply use() stuff from any decade without issues or backwards incompatibilities

you could move on with your main code and dependencies would just keep on working like they used to

albeit possibly with a performance penalty

[lusty scroll]

thinking who has done inter-process object proxies ?

[verbal escarp]

maybe we can find someone to help with that in #async-and-concurrency

[lusty scroll]

possibly

[ruby relic]

have you guys ever tried using python to record electronic instruments that are connected to a laptop \ pc?
imagine if you were to connect an electric guitar or electronic drums into the pc.
Do you think theres a specific channel in the computer i could use to record the data sent in there?

[red solar]

How are you connecting them? There’s probably a library for whatever way you’re doing it

[ruby relic]

i never tried it, but i presume people will use MIDI prolly

or USB

[red solar]

You don’t have the hardware to test it yourself?

[ruby relic]

i do, but i never tried it

i got an electronic set of a drumkit and guitar

i prolly lack the MIDI connectors

[red solar]

There’s a bunch of MIDI libraries for Python, but I would get the hardware set up before you try using the software

[ruby relic]

what kind of hardware do you mean? the electronic instruments?

or do i need a powerful Laptop?

[main lynx]

you can definitely record audio with Python, i don't have a specific recommendation on what to use but this might be helpful https://wiki.python.org/moin/Audio

[ruby relic]

i will take a look at it, thanks!

[red solar]

Ah I was thinking more of pymidi

[red solar]

I mean connect your electric instruments to your computer, download some ready made midi software to check it works, and then try to write Python to communicate with your instrument

[ruby relic]

aite thanks

[white nexus]

wow

python's current docs got their theme pretty much back in 2013

or before

https://docs.python.org/2.6/

[uncut pebble]

I was just wondering about these sizes

!e

from sys import getsizeof

print(getsizeof(2))
print(getsizeof([2]))
print(getsizeof([2, 3]))
print(getsizeof([2, 3, 4]))
print(getsizeof([2, 3, 4, 5]))
print(getsizeof([2, 3, 4, 5, 6]))```

[fallen slate]

@uncut pebble :white_check_mark: Your eval job has completed with return code 0.

001 | 28
002 | 64
003 | 72
004 | 120
005 | 120
006 | 120

[uncut pebble]

found something relevant, going through it now https://towardsdatascience.com/the-strange-size-of-python-objects-in-memory-ce87bdfbb97f

!e
from sys import getsizeof
print(getsizeof([]), getsizeof([2]), getsizeof([2, 3]))

[fallen slate]

@uncut pebble :white_check_mark: Your eval job has completed with return code 0.

56 64 72

[uncut pebble]

8 bytes for each

!e

from sys import getsizeof as g
print(g([1, 2, 3, 4, 5, 6]), g([1, 2, 3, 4, 5, 6, 7]))```

[fallen slate]

@uncut pebble :white_check_mark: Your eval job has completed with return code 0.

152 120

[uncut pebble]

???

[zenith flint]

Is it possible somehow in Windows to only send the SIGINT signal to MainProcess on KeyboardInterrupt? It works fine in Docker as it only sends the a SIGTERM to MainProcess when the container receives a stop signal, but since Windows doesn't handle processes well I have to handle signals in each process when I'm testing locally. Which is fine just bloats the code for local testing

[red solar]

If all child processes receive signals on windows, idk how to change that, but maybe set your own signal handler for SIGINT that checks which process it’s in and only raises KeyboardInterruptException in the main process?

[weary flax]

thanks, i just searched mentions on my name to see if somebody answered....

[visual shadow]

this isnt the right room, perhaps try ot rooms.

[weary hamlet]

yeah, i figured that, sry

[serene grail]

!e

[fallen slate]

Command Help

!eval [code]
Can also use: e

*Run Python code and get the results.

This command supports multiple lines of code, including code wrapped inside a formatted code block. Code can be re-evaluated by editing the original message within 10 seconds and clicking the reaction that subsequently appears.

We've done our best to make this sandboxed, but do let us know if you manage to find an issue with it!*

[serene grail]

!e print("hi")

[fallen slate]

@serene grail :white_check_mark: Your eval job has completed with return code 0.

hi

[verbal escarp]

why does b'\x9a\xe6\xa8' - a value produced by hypothesis - mess with sha256 and my JACK-encoding roundtrip?

[spice pecan]

That's because of how lists work. Internally, they are backed by an array of pointers to PyObjects. On a 64 bit system, a memory pointer would take 8 bytes, which is why they grow in increments of 8 up to a certain point. But because the backing array has to be relocated in order to grow, it would be incredibly inefficient to do it for every single append. What python does is, instead of just keeping track of the backing array's length, it also keeps track of the amount of items the list has, which can (and mostly will) be lower than the capacity. This is called amortization, which is done via overallocation, and several other languages do it as well - C#'s List does this, C++'s vector too, so on and so forth

Plus the size includes some bytes of overhead for the actual list object itself (because it too is a PyObject)

[red solar]

doesn't explain why a list of 6 ints would take more memory than a list of 7 ints tho

[verbal escarp]

python internalizes some values, the list isn't a list of values but of pointers, iirc

[red solar]

they're the same values tho...

[feral cedar]

it would still need to point to those values

[spice pecan]

That part comes from the semantics of BUILD_LIST, which is different from just calling a constructor

[verbal escarp]

i think the internalized objects don't count towards the size of objects, it's complicated

[spice pecan]

It sometimes takes a list that was recently garbage collected, for example

[spice pecan]

sizeof doesn't give you the size of the objects stored by the list

It only gives you the size of the list object itself, excluding whatever is inside

[red solar]

elaborate on how this means the second list with an extra element takes less memory?

[sturdy timber]

!e

from sys import getsizeof as g
print(g([1, 2, 3, 4, 5, 6, 7]), g([1, 2, 3, 4, 5, 6])) 

[fallen slate]

@sturdy timber :white_check_mark: Your eval job has completed with return code 0.

120 152

[feral cedar]

funny

[spice pecan]

I don't know all the semantics and rules, we'd have to look at the source for that. This wouldn't be the case for calling list, it's just the specifics of how that particular case is treated

[verbal escarp]

if python was smart, the answer could be "because it realizes that the small list is part of the longer list and points to the smaller list as part of the longer list" but i doubt python is that smart

[red solar]

can LIST_EXTEND also make the list smaller? like vector::resize() in C++?

[spice pecan]

!e from sys import getsizeof as g; print(g(list(i for i in range(1, 8))), g(list(i for i in range(1, 7)))

[fallen slate]

@spice pecan :x: Your eval job has completed with return code 1.

001 |   File "<string>", line 1
002 |     from sys import getsizeof as g; print(g(list(i for i in range(1, 8))), g(list(i for i in range(1, 7)))
003 |                                          ^
004 | SyntaxError: '(' was never closed

[spice pecan]

ughhhhh

don't type code on your phone kids

[red solar]

lol

[verbal escarp]

lol

[spice pecan]

I'm not sure actually

[red solar]

https://github.com/python/cpython/blob/51ed2c56a1852cd6b09c85ba81312dc9782772ce/Objects/listobject.c#L871 found the source

[spice pecan]

I know dicts can shrink on an insert

[red solar]

    /* Guess a result list size. */
    n = PyObject_LengthHint(iterable, 8);
    if (n < 0) {
        Py_DECREF(it);
        return NULL;
    }

why is it guessing 😂

    /* Cut back result list if initial guess was too large. */
    if (Py_SIZE(self) < self->allocated) {
        if (list_resize(self, Py_SIZE(self)) < 0)
            goto error;
    }

ah it can be smaller though

[spice pecan]

Lists probably either can't, or have a strict rule about it so it doesn't keep shrinking and growing on pop-append pairs

[spice pecan]

Because not every Iterable knows it's length, and some have an estimate, but not the exact length

[red solar]

oh

which one has an estimate? and how do you get it python?

[spice pecan]

length_hint dunder, I don't think there's a top level function for that like there is for length

...

[red solar]

lol fun on phone?

[spice pecan]

Yes

[red solar]

ok that's pretty cool tho, never knew __length_hint__ existed

[spice pecan]

I believe map is one example of an Iterable with a hint

[red solar]

but map supports len?

[verbal escarp]

"in case it lied"

[spice pecan]

Nope, it doesn't

[verbal escarp]

if (m > PY_SSIZE_T_MAX - n) {
/* m + n overflowed; on the chance that n lied, and there really
* is enough room, ignore it. If n was telling the truth, we'll
* eventually run out of memory during the loop.
*/

[peak spoke]

In [130]: operator.length_hint(map(str, "ab"))
Out[130]: 0

In [131]: len(map(str, "ab"))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-131-7ee12642acc8> in <module>
----> 1 len(map(str, "ab"))

TypeError: object of type 'map' has no len()

[spice pecan]

Nevermind, no hint dunder I'm questioning things now

[red solar]

oh map, not dict - idk what map is

[red solar]

oh lmao

i missed that

[lunar trail]

idk what map is

!d map

[fallen slate]

map

map(function, iterable, ...)```
Return an iterator that applies *function* to every item of *iterable*, yielding the results. If additional *iterable* arguments are passed, *function* must take that many arguments and is applied to the items from all iterables in parallel. With multiple iterables, the iterator stops when the shortest iterable is exhausted. For cases where the function inputs are already arranged into argument tuples, see [`itertools.starmap()`](https://docs.python.org/3/library/itertools.html#itertools.starmap "itertools.starmap").

Similar names: 2to3fixer.map

[peak spoke]

it could have a default then (as 0 doesn't really fit even as a hint)

[red solar]

oh that map (i was thinking of a class somewhere)

[peak spoke]

I think a range iterator should have a normal hint

[spice pecan]

>>> help(length_hint)
Help on built-in function length_hint in module _operator:

length_hint(obj, default=0, /)
    Return an estimate of the number of items in obj.

    This is useful for presizing containers when building from an iterable.

    If the object supports len(), the result will be exact.
    Otherwise, it may over- or under-estimate by an arbitrary amount.
    The result will be an integer >= 0.```

Yeah, that's the case

[spice pecan]

Oh, right

[white nexus]

!e ```py
l = ['a', 'b', 'c', 'd', 'e']
print(next(l))

[fallen slate]

@white nexus :x: Your eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 2, in <module>
003 | TypeError: 'list' object is not an iterator

[white nexus]

Iterator and iterables are different???

[peak spoke]

iterables create iterators

[red solar]

slightly off topic, but i just discovered pycharm has presentation mode, so that's cool 🙂

[white nexus]

How?

[red solar]

i can't believe you tried to do this on mobile...

[spice pecan]

I will throw my phone out the window at this rate

[peak spoke]

Iterators keep track of the iteration state, the iterable creates that so it's given to a for loop or the user to go through manually

e.g. you can have 5 iterators at different positions created from a single iterable list

[white nexus]

Interesting

[spice pecan]

!e py from sys import getsizeof as g print('BUILD_LIST:', f'{g([1, 2, 3, 4, 5, 6]) = }', f'{g([1, 2, 3, 4, 5, 6, 7]) = }') print('Unpack range:', f'{g([*range(1, 7)]) = }', f'{g([*range(1, 8)]) = }') print('Constructor call:', f'{g(list(range(1, 7))) = }', f'{g(list(range(1, 8))) = }') print('Constructor call with hints stripped:', f'{g(list(i for i in range(1, 7))) = }', f'{g(list(i for i in range(1, 8))) = }')

[fallen slate]

@spice pecan :white_check_mark: Your eval job has completed with return code 0.

001 | BUILD_LIST: g([1, 2, 3, 4, 5, 6]) = 152 g([1, 2, 3, 4, 5, 6, 7]) = 120
002 | Unpack range: g([*range(1, 7)]) = 152 g([*range(1, 8)]) = 120
003 | Constructor call: g(list(range(1, 7))) = 104 g(list(range(1, 8))) = 112
004 | Constructor call with hints stripped: g(list(i for i in range(1, 7))) = 120 g(list(i for i in range(1, 8))) = 120

[white nexus]

How to create an iter?

Oh

!e
l = ['a', 'b', 'c', 'd', 'e']
print(iter(l))

[fallen slate]

@white nexus :white_check_mark: Your eval job has completed with return code 0.

<list_iterator object at 0x7f3805629090>

[white nexus]

Duh

!e
l = ['a', 'b', 'c', 'd', 'e']
print(next(iter(l)))

[fallen slate]

@white nexus :white_check_mark: Your eval job has completed with return code 0.

a

[white nexus]

Well that's cool

[red solar]

this is... a lot to take in lol

[white nexus]

!d iter

[fallen slate]

iter

iter(object[, sentinel])```
Return an [iterator](https://docs.python.org/3/glossary.html#term-iterator) object. The first argument is interpreted very differently depending on the presence of the second argument. Without a second argument, *object* must be a collection object which supports the iteration protocol (the `__iter__()` method), or it must support the sequence protocol (the `__getitem__()` method with integer arguments starting at `0`). If it does not support either of those protocols, [`TypeError`](https://docs.python.org/3/library/exceptions.html#TypeError "TypeError") is raised. If the second argument, *sentinel*, is given, then *object* must be a callable object. The iterator created in this case will call *object* with no arguments for each call to its [`__next__()`](https://docs.python.org/3/library/stdtypes.html#iterator.__next__ "iterator.__next__") method; if the value returned is equal to *sentinel*, [`StopIteration`](https://docs.python.org/3/library/exceptions.html#StopIteration "StopIteration") will be raised, otherwise the value will be returned.

[spice pecan]

Basically:

• If you use a literal (including unpacking), dark magic happens, I still need to check the source code
• If you create a new list from something with a (correct) hint, there are no surprises, it allocates less for a smaller one
• If you strip the hints, it follows standard amortization rules, so in this case they turned out to be the same size

Stripping the hints (in my case, via using a generator expression) should technically make it equivalent to l = []; l.extend(...)

[red solar]

unpacking counts as using a literal?

LOAD_CONST 2 ((1, 2, 3, 4, 5, 6, 7))
tbh i'm surprised Python can LOAD_CONST a whole ass tuple (which is what's generated from the literal)

[spice pecan]

I'd put them in a similar category

You have a list literal, that just happens to contain an unpacked iterable

[red solar]

hmm

[spice pecan]

>>> from sys import getsizeof as g
>>> l = []; l.extend(range(1, 7))
>>> g(l)
152
>>> l = []; l.extend(range(1, 8))
>>> g(l)
120```
Ooooookay, I have more questions

[peak spoke]

the literal should be equivalent to an extend

[red solar]

yeah the literal is BUILD_LIST, LOAD_CONST, and LIST_EXTEND

[spice pecan]

Hmm, it would make sense for both LIST_EXTEND and list.extend to be completely equivalent

[peak spoke]

if I had to guess I'd say it grabbed a free list out of somewhere, although I'm not sure if that'd be so consistent

[surreal sun]

Are these integers cached? I’m also kinda curious

[spice pecan]

Integers -5 through 256 are cached, I believe

Them being cached has absolutely no influence on the list's size though

[surreal sun]

Ah

[spice pecan]

https://github.com/python/cpython/blob/51ed2c56a1852cd6b09c85ba81312dc9782772ce/Objects/listobject.c#L2881-L2889

[fallen slate]

Objects/listobject.c lines 2881 to 2889

static PyObject *
list___sizeof___impl(PyListObject *self)
/*[clinic end generated code: output=3417541f95f9a53e input=b8030a5d5ce8a187]*/
{
    Py_ssize_t res;

    res = _PyObject_SIZE(Py_TYPE(self)) + self->allocated * sizeof(void*);
    return PyLong_FromSsize_t(res);
}```

[spice pecan]

This is the code responsible for getting the size of a list, if anyone's interested

So it's simply its own size + the amount of pointers it can store

[red solar]

lol

also didn't know you could select multiple lines like that

[spice pecan]

I keep forgetting the syntax for that :P

I don't know if you can do it from GH directly, and if you were around to see my previous attempts to use this command, you could definitely have a laugh or two

[grave jolt]

you can select a range with Shift+clicking on lines

[red solar]

huh dope

[spice pecan]

Ohhhhhh

That makes it significantly easier

[red solar]

how were you doing it?

[spice pecan]

Getting a permalink to a line and manually adding -L...

[red solar]

lmao

[spice pecan]

Having to remember that they're separated by a - specifically, and the second num also has L prepended, was a chore, and not one I was good at

[red solar]

i wanna see how you're gonna do shift+click on mobile :p

[spice pecan]

if i really wanted to I could just connect my bluetooth keyboard and break the system

but that would involve getting out of bed and I don't support that

[red solar]

ahhh, now this all makes sense

i feel you lol

[spice pecan]

This is getting a bit off-topic, we should probably move this convo

[red solar]

https://github.com/doodspav/atomics/blob/devel-ffi/ARCHITECTURE.md any feedback? (more on the design of the project as described in this file, but feedback on the file itself is also welcome)

also gotta say, the formatting looked nicer when rendered by PyCharm :/

(bear in mind this is ARCHITECTURE, not README, and that README isn't up to date yet)

[red solar]

(it's a single small file, i promise it won't bite)

[halcyon trail]

if classes aren't meant to be constructed by the user, that's not something that should need to be documented

make them private (prefix with _) or make them ABC's

self documenting

[red solar]

i want them for type hinting tho :/

and nothing will break if they do construct the class

maybe i should clarify that they're not intended to be constructed in the sense that they're just no useful, not that they will necessarily do damage

although lemme rewrite the README first

[red solar]

can an ABC define full instance methods?

[halcyon trail]

I'm pretty sure

Some ABCs provide default implementations of things

[elder blade]

I didn't get past the first paragraph because I am trying to figure out the point of patomic..

But all I can find are huge files with hundred layer macros https://github.com/doodspav/patomic/blob/devel/src/impl/msvc/msvc.c

[red solar]

ah :/ was hoping one wouldn't need to delve into that :/

lemme finish the readme, and then see if you still need to go to patomic

[elder blade]

Can I get a Tl;Dr of what the point of them are?

[red solar]

which ones?

the atomics stuff, or the patomic stuff?

[elder blade]

Both, I don't understand why it exists basically

[red solar]

from threading import Thread

i: int = 0


def fn(n: int) -> None:
    global i
    for _ in range(n):
        i += 1


if __name__ == "__main__":
    total = 10_000_000
    # run threads to completion
    t1 = Thread(target=fn, args=(total // 2,))
    t2 = Thread(target=fn, args=(total // 2,))
    t1.start(), t2.start()
    t1.join(), t2.join()
    # print results
    print(f"a[{i}] == total[{total}]")

this isn't correct - a will not equal total

import atomics
from threading import Thread


def fn(ai: atomics.INTEGRAL, n: int) -> None:
    for _ in range(n):
        ai.inc()


if __name__ == "__main__":
    # setup
    a = atomics.atomic(width=4, atype=atomics.INT)
    total = 1_000_000
    # run threads to completion
    t1 = Thread(target=fn, args=(a, total // 2))
    t2 = Thread(target=fn, args=(a, total // 2))
    t1.start(), t2.start()
    t1.join(), t2.join()
    # print results
    print(f"a[{a.load()}] == total[{total}]")

this code will work correctly 🙂

@elder blade

(and then also a version that accepts an external buffer, so you can use it in shared memory and stuff, i have a multiprocessing example if you want)

[elder blade]

How does that work? Doesn't the GIL ensure that the first codeblock should work?

[red solar]

no - the GIL can switch threads between any 2 bytecodes

and += on an int is like 4 bytecodes

[elder blade]

I see alright this is cool

[red solar]

ty

[elder blade]

@red solar I've read it but I am still confused. See I don't know the developer reasoning behind the 3 different classes and that probably meant that nothing really stuck with me.

For the file tree, it seems like it's wrong? There is no atomics.atomics folder?

[red solar]

hmm the file tree reads as if you had installed it, so atomics.src.atomics.atomic -> atomics.atomic

ig i should also make that clear

[elder blade]

But even then that doesn't exist

[red solar]

ah crap

you're right

[elder blade]

Did you mean atomics._impl.atomic?

[red solar]

yeah lmao

is there a warning that's the opposite of deprecated? like "it's implemented but don't use it yet"?

[elder blade]

Usually you'd just not include that in a release, or through documentation say that it's in alpha/beta/pre-release.

[red solar]

By default, .is_valid calls .is_valid_recommended. The class Alignment
also exposes .is_valid_minimum. Currently, no atomic class makes use of the
minimum alignment, so checking for it is pointless. Support for it will be
added in a future release.

is this good enough documentation for that?

[elder blade]

I'd say that nicely encapsulates that it's not really meant to be used

[red solar]

Yay 🙂

btw do you know C?

[elder blade]

A little little bit

From Cython experience pretty much

[red solar]

hmm... do you know what alignment is?

[elder blade]

For structs?

[red solar]

yeah

[elder blade]

Yeah

[red solar]

https://github.com/doodspav/patomic/blob/devel/include/patomic/types/align.h can you tell me if this comment makes sense to you you? (it's a comment on a 3 element struct at the bottom)

[elder blade]

Oh then I don't know alignment. I thought you meant alignment as in how bits are aligned in a struct for minimum memory wasted.

[red solar]

this is that, but complicated :/

but ok ig i can't just reference this in my atomics documentation :/

[red solar]

https://github.com/doodspav/atomics/tree/devel-ffi#readme I updated the readme (if anyone's still on)

[white nexus]

;-;

[unkempt rock]

How can one profile a python program to determine most common Optcodes and Branches for computer architecture testing?

[prime estuary]

Well, the primary hook is sys.settrace() - you provide a function, and it'll be called for every line of code that's run. You can then write whatever statistic logic, or do debugging stuff. Of course there are lots of tools already written that do so.
https://docs.python.org/3/library/sys.html#sys.settrace

[sturdy timber]

You might find this helpful https://github.com/faster-cpython/tools/blob/main/scripts/count_opcodes.py

[verbal escarp]

you could use do warnings.warn("This function is in active development and may be changed without notice.", FutureWarning)

or maybe UserWarning

Changed in version 3.7: Previously DeprecationWarning and FutureWarning were distinguished based on whether a feature was being removed entirely or changing its behaviour. They are now distinguished based on their intended audience and the way they’re handled by the default warnings filters.

i guess that's the question you need to ask - who's your audience

if you want to warn other devs, you could subclass DeprecationWarning maybe

btw, i was thinking about the idea of using subprocess-interpreters to fully isolate certain packages - it reminds me of Eiffel or other approaches to component-based software engineering. it could allow for crash-resilient architectures where an attack could only crash a portion of the software but it would automatically reload the part that got killed

also it could allow for snapshots of critical parts and resume operation from those (thinking of numpy encountering division by zero or buffer overflows) without affecting the proper functioning of the whole

you could take a snapshot of the whole interpreter, write it to disk, if it crashed, load it from disk as if nothing happened

the rest of the program wouldn't need to care

of course it doesn't absolve bad software design and proper testing, but it would make software much more resilient against scenarios that couldn't be anticipated

[elder blade]

When would you take the snapshot though?

[verbal escarp]

the package would have to signal "i'm in a safe state" to the main program somehow

[sturdy timber]

I can't think of any case that would be helpful, surely you're just saving a state that's about to crash, so if you 'resume' it from the snapshot it would just crash again?

[verbal escarp]

not really, just think of a server that has a component that might be vulnerable (to certain inputs or just random bad luck) so you isolate it, the whole thing is stored while it is in a known stable state (for example before any actual input is handled at all)

and then it could be restored at any time if any combination of inputs caused a crash

[sturdy timber]

ah

[lusty scroll]

if modules were picklable, and you could pickle all of them including the __main__ module, that might be possible

[verbal escarp]

couldn't you just write the memory of the interpreter to disk as a whole?

[gleaming rover]

that would work if you weren't holding external resources

which you are likely to be

[grave jolt]

and, well... objects hold references to other objects. that is, memory addresses. how are you going to reconstruct them?

[flat gazelle]

well, that's possible if they are all pointers to python types. But given an arbitrary pointer made with ctypes, it is indeed impossible

[grave jolt]

it's just that - how do you know what 8-byte chunks of this heap are pointers? if it's some kind of c extension or some tricky stdlib type, you can't be sure

[flat gazelle]

hmm, yeah, IG the interpreter is more complex than just a graph of objects

[verbal escarp]

i'd imagine if the package is basically self-contained and references are only accessed via the module-attributes, they could be effectively proxied and isolated that way

in justuse we take a similar approach to swap out the implementation of single modules for reloading, but packages are way more complex

still wonder how to communicate with those isolated objects in a totally different interpreter..

i've never done anything like that before

[raven ridge]

Not all of the resources owned by an application are held in the application's memory. Things like which files and sockets are open and what physical memory pages are mapped to each virtual memory page are held in kernel memory. So, restoring the process's memory to an earlier snapshot would only restore some parts of the program's state, not all of it.

If after you save and before you restore the interpreter memory the program runs os.close(0), then calling input() after you restore will fail. If it runs munmap, then accessing memory after the restore might fail. If it runs sock.close() on a socket, then calling any method on that socket would fail after a restore. Etc, etc, etc.

[lusty scroll]

that would need to be accounted for by the serialization of the types holding resources, I think

like FileIO objects would need to store the path and position in the file, then they could reopen that file and seek to the same position upon deserialization, perhaps

the file descriptor would be different but hopefully that's just an implementation detail

[raven ridge]

that's not possible with sockets.

[flat gazelle]

not even with files, once you close a file, that file could no longer exist, have different permissions etc

[raven ridge]

Yeah. It's not really possible with files, but it's very not possible with sockets. Once a server closes a socket, the client gets a message saying "no more data is coming", and vice versa. You can't undo that.

[lusty scroll]

well, you're just SOL in either case 😄

[raven ridge]

It's not possible with anonymous memory mappings, either. The "anonymous" part is that they have no identifier referring to them, so there's no way to re-open the mapping once it's closed, because you have no way of identifying which one. (Not to mention that if you close the last reference to it, the kernel just destroys it entirely)

[lusty scroll]

how often do simple interactive sessions use anonymous mappings and long-lived sockets

[raven ridge]

very often. Anonymous mappings are one of the main ways that memory gets allocated - large chunks of memory are almost always allocated by anonymous memory mappings under the hood by malloc

[lusty scroll]

still, that would be a problem for any process snapshotting / migration mechanism

[raven ridge]

and stdin/stdout/stderr are examples of long lived pipes, which have exactly the same characteristics as long lived sockets.

[lusty scroll]

well we're talking at the python level, aren't we

there might be some complexities to be worked atound, but they are absolutely solveable with a little cleverness

you could reopen the socket / mapping / whatever, depending on the specific need (or use some other alternative that is cleanly serisalizable)

[raven ridge]

That's not true - I just explained that you cannot, and why not.

when you close a pipe or socket, the other end is immediately notified. You cannot "undo" that operation. You could delay that - don't really close, and instead do a fake close which you could undo later - but if you do that, it changes the behavior of the program in an observable way. Likewise for writing to a file or socket or pipe. Or reading from a socket or pipe.

Basically, your idea does not work with any type of external resource.

[lusty scroll]

wrong, you can reopen the socket

[raven ridge]

So, your application is a server, and it has accepted a connection from the client. You close that socket, and the client moves on with its life. Later you decide you want to reopen that socket. How do you do that? How does the client side get reattached?

[lusty scroll]

depends on what the point of the socket is

if it's for an HTTP request, you could just reissue

[raven ridge]

that only works for idempotent requests.

reissuing a request for a POST, let's say, isn't safe.

but also - you're flipping things around. in my case, I was saying that your application - the one you wanted to suspend - was the server, not the client.

[lusty scroll]

oh, well that's even easier

we don't care about it anymore in your scenario

if the client is disconnected anyway, there's npo point bothering with the socket at all

[raven ridge]

but then you haven't restored the state

[lusty scroll]

just make it a closed "dummy socket" 😁

[raven ridge]

the state before was "We're in the middle of responding to this client". The state after the restore is "oops, client gone"

the same case happens with HTTP requests on the client side, though. The state when you snapshot might be "we're in the middle of POSTing a file". The state after the restore would be "we have a brand new connection to this server". Not the same thing.

[lusty scroll]

so?

it just has to work

[raven ridge]

so... maybe I don't understand your goal at all, then. If you're OK with the idea that after "restoring", every externally held resource, from sockets to pipes to files to memory maps to locks and semaphores, might be in a different state than they were in when you snapshotted, then - what's the point of the snapshot?

[lusty scroll]

when you restore the system, you're not restoring anything that was running

just the "static state" was what I mean, like what's available to you when python is waiting for a new input... that's what I was contemplating

like if you wanted to restore a jupyter notebook for example so yoi could add code to it without losing everything

[raven ridge]

there's no such thing as "static state", because the parts that you're considering static state - like which objects exist - refer to parts that are externally held dynamic resources. A python socket.socket object is "static state" in the sense that the interpreter owns that object completely, but that object owns a file descriptor, and the kernel holds a table mapping that file descriptor to an open TCP connection. If you restore half of that but not the other half, the program is referring to kernel resources it no longer owns, and is no longer in a stable, correct state

[lusty scroll]

in some cases, sure ... then you leave that to lower level inplementation and restore objects at a higher level

there'd be some things that can't be serialized, I'm sure, but a lot can be... enough to consider it useful

[red solar]

when you say serialise... are you relying on pickle?

[raven ridge]

well, I'm quite convinced I'm right, but if you're convinced you're right, build a POC. If I'm right, I should be able to segfault that POC pretty easily.

[lusty scroll]

not pickle as it exists today, but one that can do many more things

[red solar]

i still don't see how you're going to serialise extension classes whose implementation is completely opaque

[lusty scroll]

I can segfault python easily already... what does that proove?

[raven ridge]

if I can segfault Python by using normal objects in ways that their contracts are supposed to allow after the restore, then it proves that your restore - didn't.

[lusty scroll]

alright, I'm not one to give up on things just because you insist it can't be done

[raven ridge]

I expect, for instance, that it would be pretty trivial to get into a state where iterating over a Python bytes object crashes the interpreter. That would happen because the memory owned by the bytes object was part of an anonymous mapping, and if the mapping is freed by the garbage collector after the snapshot and before the restore, then the restore would result in a bytes object that refers to memory that has been freed, and iterating over it will segfault.

either that, or you pay a pretty serious tradeoff: not allowing any object to be garbage collected after the snapshot and before the restore.

[unkempt rock]

is flask good to learn?

[raven ridge]

not really on topic for this channel, but, yes - it's a good microframework for making web services and web sites.

[unkempt rock]

Aight soz, ty

[verbal escarp]

well, i think it's very similar to how hot-reloading of a module is safe and nice as long as you only call functions and not reference state from it

if you restrict componentization to parts of the software that don't need access to external resources, i don't see a problem

you could even decouple message queuing from managing the sockets or files

so even if an attacker was killing the message queue by some exploit, it wouldn't require to drop all connections, just reload the queue from a safe state

[verbal escarp]

looking forward to it!

[white nexus]

is there a way to catch sys.exit()

using runpy to run a script i wrote from another script

[red solar]

Exit from Python. This is implemented by raising the SystemExit exception, so cleanup actions specified by finally clauses of try statements are honored, and it is possible to intercept the exit attempt at an outer level.

so except SystemExit:?

[white nexus]

thanks

wait what doc page is that

[red solar]

https://docs.python.org/3/library/sys.html#sys.exit

(i'm sure the bot has a way of getting it too)

[verbal escarp]

atexit

maybe?

!d atexit

[fallen slate]

atexit

The atexit module defines functions to register and unregister cleanup functions. Functions thus registered are automatically executed upon normal interpreter termination. atexit runs these functions in the reverse order in which they were registered; if you register A, B, and C, at interpreter termination time they will be run in the order C, B, A.

Note: The functions registered via this module are not called when the program is killed by a signal not handled by Python, when a Python fatal internal error is detected, or when os._exit() is called.

Changed in version 3.7: When used with C-API subinterpreters, registered functions are local to the interpreter they were registered in.

[white nexus]

interesting

[raven ridge]

atexit is more for libraries, while except SystemExit is something an application might do

though if the reason for catching it is to do some sort of cleanup, a try / finally or a context manager are better options.

[white nexus]

ngl I don't have a reason for it at the moment, but I may have

tldr I'm making a script which wraps a script i wrote ;-;

[white nexus]

well i don't even know what is real anymore

[prime estuary]

What you should do is change that script you want to wrap so it has a function that accepts all the options required and returns the result - the command line processing and sys.exit call should then be in a seperate function, or just in the if __name__ == '__main__' block.

Then you can call it directly.

[halcyon trail]

Basically every script I write has if name == main main()

[white nexus]

i'm using runpy and want it to be named __main__
tldr i'm writing a script to wrap my scripts so i have a launcher for my scripts ¯_(ツ)_/¯

[halcyon trail]

And main is a function that just parses arguments and calls a real function

That way you can use the real function programmatically if needed

[white nexus]

ye

i added my parse arguments under the if __name__ == ... thing

and then my main function does what the script does

so i can just provide different args to main() and not have to worry about argparse/click

[white nexus]

I made this work, but now I want to rewrite it into argparse

[halcyon trail]

I stopped doing the parse arguments and exception catching there because it's technically all global scope

Every variable you declare in that if name == main block is a global

And triggers warnings about shadowing quite often

[white nexus]

😳

where do i get those warnings

[mild flax]

!ban 645500518617317386 racism

[fallen slate]

:incoming_envelope: :ok_hand: applied ban to @stable lintel permanently.

[halcyon trail]

@white nexus pycharm

[white nexus]

ah

[halcyon trail]

But yeah even if it is just part of a script, it should feel pretty gross that every local you create there is actually a global

And there's no downside to have a main function, it's two extra lines

[white nexus]

@halcyon trail discord-modmail/modmail#111 this what I just made

[neon trout]

GitHub

scripts: add wrapper to run scripts by invoking the scripts module

[white nexus]

https://github.com/discord-modmail/modmail/blob/scripts-wrapper/scripts/__main__.py

[halcyon trail]

Don't think I really understand the purpose

[white nexus]

as of right now, its just a launcher for whatever scripts i put in it

[livid mason]

using setuptools (the standard library) can we have a monorepo?
in the js ecosystem, projects like gatsby have multiple packages per repo:
https://github.com/gatsbyjs/gatsby

is it possible to do something similar in Python? I know we have namespace
packages, but that's not what I'm looking for!

[elder blade]

Yes, see from the packager's point of view your projects folder that houses several projects with their own git repository is no different from a single project and repository.

The same way you need a package.json for each package, for Python using setuptools you need a setup.py for each package.

[livid mason]

okay, then how can I upload all packages at once?

[elder blade]

There's also one downside with this approach. For someone to install your repository directly using pip they also need to specify the specific setup.py to install

[elder blade]

That on the other hand I don't know, it would be supported for whatever software you use to upload packages

[unkempt rock]

how can i make in py sound human voice using speechrecognition module

[deft pagoda]

this is the wrong channel for your question, unfortunately

[unkempt rock]

so can you help me pls @deft pagoda

[deft pagoda]

i already offered as much help as i can, i said i've never used speech recognition before

[lusty scroll]

this is really convenient, I also do scripts this way

also if there are no command line arguments and name is __main__ i add an argument for "/dev/stdin" to sys.argv

that way it can act as a simple filter in a shell pipeline if it's a text/blob processing script

[quaint fiber]

hey guys ! i'm wondering if there is an inverse dunder method for __contains__

[elder blade]

You want a "not contains" that is not the output from __contains__ inverted?

[quaint fiber]

na i want a "is contained in"

sorry for being unprecise, by "inverse" i was meaning something like __pos__ to __add__

[unkempt rock]

no, that's not possible - unless both those types were defined by you and you make their dunder contains check the inverse case

...which is just really unexpected behaviour

[quaint fiber]

ye, and in my usecase it'll absolutely never happen

i guess i'm gonna use another operator then

or a .isin method i guess

[lusty scroll]

you mean like a __ncontains__ ?

[quaint fiber]

not "doesn't contain" but "is contained in"

like if i do A in B it triggers A.something instead of B.__contains__

[verbal escarp]

please give some example how you imagine it should work

[quaint fiber]

well i just did

[verbal escarp]

what's the expected result?

[quaint fiber]

exact same as __contains__
it's just that i have a custom class A and B isn't custom

[flat gazelle]

class InEverything:
    def __rcontains__(self, container):
        return True
assert InEverything() in [1, 2, 3]

is this the idea?

[quaint fiber]

yes exactly

[flat gazelle]

depending on the usecase, if you can make yourself compare equal to one of the elements, it should work, but it is not as powerful as a dunder would be

but to my knowledge, no such thing exists

[quaint fiber]

the only way would be to overwrite __contains__ in builtins, with stuff such as forbiddenfruit.curse
but i think that's as worse as a practice it can be

[verbal escarp]

couldn't you subclass list and add the method and use it like InEverything in L(1,2,3)?

[quaint fiber]

I could do that too ye but that would become weird versus just making a isin method

the context is i'm making a data object package for an API

[verbal escarp]

you could abandon methods and simply do it functional

that's probably the most straightforward solution

[quaint fiber]

well the whole concept of it is to be object oriented

[verbal escarp]

sigh

[quaint fiber]

i like functionnal stuff dw, i just think it doesn't apply well to my usecase

[verbal escarp]

you could hide the functional style by making a dummy 😉

like class Matcher: @staticmethod def does_contain(a, b)

🤣

i just found a problem with Enum

E = Enum("asdf", "1 2")

doesn't raise a WTFError as it should

but instead simply won't work in any way

>>> E._member_map_
{'1': <asdf.1: 1>, '2': <asdf.2: 2>}

simply no way to access the members

[flat gazelle]

you can still produce them with the getitem and iter impl on an enum class.

In [10]: E['1'] in E, list(E)
Out[10]: (True, [<asdf.1: 1>, <asdf.2: 2>])```

[verbal escarp]

oh, true

heh

[spice pecan]

I believe getattr should work too

[verbal escarp]

another nice way to make coworkers furious

setattr also allows non-identifiers, that's funky

shouldn't that be banned or something?

[flat gazelle]

it would be a major nuisance to properly sanitize

[pliant tusk]

they could call str.isidentifier on the argument but that could break older code

[flat gazelle]

but that's kind of a half measure considering you could still trivially create such attributes my manipulating dict

[grave jolt]

I can imagine that there's also a performance cost to checking it

[verbal escarp]

at least in Enum it wouldn't - is there any legit usecase where you'd want a non-identifier as an enum member?

for me dotattributes are the one big reason to even use enums

[flat gazelle]

is there a meaningful reason to forbid this though?

you can't really do it by accident

[verbal escarp]

you could typo and accidentally have a number as first char

[flat gazelle]

you could also typo and just have the wrong name as the enum member

[verbal escarp]

i think it's worse for chars that are not as obvious as numbers

some unicode chars aren't allowed in identifiers but are also not really special

or something like ```python

§ = 23
File "<pyshell>", line 1
§ = 23
^
SyntaxError: invalid character '§' (U+00A7)

shrug

[undone stump]

advanced unable to understand anything

[verbal escarp]

you'll get used to it

[undone stump]

i hope so!

[verbal escarp]

so, i was just revisiting the buffet table in justuse again, looked at it through the patma lense

behold!

def buffet_table(case, kwargs):
    case_func = {
        (0, 0, 0, 0): lambda: ImportError(Message.cant_import(**kwargs)),
        (0, 0, 0, 1): lambda: pimp._pebkac_no_version_no_hash(**kwargs),
        (0, 0, 1, 0): lambda: pimp._import_public_no_install(**kwargs),
        (0, 1, 0, 0): lambda: ImportError(Message.cant_import(name)),
        (1, 0, 0, 0): lambda: ImportError(Message.cant_import(name)),
        (0, 0, 1, 1): lambda: pimp._pebkac_no_version_no_hash(**kwargs),
        (0, 1, 1, 0): lambda: pimp._import_public_no_install(**kwargs),
        (1, 1, 0, 0): lambda: ImportError(Message.cant_import(name)),
        (1, 0, 0, 1): lambda: pimp._pebkac_no_hash(**kwargs),
        (1, 0, 1, 0): lambda: pimp._ensure_version(pimp._import_public_no_install(**kwargs), **kwargs),
        (0, 1, 0, 1): lambda: pimp._pebkac_no_version(**kwargs),
        (0, 1, 1, 1): lambda: pimp._pebkac_no_version(**kwargs),
        (1, 0, 1, 1): lambda: pimp._pebkac_no_hash(**kwargs),
        (1, 1, 0, 1): lambda: pimp._auto_install(**kwargs),
        (1, 1, 1, 0): lambda: pimp._ensure_version(pimp._import_public_no_install(**kwargs), **kwargs),
        (1, 1, 1, 1): lambda: pimp._auto_install(
            func=lambda: pimp._ensure_version(pimp._import_public_no_install(**kwargs), **kwargs), **kwargs
        ),
    }[case]
    return result

this is the thing for python <3.10

this is the same thing - with pipes, since it's possible outside of lambdas:

def buffet_table(case, kwargs):
    match case:
        case _, _, 0, 0: return ImportError(Message.cant_import(**kwargs))
        case 0, _, 1, 0: return _import_public_no_install(**kwargs)
        case 1, _, 1, 0: return _import_public_no_install(**kwargs) >> _ensure_version(**kwargs)
        case 0, 0, 0, 1: return _pebkac_no_version_no_hash(**kwargs)
        case 0, 0, 1, 1: return _import_public_no_install(**kwargs) >>_auto_install(**kwargs)
        case 1, 0, _, 1: return _pebkac_no_hash(**kwargs)
        case 0, 1, _, 1: return _pebkac_no_version(**kwargs)
        case 1, 1, 0, 1: return _auto_install(**kwargs)
        case 1, 1, 1, 1: return _import_public_no_install(**kwargs) >> _ensure_version(**kwargs) >>_auto_install(**kwargs)

i never expected that patma and pipes could make this thing so much nicer

[deft pagoda]

i've been doing this all over as well, it looks a lot cleaner

[noble kelp]

agree or disagree, any nested .get on a dictionary should include a default of an empty dict until the final get in the chain

❎ foo.get('bar').get('baz')
✔️ foo.get('bar', {}).get('baz')

[unkempt rock]

cleaner? wtf is that?

[noble kelp]

if you're talking about @verbal escarp 's message, i think it's just swapping from a dict mapping inputs to functions to using the new match case feature

[unkempt rock]

➡️ 👂 ➡️ 🧠 =

[grave jolt]

well... if you're expecting some of the gets to return None, you kinda have no choice

[halcyon trail]

it doesn't really make any sense to do nested get without a default {}

[grave jolt]

although... why do you have deeply nested dicts?

[halcyon trail]

however it does make sense to do nested []

so really, that's the choice; nested [] or nested .get(..., dict())

[noble kelp]

i wouldnt call two or three chained gets terribly deeply nested, fairly common for response jsons to be that complex

but yeah i needed the sanity check, it feels like we shouldn't ever have a bare .get

in that case

[static bluff]

What are some good books on:
The Theory of Programming (traditional)
The Theory of Programming (modern, 21st century)
How Programming Languages Work
How Python Works
Other

First on my list is the dragon book. Any others?

[halcyon trail]

To be honest I think it should be a separate function anyway

[grave jolt]

are you... returning exceptions?

[halcyon trail]

_MISSING = object()
def get_nested(top_dict, *keys, default=_MISSING):
    d = top_dict
    for k in keys:
        if k in d:
            d = d[k]
            continue
        if default is _MISSING:
            raise KeyError(f"Error, nested key {k} not in nested dict!")
        return default
        

nested_get(foo, 'bar', 'baz')

i hav a function like this in my codebase and I"d imagine many people do

[noble kelp]

yeah agreed, good call. thanks for sharing!

[lament sinew]

That's how lots of modern languages deal with errors. Just return them as a value. A type checker will make sure you always handle them.

[grave jolt]

Right, but in Python it's a very strange thing to do most of the time. Language with this style of error handling usually have tools for working with these error values.

For example, Haskell has do notation.

And Go... well, let's not talk about Go

[lament sinew]

rust

[raven ridge]

The only place I think that this pattern is reasonable in Python is moving exceptions from one thread to another.

[grave jolt]

Rust has:

• Option and Result built-in
• methods like map, map_err, and_then, or, or_else etc. on Option and Result
• match as an expression
• ?-notation
• tagged unions which are good for defining custom error types

I mean, we have dry-python stuff

but it looks a bit foreign tbh, and only works with mypy

[lament sinew]

you can get away with just pattern matching and unwrap

[raven ridge]

Forcing users to do that in a language that idiomatically uses exceptions is unwise

[lament sinew]

they can use it internally

[grave jolt]

I can't imagine how this would look with match

    fn deserialize_seq<V>(self, visitor: V) -> Result<V::Value, Self::Error>
    where
        V: serde::de::Visitor<'de>,
    {
        self.skip_spaces();
        if self.next_char()? != '[' && self.next_char()? != ' ' {
            return Err(CmdError::ExpectedLeftBracketBeforeList);
        }

        let value = visitor.visit_seq(ManyValues::new(self))?;

        self.skip_spaces();
        if self.next_char()? == ']' {
            Ok(value)
        } else {
            Err(CmdError::ExpectedRightBracketAfterList)
        }
    }

[deft pagoda]

i asked about negation pattern matching already

they didn't include it

[verbal escarp]

indeed i am

[raven ridge]

Sure. Forcing un-idiomatic interfaces on people is bad. If someone keeps the un-idiomatic stuff to the guts of their library, that's their call, but putting it into their library's interface is hostile to users.

[grave jolt]

I guess you can implement this with generators, something like this: ```py
@result
def deserialize_seq(self, visitor):
self.skip_spaces()
if (yield self.next_char) != '[' and (yield self.next_char()) != ' ':
return Err(CmdError.ExpectedLeftBracketBeforeList())

value = yield visitor.visit_seq(ManyValues(self)) 

self.skip_spaces()
if (yield self.next_char()) == ']':
    return Ok(value)
else:
    return Err(CmdError.ExpectedRightBracketAfterList())

actually the generator version doesn't look bad

so if you're ok without typing it is a way

[verbal escarp]

@grave jolt we use a function ```python
def _fail_or_default(exception: BaseException, default: Any):
if default is not Modes.fastfail:
return default # TODO: write test for default
else:
raise exception

to catch any exceptions during installation/import and return a default if the user specified one or raise the exception if no default was given

[deft pagoda]

maybe:

match self.next_char():
    case ']':
         ...
    case char if char not in '[ ':
         ...
    case _:
         ...

[grave jolt]

see, if you use match, there will be nested matches

like, 4 of them

[verbal escarp]

mainly to simplify optional imports

[deft pagoda]

if you say so

[verbal escarp]

so the user doesn't have to deal with the exceptions if they don't want to

[lusty scroll]

this would work nicely with match

[grave jolt]

?

[verbal escarp]

you can do something like graphing = use("some_optional_graph_lib", default=use("matplotlib")) or somesuch and it will first attempt to import the optional one and if that fails for any reason, fall back to one you know will work

[lusty scroll]

if it's returning an Err or some non-Err, you would have all the pieces you'd need without wrapping in a try/except

just looking at the function, not knowing the way mypy's API works

[grave jolt]

I don't think I understand you. Can you perhaps show how the function would look like with match?

[lusty scroll]

oh, I mean if one were to call your function, like

match deserialize_seq(...):
    case Err(...): ...
    case Ok(...): ...

[grave jolt]

calling is easy, I meant the body itself

[lusty scroll]

haha, I don't know, then

hu, no highlighting 😕

[grave jolt]

def deserialize_seq(self, visitor):
    self.skip_spaces()
    match self.next_char():
        case Err(e):
            return Err(e)
        case Ok('['):
            match self.next_char():
                case Err(e):
                    return Err(e)
                case Ok(' '):
                    return return Err(CmdError.ExpectedLeftBracketBeforeList())
                case _:
                    pass
        case _:
            return Err(CmdError.ExpectedLeftBracketBeforeList())

    match visitor.visit_seq(ManyValues(self)):
        case Err(e):
            return Err(e)
        case Ok(value):
            self.skip_spaces()
            match self.next_char():
                case Ok(']'):
                    return Ok(value)
                case Err(e):
                    return Err(e)
                case _:
                    return Err(CmdError.ExpectedRightBracketAfterList())
    

beautiful innit

I guess you could simplify it to ```py
def deserialize_seq(self, visitor):
self.skip_spaces()
match (self.next_char(), self.next_char()):
case (Ok('['), Ok(' ')):
pass
case Ok(_):
return Err(CmdError.ExpectedLeftBracketBeforeList())
case Err(e):
return Err(e)

match visitor.visit_seq(ManyValues(self)):
    case Err(e):
        return Err(e)
    case Ok(value):
        self.skip_spaces()
        match self.next_char():
            case Ok(']'):
                return Ok(value)
            case Err(e):
                return Err(e)
            case _:
                return Err(CmdError.ExpectedRightBracketAfterList())

[lusty scroll]

all parsers should strive to immitate that

[grave jolt]

hm?

I meant beautiful as sarcasm if that was not clear 👀

[lusty scroll]

no it's not really sarcasm 😄

it's a lot more readable than most parser generators would give you

[grave jolt]

I think the yield version is much better

well... you're not supposed to read the generated code, are you?

it's supposed to be fast, not readable

[lusty scroll]

maybe not, but i end up reading it anyways if i have to debug a parse error

[grave jolt]

true

[lusty scroll]

is there a reason this would be slower than what's usually done

[grave jolt]

not sure

I guess the parser could generate Rust code if it wanted speed 😉

[lusty scroll]

i guess it depends on how complex the cases are

that's true

[white nexus]

so uh, what happens if you don't have an __init__.py but there is a __main__.py

i have noticed that i can execute it as a module, with python -m

but not sure of anything it cannot do

in fact, i don't see a problem with not having an __init__.py

it seems like if i import it without an init py it ends up being a namespace

[main ginkgo]

!pep 420

[fallen slate]

**PEP 420 - Implicit Namespace Packages**

Link

Status

Final

Python-Version

3.3

Created

19-Apr-2012

Type

Standards Track

[main ginkgo]

yeah, per pep 420 you dont necessarily need init.py but there are some differences worth noting. Portions of namespace packages need not all come from the same directory structure, or even from the same loader. Regular packages are self-contained: all parts live in the same directory hierarchy. is the most notable i'd think.

not really sure how namespace packages handle __main__.py e.g. if you had multiple. probably just use the first one found?

[paper echo]

i kind of wish they would have picked a standard "marker" along the lines of import pkg_resources ; pkg_resources.declare_namespace instead of making them entirely implicit

but i guess they had their reasons, maybe related to the concerns about linux distros:

However, Linux distribution vendors (amongst others) prefer to combine the separate portions and install them all into the same file system directory.

[paper echo]

the spec in the pep suggests that it's whatever comes up first in the search

If not, but <directory>/foo.{py,pyc,so,pyd} is found, a module is imported and returned. The exact list of extension varies by platform and whether the -O flag is specified. The list here is representative.

i assume __main__ qualifies as one such foo

i never really considered that you could have __main__.so

[lusty scroll]

this could be "implicit namespace packages" -- I'm not sure if every tool understands them vs. just lookimg for an __init__.py

[lusty scroll]

I have seen python looks for those, if you do strace -e trace=file python ....

[white nexus]

serious question: does the help method in the repl have a python implementation?

!d help

[fallen slate]

help

help([object])```
Invoke the built-in help system. (This function is intended for interactive use.) If no argument is given, the interactive help system starts on the interpreter console. If the argument is a string, then the string is looked up as the name of a module, function, class, method, keyword, or documentation topic, and a help page is printed on the console. If the argument is any other kind of object, a help page on the object is generated.

Note that if a slash(/) appears in the parameter list of a function when invoking [`help()`](https://docs.python.org/3/library/functions.html#help "help"), it means that the parameters prior to the slash are positional-only. For more info, see [the FAQ entry on positional-only parameters](https://docs.python.org/3/faq/programming.html#faq-positional-only-arguments).

This function is added to the built-in namespace by the [`site`](https://docs.python.org/3/library/site.html#module-site "site: Module responsible for site-specific configuration.") module.

Similar names: pdbcommand.help, django.help

[white nexus]

i searched this on the website and it didn't come up...

bro wtf

pydoc

i am now going to explore the wide world of pydoc

[main ginkgo]

help(help) works in repl 😄

[white nexus]

huh

iirc

@paper echo it was you who said, that dunder variables defined that aren't defined by a pep could be overwritten

https://github.com/python/cpython/blob/2c4792264f9218692a1bd87398a60591f756b171/Lib/pydoc.py#L277-L280

[fallen slate]

Lib/pydoc.py lines 277 to 280

if name in {'__author__', '__builtins__', '__cached__', '__credits__',
            '__date__', '__doc__', '__file__', '__spec__',
            '__loader__', '__module__', '__name__', '__package__',
            '__path__', '__qualname__', '__slots__', '__version__'}:```

[red solar]

probably not a great idea tho, unless the documentation says that you can

[white nexus]

__version__ is referred to in the stdlib internally

[peak spoke]

dunders are to be used (and defined) by the implementation and the stdlib, doesn't really have to be defined by a pep

[red solar]

    def __init__(self, dist):
        """Create and initialize a new Command object.  Most importantly,
        invokes the 'initialize_options()' method, which is the real
        initializer and depends on the actual command being
        instantiated.
        """
        # late import because of mutual dependence between these classes
        from distutils.dist import Distribution

        if not isinstance(dist, Distribution):
            raise TypeError("dist must be a Distribution instance")
        if self.__class__ is Command:
            raise RuntimeError("Command is an abstract class")

        self.distribution = dist
        self.initialize_options()

        self._dry_run = None
        self.verbose = dist.verbose
        self.force = None
        self.help = 0
        self.finalized = 0

    def __getattr__(self, attr):
        if attr == 'dry_run':
            myval = getattr(self, "_" + attr)
            if myval is None:
                return getattr(self.distribution, attr)
            else:
                return myval
        else:
            raise AttributeError(attr)

how would i access self.distribution? i'm assuming __getattr__ will block me?

[peak spoke]

If it's used (even though not officially) it makes sense for a documentation module to try and show it

[spark zinc]

whats the script for playing a song into python

i just downloaded python to have fun

[red solar]

#python-discussion and https://stackoverflow.com/questions/20021457/playing-mp3-song-on-python

[white nexus]

true..

[red solar]

ah getattr is a last resort, thank god

[raven ridge]

That's how things worked before PEP 420 - you needed to have an __init__.py, and the __init__.py needed to call pkgutil.extend_path(). The problem with that was that you needed to make a package for every namespace. If you wanted to make two packages called company.team.lib1 and company.team.lib2, you would need to make a company package containing a company/__init__.py file that called pkgutil.extend_path, and a company.team package that provided a company/team/__init__.py file that called pkgutil.extend_path, and company.team would need to depend on company, and company.team.lib1 and company.team.lib2 would need to depend on company.team. It worked, but it was really hard to explain to people and maintain. And it was easy to get wrong: if you instead made both company.team.lib1 and company.team.lib2 install the company/__init__.py and company/team/__init__.py, then installing both and then uninstalling one would leave things in the wrong state, for instance.

[white nexus]

looking in the source of pydoc

and this is kinda cursed

welp can't say i hate it lmao

[spark magnet]

@white nexus why are you looking at the pydoc source?

[white nexus]

just made a history method in my repl that will use less to show my history

[white nexus]

well, help() uses the terminal pager, which is implemented from pydoc.help. So that means that in pydoc, there's a way to use the pager, and has already been written and I don't need to write it again.

That means that with readline, I can set up a history() method in my repl to use less to show my history. It works, too

[spark magnet]

i see, makes sense

[paper echo]

right, that's explained in the pep. but i guess instead of a "plain" __init__.py file, i have in mind something more like py.typed - it's empty and its contents are ignored/irrelevant, but its presence marks a namespace package. and if there is an __init__.py in the same directory, it has no effect. so that would be safe to combine with other directories + non-namespace packages

obviously this is all said and done a long time ago

but i think there is enough confusion with beginners around accidentally using namespace packages that in hindsight maybe a "marker file" like what i described might have been easier to learn and explain

[raven ridge]

so that would be safe to combine with other directories + non-namespace packages
it wouldn't - you'd still have exactly the same problem about which package provides that file.

[paper echo]

they all would

[white nexus]

I don't anticipate it to break anytime soon, but if it does, this file is just for me so ¯_(ツ)_/¯

actually, I got a lot of inspiration for my scripts from coverage and you, nedbat

[paper echo]

the reason i don't like totally implicit namespace packages is that they further blur the distinction between modules and filesystem directories, which is confusing even for programmers experienced in other languages

[raven ridge]

then you have the same problem as I said - uninstall would be broken. And Linux distro maintainers would need to do a whole lot of work to try to make that work, because systems like Debian enforce that any given file only be provided by a single package on a particular system - you can't install two packages at once if they both contain the same file.

[spark magnet]

wow, nice 🙂

[white nexus]

!pypi coverage

[fallen slate]

coverage v6.1.1

Code coverage measurement for Python

[paper echo]

hmm... isn't that currently an issue if you have one distribution that defines a namespace package, and another that defines a non-namespace package?

which i guess is slightly more pathological, but not impossible

[raven ridge]

with the same name? Yes.

but that case is already broken.

everything needs to agree on whether a package is a namespace or not.

[white nexus]

https://github.com/nedbat/coveragepy/blob/master/__main__.py this file inspired me to make a launcher for my scripts, so I can use python -m scripts <memorable script alias that doesn't match the file name>

[paper echo]

right, so if you have 10 different marker files from 10 different distro packages, who cares if they all overwrite each other? uninstalling i guess is a matter of only removing the marker file if it's the only file left in the hierarchy, which sure is more annoying

but if you're already in the business of merging file trees from different packages, surely python isn't the only place where an issue like that might arise

[raven ridge]

You could design a package manager like that - but I'm not aware of any that exists. That's definitely not how dpkg works, which is the one I'm most familiar with.

dpkg rejects installing two packages if they would both install the same file, rather than try to define semantics for it. I mean, sure, it could do something like say "well, if they're both empty", or "well, if they both have the same exact contents", then it's OK - but if it did that, they'd also need to define semantics for upgrades - what happens if a newer version of one of the packages gets a new version of that file, and it's no longer identical? Etc.

[halcyon trail]

id, I feel like 99% of my command line runs are probably by doing Ctr-R to find the last invocation, and editing what needs to be edited

so a shorter script name doesn't really matter much to me, personally. One thing that would actually be much more helpful IMHO would be shell auto-completion for python -m, which I'm sure somebody has implemented, somewhere

I've just never bothered to look for or setup

https://pypi.org/project/genzshcomp/ this is probably also fun, probably just can't be bothered because you'd need to do it for each script

[paper echo]

that's fair enough

[paper echo]

i love this idea. would be great to have it for click or clikit as well as argparse/optparse

[white nexus]

i'm pretty sure click has completion...

[white nexus]

https://github.com/click-contrib/click-completion

https://click.palletsprojects.com/en/8.0.x/shell-completion/?highlight=completion

[paper echo]

very very useful, ty

[halcyon trail]

The click stuff is afaik for arguments to the script

I'm taking about completing the module path itself

[naive saddle]

has anyone else here worked with build backends?

I just had a ride of a time trying to add support PEP 660 to setuptools and I'm not even done yet, only most of it.

(fyi if this would be better in an OT channel or #tools-and-devops lemme know)

[red solar]

i'm trying to work with setuptools now if that's relevant (it's not going well)

[naive saddle]

honestly same thing here lol

it's even worse working within setuptools.build_meta because there isn't that much information available to you to start with, you have to get it yourself by calling setup.py commands ...

[raven ridge]

a little bit...

[naive saddle]

well this is the madness I'm head deep in right now :)

I both regret and also don't regret my decision to use pep 660 as a way to jump into the setuptools codebase for the first time

[raven ridge]

heh

[naive saddle]

I have no idea what I'm doing :P

pretty fun though learning how pep 660 works ins and outs by trying to implement it

[raven ridge]

I've done some build backend stuff, but haven't messed with pep 660 at all yet.

[naive saddle]

it's interesting how they piggy backed off the wheel standard as the "communication method"

[raven ridge]

well, not that surprising - it's a whole lot easier than making a brand new standard that includes some of what wheel does, when all the build backends already know about wheels

[naive saddle]

yeah it's not surprising as it's definitely the "lower development effort" option

[white nexus]

interesting.. https://github.com/pyodide/pyodide

Pyodide may be used in any context where you want to run Python inside a web browser.

[spice bronze]

hey guys?

can i have some help with something?

[raven ridge]

This is a discussion channel, not a help channel. See #❓｜how-to-get-help

[red solar]

that looks neat

[white nexus]

hmm this takes a bit of time to start now

[white nexus]

hmmm

from the repl, is there a way that a repr method can be detected how its called

[red solar]

what does that even mean

[verbal escarp]

i'm guessing it means "can my object be made aware of the context its __repr__ is called?"

i'd say it's the same answer how to make any function aware of its caller

use inspect and go up the stack

[lusty scroll]

curious what the relationship is between making a zip archove and an editable install?

[lusty scroll]

what's the goal? if it's to prevent infinite recursion, there's probably a simpler way

another option is for the caller of the repr method to provide whatever infornation __repr__ itself needs instead of the other way around

[surreal sun]

Not that I’m aware of, I would recommend inspect.stack()[-1].code_context but that only works with code in files

[lusty scroll]

my guess this is for some kind of depth-limiting

[swift imp]

Or you implement special hooks like ipython does aka _repr_html_, _repr_png_ etc.

[manic sundial]

Using a "worker" function to process an async queue in a fully async program. Is there any reason to run this "worker" from a thread instead of a just an endless task, as in:

while True:
    await asyncio.sleep(...)
    work...

Both can be cancelled, watched, etc, so I wonder if there clear-cut reason to prefer one over the other.

[verbal escarp]

#async-and-concurrency but in general, async is easier to debug than threads

[quasi hound]

what does this error mean

i've gotten it a bunch

cyclegen is like itertools.cycle except with better syntax for what i'm doing so i made it a class

nvm just my linter being stupid

it runs

that's how you're supposed to annotate functions anyway right

something: Callable[[<arg1_type>, <arg2_type>, ...], [<returntype>]] = ...

[raven ridge]

You've got an extra pair of braces there. There shouldn't be braces around the return type

[quasi hound]

k ty

"error" doesn't go away but it still runs

all good

[grave jolt]

You can't annotate attribute assignments. Only simple single-name assignments

Also we have #type-hinting

[grave jolt]

The type checker can't really do anything with these dynamic assignments anyway.

What is TIME?

[quasi hound]

class TimeState:
    def __init__(self, name: str, repr: str, tideLevel: TideLevel) -> None:
        self.name = name
        self.repr = repr
        self.tideLevel = tideLevel
    
    __str__ = __repr__ = lambda s, _='': f'{s.name} time ({s.repr})'

TIME: Collection[TimeState] = Collection(
    Afternoon = TimeState('Afternoon', '3:00 PM', 3),
    Evening = TimeState('Evening', '6:00 PM', 4),
    Sunset = TimeState('Sunset', '9:00 PM', 3),
    Midnight = TimeState('Midnight', '12:00 AM', 2),
    Night = TimeState('Night', '3:00 AM', 1),
    Sunrise = TimeState('Sunrise', '6:00 AM', 0),
    Morning = TimeState('Morning', '9:00 AM', 1),
    Noon = TimeState('Noon', '12:00 PM', 2),
)

collection is just a dict/mapping made with kwargs

and uses getattr instead of getitem

basically an enum in this case

with a set amount of values

[grave jolt]

Right, type checkers have no idea which attributes are present.

You can use enum.Enum

Or something

[grave jolt]

...so when you do TIME.foo: int = 42, the type checker can't get any new information. It's somewhat of a bummer, but type checkers are already very complex.

[quasi hound]

got it

[ebon roost]

what is the difference between a fully connected NN and a CNN?

[raven ridge]

That's off topic for this channel, but #data-science-and-ml should be able to help answer

[white nexus]

what even is a WindowsError

like, when would it occur

https://docs.python.org/3/library/exceptions.html#WindowsError

[grave jolt]

@white nexus using windows is an error

[white nexus]

was asking for a real answer smh

[peak spoke]

windows specific modules like winreg could've used it in the past

[white nexus]

Changed in version 3.3: Several functions in this module used to raise a WindowsError, which is now an alias of OSError.

[grave jolt]

hmm I found this ```c
/* Compatibility typedefs */
typedef PyOSErrorObject PyEnvironmentErrorObject;
#ifdef MS_WINDOWS
typedef PyOSErrorObject PyWindowsErrorObject;
#endif

[white nexus]

although i don't disagree

[white nexus]

just opened python on windows....

>>> WindowsError is OSError
True

[stone field]

Things that interface with Windows libraries like sockets or DLLs can still raise WindowsError IIRC

[white nexus]

although it'll be OSError so ¯_(ツ)_/¯

they are the same thing, although that isn't documented too great

[stone field]

yeah, hence why they kept the name there for compatibility

[white nexus]

ye

[magic hawk]

lua is better

[spark magnet]

@magic hawk what is the point of this?

[magic hawk]

honestly, idk

anyway cya

[red solar]

https://tidelift.com/subscription/pkg/pypi-setuptools?utm_source=pypi-setuptools&utm_medium=referral&utm_campaign=docs

does this mean i can get a free trial and get them to solve my setuptools issue for me?

[unkempt rock]

speaking of this, does anyone understand what the numbers in the right column of the dis.dis mean? i read these answers https://stackoverflow.com/questions/12673074/how-should-i-understand-the-output-of-dis-dis/12673195 and i still dont really understand

[unkempt rock]

oh i see, thank you that makes sense

[weary flax]

is there a way to stream notifications from a website using selenium ?
what i have right now prints the whole list over and over, i need to find a way that shows the message only if its from like the last few seconds

[astral gazelle]

you cant advertise here and your page is unreachable

[lunar trail]

!warn 433548397337640960 We do not allow recruitment here, as is clearly stated in our #rules. Please don't do so in future.

[fallen slate]

:incoming_envelope: :ok_hand: applied warning to @tacit wing.

[tacit wing]

No worries. Won't happen again!

[tacit wing]

Yeah my hosting provider is having issues with their Sydney location

[red solar]

Can i return a tuple from __iter__? i'm trying to think of a reason why that might not work

nvm i can't call next() on it

there go my hopes for nice type hinting :/

[visual shadow]

Iter should return an iterator, not a sequence. Why do you want to make it return a tuple

[red solar]

check #type-hinting

[spice pecan]

Iterator[tuple[...]] might work

Actually, now that I think about it, it's really weird

[native flame]

wouldnt Iterator[tuple[...]] mean each element is a tuple

[spice pecan]

Oh, right

hmm, I'm not sure if there's a way to do this

tuple is kinda special

[red solar]

[spice pecan]

what even is the annotation for tuple's iter?

[elder blade]

A union of all possible args?

Otherwise probably special-cased

[spice pecan]

that's the only one i can come up with, but that wouldn't give you the actual types on unpacking

since union doesn't preserve order

tupleiterator[...] when

[native flame]

what about just making a separate method that returned a tuple

[native flame]

In [7]: type(iter(()))()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-925ed1e942ec> in <module>
----> 1 type(iter(()))()

TypeError: cannot create 'tuple_iterator' instances

wonder if you can actually typehint with this lol

[red solar]

but i wanna splat 😦

[native flame]

ok, exp = CmpxchgResult(True, 5).method()

[red solar]

😒

[native flame]

[red solar]

this isn't that important, i'm just sad there's no solution

import atomics


def atomic_mul(a: atomics.INTEGRAL, operand: int):
    res = atomics.CmpxchgResult(success=False, expected=a.load())
    while not res:
        desired = res.expected * operand
        res = a.cmpxchg_weak(expected=res.expected, desired=desired)

like this is how i'd use it

[visual shadow]

Could anyone help me wrap my head around what the ask is, and then what's the issue with the ask?

With the caveat that I don't really understand type hints much, having a hard time understanding what the problem itself is.

[red solar]

If i do x, y = (True, 2), type checkers can deduce that x is bool and y is int

to get that behaviour with a custom type, you need __iter__

which can't really be type hinted like that, since it just returns an iterator (seems to be the verdict)

[visual shadow]

Makes sense. How do type checkers deduce this part then. Do they still deduce this if it's defined elsewhere and unpacked after?

[red solar]

i'm guessing they have a special rule for tuple :/

[visual shadow]

Yeah, they must have been doing a static analysis on tuples, and even more specifically, just tuple literals.

[red solar]

    t = tuple((3, "hi"))
    return t


if __name__ == "__main__":
    a = atomics.atomic(4, atomics.INT)
    a.store(5)
    x, y = atomic_mul(a, 10)

even from that it deduces it

(i chopped off the function start)

[visual shadow]

Impressive, though that's still ultimately a literal that's visible

I half suspect even this won't work if you read the tuple from elsewhere where the original contents aren't visible in code. Just a hunch, don't really use type hinting much.

I guess bottom line: this is a capability of the type checkers. Even tuples never decorated their internal iter to expose the types of contents, that's just what you thought is the approach, but it's not. Ultimately I suppose it's the type checkers that are failing to connect the dots for your custom object

[red solar]

hmm... you make a good point

which is why i hope type checkers have something special for this

and that people in #type-hinting find it for me, because i couldn't

[visual shadow]

Yeah, it's also quite possible such capability doesn't exist yet, in which case this might be a feature request. I just don't know whether it's feasible to generalize or not

To me, inferring types during iteration and unpacking of a custom object sounds like a decently tricky task statically*

[paper echo]

i am hoping that with variadic generics we can make tuples less of a special case, for exactly this kind of use

[visual shadow]

is that something in the pipeline for future python versions?

found it. https://www.python.org/dev/peps/pep-0646/ looks like a proposal atm

[spice pecan]

Assuming you provide a typehint elsewhere (function argument, typing.cast, etc), it should still typecheck properly, IIRC tuples really are special-cased, which is a little sad tbh

[spice pecan]

This would be a very welcome change

[verbal escarp]

never got my head around the definition of "variadic"

[spice pecan]

In essence, "with a variable amount of parameters"

A functions is variadic if it can be called with an arbitrary amount of arguments, like print, for example

[verbal escarp]

so meta-parameters?

[paper echo]

not meta, but "any number of them"

[visual shadow]

would range not count, because it has an upper limit?

just curious

[spice pecan]

That would be considered overloads ig

[paper echo]

yeah variadic would be "zero or more"

that's just having optional arguments

[visual shadow]

got it

[verbal escarp]

with the example of print, it takes any number of strings as positional parameters, how would you describe that?

[paper echo]

print() is a variadic function

[visual shadow]

thats a variadic

[spice pecan]

Receiving *args (or params <type> args in, say, C#) would be variadic

[visual shadow]

i guess its name does align with "varargs" nicely.

variadic, that is.

[verbal escarp]

how do you describe it as a type?

[spice pecan]

I think the current way of doing this is Callable[[object, ...], None]

[verbal escarp]

also, variadic only applies to positional arguments, not to kwargs?

[paper echo]

in principle **kwargs could be variadic. in practice people usually have in mind positional arguments

[verbal escarp]

but kwargs.. ah. hu? okay..

[spice pecan]

That's a tricky one, tbh, as I can't really come up with an example of **kwargs outside of python and maybe some equally dynamic languages

[paper echo]

why not? it's an arbitrary number of arguments

[visual shadow]

to consider this, might be worth even simplifying it further. if you have a single keyword argument, ultimately what are you typehinting

still the data that needs to be passed, right

so it makes sense that being positional or keyword shouldnt be a concern for the type of the thing passed

[verbal escarp]

i would probably use **kwargs in that case anyway and then type it as Dict[str, str] or somesuch.. but you're right

[wise bough]

Hey, guys!

I'm trying to create a basic local mail system without any external tools but Python, and would like someone to try my tool.

And to discuss, refine, etc.

It's for a private network of buddies.

Assuming no access to internet or experts.

It's functional in its current state, but there's certainly more that could be done.

[verbal escarp]

local mail system? o.O

[wise bough]

Yup.

So this is my thought process:

1. I'm not a lv. 10 programmer. More like lv 0.5. Same for networking. But I could get cable and wifi repeaters from walmart and set up a local network with a share drive.

With me so far? Basic stuff like that I can do.

But how could we communicate?

So what if we create folders on a share drive for each person, and those are our inboxes.

But then users can accidentally delete their only copies of stuff and the chance they'll mess with others' things are too high.

We're not necessarily hardening against insider threats, because that's tough, but we want to limit user input to healthy stuff.

[verbal escarp]

okay, just.. why?

[wise bough]

So with Python (and with Powershell, as a separate version currently being ported to Python), we have the following options: compose a message (creates, writes, and drops file in recipient's inbox),

read your message,

download your messages to your local machine,

move your messages between folders (inbox, archive, and trash), clear trash,

create public pages (anyone can view these), and download public pages on the network.

[wise bough]

The why is weird, but I build a lot of networks from scratch on a regular basis and we don't have access to the internet when we make them. I do this for work.

[verbal escarp]

okay

you can install python dependencies?

or is that stdlib only?

[wise bough]

Also, I've realized something. I don't have good over-the-internet networking skills, or even bad ones. But if we build out a good tool for messaging and adding locally-accessible content for everyone to see assuming access to a shared folder, distribution of the network can be done over the internet with any cloud syncing application. And if we were to stop trusting a company like mega, we could just move the files over to G Drive, or Dropbox, without losing any data.

And right now it's standard features only for Python.

Here's an example console output during usage:
Enter username: don
Account for don created/found.
Press enter to continue.
//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

INBOX:
['20211111-0047-sta-t.txt', '20211110-1923-don-hey.txt']

ARCHIVE:
['20211110-2233-sta-fwd.txt', '20211110-1743-sta-which.txt']

TRASH:
[]

PUBLIC PAGES:
['test-don.txt']

OPTIONS:
[1] SEND A MESSAGE
[2] READ MESSAGE FROM INBOX
[3] DOWNLOAD INBOX, FOR WINDOWS
[4] MOVE MESSAGES FROM TRASH TO INBOX
[5] MOVE MESSAGES FROM ARCHIVE TO INBOX

[6] READ MESSAGE FROM TRASH
[7] MOVE MESSAGES FROM INBOX TO TRASH
[8] MOVE MESSAGES FROM ARCHIVE TO TRASH
[9] EMPTY TRASH

[10] READ MESSAGE FROM ARCHIVE
[11] MOVE MESSAGES FROM INBOX TO ARCHIVE
[12] MOVE MESSAGES FROM TRASH TO ARCHIVE

[13] MAKE/EDIT PUBLIC PAGE
[14] READ PUBLIC PAGES
[15] DOWNLOAD PUBLIC PAGES, FOR WINDOWS

[16] SHOW DOWNLOADED FILES, FOR WINDOWS
[16] LOG OUT
READY, don. Option:

The worst things about this right now are that I have to type in the file name to specify what message I want to manipulate. I could use tkinter on Windows, but it breaks compatibility with Android. But like I said, I might just not be doing a good job using that feature. Also, it shows the location of the source files (the originals), which exposes other users to accidental deletions.

[verbal escarp]

compatibility with android?

[wise bough]

Yeah, it's an additional goal. We can get on the network with a synced cloud drive from mobile. I didn't create an APK, I just downloaded Pydroid 3 and run the script using that.

With the script located in the cloud synced folder.

I know this is very weird, and very specific, but the machines that form the bulk of the userbase are locked down and can't access most internet site, cannot run executables (we have no admin rights).

[deft pagoda]

maybe you should move this discussion else where? this is very offtopic for this channel

[wise bough]

sites*

Sorry, I'm new here. Don't know what topics constitute advanced discussion. If you don't mind, Amogorkon, I'd like to message you.

[deft pagoda]

check the channel topic

Discussion on the use cases, implementation and future of the Python programming language including PEPs, advanced language concepts, new releases, the standard library, and the overall design of the language.

[gleaming cave]

Hey, does anyone know a fun Python website with exercises?

[visual shadow]

perhaps ask that in #python-discussion

[gleaming cave]

It depends right? Maybe i ment advanced but fun exercises ^^. But alright, thank you.

[visual shadow]

this channel is for

Discussion on the use cases, implementation and future of the Python programming language including PEPs, advanced language concepts, new releases, the standard library, and the overall design of the language.

trying to come up with a good name that captures what this channel does is the single biggest pain point we have had, but yes, it's not just about "any" advanced topic, but specifically discussion about python language itself.

[astral gazelle]

#internals-and-implementation

[visual shadow]

that might be worth suggesting to #community-meta

[paper echo]

#python-internals would be maybe good

also moving it to the topic section? idk

a few different names have been tried

[gleaming cave]

Yeah, advanced would be writing in C right, Jk.

[visual shadow]

unironically, a lot of python is written in C, so in a way you'd be right 😛

[visual shadow]

youve posted in #data-science-and-ml , this channel isnt the right channel for this. please see room topic.

[proper flame]

So sorry !!

[verbal escarp]

#python-internals sounds like core-dev-only

btw, i was just wondering, is there a good way to pre-process python code before parsing it? i could then switch generics from 3.10 to List, based on the version of the interpreter

[prime estuary]

You're probably looking for the ast module, parse to that, implement a tree visitor and you can go through the whole syntax.

[verbal escarp]

that won't help if the code is syntactically incorrect for that version

[prime estuary]

True, though the generics aren't new syntax.

[verbal escarp]

patma is

[flat gazelle]

you can use a custom encoding IIRC

[verbal escarp]

how would that work?

[raven ridge]

https://aroberge.github.io/ideas/docs/html/ - this shows a bunch of ways to do funky tricks with Python's syntax, including the coding comment approach.

[flat gazelle]

you would have a launcher which registers the encoding and then if you do

# coding: your_coding
```you will get the entire file as a stream of bytes which you can arbitrarily transform

[surreal sun]

Ooh, I love this package. Gave me inspiration to try out import hooks

[unkempt rock]

hello, i was just going through PEP 593 (https://www.python.org/dev/peps/pep-0593/) which is on the Annotated typing. could someone clarify for me what's meant by "metadata" in this PEP?

This PEP introduces a mechanism to extend the type annotations from PEP 484 with arbitrary metadata.

[raven ridge]

information about a variable other than its type

[unkempt rock]

oh, i see, thanks

[raven ridge]

I have it bookmarked for whenever someone says "I want to make Python do $crazy_thing"

[surreal sun]

Speaking of, a talk I really liked on this was https://youtu.be/xLc5xPYGGnQ

The examples were some pretty interesting stuff, like something called rwatch that will “watch” your variables or values and apply a callable to it with some black magic

Which in reality the black magic is just messing with frames

[raven ridge]

ooh, that sounds like a good talk.

[surreal sun]

Yeah, it’s a really good watch tbh

[grave jolt]

Does PyPI just ignore requests to remove malware?

it definitely seems like so

I think I've asked this here before but, well, you know

[naive saddle]

yeah pypi-support just keeps on growing and growing

quite disappointing but I guess I understand their perspective (as a maintainer of psf/black) even if the security risk element makes it hard to accept

[elder blade]

Holy! 700MB package?! https://github.com/pypa/pypi-support/issues/1458

What size limit does PyPA have on PyPI so far?

[naive saddle]

100 mb per file and 10gb per project IIRC?

https://github.com/pypa/warehouse/blob/3efc42ce44ecf9e72bfd4dcb924e00594db21f36/warehouse/forklift/legacy.py#L70-L72

[fallen slate]

warehouse/forklift/legacy.py lines 70 to 72

MAX_FILESIZE = 100 * ONE_MB
MAX_SIGSIZE = 8 * 1024
MAX_PROJECT_SIZE = 10 * ONE_GB```

[verbal escarp]

Addrmatcher is an open-source Python software for matching input string addresses to the most similar street addresses and the geo coordinates inputs to the nearest street addresses. The result provides not only the matched addresses, but also the respective country’s different levels of regions for instance - in Australia, government administrative regions, statistical areas and suburb in which the address belongs to. <- that sounds like it could be even bigger

eh.. wrong msg replied to

[naive saddle]

I was confused for a sec there 😄

[verbal escarp]

sorry

couldn't pypa have a seperate data hosting service and python integrate it on import?

doesn't sound very economical to include the data with the code

iirc nltk had a rather elegant solution

last time i checked, they fired up a tkinter gui on first import or somesuch to download all the texts, depending on user need

[verbal escarp]

@raven ridge that page you linked got me thinking. could an import hook also make variable/attribute docstrings possible?

[raven ridge]

the problem with docstrings for variables is that there's nowhere to put them. You can't store them as an attribute of the variable itself, because given something like x = 1, there's no way to set an attribute on 1 - and even if there was, that same 1 is shared for every variable whose value is 1, and they might all want different docstrings.

you could store it on the module, instead, somehow, perhaps - the closest thing to that that you're likely to be able to build as a module level dict mapping name to docs

[verbal escarp]

what about __attribute_docs__ as {qual_name: docstr}

[raven ridge]

sure. It limits it to only things that have a qualname, and you'll need to know what module a name came from (since that's not tracked automatically anywhere) - but, yeah, that's doable.

[verbal escarp]

the things that don't have a qualname are hard to address either way, i suppose

just thinking of vars in closures

[grave jolt]

VSCode supports this for variables and attributes, but yeah, you can't inspect this at runtime

[verbal escarp]

TIL

[raven ridge]

Sphinx supports the same, which allows getting it into generated docs

[grave jolt]

TIL

[raven ridge]

I don't believe pydoc supports it, though, and help() certainly doesn't.

[grave jolt]

yeah, help inspects an object, not a symbol

[raven ridge]

Sphinx has 2 ways of doing it, actually - either a docstring after the assignment, or a special doc comment after it - a comment starting with #:

[verbal escarp]

couldn't help() refer to sphinx-generated docs? ^^

[grave jolt]

help receives an object, not a symbol

[raven ridge]

For Sphinx:

For module data members and class attributes, documentation can either be put into a comment with special formatting (using a #: to start the comment instead of just #), or in a docstring after the definition. Comments need to be either on a line of their own before the definition, or immediately after the assignment on the same line. The latter form is restricted to one line only.

[grave jolt]

like, it can receive 42 and what can it do with it?

[naive saddle]

sorry for taking so long to respond, the answer is that the editable standard uses the wheel format as the communication method between the frontend and backend so they don't have to invent a new way to pass files and data to each other

the backend is responsible for setting up a wheel with all of the files necessary for the editable install (except for console scripts) in particular with a .pth file (which powers the whole editable install mechanism) and the calling frontend receives this wheel, unpacks it, and installs it

this wheel is special tho as it's not meant to be distributed, in most cases it's literally just the ${name}.dist-info metadata + the .pth file(s)

[grave jolt]

But that's not what we're talking about. We were talking about stuff like ```py
FROBNICATION_FACTOR = 42
"Factor to adjust the frobnication metafoobarization, in percents"

[deft pagoda]

if somehow help could know the alias you're calling it with

[raven ridge]

You're agreeing with fix and salt about why what amogorkon wants isn't possible, @unkempt rock

[grave jolt]

we could have some kind of Symbol object (like inspect.Symbol("re.MULTILINE")) that would act like a pointer to something?..

[verbal escarp]

actually i'm fairly happy with vscode and sphinx supporting those attribute docs, don't care that much about help()

[grave jolt]

yeah I personally don't use help

maybe ? in ipython occasionally

[verbal escarp]

but mostly for functions there, which is covered

[deft pagoda]

i use ? a lot

[peak spoke]

?? is also convenient to get the source quickly

[white nexus]

ngl that would be cool, you should write a pep for this

[white nexus]

or a string object, and then matches that up internally

!e import pydoc ; pydoc.help('42')

[fallen slate]

@white nexus :white_check_mark: Your eval job has completed with return code 0.

001 | No Python documentation found for '42'.
002 | Use help() to get the interactive help utility.
003 | Use help(str) for help on the str class.

[white nexus]

interesting...

and its wrong lmfao

!e import pydoc ; pydoc.help('str')

[fallen slate]

@white nexus :white_check_mark: Your eval job has completed with return code 0.

001 | Help on class str in module builtins:
002 | 
003 | class str(object)
004 |  |  str(object='') -> str
005 |  |  str(bytes_or_buffer[, encoding[, errors]]) -> str
006 |  |  
007 |  |  Create a new string object from the given object. If encoding or
008 |  |  errors is specified, then the object must expose a data buffer
009 |  |  that will be decoded using the given encoding and error handler.
010 |  |  Otherwise, returns the result of object.__str__() (if defined)
011 |  |  or repr(object).
... (truncated - too many lines)

Full output: too long to upload

[white nexus]

eg, it'll take a str and match it up in the globals, or import it, and do it that way

but it'd be nice to have a way that defines it to be able to inspect it for a variable, even if its not stored with said variable

attrs does something like this with its classes
every class it makes has one attribute that it uses internally __attrs_attrs__

iirc that stores a class or dict of the fields on the object, with their metadata of their field object, and annotations

the way to access from it, is to use attrs methods

iirc its something like attr.fields(instance_of_attrs_class)

[unkempt rock]

lol https://github.com/uts-mdsi-ilab2-synergy/addrmatcher/tree/main/data/Australia

[clear coral]

thoughts on doing this?

import sys

class SomeClass:
  if sys.version_info[:2] < (3, 9):
    def method(self):
      raise Exception("This method is only supported on python versions 3.9 and up.")
  else:
    def method(self):
      # original functionality

also what exception type would be appropriate?

[raven ridge]

why not the more obvious

import sys

class SomeClass:
    def method(self):
        if sys.version_info[:2] < (3, 9):
            raise Exception(...)
        # original functionality

[raven ridge]

NotImplementedError

you need to check sys.version_info, not sys.version, for the way you're using it, also.

[clear coral]

yep, fixed the version_info already

idk, it just occurred to me as a different way of doing what you showed in your snippet.

it doesnt seem to have any obvious downsides

but maybe im missing something?

[raven ridge]

your way has the pro of being slightly less overhead per call, but the con of being more difficult for others to read, and causing the entire body of the function to need to be indented further, and the signature needing to be duplicated

[clear coral]

cant say i really agree with being more difficult to read but the other stuff is fair enough

[raven ridge]

most people are very surprised to learn that an if can go into a class body the first time they see it.

the first time users see your pattern, it will surprise them.

If you were OK with the exception being an AttributeError, you could also just do:

import sys

class SomeClass:
    pass

if sys.version_info[:2] >= (3, 9):
    def method(self):
        # original functionality

    SomeClass.method = method

[clear coral]

i think this wont provide the correct context for method, e.g. itll be missing the class cell and super calls wont work as a result

[raven ridge]

that's true, if you did this you'd need to use the two-argument super() call, not the zero-argument super() call.

[clear coral]

guess the only real place it could make more sense is if the function definition itself was version dependent, like if you had something in the signature that would need a newer stdlib version

(i mean the general idea of using an if to a seperate version-gated definition of the same function, not specifically the AttributeError solution)

[raven ridge]

well, in that case, you probably couldn't use this technique, because both the true and false branches of the if need to be parsed and compiled

instead, you'd put the code that requires a new version into a separate module, and conditionally import it only if you're on the new Python version. Or you'd put it in a string literal and use exec(), perhaps.

[clear coral]

thinking of something like this:

def some_function(arg=some_stdlib_function_or_property_that_doesnt_exist_in_all_versions()):
  ...

[raven ridge]

hm, sure, for a default value, I suppose.

I was thinking more like * or / for keyword-only or positional-only args.

[clear coral]

or annotations using types that arent in all versions

[raven ridge]

for annotations I'd just from __future__ import annotations

[clear coral]

though you could just stringify those i guess

or that, yeah

[raven ridge]

or that, yeah 🙂

[amber nexus]

My favorite thing to abuse 👀

[ruby pilot]

looks like a job for a decorator

[clear coral]

decorator cant handle the case of something version dependent in the signature

[white nexus]

;-;

bottle is broken on 3.10

[astral gazelle]

arent most things broken on 3.10

what about numpy/pandas

[white nexus]

its broken because of legacy imports

since 3.2

wait

why did i get a deprecation warning

ah, something i had was not using the latest version of bottle lmao

[red solar]

legacy imports?

i'm confused, what breaks things on py310?

[radiant fulcrum]

They all work

Most of the big well known libs support 3.10 now atleast all your big data science libs e.g. Scipy, pandas, numpy, etc...

The error libs run into is distutils.utils not being a thing anymore

[red solar]

oooh

well thank god i don't use distutils 🙂 setuptools ftw

although i do run into issues with only recent versions of pip knowing about pyproject.toml and thus failing to install from source

not sure how to deal with that

[lusty scroll]

seems like there's been a skull and crossbones around distutils for a while 😄

[bleak star]

Hey!

[verbal escarp]

it just occurred to me that if python followed semver, we'd be on version 5 or 6 by now ^^"

with all those backwards-incompatible syntax changes over the last 10 "minor" versions

[feral cedar]

it's not backwards incompatibility though. it's about breaking things that were valid before

[verbal escarp]

hm.. okay, got a point

[visual shadow]

Yeah python tries very hard to stay backwards compatible. Ie it won't break existing code when it adds new things

[flat gazelle]

I wouldnt say very hard. There is an effort made, but C, C++, PHP etc only break far less code than python does

[elder blade]

JavaScript is probably the best

Because it can literally not have brewing changes in any shape or form

[unkempt rock]

Ñ

[verbal escarp]

well, if/when we realize subprocess-interpreters in justuse, maybe it'll become possible to run python2 code with a python3 main program

then python has to worry about backwards compatibility even less

[red solar]

Where did people put build dependencies before pyproject.toml?

[flat gazelle]

setup.py IIRC