#internals-and-peps

1 messages · Page 24 of 1

halcyon trail
raven ridge
#

the easiest option would be to add a breakpoint() call there, and run the process under gdb, and then just hit ctrl-c at the pdb prompt to drop into a gdb prompt

vital urchin
raven ridge
#

yes, exactly

vital urchin
#

with signal

raven ridge
#

oh - sure, that works

#

that's exactly what doing breakpoint() and hitting ctrl-c at the prompt would do, too, though

vital urchin
#

oh i see

raven ridge
#

but yes, you could os.kill(os.getpid(), signal.SIGINT) to get the same effect

vital urchin
#

nice i see ok, ty

jade raven
#

what for?

vital urchin
#

is it me or building --with-address-sanitizer is extremely long? or did my build hang heheh

vital urchin
#

anyone got suggestions

#

DEADLYSIGNAL is also slightly intimidating ngl

#

getting this across multiple python versions

#

tried multiple docker base images

lucid snow
#

so, what would be some issues with .pop accepting slice objects?

raven ridge
#

It would be strange for the return type to change depending on the argument that was passed

faint river
raven ridge
#

Yes, but I think that would be more surprising for a function

#

Also, how would it handle subclassing? If you subclass list, would o.pop(slice(0, 10)) return an instance of list or of the subclass?

flat gazelle
#

I mean, that's the exact same problem everything else has too, UserList can deal with it.

faint river
#

slicing a subclass of list returns a list

#

So popping from a subclass of list with a slice should return a list

flat gazelle
#

It seems fine-ish to me from a technical standpoint, but I don't think the symmetry with list indexing really makes sense without the : sugar

raven ridge
#

I think that, in the world with type annotations, we've moved away from adding functions whose return types can only be described using @overload because they depend on the input types

swift imp
#

As a result Pandas has a whole bunch of stuff you gotta implement so it knows how to repack your data after operations, when you make your own subclass

#

I also think that list is too crucial of a container that this shouldn't be supported

#

it'll add too much overhead in the general case

#

Whats wrong with just [original.pop(idx) for idx in range(10)]

faint river
swift imp
lucid snow
#

alright, here's a crazier idea:

lst.pop[start:end:step]

👀

faint river
#

Lock this man up

swift imp
lucid snow
#

sth like that ig

faint river
#

Or just allow slice literals in more spots

lucid snow
#

alr, I can sense this potentially causing issues for backwards compat, soo, to be in line with the slice notation we get when using [], how about sth like .pop_slice[start:end:step]?

lucid snow
swift imp
#

the new class just needs to implement __call__ and __getitem__

lucid snow
#

I can't think of any particularly obscure reasons right now, but like...

faint river
swift imp
#

eh

#

I would think it would have to be a property

faint river
#

why a property?

swift imp
#

and the property handles that, isntead of making Pop a descriptor

faint river
#

User-defined functions are all descriptors

swift imp
#

Because then __get__ is just always returning itself whether accessed from class or instance, but realistically you'd probably want adifferent Pop instance on each attribute access

lucid snow
#

wait, why?

faint river
#

That doesn't make sense

lucid snow
#

well, that's breaking backwards compat

swift imp
#

like x = [1,2,3]; print(id(x.pop) != id(x.pop)) would return True is the correct way to do it

#

imo

lucid snow
#

why though?

#

that's a potential breaking change

swift imp
#

I cant think of a good reason

#

Other than, i believe thats how pandas does it with loc

lucid snow
#

I think this can be just solved using composition really, where you pass the list to Pop obj during __init__ and then just simple attribute access via list.pop that returns that composite Pop

swift imp
lucid snow
#

oh, that, yeah, but that's not the same as x.pop != x.pop

#

the ids I mean

swift imp
#

wouldn't the id of 2 different instnaces be different?

raven ridge
#

!e You can already do this: ```py
s = slice(1, 3)
l1 = [1, 2, 3, 4]

l2 = l1[s]
del l1[s]

print(f"{l1=} {l2=}")

fallen slateBOT
#

@raven ridge :white_check_mark: Your 3.12 eval job has completed with return code 0.

l1=[1, 4] l2=[2, 3]
lucid snow
#

but that's not an expression

raven ridge
#

Having .pop() accept slices doesn't seem to add much, since you can always just slice to get the items into a new list and then del them from the original list

raven ridge
# 
def pop_slice(lst, slice):
    ret = lst[slice]
    del lst[slice]
    return ret
lucid snow
#

that adds extra overhead... no, but like is there any harm in adding this new shiny feature?

flat gazelle
#

It's more stuff in a language already fairly full of stuff

#

It's quite hard to justify adding something just to avoid writing two lines

lucid snow
#

I mean, yeah, more code to maintain... is it gonna add thaaat much more code though? (well, yes, because writing a Python class in C is not quite the same simple as in Python)
alright, ig my dreams are crushed here bread_pensive

swift imp
fallen slateBOT
#

pandas/core/indexing.py line 304

def loc(self) -> _LocIndexer:```
swift imp
flat gazelle
#

Not really

wary aspen
#

is multiprocessing.connection doing something about creating new processes? why is it in multiprocessing?
for example here https://stackoverflow.com/a/63547025/964478 the server listening for the client messages does not create any processes, right?

Stack Overflow
how can I disable multiple instances of a python script?

I made a python program and I want to disable the option to launch it multiple times in parallel.
I searched the web and found that I need to do it with socket and ports, but I didn't find any exam...

flat gazelle
pliant tusk
#

!e ```py
from fishhook import *

@hook(type(list().pop))
def getitem(self, slc):
if not (isinstance(self.self, list) and self.name == 'pop'):
return orig(self, slc)
S = self.self[slc]
del self.self[slc]
return S

x = [1,2,3,4,5]
p = x.pop[0:2]
print(p, x)```

fallen slateBOT
#

@pliant tusk :white_check_mark: Your 3.12 eval job has completed with return code 0.

[1, 2] [3, 4, 5]
pliant tusk
lucid snow
#

if I just wanted to make it possible in my codebase in some way, I'd probably do the reasonable way and subclass list instead or use a function, I just wanted it be part of standard Python firPensive

idle oasis
#

print ("hello world")

mortal blade
#

wat is the answer

#

@west lily

faint river
#

also please use a proper IDE or code editor

mortal blade
#

wdym

faint river
# mortal blade wdym

Discussion on the use cases, implementation and future of the Python programming language including PEPs, advanced language concepts, new releases, the standard library, and the overall design of the language.
this is the topic of this channel.

grave jolt
unkempt rock
#

or just ask

gray galleon
#

will logging have pep8-compliant aliases

feral island
mild cobalt
#

Have a discussion link on that? I’ve often wondered why it doesn’t

spark magnet
merry bramble
mild cobalt
#

Yes, I’m sure there have been, which is why I asked for a link to the discussion instead of asking to rehash. I wasn’t sure if there’s a canonical pep or something similar that attempted this.

merry bramble
#

There have been more, it feels like this comes up every 3 months or so

quick sentinel
#

where are the python standard libraries located?

grave jolt
quick sentinel
#

thanks im just using the typical python installed by the installer from the website

grave jolt
#

that's CPython yeah

quick sentinel
#

ok thanks

#

im trying to make my own portable version of python

#

i dont have a modules folder

#

does that mean they are just in the lib folder

jade raven
#

it should be site-packages for third party

grave jolt
quick sentinel
#

right. welp pygame seems to work fine which is all i needed really. it is for distributing my games

jade raven
quick sentinel
#

probably github. ill like make a "packager" using something like pysimplegui. you just then specifiy which packages you need. i havent started working on my game yet. well not much of it. i want a solid way to distribute it before making it

mild cobalt
quick sentinel
#

the problem with pyinstaller is i didnt really like how it packaged the projects. i made a small game once before but it refused to launch because it couldnt find the assets folder i had to move it from where it had been automatically placed

pliant tusk
#

would this assertion be considered a security detail of os.path?

fallen slateBOT
#

Lib/ntpath.py line 225

# join(head, tail) == p holds.```
pliant tusk
#

or is it just a generalized assumption?

quick sentinel
#

does anyone know what the platform independant libraries are?

#

for context ```
Could not find platform independent libraries <prefix>

#

when running my portable version of python

#

i think i might be missing some folder from the usual python install

lucid snow
#

Not sure how relevant this channel even is for questions about hypothetical Python features

#

what about a .= augmented . operator?
instead of sth like

string = string.replace("this", "that")

you could do

string .= replace("this", "that")
#

basically just syntactical sugar

#

like, no __igetattr__ special method or anything like that

tacit hawk
#

I think it makes reading harder because "replace" lookup scope is not clear

dusk comet
#

yeah, this looks very weird

#

what about this?

string = $.replace("this", "that")
``` `$` is a sugar for lhs of an assignment
tacit hawk
#

creating implicit vars is always bad since it may break lots of code

dusk comet
#

what would happen in this example? py x[print('hi')][print('world')] .= a if this is equivalent to x[print('hi')] = x[print('hi')].a, then hi and world would be printed twice
i dont think this is a good idea
this might be a better alternative:

x[print('hi')][print('world')] .= a
###
lhs = x[print('hi')]
#     ^^^^^^^^^^^^^^ side-effect here
key = print('world')

lhs[key] = lhs[key].a
#^^^^^^^ this executes __setitem__
#          ^^^^^^^^ this executes __getitem__

x.a.b .= c
###
lhs = x.a
#     ^^^ this might contain side-effect, but it is executed exactly once
lhs.b = lhs.b.c
#^^^^ __setattr__ - if there is a side-effect, this is fine
#       ^^^^^ __getattr__- if there is a side-effect, this is fine too
#           ^^^ __getattr__ - desired side-effect
tacit hawk
#

Yes, I say it's a bad idea to use the .= or the alias $

lucid snow
dusk comet
#

yeah, $ suffers from the same problems as .= because it is not clear what code actually will be executed
and the fact that x in x=y and y=x behaves differently makes it even worse

dusk comet
lucid snow
#

idk, I think it's fine
you'd "save" the output of the expensive function in a temp variable anyway, so it would make sense to also do it when using .=
or it would evaluate only once always...
hmmm

dusk comet
#

obj.compute_very_expensive_stuff().a .= b - this looks like expensive stuff happens only once, but actually it happens twice, and second one is hidden behind the syntax
python is usually pretty explicit and does only stuff that you ask for, and i dont think .= syntax fits well into language

lucid snow
#

mmm, maybe, I still would love to have it, at least so it can be used for simple enough stuff... and evalute only once

tacit hawk
#

that just reduces typing a little bit, I think it's already well solved by autocomplete tools of IDEs

tacit hawk
# dusk comet `obj.compute_very_expensive_stuff().a .= b` - this looks like expensive stuff ha...

I don't think it would be evaluated twice unless you expand

Below I use the + op as example:

class Value:

    def __init__(self, value: int):
        self.value = value


class Foo:

    def compute_something_expensive(self) -> Value:
        print("Computing something expensive...")
        return Value(0)
    
foo = Foo()
foo.compute_something_expensive().value += 1
print("Done!")

Computing something expensive...
Done!

now expanding like you did

foo = Foo()
foo.compute_something_expensive().value = foo.compute_something_expensive().value + 1
print("Done!")

Computing something expensive...
Computing something expensive...
Done!
faint river
#

!e ```py
class Value:
def init(self, a):
self.a = a

class OtherValue:
def init(self, b):
self.b = b

class Foo:
def compute_something_expensive(self):
print("Computing something expensive...")
return Value(OtherValue(0))

foo = Foo()
temp.a = (temp := foo.compute_something_expensive()).a.b
print("Done!")

fallen slateBOT
#

@faint river :white_check_mark: Your 3.12 eval job has completed with return code 0.

001 | Computing something expensive...
002 | Done!
rose schooner
#

otherwise augmented assignments wouldn't have a use other than shortening code

rose schooner
#

...which seems to be the entire argument for .= so far

spark magnet
#

idk if it was covered above, but .= is significantly different than other augmented operators. In s .= replace("old", "new"), how is replace evaluated?

rose schooner
#

hmm yes that was mentioned

gray galleon
radiant garden
#

it is markedly different (other operators evaluate their arguments before the actual operation)

gray galleon
#

why is ... not an alias for None but its own object

faint river
#

because it is meant to be used as a value, in slicing/indexing syntax

#

its usage in typestubs came after

gray galleon
#

like lst[...]?

faint river
#

I recommend to explore how it is used in numpy

gray galleon
#

ok so its for sugar

#

a[0, :, :] == a[0, ...]

faint river
#

so is the binary operator @

#

the matmul operator was added in 3.5 specificially because of NumPy. No builtins use it.

gray galleon
#

is the ellipsis motivated by numpy
or does numpy just conveniently use it

faint river
#

not sure on that one

#

oh yeah, ... is used in Callable for typehinting, when the argument type(s) doesn't matter

#

but idk since when that has existed

gray galleon
gray galleon
faint river
#

no, because ... in Callable args represents any amount as well

rose schooner
faint river
#

is there a PEP(s) for Ellipsis?

rose schooner
faint river
#

i c

worthy sandal
#

how to flatten this two dimensional set py f = {{1,4},{2,5},{8,10},{6,7}}

quick snow
#

But assuming the inner sets are frozen, set.union(*f) should do it.

worthy sandal
#

Still getting the error TypeError: unhashable type: 'set'

quick snow
#

Sets can only contain hashable items. Since sets are mutable (they can be changed), they aren't hashable.
If you make the inner sets a frozenset, it will work.

worthy sandal
#

Just had microsoft interview, they gave the above set and asked to merge intervals

#

my approach for intervals was right but the typeerror flopped my interviewlemon_angrysad

worthy sandal
#

@quick snow any idea how to flatten it if we face this situation again?

dusk comet
faint river
swift imp
tropic fulcrum
#

Has anyone been successful in building Python 3.13a05 with the experimental jit enabled? I tried on Ubuntu 22.04LTS and it died looking for clang-16.
(apologies in advance for cross-posting, but this got lost in #python-discussion)

boreal umbra
#

Is it considered part of the language spec that, for some non-immortal object named x, where x is the only reference to the object: del x will cause the object to be deleted before proceeding to the next statement?

#

from what I understand, that is the behavior in cpython, but I'm not sure if it's specified per se.

tropic fulcrum
#

I don't think it is part of the language spec, and even in CPython, while this is probably true for 99% of typical code and typical classes, there can be corner cases (if __del__ is implemented and must be run, for instance). For the 99%, I expect CPython's reference counting kicks in and does what you describe. And it is very easy to overlook the creation of new references to an object (was it used as an argument to a function wrapped with lru_cache? There is another reference.). So that might be a more relevant question to ask yourself - is the refcount to x really just 1?

quick snow
#

Objects are never explicitly destroyed; however, when they become unreachable they may be garbage-collected. An implementation is allowed to postpone garbage collection or omit it altogether — it is a matter of implementation quality how garbage collection is implemented, as long as no objects are collected that are still reachable.

dusk comet
#

no, it is not specified
and it might not be true in other implementation

#

but it was like that in cpython for ages, so there is a lot of code out there that relies on that

tropic fulcrum
#

If you are looking for some kind of guaranteed cleanup, that is what context managers are for.

boreal umbra
#

My question is answered ✅ praygeBlessed

dusk comet
#

this is possible iff implementation does RC (ref counting), which is not the case for most of them
pypy doesnt do RC, iirc
micropython doesnt do RC
jython, brython probably too

quick snow
boreal umbra
#

(didn't want it to get burried)

final geode
#

Let me know if you have any issues.

mental smelt
#

now that pep 668 is widely implemented. whats your way for quick scripting. i used to use python to run http requests with the requests package but no it seems that i have to create a venv for max 5 lines of code. whats the expected approach?

spark magnet
boreal wedge
#

Here is what i wanna say about Python i think it greatly helps people make programs and take a good run at programming.

#

I feel skeptical about selling and being a programmer is way farther away everyday for most but certain keep pushing the envelloppe and that's brilliant.

#

I am all for open source softwares and bringing people forward to help with issues in daily life with the amount of stuff we have to do on our own now we can still do better and i love that.

#

That's my opinion and i could be wrong but i just wanna say that because i love people that do good things especially when i see myself reflected in their project but that's just a small bit of the projects i am unable to create and sometimes i just wanna get people excited about an idea and that's where i feel the folks at home bring joy unbeknowst to me or something or other
love liam

tropic fulcrum
final geode
tropic fulcrum
#

Let me know if you want a performance suite - pyparsing does a decent job, but parse performance has always been a weak spot.

final geode
#

What sort of a speedup are you seeing vs. 3.12?

#

Or vs 3.13 (no JIT)

tropic fulcrum
#

I'm actually just getting dressed for an event, so I need to break off else I'll be late. I'll post some numbers later this evening.

tropic fulcrum
#

Is there a way to detect the presence of the JIT, via the sys or platform modules?

final geode
#

Do you mean like “is it on”, or “am I literally in jitted code right now”?

tropic fulcrum
#

"Am I running Python with the JIT enabled?"

final geode
#

If the former, the JIT is always enabled if you built with the flag.

#

But no way to tell between different builds.

tropic fulcrum
#

Yes. But in my tests, I run many Python builds and versions.

#

So it would be nice if in the test output I could log if JIT'ing or not (like I currently log the Python version, and the version of the package under test).

final geode
#

Yeah, sorry. I forget if the configure options or compiler flags end up in sysconfig somewhere…

tropic fulcrum
#

"Am I running a Python built with the enable JIT flag?" might be a better wording of my question.

#

This looks promising:

>>> pprint(sysconfig.get_config_vars()["PY_CORE_CFLAGS"])
('-fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -D_Py_JIT '
 '-fno-semantic-interposition -flto -fuse-linker-plugin -ffat-lto-objects '
 '-flto-partition=none -g -std=c11 -Wextra -Wno-unused-parameter '
 '-Wno-missing-field-initializers -Wstrict-prototypes '
 '-Werror=implicit-function-declaration -fvisibility=hidden -fprofile-use '
 '-fprofile-correction -I./Include/internal -I./Include/internal/mimalloc -I. '
 '-I./Include -DPy_BUILD_CORE')
#

I can search for -D_PY_JIT. The leading _ implies this might change in future, but for right now, it will help me distinguish if I'm running the Py3.13 built with the JIT enabled or not.

final geode
#

You may be able to find the configure options and search for --enable-experimental-jit too. Of course, I’m not sure if either of these work with Windows…

tropic fulcrum
#

Yes, there is a CONFIG_ARGS item in that same dict, and it has the configure tag.

Right now I'm only testing on Ubuntu, and I doubt I'll do any of this on my WIndows environment.

#

Thanks for the pointer, I've never used the sysconfig module before.

#

As for my tests, they may not be very good benchmarks - I'm not really seeing any significant difference across 3.12, 3.13, and 3.13-JIT, for both my littletable and pyparsing test suites.

# 
>>> '--enable-experimental-jit' in shlex.split(sysconfig.get_config_vars()["CONFIG_ARGS"])
True
worthy sandal
swift imp
grave jolt
#

!rule 6 4 @pulsar island don't advertise on this server

fallen slateBOT
#
Rules

4. Use English to the best of your ability. Be polite if someone speaks English imperfectly.

6. Do not post unapproved advertising.

quick snow
#

See above, this is not an ad board.

final cypress
hot grove
#

hello !

jade raven
#

is it possible to detected type aliases like ```py
typed_asyncio_queue = asyncio.Queue[tuple[str, str]]

quick snow
#

Not using the AST walker: Syntactically this is no different from any other indexing operation. You can check if the value is an instance of types.GenericAlias though.

jade raven
#

huh, mypy's stubgen annotates that as ```
typed_asyncio_queue: Incomplete

quick snow
jade raven
#

(type alias) typed_asyncio_queue: type[Queue[tuple[str, str]]]

#

yet it's stubgen doesn't write it correctly

quick snow
fallen slateBOT
#

mypy/typeshed/stdlib/asyncio/queues.pyi line 22

class Queue(Generic[_T], _LoopBoundMixin):  # noqa: Y059```
jade raven
#

nice, i'm writing an stub generator using only AST

#

but i guess not everything can be parsed correctly with it.

jade raven
#

regex might work for this actually

urban sandal
#

There are non-expressible concepts in python's type system, but that's a seperate discussion

#

parsing and traversing the AST, while resolving types is sufficient

quick snow
urban sandal
#

You told a user it can't be done by walking the AST, and led with "yes, ..." when someone said that would mean they needed to execute code.

quick snow
#

Can't be done by walking the AST: Purely on the AST level, you can't distinguish a type alias like this from any other item access.
My "Yes" was in the context of doing isinstance(the_thing, types.GenericAlias).
And they would need to execute code in order to be always correct with the assessment.

jade raven
urban sandal
#

No. Static analysis should not require a runtime isinstance.

jade raven
#

so how would you try to parse that type alias?

quick snow
#

You should make the same assumptions as mypy does, trust in the stubs etc. and accept that you're not 100% correct and complete.

urban sandal
#

you're as correct as the type system supports if you do that, which is constantly improving. when static analysis tools have full view of an application, and there are no reachable code paths that use untypable constructs, then short of pathlogogical things like intentionally replacing with a mismatched type using a debugger at runtime (Something static analysis can't predict), you have good information.

jade raven
#

out of something like this?

urban sandal
#

Yes, you'd then use the ast to resolve types.

jade raven
#

but then how do i differentiate it from a normal assignment?

urban sandal
#

If it's a module level assignment of a type to an identifier, it can be treated as a typealias. At runtime, this is also just normal aliasing.Prior to 3.10/3.12, this was the only option for how type aliases like this would be created

in 3.12+, there's the dedicated type statement syntax (3.12+)
There's also the ability to explicitly declare this by annotating it as a TypeAlias. (3.10+ or typing_extensions)

you can read more about it in the specification here. https://typing.readthedocs.io/en/latest/spec/aliases.html

jade raven
urban sandal
jade raven
#

currently i believe i'm making a lot of wrong assumptions, but the output is technically correct

#

i was initially just going to use mypy's stubgen, but it's very coupled with mypy itself

halcyon trail
#

What's the best channel to talk about package managers? Packaging and distribution seems more from the perspective of someone who wants to distribute a package than someone who wants to setup an environment

merry bramble
sturdy timber
halcyon trail
halcyon trail
jade raven
halcyon trail
#

I will probably try asking there because "why not", but my point is that someone there could easily have some extremely technical issue with micromamba, that is totally valid, but not really representative at all of the reason why it's not more widely used.

#

Really I just want to understand what exactly ordinary python devs mean when they complain about the state of python package management.
I feel like I'm in many convos/interactions where it's taken as a given that python's package management situation is bad, and I want to understand why people feel that way, exactly.

jade raven
#

i would think package maintainers would be accutely aware of the issues most end users would have with their tools

halcyon trail
#

well, anyhow, I will ask there, and see what shakes out, if anything 🙂
since we're in this already do you have any thoughts?

jade raven
#

i wonder if there's a way to find the import name of all pip installed modules (programatically)

dusk comet
#

importlib should be able to find all importable modules

jade raven
#

basically a pip install name isn't neccesarily the name they're imported with

#

my end goal is to basically pip freeze and then write their importable names to a main.py script if that makes sense

steel solstice
#

you want the importlib.metadata probably then

merry bramble
# jade raven i wonder if there's a way to find the import name of all pip installed modules (...

You can do something like this to get a list of importable top-level modules provided from a distribution that's been pip-installed:

import importlib.metadata
import inspect

def get_pkgs_associated_with_requirement(req_name: str) -> list[str]:
    dist = importlib.metadata.distribution(req_name)
    toplevel_txt_contents = dist.read_text("top_level.txt")
    if toplevel_txt_contents is None:
        if dist.files is None:
            raise RuntimeError("Can't read find the packages associated with requirement {req_name!r}")
        maybe_modules = [f.parts[0] if len(f.parts) > 1 else inspect.getmodulename(f) for f in dist.files]
        return [name for name in maybe_modules if name is not None and "." not in name]
    else:
        return toplevel_txt_contents.split()
jade raven
#

!e ```py
import importlib.metadata
import inspect
import pkg_resources

def get_pkgs_associated_with_requirement(req_name: str) -> list[str]:
dist = importlib.metadata.distribution(req_name)
toplevel_txt_contents = dist.read_text("top_level.txt")
if toplevel_txt_contents is None:
if dist.files is None:
raise RuntimeError("Can't read find the packages associated with requirement {req_name!r}")
maybe_modules = [f.parts[0] if len(f.parts) > 1 else inspect.getmodulename(f) for f in dist.files]
return [name for name in maybe_modules if name is not None and "." not in name]
else:
return toplevel_txt_contents.split()

installed = pkg_resources.working_set
results = set()
for pkg in installed:
results.update(get_pkgs_associated_with_requirement(pkg.project_name))

print(results)```

fallen slateBOT
#

@jade raven :x: Your 3.12 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "/home/main.py", line 3, in <module>
003 |     import pkg_resources
004 | ModuleNotFoundError: No module named 'pkg_resources'
jade raven
#

ah

merry bramble
#

It's best to avoid pkg_resources where possible; it's deprecated and has very bad performance characteristics

#

It should be possible in nearly all cases to do what you want with importlib.metadata, but it's sometimes annoyingly hard to find out what the equivalent idiom is...

jade raven
#

i don't quite like how it includes pycache however

tight parrot
#

I’m quite interested in a comprehensive python language test suite, similar to ruby/spec (not from the viewpoint of BDD). The number of python compilers/interpreters is growing pretty quickly, mostly all of them have some quirks and differences with the reference implementation (cpython). And the only language reference is the cpython source code itself, not even the documentation. Not sure if this is the best channel for such discussion, but maybe you folks know some existing suite or can point me to some community that would be interested in creating a new one?

feral island
spark magnet
restive willow
#

How do i compare the first letter of two or more words and print the one that is first alphabetically ?

faint river
#

although just asking for the answer isn't really a helpful question to your learning

restive willow
#

oh thanks

restive willow
faint river
# restive willow hmm than how should I ?

provide what code you have written to try to solve the problem, and also include what issue you ran into. Then people can help you understand why the issue happens.

restive willow
#

Well the issue is that I have not thought of any way to do it. Thats the only reason why I came here to look for help

faint river
#

ok, continue in a different channel

#

try to solve part of the problem if you don't know how to solve the whole problem

restive willow
#

hmm I will try I guess

jade raven
jade raven
#

<@&831776746206265384>

loud summit
#

!pban 1036695883217117225 14d We are not a chain mail server. Don't post useless spam here.

fallen slateBOT
#

:incoming_envelope: :ok_hand: applied ban to @merry field until <t:1713005560:f> (14 days).

fast bloom
#

hello how can i setup flask on pycharm 2023 community eddition

#

i downlooded the library and used the template code but it gave me a lot of errors

safe basalt
fast bloom
stone elbow
#

🤨

naive saddle
#

We are not going to help you with malware here, full stop.

#

I'm not sure how packaging malware into an exe relates to security research.

stone elbow
#

sir this is a wendys

stone elbow
#

Am i good guy? : False

#

well that clears it up

naive saddle
#

we aren't going to condone this sort of vigilantism here

stone elbow
#

also this is the wrong channel

sterile totem
jade raven
#

you here to troll?

civic marsh
#

no just a super sigma guy working on malware 🔥

quick snow
#

Wasn't there a PEP about arbitrarily tagged strings, something like s[sql]"SELECT * FROM table"? Or did I dream that?

grave jolt
fallen slateBOT
quick snow
grave jolt
#

well... it is deferred 🙂

grave jolt
quick snow
# grave jolt what did you want?

A string tagged with some arbitrary other string (e.g. a language tag). Such that editors can syntax-highlight my SQL string inside my Python code correctly.

#

I think copying Markdown syntax would be great:

conn.execute(
    `‌``sql
        DROP TABLE students;
    `‌``
)
grave jolt
#

If it's just for editors, I think picking a comment style like # syntax: sql might be better than adding a new syntax element

quick snow
tulip burrow
half wolf
half wolf
stuck apex
#

Hi, I'm not sure if I'm supposed to ask this here or in python-help, I have a question regarding formatting long key-value pairs in dictionaries according to PEP 8:

PEP 8 lays out guidelines on how to indent code as well as where to put the closing bracket in a construct, but how does this all apply to dictionaries?

Which one of the following (if any) would be correct?

image.png image.png
#

“python glossary = { "Operating System": ("An operating system (OS) is system software " "that manages computer hardware and software " "resources, and provides common services for " "computer programs."), } “

#

“python glossary = { "Operating System": ("An operating system (OS) is system software " "that manages computer hardware and software " "resources, and provides common services for " "computer programs." ), } “

#

Thanks in advance for any help ❤️

raven ridge
#

It formats this as: ```py
glossary = {
"Operating System": (
"An operating system (OS) is system software "
"that manages computer hardware and software "
"resources, and provides common services for "
"computer programs."
),
}

grave jolt
#

This data seems like it might belong in a file, or something

merry bramble
grave jolt
#

that is true 🙂

#

I meant something like a separate data file that is formatted in a way that makes sense for mostly-text data

stuck apex
# raven ridge It formats this as: ```py glossary = { "Operating System": ( "An ope...

Okay. Thanks for the response. I agree, that output makes sense in terms of readability.

I've heard of black and other formatting solutions, but right now I'm just starting learning Python and I'm intentionally just working through a CLI with stuff like Vim so that I completely understand what I'm doing and if something doesn't run or I have a formatting issue I'm the only source for a problem, so I can hopefully learn.

stuck apex
raven ridge
#

even if you don't want to run Black, my formatting recommendation is still just "do what Black does". You can do that manually by imitating its style, it's just more work 🙂

stuck apex
stuck apex
halcyon trail
#

there's only one gotcha with formatting with black/ruff, which is that it really does a horrible job with multi context managers

#

as of python 3.10 or 3.11, you can insert parens around all of the context managers, and then it formats it nicely

#

but for whatever reason, black/ruff will not do this for you

#

so it's kind of a trick you have to know. doesn't come up often though

stuck apex
pliant tusk
#

!e py import dis dis.dis('a and b and c')
is the optimizer meant to optimize out multiple returns like this?

fallen slateBOT
#

@pliant tusk :white_check_mark: Your 3.12 eval job has completed with return code 0.

001 |   0           0 RESUME                   0
002 | 
003 |   1           2 LOAD_NAME                0 (a)
004 |               4 COPY                     1
005 |               6 POP_JUMP_IF_FALSE        8 (to 24)
006 |               8 POP_TOP
007 |              10 LOAD_NAME                1 (b)
008 |              12 COPY                     1
009 |              14 POP_JUMP_IF_FALSE        3 (to 22)
010 |              16 POP_TOP
011 |              18 LOAD_NAME                2 (c)
... (truncated - too many lines)

Full output: https://paste.pythondiscord.com/ZXIKAA5XMEIVSHJD4LRPFRVGHM

pliant tusk
#

the last 3 instructions are all RETURN_VALUE

dusk comet
#

optimizer optimizes for code speed (pretty poorly), not for code size

#

code deduplication is not a big concern, but requires some work to be done in the compiler

jaunty radish
#

i need help in RASA chat-bot api

pliant tusk
flat gazelle
#

Code duplication is also occasionally helpful for the adaptive interpreter, since it allows the same bit of code to adapt in different ways (not that that's useful with RETURN_VALUE specifically).

dusk comet
# 
def add(x, y):
    if isinstance(x, int):
        return x + y
    else:
        return x + y
terse steppe
rose schooner
#

python does that in bytecode

prime estuary
#

Even with duplicate code like this, the compiler can't get rid of one because of tracing/profiling support. If you enable a debugger, coverage etc it needs to ensure correct line numbers are produced. If you disassemble a try: block, you'll see a NOP that had to be added for the same reason.

feral island
#

!pep 659

fallen slateBOT
feral island
#

Because the specializer can turn the first x + y into a specialized instruction for adding ints

#

In this case, it's unlikely to be a net positive because the cost of the isinstance() will dwarf the gain of specializing for int addition, but I can imagine this improving performance if you perform lots of additions in a function

rose schooner
# feral island Because the specializer can turn the first `x + y` into a specialized instructio...

seems like it ```pycon

def add(x, y):
... if isinstance(x, int):
... return x + y
... else:
... return x + y
...
for _ in range(1_000): add(5, 4) and None
...
for _ in range(1_000): add("a", "b") and None
...
from dis import dis
dis(add, adaptive=True)
1 0 RESUME 0

2 2 LOAD_GLOBAL_BUILTIN 1 (NULL + isinstance)
12 LOAD_FAST 0 (x)
14 LOAD_GLOBAL_BUILTIN 2 (int)
24 CALL_NO_KW_ISINSTANCE 2
32 POP_JUMP_IF_FALSE 5 (to 44)

3 34 LOAD_FAST__LOAD_FAST 0 (x)
36 LOAD_FAST 1 (y)
38 BINARY_OP_ADD_INT 0 (+)
42 RETURN_VALUE

5 >> 44 LOAD_FAST__LOAD_FAST 0 (x)
46 LOAD_FAST 1 (y)
48 BINARY_OP_ADD_UNICODE 0 (+)
52 RETURN_VALUE

unkempt rock
#

side note

#

made this with python

faint river
plain lichen
#

I wonder if there's any tool(s) that can provide CPython-level stack trace. For example, l = [1,2,3] in Python may call PyList_New() in CPython, and I'd like to see PyList_New() in the stack trace reported by the tracer tool(s).

I found Py3.12 has support for Linux perf (https://docs.python.org/3/howto/perf_profiling.html), which basically gives what I wanted. But when I tried it on Ubuntu 22.04 with Python3.12.2, it didn't always work - symbols were hex numbers instead of human-readable strings. Not sure if this functionality was not fully supported yet.

Thanks in advance.

raven ridge
#

You should get human readable strings if you have debug symbols available. You may be able to get that by installing a python3.12-dbg package, or possibly by setting DEBUGINFOD_URLS=https://debuginfod.elfutils.org/ as an environment variable

raven ridge
plain lichen
#

Okay, I used deadsnakes PPA but didn't install debug symbols. With debug symbols, perf tool worked. Thx.

plain lichen
#

I think python-level tracing is pretty clear, with those builtin profiling modules. But the C-level tracing seems hard

raven ridge
raven ridge
# plain lichen I found pystack pretty interesting. Just curious, how's that implemented? Maybe ...

Much like gdb. A remote process's memory is read using either ptrace or process_vm_readv, a core file's memory is read by interpreting the ELF LOAD segments it contains. From there, the tool uses version specific implementation details of the CPython interpreter to determine the Python stack, GIL state, local variables, etc, and uses elfutils (libelf and libdw) to unwind and symbolify C stacks

#

And then uses more CPython implementation details to stitch the C stack and Python stack together, at least for modern CPython versions

plain lichen
#

I see. Lots of things within a few sentences lemon_s_winter I'll explore it more if I get a chance. Thanks for the help!

raven ridge
#

The ELI5 version is that it works much like a C debugger like gdb or lldb does, with a bunch of CPython specific black magic sprinkled in

sharp plover
#

Hello. I was wondering if there's any reason why heapq functions couldn't be applied to array.array objects? Currently these functions require the heap to be represented as a list, although arrays implement essentially the same interface as lists so it should just work as far as I can tell. Someone was asking in #algos-and-data-structs.

dusk comet
#

i guess there is a hardcored if not isinstance(arg, list): raise TypeError somewhere inside
iirc, heapq is implemented in python
you can try removing the check, and if it still works you can open issue/PR

sharp plover
gray galleon
#

should heapq have its own heap class

flat gazelle
#

probably, yeah.

gray galleon
#

having a different interface to do operations on heaps compared to other collections is awkward

native flame
#

!d queue.PriorityQueue exists but ive never seen people actually use it in place of heapq

fallen slateBOT
#
queue.PriorityQueue 

class queue.PriorityQueue(maxsize=0)```
Constructor for a priority queue. *maxsize* is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If *maxsize* is less than or equal to zero, the queue size is infinite.

The lowest valued entries are retrieved first (the lowest valued entry is the one that would be returned by `min(entries)`). A typical pattern for entries is a tuple in the form: `(priority_number, data)`.

If the *data* elements are not comparable, the data can be wrapped in a class that ignores the data item and only compares the priority number:
grave jolt
#

Given that it was written 17 years ago, it's more or less dead

#

"generic functions" as described in that PEP are available as functools.singledispatch

#

typing.overload is only for static analysis purposes though

#

I'd probably just make two different functions

#

Or you could have a single function that accepts an integer or a date as an argument

dusk comet
#

!pep 744

fallen slateBOT
dusk comet
# 
>>> b''.splitlines(str)
[]
>>> b''.splitlines(print)
[]
>>> b''.splitlines(...)
[]
``` bug?
#

it is not supposed to take these arguments, it should have raised TypeError for too many arguments

#

!d str.splitlines

fallen slateBOT
#
str.splitlines 

str.splitlines(keepends=False)```
Return a list of the lines in the string, breaking at line boundaries. Line breaks are not included in the resulting list unless *keepends* is given and true.

This method splits on the following line boundaries. In particular, the boundaries are a superset of [universal newlines](https://docs.python.org/3/glossary.html#term-universal-newlines)...
feral island
#

I guess it converts everything to a boolean

#

not great but probably not something we can change

dusk comet
#

that makes sense

merry bramble
#

later discussed in https://discuss.python.org/t/boolean-arguments/21662/14

Discussions on Python.org
Boolean arguments

I’m not sure if that it’s a good idea. The int type has a bool() method. Does it mean that 0 and 1 are considered as boolean? Other examples: >>> (3.14).bool() # float True >>> (5j).bool() # complex True The numpy.bool_ does not inherit from Python built-in bool type. While I’m not surprised by flag=1 instead of flag=True, usin...

void storm
#

I was just experimenting with memory leaks in python, I wanted to learn about how it works in python.

So I made a memory leak program, two functions that leverages two different methods to leak memory

Closure:

def leak_memory():
    leak = []

    def inner():
        while True:
            leak.append("Bug")

    inner()
    return leak


leak_memory()

Global variables

def leak_memory():
    global leak
    leak = []
    while True:
        leak.append("Bug")


leak_memory()

And I got some interesting results that I want to learn about.

First thing is that both of these methods take different times to fill up the memory, and from my testing, closure method is a little more than twice as fast as the global variable method.

For the next point a little context is, both of these methods take negligible amount of CPU usage
Second thing is in which I am more interested in is that, when Linux kernel intervenes after the memory is full, the CPU usage peaks.
With the closure method the CPU usage maxes out at 100% and the system freezes for rougly 2 seconds.
But with the global variable method, the CPU usage caps out at about 84%, and the crash is almost instant.

So can someone explain this behaviour to me please? (if you are replying please ping me! and also thank you ♥️ )

grave jolt
#

closure method is a little more than twice as fast as the global variable method.
That's likely because accessing a closure variable is faster than a global variable

#

accessing a global variables requires finding an entry by key in the globals() dictionary, while the location of a closure variable (also known as "cell variable") is more static, just like with local variables

dusk comet
grave jolt
void storm
void storm
#

It is when the kernel intervenes that CPU usage peaks

void storm
grave jolt
#

as I said, both eat 100% of a CPU core for me while they're running

void storm
#

while running?

#

Well that's strange

#

okay so pardon me, my initial observation was wrong. It does take up some CPU usage, 8.3% to be exact. Both methods take up exactly 8.3% CPU usage.

#

But throughout

#

There is no going above 8.3% ever

#

So my guess is this probably means system i.e., the kernel is using the rest of the CPU

flat gazelle
#

does your system have 12 cores?

void storm
#

yes it does

#

That's intersting though! How'd you guess?

grave jolt
#

!e

print(1/12)
fallen slateBOT
#

@grave jolt :white_check_mark: Your 3.12 eval job has completed with return code 0.

0.08333333333333333
void storm
#

Sorry it doesn't have 12 cores, but 12 threads

flat gazelle
#

the CPU usage is for the entire CPU, not per thread

#

so 8.3% means 100% of a single thread

feral cedar
#

this wouldn't be called a memory leak, though. it's just using up all the memory

void storm
grave jolt
void storm
void storm
void storm
grave jolt
#

If you use top, it will show you the "CPU usage" (in the %Cpu line), that's the whole CPU, all the 12 hyperthreads. But every process line lists CPU usage such that 100% is a single core

image.png
void storm
#

oh that's what you mean by 100% CPU usage

#

I got it

grave jolt
#

so a particularly hungry multithreaded program (like ffmepg) will shows as almost 1200% in the %CPU column

void storm
#

Alright thanks for explaining that

#

But I don't understand why does system uses max CPU to terminate the program

#

like all threads

feral island
#

maybe use a profiling tool to figure out what's happening

#

maybe GC gets heavily triggered around the time it gets killed or something

gray galleon
void storm
feral island
#

profile the program with a C-level profiler like perf

feral cedar
void storm
raven ridge
dusk comet
#

OS always keeps track of any memory, and is prepared to free it as soon as you kill the process
so memory leaks are impossible 😉

#

OS is the best GC

raven ridge
hazy patio
#

Oops

fallen slateBOT
#

:incoming_envelope: :ok_hand: applied timeout to @hazy patio until <t:1713128668:f> (10 minutes) (reason: duplicates spam - sent 4 duplicate messages).

The <@&831776746206265384> have been alerted for review.

clear nexus
#

Hi , could you please help me with the error is giving me

jade raven
jade raven
#

please check out the channel description and make sure you stick to the topics

unkempt rock
unkempt rock
#

or screenshot server

#

not server

#

mb i emant channel

jade raven
#

no, but you can send images in any of the offtopic channels.

unkempt rock
#

hi

rapid sigil
#

hi! i've got a tricky case of accessing internal attribute dictionary when self.__dict__ was overridden, how can I access it?
#1230932562499862578 message

radiant garden
#

one easy method is to use the default pickling methods (3.11+)

class MyClass:
    def __init__(self) -> None:
        super(MyClass, self).__setattr__("attribute", False)
    __dict__ = {}

obj = MyClass()

print(obj.__dict__) # {}
print(obj.__getstate__()) # {"attribute": False}
craggy heath
#

!e
print(1/12)

fallen slateBOT
#

@craggy heath :white_check_mark: Your 3.12 eval job has completed with return code 0.

0.08333333333333333
craggy heath
#

!e
print(1/1112)

fallen slateBOT
#

@craggy heath :white_check_mark: Your 3.12 eval job has completed with return code 0.

0.0008992805755395684
quick snow
fallen slateBOT
#

:incoming_envelope: :ok_hand: applied timeout to @normal cliff until <t:1713708211:f> (10 minutes) (reason: burst spam - sent 8 messages).

The <@&831776746206265384> have been alerted for review.

spark verge
#

Anyone know why async_generator.athrow().close() and async_generator.asend().close() are No-ops? I think they should throw a GeneratorExit into the async generator and raise RuntimeError if the underlying generator doesn't close

dusk comet
#

i guess x=async_generator.athrow(); x.close() closes x, not async_generator

spark verge
#

But then the AsyncGenerator is broken

spark verge
#

Consider calling await anext(async_generator)

#

And then someone closing the current Coroutine

#

You'd expect a GeneratorExit thrown into the async generator

#

I think

spark verge
#

Or worse someone throwing GeneratorExit into the coro

rose schooner
#

python/cpython#118130 wow

neon troutBOT
rose schooner
#

python sure is unquestionable about its loyalty to monty python

flat gazelle
half vessel
#

can some one help me with an assigment im stuggling on

flat gazelle
half vessel
#

im trying to do it but im stuck

#

?

#

and the help channel dosent have what I need so im kinda at my wits end right noiw

misty oxide
#

How can I get a list of direct referents from the garbage collector? The gc module only lets you get all referents. I'm willing to write C to get only the direct ones.

#

Is inspect.getmembers() what I want?

spark magnet
misty oxide
#

I just figured that out.

#

I'm trying to write a GUI for inspecting references.

#

@spark magnet

#

To do what I originally thought gc.get_referents() did, it's just:

def find_all_reachable_objects(*objects) -> list:
    seen = set()
    stack = list(objects)
    reachable = []
    while stack:
        obj = stack.pop()
        if id(obj) not in seen:
            seen.add(id(obj))
            reachable.append(obj)
            stack.extend(gc.get_referents(obj))
    return reachable
spark magnet
#

sounds like a cool project. It could get into some tricky edge cases.

misty oxide
#

I am experiencing that :)

#

There are objects for which calling getattr() attempts to import cffi.

#

Functions also all hold a reference to globals()

#

So there's really no way to contain the scope without nailing every edge case, but it's unclear if nailing every edge case is even a good idea.

spark magnet
#

yes, and your code will have references to things.

misty oxide
#

The correct approach, I think, is bottom up rather than top down.

#

But then you have the issue of figuring out how to cut off every reference in your own tracing code. Still probably the better approach though.

spark verge
spark verge
misty oxide
dark umbra
#

Can anyone plz suggest me data analysis course which is both rigorous and indepth i saw few courses from harvard and they are not in depth and fully explain ain and out of concept

halcyon trail
#

anybody know why dict's operator |

#

requires a dict on the RHS as well as the left

feral island
halcyon trail
#

Why did list do it then

feral island
raven ridge
#

Both lists and tuples support + and it would be strange if [1, 2] + (3, 4) == [1, 2, 3, 4] but (1, 2) + [3, 4] == (1, 2, 3, 4)

#

that particular example can be fixed by making the two types know about each other and coordinate on what the type of the returned object should be, but that doesn't scale to supporting arbitrary sequences

#

!e for an even more fun example: ```py
l = []
l += {1: 2}
print(l)
l = l + {3: 4}

fallen slateBOT
#

@raven ridge :x: Your 3.12 eval job has completed with return code 1.

001 | [1]
002 | Traceback (most recent call last):
003 |   File "/home/main.py", line 4, in <module>
004 |     l = l + {3: 4}
005 |         ~~^~~~~~~~
006 | TypeError: can only concatenate list (not "dict") to list
raven ridge
#

would you want [] + {3: 4} to be [3]?
What about {3: 4} + []?

#

I think commutativity, rather than associativity, is probably the reason. It's weird for a + b to differ from b + a

halcyon trail
thin hinge
raven ridge
#

sure, true. + is used for both sequence concatenation and addition in Python. but at least the result of both of those is the same type

unkempt rock
#

guys i accedienty pinged everyone

#

oopsies

faint river
unkempt rock
#

yeah ik

#

but i wrote @ everyone

#

and then

#

i got a dm

#

saying you shouldnt do that

#

and gave me a warning

faint river
#

ok so why are you talking about this in this channel?

unkempt rock
faint river
#

next time, pick a more general channel instead of a topical channel

unkempt rock
#

oh wait

#

so

#

i havea question

faint river
unkempt rock
#

so yes

#

is variable_name same as "variable name"

#

or

faint river
#

there aren't topical channels for the bare basics of the language

merry obsidian
merry obsidian
quick snow
fallen slateBOT
#
Rules

6. Do not post unapproved advertising.

rustic jungle
grave jolt
#

This is a Python server

quick snow
#

!pep 667 was accepted, nice. No more PyFrame_LocalsToFast

fallen slateBOT
cerulean siren
#

Hey Everyone!
I just recently updated my windows.
Yesterday, I clicked "update and shutdown" option.

Today when I turned on my laptop, my data was lost as I reset my windows. My lock screen had the same wallpaper. Then I restarted my laptop and my Lock screen didn't have the same wallpaper.

gray galleon
grave jolt
obtuse wraith
covert fiber
#

anyone have him added or friend w him?

image.png
quick snow
misty oxide
#

Is there an explanation anywhere of why comprehensions are parsed and compiled as their own functions?

feral island
#

!pep 709

fallen slateBOT
misty oxide
#

Excellent, thank ya ❤️

feral island
#

Though this mostly covers an implementation detail that changed in 3.12. The reason they use separate functions generally is so that they can have their own scopes. Carl's innovation in 3.12 allows the user-visible scoping behavior to stay the same while still preserving the separate scopes.

misty oxide
#

This was exactly the implementation detail that I was wondering about. It seems incredibly wasteful to create a frame for such a thing.

boreal umbra
#

Has there ever been discussion of a str.isempty method that returns True for empty strings and space-only strings? It's trivial to get with not s.strip(), but isn't very self-documenting.

feral island
#

which of course is also not a very self-documenting name

boreal umbra
feral island
#

I don't recall any discussion about this

feral island
boreal umbra
#

though I think that isspace is as self-documenting as isupper, islower, etc.

grave jolt
#

I really don't like the noseparatorcase

#

isupper 🍲
islower ducky_snucky

quick snow
#

isprintable 🏃

#

isspace 🛰️ ⏲️

dusk comet
#

half of sys module is in snake_case, half is in nosepcase

eager lava
#

!traceback

gray galleon
rose schooner
wanton flame
rose schooner
graceful dirge
#

@warm cape

warm cape
faint river
#

or at least post images in #bot-commands or an ot channel

graceful dirge
warm cape
graceful dirge
worthy sandal
#

Hi Guys, I am new to api development in python. could anyone please suggest me some good resources where can i learn the concepts of api development with python

dusk comet
#

step 1. go to the correct channel
step 2. check pins
step 3. run !resources in #bot-commands

spice escarp
#

Dose anyone know how to make a esp for fivem

dusk comet
fallen slateBOT
#

Lib/_pyrepl/pager.py line 26

return plainpager```
feral island
minor island
#

Any landing page idea

spark verge
minor island
#

Ok

spark verge
grave jolt
#

!cban 1176127939088232498 shady advertisements

fallen slateBOT
#

:incoming_envelope: :ok_hand: applied ban to @burnt flicker permanently.

merry venture
#

given a frame frame, would you agree that frame.f_locals is frame.f_globals is an ultimate way of determining whether the frame refers to module-level code, not one inside a function for example? do you have an idea for a better way of examining that?

# 
def inside_f() -> None:
    print(locals() is globals())  # False

print(locals() is globals())  # True
inside_f()

EDIT: hm, actually, I want to also include cases like ```py
class Foo:
print(locals() is globals()) # False (obvsly)

since they fire on module execution, so nvm
#

also, two questions regarding the current type system

  1. unpacking unions of tuples
    don't you think it should be possible to unpack same-size unions of tuples?
    currently, it's not possible to do something like
# sorry, I'm writing for 3.8+ here since OSS forces me to... sort-of
from __future__ import annotations
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    # OptExcInfo is a union of tuple[None, None, None] and tuple[type[BaseException], BaseException, types.TracebackType]
    from _typeshed import OptExcInfo 
    from typing_extensions import Unpack

class CM:
    def __exit__(self, *args: Unpack[OptExcInfo]) -> None:
        pass
fallen slateBOT
#

configzen/data.py lines 93 to 97

# Unpack[DataFormatOptionsType] cannot be used here,
# because this functionality is not supported by mypy yet.
# Override the **options annotation in your subclass of DataFormat with
# the subclass of DataFormatOptions corresponding to your subclass of DataFormat.
def configure(self, **options: Unpack[DataFormatOptions]) -> None:```
merry venture
#

@feral island do you think it's worth a pep? "Extending typing.Unpack capabilities"?
the use cases are I think quite niche, but on a more advanced level would be very cool and I don't think they're very hard to implement...

#

and actually if there's a more useful thing that needs someone to pick up and contribute a solution, let me know...

feral island
feral island
merry venture
#

oh, great! I'll pick up some then. thanks for such a quick reach out!

feral island
#

though I'd caution that for mypy the limiting factor is often reviewer time. I'll try to review PRs especially related to support for new typing features, but in many parts of mypy I'm out of my depth

merry venture
#

if it's welcome, I'll review when I have time

feral island
#

definitely!

wispy sable
feral island
#

LOAD_NAME is a useful opcode too

#

That sounds like you're doing something very advanced that needs a careful understanding of the bytecode (a debugger?)

#

It's going to be hard to suggest an alternative without knowing more about what you're trying to achieve

#

yes, loading a different global should work with changing co_names. Other changes to the bytecode would need more work. Possibly the dis module can help but it's going to get hairy

pliant tusk
#

you could technically abuse some other opcodes to load globals by abusing where the globals dict pointer is stored in a given frame, but it would be unstable

vital heath
#

is there a way to keep track of status of pep 703?

spark magnet
vital heath
#

oh ok - so it's targeted to 3.13...thx

feral island
spark magnet
wanton flame
spark magnet
spark magnet
#

this sounds like it runs counter to the server rules.

wanton flame
spark magnet
spark verge
#

When did async generators stop being provisional? They were released provisionally in 3.6

#

PEP 525, Asynchronous Generators (provisional)

vagrant wolf
#

Hello Everyone. I am new to python and coding. Could anyone suggest me a good source to learn python briefly and quickly. I also wanna learn pandas in python.

quick night
halcyon trail
#

I'm only now learning that if you add an hour to a python datetime, you don't necessarily get a datetime an hour later 😢

dusk comet
#

huh?

#

timezone shenanigans i suspect

halcyon trail
#

Yeah

#

python's interpretation here is ultra weird, imho. it just increments the clock by whatever amount, as though DST didn't exist

#

So like, if you decide to add exactly 30 days (i.e. 30 * 24 * 60 * 60 seconds, not incrementing the date 30 times) as as time delta to a typical zoned time in late february

#

what you get is probably not what you want; and the more you understand timezones, the less likely it is to actually be what you want (I would say)

#

it's incredibly unfortunate that python doesn't have separate classes for naive, and timezone aware, datetimes, imho

dusk comet
halcyon trail
#

hah

#

its' just not going to be that time interval later, which is pretty bizare. when people do stuff with timedeltas, that's generally what they expect

raven ridge
#

!e This works the way I'd expect: ```py
import datetime, zoneinfo
dt = datetime.datetime(2024, 11, 3, 1, tzinfo=zoneinfo.ZoneInfo("America/New_York"))
print(dt)
print(dt + datetime.timedelta(hours=1))

fallen slateBOT
#

@raven ridge :white_check_mark: Your 3.12 eval job has completed with return code 0.

001 | 2024-11-03 01:00:00-04:00
002 | 2024-11-03 02:00:00-05:00
raven ridge
#

wait, no, actually - no, it doesn't. Huh.

#

I was expecting 01:00:00-05:00. Hm.

halcyon trail
#

Oops sorry

#

Didn't see your last message

raven ridge
halcyon trail
#

I thought that the jump occurs between 2 and 3

#

The simplest way really is just to add a day, on the Saturday before dst

#

You'll get the same time, but on Sunday, which is IMHO pretty much not what most people experienced with such libraries expect

raven ridge
halcyon trail
#

Right

raven ridge
#

so, yeah, I agree with you - the behavior isn't what I expect, and somehow I never noticed that... 🤯

halcyon trail
#

Yep

#

That was my reaction

dusk comet
#

what happens when DST moves time 1 hour back?
lets assume today is a normal day, and tomorrow DST will start
what time will be after 23:59? 23:00 of the same day?

halcyon trail
#

Exactly

halcyon trail
#

And you have a one hour range of times that are ambiguous

#

There's no way to unequivocally map them back to UTC

dusk comet
#

so there are two moments that are denoted using two indistinguishable times?

halcyon trail
#

Yes

dusk comet
#

i cant express by emotions and feelings right now...

halcyon trail
#

So, in New York time zone for example, in November there's a Sunday where you repeat 1am twice

raven ridge
#

at least for timezone-aware datetimes

dusk comet
#

eww

raven ridge
#

1 hour after 1 AM November 3rd EDT is 1 AM November 3rd EST

#

I mean, sure, "eww", but that's how DST works

halcyon trail
#

The time zone offset isn't specified though anywhere. Usually the model is that the time zone offset is an output, not an input

raven ridge
#

it's an input via fold

halcyon trail
#

Sure

#

But in most real world problems of course that's not really specified.

raven ridge
#

sure

halcyon trail
#

Anyhow I think personally this behavior is just dead wrong

#

There's 3 choices a datetime library can make her and this is by far the worst one

#

C++ seems to just disallow arithmetic on zoned times, afaict

#

Which is probably my preferred choice, though I suspect a lot of developers first reaction to that will be to complain about verbosity

dusk comet
#

we need better calendar, current one is awful

#

also, ban DST

halcyon trail
#

As far as I can see the most reasonable mix of correctness and maturity is pendulum

#

Pendulum though repeats the really awful mistake of having a single type for both naive and aware datetimes

urban sandal
#

The best option is to not use datetime objects at all. Store the original text input you got from a user about their time and localize it on demand. Do any time shifting yourself. This avoids most of the problems, including the one where you stored a timestamp and not the time they specified + their timezone to then between when they gave you that input to schedule something, and months later a government decided "actually, DST is postponed a week" making the precalculated timestamp wrong.

halcyon trail
#

Uh, for anyone reading, that is really terrible advice, especially "do the time shifting yourself"

quick snow
#

It depends. One of the problems with time is that in different contexts, we mean different concepts by it. Sometimes it's "a globally unique point in time", sometimes "a human-specified label for an event time".

If it's the former, ideally you convert into UTC at the earliest possible moment, and convert to local time for display purposes only.

If it's the latter, then it depends.
For example, if I go on a transatlantic business trip, I want my online meeting times (which didn't change) to just be displayed in local time. But my alarm clock shouldn't be displayed in local time, but be reinterpreted in local time.

And for the last case @urban sandal mentioned he is absolutely right. Although I might save the entry as a combination of naive datetime (event time) and aware datetime (scheduling time) instead of a string in that case.

#

TL;DR: Time is complicated not because of Python design decisions, but because of the problem domain being complex.

urban sandal
#

there would never be a single correct way to handle this problem that works for all kinds of applications, but there's also no way to fix it now, the presence of operators with expected behaviors, rather than functions which can be chosen for the correct kind of behavior needed ("should this be added absolutely, or added as natural wall time, and how to handle ambiguous timezones?") somewhat precludes fixing it. It isn't something that a single math operator can handle for all use cases.

urban sandal
#

Gonna add a clarification based on a dm I got about this: doing it yourself does not need to mean reinventing the wheel from first principles, but the moment you care enough about timezones and user perceptions of time to start looking into aware datetime objects, you should consider how much the tools you have already do what is right for your use case, and which behaviors if any you should create something else for.

Its also perfectly okay to decide "im okay with the behaviors other people picked for this" even if it isnt perfect. Design vs time tradeoffs are a fair and valid concern

halcyon trail
#

Time is complicated, and datetime has some really bad design choices

#

Believe me that me, and the author of the linked article are aware of this one paragraph shpiel about different concepts in time. I'd suggest focusing on the specific points

#

And you said yourself that you wouldn't store as a string. Would you "shift yourself"?

errant oak
#

For anyone else following along, this is the article quicknir linked earlier: https://dev.arie.bovenberg.net/blog/python-datetime-pitfalls/#1-incompatible-concepts-are-squeezed-into-one-class

Arie Bovenberg
Ten Python datetime pitfalls, and what libraries are (not) doing ab...

It’s no secret that the Python datetime library has its quirks. Not only are there probably more than you think; third-party libraries don’t address most of them! I created a new library to explore what a better datetime library could look like.

celest smelt
#

Can anyone guide me how do I start my DSA in Python

safe basalt
hybrid relic
#

Hello again, I've recently come across the option for enabling the experimental JIT compiler, but the parameters it accepts is making my head swim:
For configure.ac:
no (aka --disable-experimental-jit with no argument)
yes (aka --enable-experimental-jit with no argument)
yes-off
interpreter
interpreter-off
For Windows build.bat
--experimental-jit
--experimental-jit-off
--experimental-jit-interpreter
--experimental-jit-interpreter-off
configure.ac also says that interpreter-off is apparently a secret option as well. But that doesn't mean much to me, since I have no clue what it does. What does each option actually mean? Even more confusingly, when I run build.bat with --experimental-jit-off the build seems to try to build the JIT even when I specifically said not to include the JIT, I'm not sure whether that's a bug or whether it's just me being stupid

naive saddle
#

experimental-jit-off is likely yes-off:

Build the JIT, but do not enable it by default. PYTHON_JIT=1 can be used to enable it at runtime.

hybrid relic
#

ahhh, so I was being stupid after all

#

Thanks!

#

I'll admit I was completely thrown off by "JIT interpreter"

#

well, at least at first

tidal junco
#

I'm looking for core contributor reviews of my PR to expand pickle importing for PyModule_Create modules to solve a long-standing issue in PyTorch: https://github.com/python/cpython/pull/119152

I've created a discussion post as well: https://discuss.python.org/t/request-for-review-of-gh-87533-expand-pickle-importing-to-support-non-package-c-modules/53833

Looking forward to receiving feedback and hopefully solving this issue once and for all!

GitHub
gh-87533: Expand pickle importing to support non-package C-modules ...

There have been recurring issues with PyModule_Create modules in PyTorch. When trying to serialize attributes of these C-modules, pickle fails to import the C-module because they are not a packages...

Discussions on Python.org
Request for review of "gh-87533: Expand pickle importing to support...

PR: Expand pickle importing to support non-package C-modules I am looking for reviewers for a patch I desire to how pickle loads non-package modules, like those created via PyModule_Create. Erroring while trying to serialize PyModule_Create attributes has been a recurring issue in PyTorch for at least 4 years: Functions in torch._C._nn and to...

raven ridge
#

It seems to me that teaching pickle to import modules that can't be normally imported is worse than fixing those modules to actually be importable

tidal junco
#

I understand your argument. I'll look into what a solution like this might look like. Thanks for the links!

raven ridge
#

It seems to me that the issue is that torch is creating things that are "modules" in one sense (instances of PyModule_Type) but not in another much more common sense (things that you can import from)

tidal junco
#

Instances of types.ModuleType are exactly analogous to this

raven ridge
#

I mean that modules are supposed to be used with import, but they've created modules that can't be. That's the weird thing here. Pickle is just a victim of that. It's doing a thing that's supposed to work, but it doesn't, because these aren't modules in the normal sense of the word

faint river
#

<@&831776746206265384>

peak spoke
#

I've been playing around with str.translate and it's getting weirdly high runtimes when a character is mapped to "", any idea what's going on and why it's so much slower than the None case or replacements with 1 char strings strings? It's also not particularly fast when the replacement strings are longer but that's somewhat expected

In [24]: a = "abc"*50_000

In [25]: %%timeit t = str.maketrans({"a": None})
    ...: a.translate(t)
76.4 µs ± 807 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

In [26]: %%timeit t = str.maketrans({"a": ""})
    ...: a.translate(t)
5.83 ms ± 43.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [27]: %%timeit t = str.maketrans({"a": "", "b": ""})
    ...: a.translate(t)
3.66 ms ± 6.28 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [28]: %%timeit t = str.maketrans({"a": "", "b": "d"})
    ...: a.translate(t)
3.88 ms ± 54.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [29]: %%timeit t = str.maketrans({"a": "e", "b": "d"})
    ...: a.translate(t)
58.6 µs ± 500 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
quick snow
fallen slateBOT
#

Objects/unicodeobject.c lines 8692 to 8695

if (item == Py_None) {
    /* deletion */
    translate[ch] = 0xfe;
}```
somber thunder
#

Cool

ripe vale
#

!rule

fallen slateBOT
#
Rules

The rules and guidelines that apply to this community can be found on our rules page. We expect all members of the community to have read and understood these.

gray galleon
#

why is```py

issubclass(ZeroDivisionError, ValueError)
False

faint river
#

!e print(ZeroDivisionError.mro())

fallen slateBOT
# faint river !e print(ZeroDivisionError.mro())

:white_check_mark: Your 3.12 eval job has completed with return code 0.

[<class 'ZeroDivisionError'>, <class 'ArithmeticError'>, <class 'Exception'>, <class 'BaseException'>, <class 'object'>]
gray galleon
#

isn't ValueError for giving invalid arguments
so that covers division by zero as well?

spark magnet
grave jolt
#

Yeah, it doesn't matter too much. If you want to handle an error from a certain operation, you need to go to the documentation and see what exceptions it raises.

umbral plume
#

by extension, TypeErrors, IndexErrors, and KeyErrors could also all be considered categories of errors that stem from an invalid value being passed to something somewhere

gray galleon
#

fair enough

formal ore
#

cant you do except (ZeroDivisionError, ValueError):

spark magnet
jade raven
#

why aren't variables as keys invalid but valid for values?

#

!e ```py
sample = (0, 1, 2)

sample2 = {sample: "one", 1: 0, "henlo": {0: 1, 1: 0}}

match sample2:
case {sample: work, 1: 0, "henlo": {0: 1, 1: 0}}:
print(work)```

fallen slateBOT
jade raven
#

!e ```py
variadic = {0: 1, 1: 0}

sample = {0: 1, 1: 0, "henlo": {0: 1, 1: 0}}

match sample:
case {0: 1, 1: 0, "henlo": z}:
print("0:1, 1:0, henlo:", z)

fallen slateBOT
round path
grave jolt
rose schooner
grave jolt
jade raven
#

Since I was able to use z as a value

dusk comet
#

#esoteric-python message

>>> import gc
>>> class X: ...
...
>>> x = X()
>>> gc.get_referents(x)
[X]
>>> x.a = 1
>>> x.b = 'f'
>>> gc.get_referents(x)
[1, 'f', X]
>>> x.__dict__
{'a': 1, 'b': 'f'}
>>> gc.get_referents(x)
[{'a': 1, 'b': 'f'}, X]
>>> x.__dict__ = {}
>>> gc.get_referents(x)
[{}, X]
>>> x.c = 42
>>> gc.get_referents(x)
[{'c': 42}, X]

what is going on?
why isn't x.__dict__ always mentioned in gc.get_referents(x) ?
why does accessing x.__dict__ make it appear in gc.get_referents(x) ?

feral island
# dusk comet https://discord.com/channels/267624335836053506/470884583684964352/1245237262988...

instance dicts are lazily materialized. Not sure whether there's good documentation of this, but here's a related issue: https://github.com/python/cpython/issues/106485

GitHub
Reduce the number of materialized instance `__dict__`s · Issue #106...

Accessing the dict attribute of an object causes the dictionary to be materialized. This has a negative impact on performance. We should provide means to access the dict without forcing mat...

quick night
# 
>>> a = 10
>>> b = 3.1415
>>> c = True
>>> d = False
>>> print(f'{a =:>10}')
a =        10
>>> print(f'{b =:>10}')
b =    3.1415
>>> print(f'{c =:>10}')
c =         1
>>> print(f'{d =:>10}')
d =         0
>>> print(f'{c = }')
c = True
>>> print(f'{d = }')
d = False

why do booleans get converted to int here?

feral island
quick night
#

oh, they don't defined their own?

#

ah nvm, got it

faint river
#

this will apply formatting as a string

#

and can do !r for repr

#

!e print(f"{True = !s:>10}")

fallen slateBOT
quick night
boreal umbra
#

Has anyone here actually used name mangling? I've only seen it used by students whose instructors have misguided ideas about Python OOP. I've never actually seen it used "correctly".

swift imp
boreal umbra
swift imp
#

Hm then I guess I haven't

#

I thought a single leading would cause it not show up in dir

feral island
#

It does. Double underscores in a class body cause the name to be mangled, which provides some protection against reuse of the name in subclasses

#

And no, I never use it on purpose either

swift imp
#

Til

#

I don't think I've ever wrote something that was meant to be subclassed publicly to worry about this. In other words if I wrote something that supposed to be subclassed then I will do the subclassing and therefore know not make the mistake of overwriting

dusk comet
#

I see only two situations when you might want to use mangled names:

  1. if codebase you manage has pretty complicated inheritance tree, and there is a chance of accidental overriding of a name
  2. if your class is meant to be subclassed by user of your library

and you dont need mangled names in these cases:

  1. if your inheritance is so complicated, you are probably doing something wrong. And methods that you want to make mangled probably can be free functions as well
  2. usually in situations like this you are supposed to override only specified set of methods to provide necessary behaviour, and base class will act like a convenient glue that will make magic happen. Putting too many in such classes is probably not a good idea, you can extract everything else into different class/free functions
#

Sometimes there is a name that should be private, must not be accidentally overrided (otherwise it will break everything), and should be accessible from several classes that need it.
This name cannot be mangled, it will make it inaccessible from several classes, so one of the best solutions is to mangle it yourself, using library name instead of class name, like this: _mylib_crucial_method or this: __mylib_crucial_method__

this happens pretty often in many libraries:

  • __mypyc_XXX__ in mypyc-generated code
  • __array_XXX__ in numpy
  • __dataclass_XXX__ in dataclasses
  • __attrs_XXX__ in attrs
  • __ctypes_XXX__ in ctypes
errant oak
# boreal umbra Has anyone here actually used name mangling? I've only seen it used by students ...

Here's a great explanation of the purpose of name mangling and how it's supposed to be used.

https://www.youtube.com/watch?v=HTLu2DFOdTg&t=2035s

The whole talk is well worth watching, but this links directly to the part that talks about the rationale behind name mangling. (There's a minute or two of buildup.)

YouTube
Next Day Video
Python's Class Development Toolkit

Raymond Hettinger
This is a short, but thorough tutorial on the Python's built-in toolset for creating classes. We look at commonly encountered challenges and how to solve them using Python.

▶ Play video
final knoll
feral island
#

That would be a compatibility break

#

More likely we'd simply document that WASI has different behavior

final knoll
feral island
#

possibly we can make the default platform-dependent

#

but again, I don't know enough about this area to have a confident opinion

round path
#

Usually the outcome of something like this is to skip the test on the exceptional platform I think?

final knoll
#

Skip ALL os.link related test in WASI? Seems a little bit drastic,

#

And is just leave os.link(src, dst) always a error in WASI a good choice? I may think not.

#

If so, currently only you write os.link(src, dst, follow_symlinks=False) make it work in WASI.

urban sandal
#

There's another option: implement the logic required on WASI to follow the link, and error if it can't be followed and following was requested (wasi doesn't support absolute symlinks last I checked)

small sand
#

!pastebin

fallen slateBOT
#
Pasting large amounts of code

If your code is too long to fit in a codeblock in Discord, you can paste your code here:
https://paste.pythondiscord.com/

After pasting your code, save it by clicking the Paste! button in the bottom left, or by pressing CTRL + S. After doing that, you will be navigated to the new paste's page. Copy the URL and post it here so others can see it.

rose schooner
#

how does the cpython parser's memoization work exactly?

boreal umbra
#

Is there anyone we should want to nominate to the PSF board?

harsh sandal
feral cedar
#

why is it called a roadmap if there is no road

swift imp
# harsh sandal

Where does async fall under this? Its one of the topics im not familiar with at all in python. I've had no reason to learn it and wouldn't even know how to start.

#

No web framework, network automation, and the DSA part was more using those data structures than developing them. What does the DSA portion even mean. Like you've developed them or just used them.

grave jolt
#

This roadmap is... confusing

raven ridge
#

it doesn't seem on topic for this channel either way. And it doesn't seem particularly universally applicable. I've been a professional Python programmer for a decade and only have experience with about 3/4ths of these things

halcyon trail
#

I've been avoiding regex for a decade and this roadmap won't stop me

#

The "Advanced" section also seems more miscellaneous than anything

spark magnet
#

also, putting expressions under "Advanced" seems odd?

dusk comet
#

putting "Exceptions" under "Basic" also seems odd to me

#

i guess they are mixed up with "Expressions" 😄

flat gazelle
#

I mean, knowing how to read error messages is basic.

crisp locust
harsh sandal
harsh sandal
#

That is help to know some procedure that's all

#

And others are packages, libraries, concepts etc..

gray galleon
soft wren
#

guys i have learned basics of python but due to my college exams i have not practiced since two months so how should i start over again ?(i have my notes though)

dusk comet
#

!resources

fallen slateBOT
#
Resources

The Resources page on our website contains a list of hand-selected learning resources that we regularly recommend to both beginners and experts.

dusk comet
soft wren
#

ok cool

hollow hemlock
#

Good night,

I've created a virtual environment, installed pandas, and I'm trying to import it, but it's not working. Any insights? I'm just a newbie here.

image.png
sage gust
#

probably you need to choose "my venv" in editor, you are using Python 3.115. Try to click it 🙂

hollow hemlock
#

manhy thanks @sage gust

sage gust
#

No problem, bro 😉

tiny whale
#

hello, please let me know if wrong channel. I am trying to make a tool that instruments over all sources of external data in a program, writing them to an external database / log store. Examples are anytime a socket is read, file is read, current time is read, current memory usage read, etc. The goal is to be able to reproduce the state of a python program (especially long-running, such a server) at any time from the smallest data footprint. Hence why I only want to store external inputs into the system, intermediate pure computations don't need to be stored. At the same time, I need to make sure I capture every possible source of external data, which in some cases will be at module initialization and therefore at import time. What would be the best layer to do this instrumentation? The options I see are 1) at the python level, looking out for standard library functions that introduce external state, 2) looking at the python bytecode level, or 3) looking at linux system calls. This would be a decorator, so I can work with the target function as an argument.

feral island
# tiny whale hello, please let me know if wrong channel. I am trying to make a tool that inst...

It's going to be very hard to do this fully reliably from the Python level, since there are many different ways an application could ultimately do IO. One option would be to look at audit events (https://docs.python.org/3/library/audit_events.html#audit-events). Using monitoring of system calls could also work.

Python documentation
Audit events table

This table contains all events raised by sys.audit() or PySys_Audit() calls throughout the CPython runtime and the standard library. These calls were added in 3.8 or later (see PEP 578). See sys.ad...

tiny whale
feral island
#

Yes, I think the audit API is designed to tell you that an event occurred but not necessarily what it returned

#

I guess it depends on how robust you want this to be, but I don't think anything you do at the Python layer can really do this robustly.

tiny whale
#

Makes sense, appreciate the help.

raven ridge
# tiny whale hello, please let me know if wrong channel. I am trying to make a tool that inst...

if you want to see when the app has read the time, and what time it read, I don't think any of the 3 layers you proposed will do the trick. Looking at the Python source code or Python byte code, you'd have no idea if the Python interpreter itself read the clock (maybe as part of initializing a PRNG or something). Or if a native extension module like tensorflow did.

And reading the current time doesn't necessarily cause a system call on Linux. It makes a call into the vDSO. I maintain a profiler (Memray) that works by intercepting PLT entries pointing to functions it needs to detect calls to. I think that's the approach you'd need to take - you'd need to patch the executable and every loaded shared library to call your function instead of the vDSO to get the current time, and your function would call the vDSO and then note the result it got. You would also need to intercept the syscalls - they still exist, and something could choose to use them instead of using the vDSO.

#

you might also want to look into time travel debuggers for inspiration. Time travel debuggers need to detect IO so that they can replay it if you rewind and then redo

#

you might also be able to get something working using eBPF

tiny whale
raven ridge
#

yes, that is the complete list. In principle more can be added at any time, but in practice the things that have been added historically have been things that applications tend to call very, very often

tiny whale
#

as for the sys calls, I found ptrace which allows intercepting them, w/ access to both data and metadata. i'll see if the time travel debuggers also use that. thanks for all the guidance

raven ridge
#

ptrace is one way to intercept syscalls. The usual downside to it is that you can't be selective - it fires for every syscall, which slows the program down quite a lot. But if your goal is truly to see all IO that the program is doing, that seems like it's not a downside for you at all - there are practically no syscalls you would want to ignore. So yeah, ptrace sounds like it might be just the thing.

#

eBPF is another way to do it

#

also - do you know about strace?

#

if all you need is logging, you might be able to get literally everything you need to know using strace -o strace.out -s 1024 python yourprogram.py

tiny whale
#

@raven ridge I'll also take a look at eBPF and strace, Logging is one component, and it'd be the default (run asynchronously, with no strict schema) for all IO in calls such as data = sock.recv(64). However, if user does data: Annotated[bytes, InDatabase()] = sock.recv(64) then I would create a relational DB table ahead of time with two fields, timestamp of talking to the socket and the bytes data. Then, would write to it on each execution and would actually want to block the declaration of data until after its written, to get the same blocking behavior as if the database persistence was written imperatively. So ultimately I might settle on a mix of solutions

misty oxide
#

Does Py_BuildValue("s", str) (which calls PyUnicode_FromStringAndSize) intern the python string? How can I convert a cstring that will for sure be interned?

#

I have just discovered PyUnicode_InternFromString.

misty oxide
#

Is there an ABI stable way to create a class from inside a C extension?

errant oak
still star
#

!rule 5

fallen slateBOT
#
Rules

5. Do not provide or request help on projects that may violate terms of service, or that may be deemed inappropriate, malicious, or illegal.

jovial flame
#

Hello, your message was removed for violating server rules.

fluid crane
#

has anyone worked with power apps and model driven apps before?? I need help

crisp locust
dapper lily
#

wrong channel

hybrid relic
merry bramble
raven ridge
#

They don't, no

hybrid relic
#

interesting

#

seems like a bit of a waste though

raven ridge
#

Why's that?

final geode
misty oxide
#

Does any tooling exist for running down refcount issues in C extensions?

raven ridge
# misty oxide Does any tooling exist for running down refcount issues in C extensions?

Too many increfs or too few decrefs will lead to a leak. You can use gc.set_debug(gc.DEBUG_LEAK) or a memory profiler (like Memray, which I maintain, or valgrind) to try to figure out which objects are being leaked.
Too many decrefs or too few increfs will lead to objects being destroyed before you meant them to, and use-after-free bugs. That'll likely cause crashes, hopefully relatively soon after the problem occurred. You can catch some of those with the PYTHONDEVMODE=1 environment variable, or with python -X dev. You might catch more of them with PYTHONMALLOC=malloc+debug. Valgrind can help here. If you're on Linux, export MALLOC_CHECK_=3 can help, too.

#

these tools aren't going to get you any further than telling you which object's reference count was mismanaged, where it was first created, and where it was destroyed (if it was destroyed and you've got a use-after-free bug). Once you know which object's reference count was screwed up, you'll need to audit all your code that deals with that object to figure out which piece of code was responsible for screwing it up

misty oxide
jade raven
#

sorry, you can ignore the nuitka column, but is there a reason 3.7 seems to be faster re: for loops vs other python versions?

image.png
#

the numbers are in seconds in runtime

dusk comet
#

i dont see anything that might cause this on "What's new in 3.8" page

rose schooner
jade raven
fallen slateBOT
#

pyperformance/data-files/benchmarks/bm_concurrent_imap/run_benchmark.py lines 13 to 22

def bench_mp_pool(p: int, n: int, chunk: int) -> None:
    with Pool(p) as pool:
        for _ in pool.imap(f, range(n), chunk):
            pass


def bench_thread_pool(c: int, n: int, chunk: int) -> None:
    with ThreadPool(c) as pool:
        for _ in pool.imap(f, range(n), chunk):
            pass```
jade raven
#

but that's the core bit of what's being benchmarked

fallen slateBOT
#

pyperformance/data-files/benchmarks/bm_concurrent_imap/run_benchmark.py lines 9 to 10

def f(x: int) -> int:
    return x```
rose schooner
uneven raptor
#

why isn’t _typeobject part of the limited abi?

feral island
raven ridge
#

"change" how? PyObject is part of the limited abi, though only ob_type and ob_refcnt are exposed. Couldn't PyTypeObject also be exposed, to allow direct access to its slots?

#

adding PyTypeObject to the limited API wouldn't make it impossible to add new fields to PyTypeObject in the future, it would just be a guarantee that some subset of the fields that currently exist would always exist

#

and that guarantee is already sort of being made via the slot IDs - PyTypeObject.tp_as_number.nb_int isn't part of the limited API, but the limited API is still guaranteeing that it will always exist because Py_nb_int is part of the limited API and defined to be used to get/set PyTypeObject.tp_as_number.nb_int

raven ridge
#

I guess the question boils down to: given that the type slots are guaranteed by the stable ABI to exist, would it be too onerous to keep their offsets within the structure stable?

final geode
#

Granted, I’m not an expert on a lot of this C API stuff, but in general I don’t think “let’s do it because we can” is a great motivation for this sort of stuff. To turn the question around: would it be too onerous to keep the structure unstable?

raven ridge
#

The cost of keeping it unstable is that the way to do lots of things with the limited API is different than how they'd be done with the unstable API, which slows adoption of the limited API. There's lots of docs about how to do things with the unstable API, and most of those docs don't also say how to do things with the stable API, so every difference is a point of friction that makes it harder to learn to use the stable API

final geode
#

Eh. I’d argue that we should probably be moving away from getting and setting PyTypeObject members directly. Heap types have lots of advantages, and the few cases where touching type members is needed (tp_dealloc comes to mind) should probably have dedicated functional APIs to abstract them away.

#

I get what you’re saying though.

raven ridge
#

Perhaps it'd be good to take a pass over the examples in the C API docs to prefer using PyType_GetSlot over direct slot access where possible. That'd be another way to bring the two APIs closer - encourage using the limited API subset even wherever possible

final geode
#

Yeah, even if only to call out that the limited API approach exists and provide an example.

raven ridge
#

When you step back and think about it, it's quite weird that the unstable ABI that changes yearly is documented as the default way to do things, and the stable ABI that lets you build forwards compatible extension modules is the poorly documented alternative...

humble wadi
#

Wow, u r cpython core dev

#

that is amazing

hexed latch
#

hlo

#

is there any free api avaialable to get flight info using aircraft number ?

wheat compass
#

Hello! I'd like to contribute to Cpython. Where do I start? Is there newcomer friendly tags in the issue tracker?

dusk comet
rain trellis
#

I've put together an introductory blog post on building CPython with the JIT (for POSIX systems), and looking at its guts with PYTHON_LLTRACE and -X pystats - would anyone be interested in taking a quick look and making sure I'm not saying anything outrageously wrong? @final geode

final geode
radiant topaz
#

i'm trying to figure out if pep 234 cares if an iterator is destructive or not -- i have an api that essentially iterates records in a paginated table via rest -- and what is the pep expectation on holding old pages in memory or not. Like, what is right -- and how would one augment or change this behavior? like if my iter today starts being destructive when yesterday it wasn't... big change

grave jolt
radiant topaz
#

meaning if you can iterate the object twice or not, in 234 it points out file's are like this (or, to quote, "some iterators are destructive: they consume all the values and a second iterator cannot easily be created that iterates independently over the same values.")

grave jolt
#

ah

#

I guess it's more of a property of an itera_ble_ than an iterator

#

In case of a file the file is an iterator, so it has to be non-reusable in iteration

#

(unless you call file.seek(0))

radiant topaz
#

would that imply the expected behavior of all iterables is that you 'drop' previous values once consumed?

grave jolt
#

I'm not sure why the destructibility phrase was even included in the pep

#

it's odd

radiant topaz
#

i'm mostly confusing myself, i think.

grave jolt
#

e.g. if you have a list foo, you can call iter(foo) as many times as you want to iterate over it. You can't do the same with (arbitrary) file objects

#

It's not really a property of the iterator itself

radiant topaz
#

fair, but if I have an object bar which is an iterator... or an inerable?

grave jolt
#

wdym?

grave jolt
radiant topaz
#

say class Bar: __iter__(): blabla __next__() blabla -- i could for x in iter(Bar()) all day but only for x in Bar() once?

grave jolt
#

an __iter__ of an iterator should return itself

#

i.e. ```py
def iter(self):
return self

radiant topaz
#

maybe this is where i shot myself in the foot, then:

    def __iter__(self):
        self.__is_iter = True
        self.rewind()
        return self
grave jolt
#

Yeah that would be confusing

radiant topaz
#

follow up question, is doing with Bar() as b: to have non-rewind iteration an offense of sensibilities?

grave jolt
radiant topaz
#

meaning it'd look something like

def __iter__(self):  
 if not self.is_in_with_statement:
   self.rewind()
 return self
grave jolt
# radiant topaz maybe this is where i shot myself in the foot, then: ``` def __iter__(self...

The reason this would be confusing is that iter() is usually expected to be a no-op for an iterator. Suppose you have a function/class accepting an iteratble, like map. It will definitely call iter() on the argument. So if you do e.g. py def deal(cards): it = iter(cards) burned_card = next(it) hand = itertools.islice(it, 5) print(burned_card, hand) it would be really surprising if islice also included the "burned card" in the output, which would happen if cards.__iter__ did some rewinding

radiant topaz
#

yeah, seems like the true way forward is a major version release and breaking backward compat here.

grave jolt
#

If you're changing the behaviour, it is a breaking change yeah

#

Well, what is and isn't backwards compatible is a philosophical topic... but going from rewinding to non-rewinding and vice versa seems like it

radiant topaz
#

yeah, it's just... do i honor the PEP or just have a quirk. Unless you know of some PEP on seekable/rewindable iterables...

raven ridge
round path
rain trellis
round path
#

Yup those are different DPRINTFs !

rain trellis
#

I'll clean that up 😅

hybrid relic
#

PHP initially also directly emitted native code straight from it's opcodes, passing it to the DynASM backend

#

but after a while they ripped out the code that directly emitted native instructions and moved to an IR framework more akin to traditional JITs

thick hemlock
#

there's also no good way to type that afaik

#

but it's not very comprehensive

#

the general consensus, i think, was that an Iterable is not promised to be repeatble (but it's okay for it to be) but a Collection usually is

thick hemlock
# thick hemlock but it's not very comprehensive

Especially the statement

The whole point of Iterable is that you can get an iterator on it, and arguably it was a mistake to make Iterator support Iterable in the first place. It's not perfect, but is simple, which is usually more "pythonic" than a more technically correct annotation that is very complicated.

Which is very opinionated but I haven't found any more discussion about it unfortunately

#

maybe some typing experts have an opinion on that though

radiant topaz
#

it feels like it's just open to interpretation

halcyon trail
#

not sure where this is written "formally"

thick hemlock
#

I don't think any type checkers or linters care about it since its not formal

#

it would be nice to be able to catch errors like that

halcyon trail
#

Well, eventually a person needs to verify something 😛

#

a type checker (in python at least) also cannot verify that Mapping isn't mutating stuff

#

even though that's clearly against the spirit of the Mapping interface (which is why MutableMapping exists)

#

but surely no type would be so insiduous as to syntactically satisfy Mapping while actually performing mutations - oh no, wait, defaultdict says hi 😂

thick hemlock
#

yeah yeah obviously there are always bypasses i just hoped there would be a way to signify the type of inp here


def iterate_twice(inp: ...):
   for a in inp:
      pass

   for b in inp:
      pass

such that if inp is NOT a subclass of this type (for example Iterable you would get a warning

#

i mean i know theres no way to promise that it WILL work i just hope theres a way to warn on some types that clearly won't always work

#

(and yes i know i can use tee for this)

dusk comet
# 
def iterate_twice[T](inp: Iterable[T] & ~Iterator[T]):
   yield from inp
   yield from inp
 ``` 🥺
halcyon trail
#

well, negative bounds are sketchy, but also I would say that's just not sufficient guarantee anyway

thick hemlock
halcyon trail
#

Collection seems like the most reasonable solution

thick hemlock
#

yeah collection works

#

but the problem is that i want an error when i forgot to write Collection

#

and said iterable instead

halcyon trail
#

python's type checker cannot check if you used something more than once

#

you'd need rust's type checker for that 😛

thick hemlock
#

i see

#

is this something that a tool like ruff could handle

#

or is this beyond static analysis?

halcyon trail
#

but you need to design the language around it from day 1, I would say

#

for it to work rigorously

thick hemlock
#

yeah I meant 3rd-party-analysis

halcyon trail
#

obviously, Ruff could have a heuristic that could catch common cases

thick hemlock
#

yeah