#internals-and-peps

1 messages Β· Page 161 of 1

dapper lily
#

oh. I meant x, y = 5, z isn't really "swapping" anything

timber dagger
#

load 5 ([] -> [5])
load z ([5] -> [5, z])
swap ([5, z] -> [z, 5])
assign x ([z, 5] -> [5])
assign y ([5] -> [])

rose schooner
#

pretty sure changing the order of 5 and z on the stack is called "swapping"

dapper lily
#

I see

rose schooner
#

should there be a special case that replaces UNPACK_SEQUENCE/SWAP with just the store instructions in the right place

prime estuary
#

It's not always legal to reorder things, since one of the two loads could raise an exception, and it's important that all of the RHS happens before the LHS.

rose schooner
#

i'm talking about reordering the STORE_*s

prime estuary
#

Hmm that could be doable, not sure if tracing etc could make that observable. Hmm also __del__ could be sneaky...

feral island
swift imp
#

What was the pep that allowed this syntax @list_of_callables[0] or @obj.attribute_which_is_callable?

#

Previous versions you couldn't use attribute or item access with @ during decoration

timber dagger
#

pep 614

#

!pep 614

fallen slateBOT
#
**PEP 614 - Relaxing Grammar Restrictions On Decorators**

Link

Status

Final

Python-Version

3.9

Created

10-Feb-2020

Type

Standards Track

elder blade
grave jolt
#

yeah, you could

swift imp
sour thicket
#

Hi I am trying to find people who are interested in efficient coding, My mission is to find solutions for coding problems with the minimum time complexity possible. If anyone's interested DM me. Thanks!

sacred yew
fallen slateBOT
#
Rules

6. Do not post unapproved advertising.

sour thicket
astral gazelle
#

Its offtopic for the channel, ask in offtopic

quick snow
#

Found this in the help of array.array:

NOTE: The 'u' typecode corresponds to Python's unicode character. On
narrow builds this is 2-bytes on wide builds this is 4-bytes.
AFAIK, narrow/wide builds aren't a thing anymore since 3.3. Should this just be removed and changed to always be 4?

feral island
quick snow
fallen slateBOT
#

Modules/arraymodule.c line 562

{'u', sizeof(wchar_t), u_getitem, u_setitem, u_compareitems, "u", 0, 0},```
eternal glacier
#

Where can I get questions for a python dev interview?

#

any websites?

spark magnet
eternal glacier
spark magnet
eternal glacier
#

ok

#

thanks

white nexus
native flame
rich cradle
#

That's an interesting take

swift imp
# white nexus any idea why https://peps.python.org/pep-0505/ was deferred?

I certainly do like a lot of that pep, especially the user defined null protocol. I know singletons are frowned upon but I use them quite a bit at work in certain parts bc it just makes development cleaner in certain aspects. Passing around None is too much of a catch all and you lose context some times. It's like using np.nan as both a signaling that a floating point computation went bad and to represent missing data. Can't tell what the original context was without looking backwards

civic hamlet
#

It looks like the internals for the ABCs (collections.abc) were re-written in C for Python 3.7, but the Python implementation remains in _py_abc.py. Does anyone know if/when this Python implementation is still used? It seems like the C implementation has had some changes since then and they could behave differently.

feral island
civic hamlet
#

Sure - I just see the difference by reading the code but without knowing when the Python implementation is used I couldn't actually test or make it happen. It looks like it will fall back to the Python implementation if it fails to import the C module, so maybe I can force that to happen or just monkeypatch a proof-of-concept.

#

It's a difference in subclass registration and setting Py_TPFLAGS_MAPPING/Py_TPFLAGS_SEQUENCE - I'm not sure this is relevant for PyPy because I imagine they have their own implementation of these flags or something like it.

peak spoke
civic hamlet
#

Thanks - good call.

civic hamlet
feral island
boreal umbra
lofty rover
#

I'm looking for some general feedback on code style/legibility. E.g. part of this:

def _createFiles(imagelist):
    if len(imagelist) == 1:
        print('WARNING: single image found - sequential pipeline')
        raise NotImplementedError
    else: 
        p = lambda x:(str(Path(config.dataset_path) / x))
        originalFileNames = [i.get('file_name') for i in imgs_seq_lookup.get(imagelist)]
        baseFilePaths = [p(i) for i in originalFileNames]

        cats = [i.get('category_id') for i in imgs_seq_lookup.get(imagelist)]
        ids = [i.get('image_id') for i in imgs_seq_lookup.get(imagelist)]

        for baseFilePath, categoryId, imageId in zip(baseFilePaths, cats, ids):
            if config.generate_histlp: 
                lbp = colorhistslbp.getLpb(baseFilePath)

This is not PEP-8, but is it fine? Should I adhere more to standards?

quick snow
#

You should.
Is colorhistslbp your code? Make get_lpb() take a path-like object.
You don't need all those .get(), since your code would anyways fail if those values were None.
If you're assigning a lambda to a name, you're doing something wrong.

#

First approximation of improvement:

def create_files(imagelist):
    if len(imagelist) == 1:
        logger.warn("single image found - sequential pipeline")
        raise NotImplementedError

    for img in imgs_seq_lookup[imagelist]:
        base_path = Path(config.dataset_path) / img["file_name"]
        cat_id, img_id = img["category_id"], img["image_id"]

        if config.generate_histlp:
            lbp = colorhistslbp.get_lpb(base_path)
dusk comet
#

instead of

        logger.warn("single image found - sequential pipeline")
        raise NotImplementedError

do ```py
raise NotImplementedError("single image found - sequential pipeline")

cosmic oak
#

Hi guys. I'm interested in learning more about the garbage collector in Python. Just started reading the Garbage Collection Handbook by Jones, Hosking & Moss (2011) but was wondering if anyone had other recommendations / more recent texts. Sorry if this is the wrong channel.

feral island
cosmic oak
#

Nope. I didn't know there was a devguide. Ty!

onyx drift
#

I wonder whether there should be a link from the gc module docs to that devguide article; it's a fantastic write-up, and it's not as discoverable as it deserves to be.

frigid bison
#

How can I call internal cpython functions in python

#

I want to call the _PySys_ClearAuditHooks

#

Since there is no variant of it in the sys module

frigid bison
fallen slateBOT
#

Python/sysmodule.c line 322

* and as such the function is not exported.```
frigid bison
#

This is the problem

#

It's not exported

#

So is there any way I can still call it?

feral island
#

anything is possible using ctypes, but you may get segfaults instead

frigid bison
#

I also don't see why they would remove the feature. Doesn't it make sense that if you can add an audit hook you should also be able to remove it?

raven ridge
quick snow
#

I mean.. there is an audit hook for _PySys_ClearAuditHooks, so at least that could be logged

raven ridge
#

ah true, so at least it wouldn't be silent.

frigid bison
verbal escarp
#

just got the idea in python-general, but i guess here's a better place for that: what would be required (syntax-wise?) to have x = mylist[0] as a lazily evaluated variable?

sacred yew
verbal escarp
#

i was thinking of a global, but a descriptor could work

sacred yew
#

ooh forgot about descriptors

#

that makes it easier

#

do those work in non-attribute variables?

quick snow
#

Can you explain what you mean by "lazily evaluated variable"?

verbal escarp
#

like a label in a UI

quick snow
#

Understood now

sacred yew
#

just make a proxy class
that just forwards every method call to the current element of the list

#

I think that should work?

quick snow
#

Do you require x is 1, or is x == 1 fine?

verbal escarp
dapper lily
#

like dict_values, I see

quick snow
#

!e

class LazyList(list):
    def __getitem__(self, item):
        return ItemProxy(self, item)

class ItemProxy:
    def __init__(self, parent, item):
        self.parent = parent
        self.item = item
    def __eq__(self, other):
        return list.__getitem__(self.parent, self.item) == other

my_list = LazyList([1,2,3])
x = my_list[0]
print(x == 1)
my_list[0] = 2
print(x == 2)
fallen slateBOT
#

@quick snow :white_check_mark: Your eval job has completed with return code 0.

001 | True
002 | True
verbal escarp
#

@amber nexus gave a different approach in py-gen: ```py
import ctypes
import builtins

base = ctypes.py_object.from_address(id(globals()) + 8)

class fglobals(dict):
slots = ()
def getitem(self, key, dict=dict, base=base, builtins=builtins, getattr=getattr, print=print):
print(f"Looking up {key}")
try:
base.value = dict
return self[key]
except:
try:
return getattr(builtins, key)
except:
return key
finally:
base.value = class

base.value = fglobals

a = 10

print(a)

#

forward the lookup

#

lots of work for a "simple thing" πŸ˜„

amber nexus
# 
def __getitem__(self, key, dict=dict, base=base, builtins=builtins, getattr=getattr, print=print):
```Is rather cursed
#

And gives me an idea πŸ‘€

verbal escarp
#

can't be the only one who ever wondered about lazily evaluated variables..

#

@amber nexus base = ctypes.py_object.from_address(id(globals()) + 8) throws me off, why the +8?

quick snow
verbal escarp
verbal escarp
amber nexus
verbal escarp
verbal escarp
quick snow
#

For your usecase using lazex is a bit explicit, which I think I read somewhere is better than implicit, can't remember where.. :P

x = lazex.Expression("my_list[0]")
#

(it came from a desire to be able to do stuff R can do)

verbal escarp
#

i'm guessing you're eval-ing?

#

ah, parse and ast-juggling

#

crazy monkey

#

huh

#

waitasecond

#

if annotations aren't evaluated, couldn't you use that?

#

or variable docstring

# 
x = lazycrazy()
"""
my_list[0]

do not try this at home!
"""
quick snow
#

Yes. But you have to keep track of the locals/globals of the frame.

flat gazelle
#

I mean, you could do lambda: my_list[0]

verbal escarp
flat gazelle
#

indeed

quick snow
#

You can replace __annotations__ with something that helps you with that (remembering locals/globals)

verbal escarp
quick snow
pliant tusk
rose schooner
# 
struct _object {
    _PyObject_HEAD_EXTRA
    Py_ssize_t ob_refcnt;
    PyTypeObject *ob_type;
};
#

btw the size of object can be anywhere from sizeof(void *) + sizeof(Py_ssize_t) to sizeof(void *) * 3 + sizeof(Py_ssize_t)

spark magnet
#

can't it be larger than that? Tuples allocate their elements in the memory after the head struct.

raven ridge
#

sizeof(_object), they meant. Because _PyObject_HEAD_EXTRA can be 0 pointers or 2 pointers, depending on whether it's a debug build of Python.

spark magnet
#

i see

raven ridge
fallen slateBOT
#

Include/object.h lines 65 to 76

#ifdef Py_TRACE_REFS
/* Define pointers to support a doubly-linked list of all live heap objects. */
#define _PyObject_HEAD_EXTRA            \
    PyObject *_ob_next;           \
    PyObject *_ob_prev;

#define _PyObject_EXTRA_INIT _Py_NULL, _Py_NULL,

#else
#  define _PyObject_HEAD_EXTRA
#  define _PyObject_EXTRA_INIT
#endif```
raven ridge
#

it's part of the reference tracing built into debug builds.

void copper
#

Does anyone know of any free hotel or flight Api's to get prices?

raven ridge
void copper
raven ridge
verbal escarp
pliant tusk
#

The first 8 bytes are the ref count. So the max ref count of an object is the max signed 8 byte number

verbal escarp
#

so, can there be an overflow of references?

raven ridge
pliant tusk
#

In theory yes, but that would be very difficult and slow in practice

raven ridge
#

instead, the first 8 bytes are, so the maximum number of references is 18446744073709551616

pliant tusk
#

It's signed @raven ridge

verbal escarp
raven ridge
pliant tusk
#

So this would be the max 9,223,372,036,854,775,807

verbal escarp
#

is there a test against this potential vulnerability?

raven ridge
#

that is, an object can have at most 9,223,372,036,854,775,807 references, and each of those references uses 8 bytes, and a 64-bit program can use at most 18,446,744,073,709,551,616 bytes of memory, so that can't happen.

#

unless something miscounted its references, but then you were in undefined behavior territory anyway.

verbal escarp
#

hmm

verbal escarp
raven ridge
#

because a reference is a pointer.

#

the reference count of an object is how many shared ownership pointers to that object exist.

verbal escarp
#

okay

pliant tusk
#

But that puts you in undefined behavior anyways

raven ridge
#

right.

#

it's really easy to do with the C API, as well - just incref a borrowed reference, or fail to decref a strong reference

#

but in either of those cases, you've screwed up badly, anyway - you've leaked an object, it will never be destroyed.

native flame
pliant tusk
#

πŸ™‚

raven ridge
#

the refcount field is also a public field of the object structure - you can assign whatever you want to it from the C API. It's wrong to ever do that, but nothing stops you.

pliant tusk
#

!e ```py
from ctypes import *
import sys

x = 255
c_ssize_t.from_address(id(x)).value = -2
print(sys.getrefcount(x))```

fallen slateBOT
#

@pliant tusk :white_check_mark: Your eval job has completed with return code 0.

-1
raven ridge
#

like that πŸ˜„

verbal escarp
#

oh my.

native flame
#

that's cheating!!

pliant tusk
#

(I picked 255 because most other objects would have crashed it lol)

raven ridge
native flame
#

that's comforting

pliant tusk
#

I mean I can use hand crafted bytecode or one of my handful of memory bugs to do it

#

There are fun things you can do when the gc runs at unexpected times

raven ridge
#

both of those are just bugs (well, one might be undefined behavior rather than a bug, but that's splitting hairs)

#

the normal way to manipulate reference counts, even when you're using the C API, is to use functions that increment/decrement the reference count by 1, and when it reaches 0 the object is destroyed. Getting it to go negative would require doing abnormal stuff.

verbal escarp
raven ridge
#

Unlikely, but if we ever need to, we can. 32-bit address spaces were the norm until about 15 years ago, and those only supported up to 4 GB

verbal escarp
#

good ol' times 😐

raven ridge
#

we've already gone through 8 bit, 16 bit, 32 bit, and 64 bit computers. We might hit 128 bit one day.

verbal escarp
#

okay, concerns are mitigated for now, thanks πŸ™‚

south hearth
#

hi

#

i noticed that in python when u create a file within a directory that doesnt exist python ends up with an error, would it be an improvement if i create a issue and a pr relating to this, in the pr the directory being cretaed if it doesnt exist and then the file is put into it

dapper lily
#

it's not really Python's responsibility to create directories for users to put files in

glass lagoon
#

what if someone has mistyped the directory name? wouldn't it just end up creating redundant dirs

dapper lily
#

just like every other language

raven ridge
#

There's also good reasons not to automatically create missing directories - ownership and permissions, for one.

south hearth
#

well true

#

guess i didnt look at the wider sight

grave jolt
#

quite annoying

#

Actually, I think that's docker's behaviour, compose doesn't change it. If the target directory doesn't exist when mounting a volume, it's created.

peak spoke
#

should the set abc also require the named methods like set has to be defined? Currently __le__ etc. are required, but usages of some of those feel confusing to be compared to just doing e.g. issubset

swift imp
peak spoke
#

yeah I'm not sure how I feel about the operators themselves other than maybe union/intersection/difference

swift imp
#

The use of logical operators to do set ops is fine bc they're all the same thing

#

Like there's very uniform definition between logic, predicate calculus, and set theory

peak spoke
#

it also looks like it's impossible to check a proper subset without operators on a plain set, that could me a kwarg on the issubset method

swift imp
#

So gt is superset and lt is sunset? I've never even thought of using the relationship operators for subsets

peak spoke
#

is expanding an abc like that something that's viable?

hazy pivot
#

hey i need help with turtle

spark magnet
swift imp
round oxide
#

why do we use isalpha instead of is_alpha()

charred wagon
#

Because a consistent coding style was not established at the time it was added, and renaming it would be a breaking change.

quaint sedge
#

Can someone help me im new to python

unkempt rock
#

Would someone kindly be able to help me with micropython?

quick snow
fair plume
#

does anyone knows
how can a app be designed such that it activates only in location tracker range..

vast saffron
rocky jungle
#

Hi

visual shadow
#

Hi. I had someone ask me if len(gc.get_referrers(thing)) == sys.getrefcount(thing) should always be True, is this a reasonable expectation or no?

deft pagoda
#

dunno, to weak_refs count as referrers

inland acorn
fallen slateBOT
#

@inland acorn :white_check_mark: Your eval job has completed with return code 0.

001 | 2
002 | 4
dapper lily
#

4? pithink

inland acorn
#

Β―_(ツ)_/Β―

#

The var, 2 in the list, and...

timber dagger
#

!d sys.getrefcount

fallen slateBOT
#
sys.getrefcount 

sys.getrefcount(object)```
Return the reference count of the *object*. The count returned is generally one higher than you might expect, because it includes the (temporary) reference as an argument to [`getrefcount()`](https://docs.python.org/3/library/sys.html#sys.getrefcount "sys.getrefcount").
inland acorn
#

There we go

dapper lily
#

πŸ€¦πŸ»β€β™‚οΈ

round oxide
#

imagine a squre and in that square there are doors to each side.
then you go to each room and start boiling water in the room then move to another room to start to boil another water until they are all done.

#

is this an example of async?

#

you are python
boiling water is the task
then each function is the room?

visual shadow
#

Yeah, especially if you emphasize that you go to first room, andbset it to start boiling and immediately move into the next room and set it to start boiling. Maybe call it turning the gas on or something like that

round oxide
round oxide
inland acorn
#

There is some other stuff where you can get notified when it is boiling so you know to check that room first when you can

peak spoke
#

for normal IO, the OS tells you it finished

round oxide
#

like room 1 first then room 2...

#

can it skip rooms?

visual shadow
#

The system for checking can be designed as you see fit

#

Generally implementations keep track of when pending tasks were checked last, if it's a check based system. And then items that haven't been checked in a while start getting higher prio for next check

round oxide
#

we went from 1-100 real quick

visual shadow
#

Order isn't guaranteed in the real world. To keep with your example, what if the amount of water in each room is unequal? Then room will take unequal amounts to reach boiling. Your scheduler just keeps checking and ultimately as rooms get finished you keep moving on

#

The main thing though, is that essentially any task where the actual work doesn't require you, you can have it "start working" and then then you're free to do whatever while it keeps going

#

This is analogous to io bound tasks, where something external is doing the work

round oxide
visual shadow
#

Depends on how you design it

#

Let's just say, no, order isn't guaranteed

#

You can design your checker in such a manner that it always checks tasks in order

#

But often there's smarter ways to do things

#

Since different tasks may take different amounts of time, or effort

round oxide
visual shadow
#

It's honestly not even that, I'm just trying to keep things simple

round oxide
visual shadow
#

Ultimately the scheduler could even start the tasks out of order

#

It's like giving a command, turn gas on in room 1, turn gas on in room 2, etc

#

But then your assistant is the one turning gas on, and maybe they just like doing things out of order

#

Bottom line is, with async, you give up guarantees about what order things happen in, but you gain benefits of having multiple tasks, as a whole, finish faster because wait time is reduced.

#

That make sense?

round oxide
#

I'm trying to but this doest make sense to me code wise lemon_thinking

visual shadow
#

Code wise, it's like a queue of tasks to start, and a queue of ongoing tasks to check, maybe with some kind of "last checked" indicator

inland acorn
#

(the rust book is really good at teaching async, it even has you code an async implemention from scratch! You just need to have read the rest of the book first, I.e learn rust)

visual shadow
#

That could be one simple implementation

round oxide
round oxide
round oxide
#

now it all makes sense

visual shadow
#

Yay

inland acorn
#

πŸ₯³

visual shadow
#

I should probably mention that it may still be worth reading some kind of resource on this

round oxide
visual shadow
#

It's definitely a different style of programming, and you just want to let the concept sit in your mind for a bit

inland acorn
round oxide
inland acorn
visual shadow
#

I actually remember a python talk, that started very innocently with generators, and ended up essentially building a full async system

#

Let me see if I can find it

round oxide
raven ridge
round oxide
#

my brain can't read lemon_angrysad

raven ridge
safe hedge
#

I am using a dataclass to store some counts and aggregated values and have a few properties to return average or calculated values. In a code review someone has suggested that where I increment the counts in a method I should implement a method on the class which does the increment. This feels hugely unnecessary since Python let’s us increment the attributes directly. Am I missing something?

ie this:

def increment_val(self):
    self.val +=1
visual shadow
#

I'd talk it out with them, hard to say without context.

safe hedge
#

There isn’t much more context, I mainly just wanted to try and understand if there are generic compelling reasons not to directly increment attributes before I shoot down the idea?

feral cedar
#

maybe for logging? would be easier to add that to a function. also easier to change 1 function than change like 30 individual increments

visual shadow
#

I usually go the other way, simple implementation first, and refactor if it ends up being harder to maintain

quick snow
#

Fully agree. If for whatever reason you really need custom logic later for incrementation, then you can always replace the simple attribute with something more complicated that still works with the normal incrementation syntax.

#

!e

class Foo:
    def __init__(self):
        self._var = 0

    @property
    def var(self):
        return self._var

    @var.setter
    def var(self, new_value):
        if new_value - self.var != 1:
            raise ValueError("only allowed fo increment!")
        self._var = new_value

f = Foo()
print("var is", f.var)
f.var += 1
print("var is", f.var)
f.var += 2
print("var is", f.var)
fallen slateBOT
#

@quick snow :x: Your eval job has completed with return code 1.

001 | var is 0
002 | var is 1
003 | Traceback (most recent call last):
004 |   File "<string>", line 19, in <module>
005 |   File "<string>", line 12, in var
006 | ValueError: only allowed fo increment!
rare compass
#

you can also use +=1

frigid bison
#

yes he was aware of that, read his original message

unkempt rock
#

Is it possible for us to put a "-" in a string. after a word in a long string(sentence). The main issue for me that me I'm not sure it is even possible or not.

brave barn
#

you mean you want to replace every

#

whitespace in a string with a dash?

granite ravine
#

Has anyone worked on "recommendation systems" ?
need advice on how to determine how much a user has liked certain content when i am not taking single feedback(like in movie we rate out of 5) but the feedback is indirectly given based of :
duration of uptime
liked/dislike
shared the content or not etc.

#

the base of easier problem is we have movie to rating dataset but how do i determine rating in my case?

#

of idea is as follows, AM i right?

#

except score everything is bool

west tinsel
silk pawn
#

as for internals of python, is there a no-op bytecode instruction? i want to make it so that two or more identical compiled code objects can hash to different values (long story as to why but suffice it to say i cant change the functionality of the code objects or add the two+ bytecode objects to a list of sorts)

charred wagon
#

There is a NOP instruction documented

silk pawn
#

oh whoops, i was searching no-op

#

i see it now, thanks!

Screenshot_from_2022-07-18_16-06-51.png
charred wagon
#

You're welcome

frigid bison
timber dagger
#

yeah it would probably be best to put them at the end

silk pawn
#

oh yeah i was going to add them at the end of the function

silk pawn
timber dagger
#

or put anything at the end really since i don't think you ever fall off the end of a code object without encountering a RETURN_VALUE so it would never be ran

silk pawn
#

i bet there's an esoteric way to not have a return value but yeah that assumption is probably safe for most cases

frigid bison
frigid bison
#

in what case would that happen?

elder blade
#

!e ```python
import dis

def test():
pass

dis.dis(test)

fallen slateBOT
#

@elder blade :white_check_mark: Your eval job has completed with return code 0.

001 |   3           0 RESUME                   0
002 | 
003 |   4           2 LOAD_CONST               0 (None)
004 |               4 RETURN_VALUE
elder blade
#

Oh, alright.

feral island
#

!e import dis def test(): raise Exception dis.dis(test)

fallen slateBOT
#

@feral island :white_check_mark: Your eval job has completed with return code 0.

001 |   2           0 RESUME                   0
002 |               2 LOAD_GLOBAL              0 (Exception)
003 |              14 RAISE_VARARGS            1
dawn salmon
#

Hi experts, I have a question and I think it might be best suited for this channel:

Python by default builds the library libpythonMAJOR.MINOR.a and statically links it into the interpreter. Also it has an --enable-shared, (https://docs.python.org/3/using/configure.html#cmdoption-enable-shared) flag, which will build a shared library libpythonMAJOR.MINOR.so.1.0, and dynamically link it to the interpreter.

Are there any benefits of a shared build python, compared to a static build python? Is there any performance difference?

I found a related question on SO and is it like he said (https://stackoverflow.com/a/73035776/5983841), shared built vs statically built python only affects the "Embedding Python in Another Application" scenario ?

Stack Overflow
What's the benefit of a shared build python vs a static build python?

This question has been bothering me for two weeks and I've searched online and asked people but couldn't get an answer.
Python by default build the library libpythonMAJOR.MINOR.a and statically lin...

winged yarrow
#

hello

#

do you think it would work if most python builtin libaries are rewritten in c?

#

and maybe even more things

feral cedar
#

many already are

winged yarrow
#

ah

#

like the "expensive" ones?

feral cedar
#

yeah, things like math, list, dict are C

spark magnet
feral island
spark magnet
#

functools? bisect?

#

operator probably

#

is datetime time-critical?

feral island
#

at least parts of operator can't be written in Python

spark magnet
timber dagger
#

wait which parts? it looks to me like it's all just functions that are equivalent to operators

feral island
timber dagger
# 
def indexOf(a, b):
    for i, el in enumerate(a):
        if el == b:
            return i
    return -1```
feral island
timber dagger
#

no it would do literally the exact same thing

#

i checked the source code, it calls _PySequence_IterSearch which just uses the C equivalents of iter and next

feral island
#

the C version throws an error if the index exceeds the possible range of a C int

#

granted, not a very likely condition πŸ™‚

timber dagger
# 
import sys

def indexOf(a, b):
    for i, el in enumerate(a):
        if i > sys.maxsize:
            raise OverflowError("index exceeds C integer size")
        if el == b:
            return i
    return -1```
spark magnet
#

... i didn't mean to start a fight...

raven ridge
#

With the existence of non-entry frames in 3.11, making functools pure-Python might actually speed up partial rather than slow it down.

#

at the very least it would save a bunch of frames on the C call stack per call to a partial

verbal escarp
#

what did i miss?

grave jolt
#

!e

print(__import__('sys').maxsize)
fallen slateBOT
#

@grave jolt :white_check_mark: Your eval job has completed with return code 0.

9223372036854775807
grave jolt
#

that's gonna be one hell of a loop πŸ˜›

feral island
#

operator.index(range(10 ** 1000), 10 ** 999)

inland acorn
#

so I was looking at cpython source (specifically: https://github.com/python/cpython/blob/main/Lib/operator.py#L339-L355=), because I was wondering about the difference between iadd and iconcat, and it seems to just be that iconcat has an error check? is there a reason for this or? ```py
def iadd(a, b):
"Same as a += b."
a += b
return a

def iconcat(a, b):
"Same as a += b, for a and b sequences."
if not hasattr(a, 'getitem'):
msg = "'%s' object can't be concatenated" % type(a).name
raise TypeError(msg)
a += b
return a```

peak spoke
#

concat is for a different c api method, but the c level methods also both fall back to each other

inland acorn
#

ah, I forgot stuff is sometimes implemented in C and python

peak spoke
#

I asked about why it is a thing a while back, and I think it pretty much was just because it was implemented in the C operator module when it was added, even though the two are pretty much the same when used from python

grave jolt
#

that's one big leaky abstraction πŸ‘€

raven ridge
peak spoke
#

it still needlessly creates and destructures args and kwargs into the arguments

raven ridge
#

you sure that's needless? Seems it'd be pretty tricky to get print(locals()) to do the right thing otherwise

#

maybe not impossible, but it's certainly not as easy as just inlining the bytecode from the called function into the caller

peak spoke
#

needless for the partial

raven ridge
#

partial isn't exactly a hot spot in typical code - optimizations that might be specific to partial don't seem terribly worth pursuing. I only pointed it out because it's interesting that moving it to Python might actually make it faster, since it doesn't benefit much from being in C, and it's missing out on that inlined frames optimization as a result

peak spoke
#

on3.11.0b4

In [2]: a=functools.partial(lambda:None)

In [3]: %timeit a()
62.2 ns Β± 0.39 ns per loop (mean Β± std. dev. of 7 runs, 10,000,000 loops each)

In [4]: sys.modules["_functools"] = None

In [5]: importlib.reload(functools)
Out[5]: <module 'functools' from 'C:\\Users\\miso1\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\functools.py'>

In [6]: a=functools.partial(lambda:None)

In [7]: %timeit a()
229 ns Β± 1.73 ns per loop (mean Β± std. dev. of 7 runs, 1,000,000 loops each)

while probably not a hotspot, I'd also expect it to be as cheap as it can be. Some callbacks can be triggered a lot, which is where a partial also fits in perfectly

raven ridge
#

wow, that's much more of a difference than I would have expected

#

thanks for checking my conjecture with data πŸ˜„

swift imp
#

That's really good speed up

#

I know 3.11 isn't going to be ground breaking but I think it's a great step forward

raven ridge
#

that's not really related to 3.11 - that's showing the speed difference between the C functools.partial implementation and the Python functools.partial implementation

swift imp
#

Oh I misread that at a glance

rose schooner
#

we're getting inlines in python 3.14??

spark magnet
rose schooner
#

well i mean we're starting to do a lot of optimizing in 3.11 and above

halcyon trail
#

I remembered I read a blog post once that explained how, because of implementation choices that were being made anyways

#

it was very cheap, nearly free, to provide insertion ordering

#

on python 3's dict

#

but I cannot find it

#

anybody know of a blog post that explains a bit about this?

peak spoke
halcyon trail
prime estuary
# rose schooner we're getting inlines in python 3.14??

It's already in 3.11 - though inlining here just means the interpreter pushes a frame, then jumps back to the beginning of the eval loop to execute the new function, without needing to call a new C function. There's also several specialisations implemented so that things like __getitem__, properties and even __init__ can use this to quickly run the Python code.

rose schooner
#

but if that's just that then what does he mean by "inline the property call"

prime estuary
#

It does sound like potentially actually inlining the actual code for simpler function calls, which would be very nice.

heady mauve
#

Builtins are defined within the interpreter, but standard libraries and modules are defined in the library folders that ship with the interpreter, correct?

dapper lily
#

builtins is also a module

#

which gets auto imported when you run the interpreter interactively (i.e. the Python REPL)

heady mauve
#

So the interpreter imports int and the like from a .py file somewhere?

timber dagger
#

it's not a .py file, it's a module that's written in C

heady mauve
#

ok

peak spoke
#

the module or builtins are not really imported into your modules (unless you import it yourself), instead it serves as the namespace from where python looks up the builtins when you try to get them

#

for a generic name lookup, cpython will first look at the locals dict, then the globals dict if locals don't have the key, and builtins if the global lookup fails

dapper lily
#

ah. right.

heady mauve
#

Ok

peak spoke
#

also why you may sometimes find something like def f(int=int) as it'll turn the "slow" builtin lookup (first tries to find global, then the builtin) into a fast locals lookup

heady mauve
#

I see... That way every time code inside f references int, it's in locals. I guess it only has to try as far as builtins once when called then?

peak spoke
#

Yes; if it's used in a tight loop it may be a noticeable speed benefit, but on the other hand a loop where that may help is probably better of not being in pure python

heady mauve
#

right

#

I'm designing a hardware implementation of Python, so I have plenty of questions that I haven't been able to work out through the docs and other resources. When i call import foo, __loader__ is instantiated to try and find a module or package by the name foo?

#

According to the REPL __loader__ is _frozen_importlib.BuiltinImporter.

spark magnet
ashen kelp
#

Hey all how can I automate running my python function in a jupyter notebook? I know I can do with schedule library in a python file but I don't know how to proceed with jupyter notebook

boreal umbra
#

you should think of jupyter notebooks as the end of the line. if you need anything in a jupyter notebook outside of that exact notebook, at any time, it should be defined in a py file outside of the notebook, and be imported into it.

swift imp
#

@ashen kelp see link

heady mauve