internals-and-peps

[grave jolt]

Because if str.join encounters something that's not a list or tuple, it copies it into an internal temporary array

so no laziness can occur

[boreal umbra]

is that so that it can check that every element is a string, or what?

[grave jolt]

i don't rember

[feral island]

in general creating the list at once tends to be faster than a genexp because there's less call overhead

[boreal umbra]

more generally, does that mean that lazy evaluation is only more performant if all the data returned by the generator would consume too much memory, or if there's a chance that the consumer of the generator might stop early?

[feral island]

probably, though you should benchmark to be sure

[boreal umbra]

!e

import timeit
print(timeit.timeit("list(itertools.count(0))", setup="import itertools"))

[fallen slate]

@boreal umbra :warning: Your 3.11 eval job timed out or ran out of memory.

[No output]

[boreal umbra]

😄

[raven ridge]

it's to allow str.join to preallocate the output string

[feral island]

on 3.11rc2 ```In [74]: %timeit x = (i for i in range(10)); sum(x)
726 ns ± 4.12 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [75]: %timeit x = [i for i in range(10)]; sum(x)
574 ns ± 6.77 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

[raven ridge]

https://github.com/python/cpython/blob/main/Objects/unicodeobject.c#L9461-L9488 is the first pass, https://github.com/python/cpython/blob/main/Objects/unicodeobject.c#L9505-L9547 is the second.

[rose schooner]

this is so much better ```py
raise TypeError("sequence item 0: expected str instance, int found")

unless of course xs is empty or xs.__iter__().__next__() returns a string or a subclass of str or an object that overrides __pow__ to return a string or a subclass of str

[rose schooner]

i also managed to make setting much faster
i don't know why i did __slots__ ```py

class P:
... slots = ('x', 'y')
... def init(self, x, y):
... self.(x, y) = x, y
...
p = P(2, 3)
from timeit import main
main(['-s', "from main import p", "p.x = -2; p.y = 7"])
2000000 loops, best of 5: 120 nsec per loop
main(['-s', "from main import p", "p.(x, y) = -2, 7"])
5000000 loops, best of 5: 98.3 nsec per loop

>>> def a(): p.x = -2; p.y = 7
... 
>>> def b(): p.(x, y) = -2, 7
... 
>>> for _ in range(100000): a()
... 
>>> for _ in range(100000): b()
... 
>>> from dis import dis
>>> dis(a, adaptive=True)
  1           0 RESUME_QUICK             0
              2 LOAD_CONST               1 (-2)
              4 LOAD_GLOBAL_MODULE       0 (p)
             16 STORE_ATTR_SLOT          1 (x)
             26 LOAD_CONST               2 (7)
             28 LOAD_GLOBAL_MODULE       0 (p)
             40 STORE_ATTR_SLOT          2 (y)
             50 LOAD_CONST               0 (None)
             52 RETURN_VALUE
>>> dis(b, adaptive=True)
  1           0 RESUME_QUICK             0
              2 LOAD_CONST               1 ((-2, 7))
              4 LOAD_GLOBAL_MODULE       0 (p)
             16 STORE_MATTR_SLOT         2 (('x', 'y'))
             34 LOAD_CONST               0 (None)
             36 RETURN_VALUE

[rough token]

Hi guys, do you think this filter is pep-8 friendly?

return (m.author == ctx.author
            # Author of message is command runner
            and m.channel.id == ctx.message.channel.id
            # Message is on the same channel than context
            and not m.reference
            # Mention is not a reply reference
            and len(m.content.split(' ')) == 1
            # Message contains one element
            and (len(m.mentions) == 1 or 
                 await search_id(bot, m)))
                 # Message contains a digit or a mention

[quick snow]

I'd break it differently:

return (
    m.author == ctx.author
    # Author of message is command runner
    and m.channel.id == ctx.message.channel.id
    # etc.
)

Or use all:

return all([
    m.author == ctx.author,
    m.channel.id == ctx.message.channel.id,
    ...,
])

[rough token]

ohhhh this !!!
thank youuu

[quick snow]

just to be clear: it's not 100% equivalent. For example, if m.author != ctx.author, the rest is not evaluated in your original version, whereas it is when using all

[rough token]

return all([
    m.author == ctx.author,
    m.channel.id == ctx.message.channel.id,
    not m.reference,
    len(m.content.split(' ')) == 1,
    any([
        all([m.content.isdigit(),
            not m.mentions]),
        all([len(m.mentions) == 1,
        not m.content.replace(f'<@{m.mentions[0].id}>', '')])])])

Is this better then? :D

Oh ok I tested this and it works ! Thank you so much !

[quick snow]

)])])]) <- this looks bad

[rough token]

Oh I should break it in multiple lines?

[quick snow]

Just run it through black, that will usually make the code look decent

[rough token]

I don't want to, I prefer getting used to pep-8 manually. And then i'll start using black :D

[quick snow]

understood, you don't have to use black everywhere, just treat it as a colleague you ask for style help.

[rough token]

is it 100% pep8 friendly and perfectly?

[quick snow]

like if you're unsure about a section, run the file through black, and then maybe take that section from black.

[quick snow]

PEP-8 is — intentionally — wishy-washy and not 100% unambiguous

But.. yes.

[rough token]

aw I see :/

async def filter_user(ctx, m):
    # Returns True if m.content is a digit or a mention
    return all(
        [
            m.author == ctx.author,
            m.channel.id == ctx.message.channel.id,
            not m.reference,
            len(m.content.split(" ")) == 1,
            any(
                [
                    all([m.content.isdigit(), not m.mentions]),
                    all(
                        [
                            len(m.mentions) == 1,
                            not m.content.replace(f"<@{m.mentions[0].id}>", ""),
                        ]
                    ),
                ]
            ),
        ]
    )

this is what black returned :o

[quick snow]

looks good to me

[rough token]

I agree of too much ])

Maybe I should use and or in the smaller places

async def filter_user(ctx, m):
    # Returns True if m.content is a digit or a mention
    return all(
        [
            m.author == ctx.author,
            m.channel.id == ctx.message.channel.id,
            not m.reference,
            any([m.content.isdigit() and not m.mentions]),
            len(m.mentions) == 1
            and not m.content.replace(f"<@{m.mentions[0].id}>", ""),
        ]
    )

MUCH better

Also I removed a useless condition

[rough token]

ty for help <3

[gray galleon]

you can remove the any ```py
async def filter_user(ctx, m):
# Returns True if m.content is a digit or a mention
return all(
[
m.author == ctx.author,
m.channel.id == ctx.message.channel.id,
not m.reference,
any(m.content.isdigit() and not m.mentions,
len(m.mentions) == 1 and not m.content.replace(f"<@{m.mentions[0].id}>", "")),
]
)

[rough token]

it's supposed to be ([m.content.isdigit() and not m.mentions]) or (len(m.mentions) == 1 and not m.content.replace(f"<@{m.mentions[0].id}>", ")

[gray galleon]

so ths

[rough token]

Ohhh it's wrong ! you're right

async def filter_user(ctx, m):
    # Returns True if m.content is a digit or a mention
    return all(
        [
            m.author == ctx.author,
            m.channel.id == ctx.message.channel.id,
            not m.reference,
            m.content == "cancel",
            any(
                [
                    m.content.isdigit() and not m.mentions,
                    len(m.mentions) == 1
                    and not m.content.replace(f"<@{m.mentions[0].id}>", ""),
                ]
            ),
        ]
    )

here

[grave jolt]

that looks quite complex to me

what is m and what is m.reference?

[fallen slate]

Hey @warm mauve!

You either uploaded a .txt file or entered a message that was too long. Please use our paste bin instead.

[dapper lily]

probably a discord.Message object from the looks of it

[warm mauve]

https://paste.pythondiscord.com/raw/evovajezej

[grave jolt]

ah, and reference is the message it's in reply to

[warm mauve]

i need help on this project this don't return me occupate places

[dapper lily]

#❓｜how-to-get-help

[feral island]

if m.content == "cancel" how can it also .isdigit()

[grave jolt]

yeah

and if m.content == "cancel", then m.content.replace(f"<@{m.mentions[0].id}>", "") doesn't do anything

[rough token]

Indeed, I corrected it

async def filter_user(ctx, m):
    # Returns True if m.content is a digit or a mention

    return m.content.lower() == "cancel" or all(
        [
            m.author == ctx.author,
            m.channel.id == ctx.message.channel.id,
            not m.reference,
            any(
                [
                    m.content.isdigit() and not m.mentions,
                    len(m.mentions) == 1
                    and not m.content.replace(f"<@{m.mentions[0].id}>", ""),
                ]
            ),
        ]
    )

Thank you !

[grave jolt]

this will accept a cancel message from any user in any channel

[rough token]

ah indeed lemme rewrite it

async def filter_user(ctx, m):
    # Returns True if m.content is a digit or a mention

    return all(
        [
            m.author == ctx.author,
            m.channel.id == ctx.message.channel.id,
            not m.reference,
            any(
                [
                    m.content.lower() == "cancel",
                    m.content.isdigit() and not m.mentions,
                    len(m.mentions) == 1
                    and not m.content.replace(f"<@{m.mentions[0].id}>", ""),
                ]
            ),
        ]
    )

[grave jolt]

This is how I'd do it, probably

async def filter_user(ctx, msg):
    is_cancel = msg.content.lower() == "cancel"
    is_digit = msg.content.isdecimal()
    if len(msg.mentions) == 1:
        is_single_mention = msg.content == f"<@{msg.raw_mentions[0]}>"
    else:
        is_single_mention = False
    return is_from_here(ctx, msg) and msg.reference is None and (is_cancel or is_digit or is_single_mention)

I assume the msg.author == ctx.author and msg.channel.id == ctx.message.channel.id thing occurs in a lot of places, so I extracted it into a separate function

If you want laziness there's also this pattern

async def filter_user(ctx, msg):
    def matches():
        yield msg.content.lower() == "cancel"
        yield msg.content.isdecimal()
        yield len(msg.raw_mentions) == 1 and msg.content == f"<@{msg.raw_mentions[0]}>"
    return is_from_here(ctx, msg) and msg.reference is None and any(matches())

[rough token]

Yooooo this is next level coding for me I like it !
I also like the yield pattern but i don't know which one feels more readable and simple

[dusk comet]

Wow, never seen that
That is very interesting pattern

[rough token]

Thank you very much to be honest !

I found out that yield len(msg.raw_mentions) == 1 and msg.content == f"<@{msg.raw_mentions[0]}>" can be reduced to just msg.content == f"<@{msg.raw_mentions[0]}>"

[grave jolt]

that won't work if the message has no mentions

[rough token]

then I don't want it to be captured by this yield no?

if it has no mentions it will be filtered by the other conditions

[grave jolt]

not necessarily

if I send a message containing the text foo it won't work

[rough token]

    def matches(ctx, msg):
        yield msg.content.lower == 'cancel'
        yield msg.content.isdigit()
        yield msg.content == f'<@{msg.mentions[0].id}>'

here foo will return false won't it?

if it does, then matches() worked the way I want

[grave jolt]

what is msg.mentions[0] if the content is "foo"?

[rough token]

ohhhhh right that'll raise an error

index error... you're right x.x

what about this: yield msg.mentions and msg.content == f'<@{msg.mentions[0].id}>'

saves some horizontal space in the code

that == 1 has been taken out

uh sorry i meant len(msg.mentions)

[grave jolt]

I guess, although IMO my version is more explicit

[grave jolt]

no that's not necessary

[rough token]

an empty list is considered False or None ?

[grave jolt]

an empty list if considered to be falsey

[rough token]

you mean it's better for readability and more professional?

[grave jolt]

"professional" sounds judgemental 🙂

but yeah I think it's more clear

[rough token]

I just don't want to dissapoint the work that I'll start in january

[rough token]

okay I'll stick with it, sounds precise indeed

[grave jolt]

actually ```py
yield len(msg.mentions) == 1 and msg.content == msg.mentions[0].mention

anyway... this is probably off-topic here

[rough token]

I don't think so !

We're talking about clean code. My code is working already

The bot is active in a 12k people community

This channel is about cleaning my code not figure out how it should work so It's the right place. Coding is about many topics at the same time sometimes :D

[rough token]

SLAY I love it !

Although I prefer yield msg.mentions and msg.content == msg.mentions[0].mention
because i can clearly read it as "if there's any mention, and the content is the first mention"
whereas yield len(msg.mentions) == 1 and msg.content == msg.mentions[0].mention sounds like "if there is one mention, and the content is the first mention" which doesn't really make difference in my very very humble opinion, what do you think?

Also I want to know why you chose isdecimal() method instead of isdigit()?

def is_from_here(ctx, msg):
    return msg.author == ctx.author and msg.channel.id == ctx.message.channel.id

async def filter_user(ctx, msg):
    # Returns True if msg.content is a digit or a mention
    def matches():
        yield msg.content.lower == 'cancel'
        yield msg.content.isdigit()
        yield msg.mentions and msg.content == msg.mentions[0].mention
    return is_from_here(ctx, msg) and not msg.reference and any(matches())

I really love the result thank you so much !!!! 😭

[paper echo]

lisp programmers are a funny bunch. we have TAGBODY which is essentially a local goto block, but it's considered low-level and people rarely abuse it, it's mostly used for implementing higher-level control flow macros. i think however that lisp programmers nowadays seem to be a self-selected highly disciplined group and i can only imagine the spaghetti horrors that we'd be helping people with in this server if we had goto

presumably that's also one reason we don't have labeled loops and multi-line lambdas, it discourages complicated code

[flat gazelle]

Yeah, it is needed if you want users to handroll control flow.

[unkempt rock]

Does Python have official names for the 3 numbers of a version? like 3.10.8 = major.minor.patch (akak semver https://semver.org/). I see major.minor.micro on https://peps.python.org/pep-0440/#final-releases so I'll probably go with that, but I've also seen it as 3.major.minor.

[rose schooner]

3.major.minor?

[raven ridge]

They call it micro. https://docs.python.org/3/c-api/apiabiversion.html

[rose schooner]

https://github.com/python/cpython/blob/main/Include/patchlevel.h#L19-L26

[fallen slate]

Include/patchlevel.h lines 19 to 26

#define PY_MAJOR_VERSION        3
#define PY_MINOR_VERSION        12
#define PY_MICRO_VERSION        0
#define PY_RELEASE_LEVEL        PY_RELEASE_LEVEL_ALPHA
#define PY_RELEASE_SERIAL       1

/* Version as a string */
#define PY_VERSION              "3.12.0a1+"```

[unkempt rock]

Micro it is

[rose schooner]

<major>.<minor>.<micro>[<level><serial>[+]]

[rough token]

I have a question that might look silly lol
I'm wondering if my new commenting and clean coding looks silly:

#------------------Width counter (72 chars)-----------------------------

import os

# DISCORD LIBS
import discord
from discord.utils import get
from discord.ext import commands
from discord.ext.commands import cooldown

# INTERNAL LIBS
from config import *
from ressources import (add_report, update_report, 
                        remove_report, search_report)
from ressources.add_functions import select_user

# VARS
bot = commands.Bot(command_prefix = '?',
                   intents = discord.Intents.all())

#-----------------------
# CLIENT INITIALIZATION
#-----------------------
@bot.event
async def on_ready():
    print("Bot connected")

I've read the Clean Code book from uncle bob and he said that we spend more time reading our code than writing it and... with all those advices I decided to improve my coding cleaning to the best possible. How would you rate this?

[boreal umbra]

this would be a question for #software-architecture, but star imports are generally disliked because they put stuff in the namespace that you can't see.

[rough token]

imports are so overwhelming :(
okay i'll ask in software-design tyyy

[unkempt rock]

I notice that the Python 3.11 (and I think 3.10) installers don't have an option to customize the install location like old installers did. Is that a new thing they intend to stick with do we know?

(I like it since installing in C:/ often had permissions issues when updating pip packages)

[neat delta]

is that not what

Customize installation
Choose location and features
does?

[unkempt rock]

Oh, I'm dumb I didn't click Next to find these advanced options

[gray galleon]

not having multiline lambdas also reduce convenience when writing event driven code

[meager arch]

you see, you can abuse esoteric tricks to fit all the logic in one line!
clean code

[languid pebble]

apparent regression (it was suggested i post it here for discussion): this code finishes in python 3.10.4, but it silently crashes the python process in 3.11.0 on the last call to "test" (i'm also using windows 11)

import mmap, os
from os.path import join, expandvars, isfile
from sys import version, argv
print('Version:', version)
path = join(expandvars('%APPDATA%'), 'PyRegressionTest.bin')
print("Test file path:", path)
def test(file_length, to_search):
    print("Making test file.")
    with open(path, 'wb') as f:
        f.write(b'\0' * file_length)
    with open(path, 'rb') as f:
        m = mmap.mmap(f.fileno(), 0, access = mmap.ACCESS_READ)
        print(f"Finding \"{to_search.hex().upper()}\" in 0x{file_length:X} bytes.")
        location = m.find(to_search)
        print("Find call finished successfully.")
if isfile(path):
    os.remove(path)
    print("Removed old test file.")
input("Hit enter to begin testing.")
test(file_length = 0xFFF , to_search = b'\x25\xFF\x00\xFF')
test(file_length = 0x1000, to_search = b'\x25\xFF\x00')
test(file_length = 0x1000, to_search = b'\x25\xFF\x00\xFF')
input('All tests finished. Enter to exit.')```
this is true on at least my computer and the computer of the person who had a problem with one of my programs, but can someone else confirm?

the line containing m.find(to_search) is what's crashing the process on 3.11.0

[boreal umbra]

This is for talking about the Python language itself. Kindly remove your comment from this channel and ask in #user-interfaces

[halcyon kernel]

ok thanks

[dusk comet]

i can confirm that, im using windows 10
on CPython3.11.0 it crashes on the third call
on CPython3.10.6 it works:

>py310 _.py
Version: 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Finding "25FF00FF" in 0xFFF bytes.
Found.
Finding "25FF00" in 0x1000 bytes.
Found.
Finding "25FF00FF" in 0x1000 bytes.
Found.
Finished. Enter to exit-

upd: can you edit your code to delete %APPDATA%/PyRegressionTest.bin file please?

[languid pebble]

i'm not sure if i can, mmap requires the file to exist, and the regression terminates the process so error handling is out of the question

[raven ridge]

you could remove if it exists the next time the script is run.

[languid pebble]

revised it a bit, it deletes the file and prompts to start testing

[boreal umbra]

@proven bramble is of the impression that for loops are inherently faster than while loops because for loops are "implemented in C". Perhaps someone could explain why that doesn't add up.

[proven bramble]

import time


def for_loop(iters):
    i = 0
    for _ in range(iters):
        i += _


def while_loop(iters):
    i = 0
    _ = 0
    while _ < iters:
        i += _
        _ += 1
        

def while_for_mix_loop(iters):
    range_iter = iter(range(iters))
    i = 0
    try:
        while True:
            i += next(range_iter)
    except StopIteration:
        pass
    
        

def timer(f, iters = 1000000):
    start = time.perf_counter()
    f(iters)
    return time.perf_counter() - start

print("Number of iterations: {100}")
print(timer(for_loop,   100))
print(timer(while_loop, 100))
print(timer(while_for_mix_loop, 100))

print("Number of iterations: {1_000}")
print(timer(for_loop,   1_000))
print(timer(while_loop, 1_000))
print(timer(while_for_mix_loop, 1_000))

print("Number of iterations: {100_000}")
print(timer(for_loop,   100_000))
print(timer(while_loop, 100_000))
print(timer(while_for_mix_loop, 100_000))

Number of iterations: {100}
5.263000275590457e-06
5.8670002545113675e-06
7.407000339298975e-06
Number of iterations: {1_000}
4.950799939251738e-05
5.7083000683633145e-05
5.634699937218102e-05
Number of iterations: {100_000}
0.0038534739996975986
0.006391976000486466
0.006662508999397687```

actually do explain how while loops are implemented and how for is implemented

cause the numbers dont add up

[boreal umbra]

This appears to be demonstrating that writing out the logic for the for loop spec in pure Python is slower than just using a for loop, but that's not the same as saying that for loops are inherently faster than while loops in general.

[proven bramble]

so while/for are "implemented in C" :)

imma stop now, do tag me if we find some insight on why exactly for is faster (for "small computations" cause this really wouldnt matter for "big/large computations")

[spark magnet]

@proven bramble for loops are faster probably because they compile to fewer bytecodes, and more of the work is in C (like incrementing the value).

[proven bramble]

That's interesting, is there a way to inspect the bytecode that is generated?
I always forget that byte code is a thing

[spark magnet]

Sure, the dis module.

[boreal umbra]

(though that doesn't mean that rewriting any arbitrary while loop as a for loop is a viable optimization technique, which was what we were debating initially.)

[spark magnet]

In your code, import dis, then dis.dis(for_loop)

[spark magnet]

i missed the start of the discussion

[boreal umbra]

I know, that's okay

[proven bramble]

Follow up question, so can we say that python (the interpretor) compiles and optimises the code ? (Offcourse not at the level of say for c++)

[spark magnet]

Python compiles your source to bytecode, with some minimal optimizations

[proven bramble]

Yeah, that's fair

[proven bramble]

So why don't they optimise it further?
Is it a time to compile thing ?

Wait pypy exist

Nvm
It's probably a compile time thing

[spark magnet]

Python's semantics are very dynamic, so it's hard to optimize more without more information.

PyPy has a JIT, cython has extra syntax to add information, mypyc uses type annotations, etc..

[boreal umbra]

Yes, even things as seemingly trivial as getting an attribute of some object might trigger a function call, and that can't even be known for sure until runtime.

[proven bramble]

Ummm, type annotations would help tho right ?
As in if a function has type hints, just optimise it completely

But that would kinda restrict the dynamic nature of python

[quick snow]

No, because type annotations may lie

[proven bramble]

Yeah, i am saying that for functions with annotations, the compiler will have to ensure that when ever the function is called, the type of the objects must be same as what's annotated

Kinda making python a semi compiled language

That would be intresting

[quick snow]

But now you're making the function slower, because now the incoming types have to be checked at runtime.

[boreal umbra]

That would still require runtime type checking, and that would be especially difficult for container types (or any type with generics).

[proven bramble]

Nono
At "compile-time", when it's being transpiled to byte code

[quick snow]

What should happen when I hand in a wrong type then?

[boreal umbra]

You'd have to have static typing all through the language for that to work, or the type can't be known until runtime.

[proven bramble]

Compiler error ?

[proven bramble]

Yeah

[quick snow]

Good luck checking for something like Iterator[int], even at runtime you can't check that.

[proven bramble]

Wait that's not python anymore

[boreal umbra]

Right. Dynamic typing is baked into the language spec.

[proven bramble]

Yeah that's probably not possible without runtime type checking

[quick snow]

Runtime type checking is (in general) impossible

[proven bramble]

Only if python has a static mode/option

[boreal umbra]

So, two languages

[proven bramble]

Fair enough that's a lot of work

[boreal umbra]

I think it's what mypy originally set out to be

[quick snow]

def foo():
    while True:
        if requests.get("https://api.l3vi.de/bar").json()[":)"]:
            yield 1
        else:
            yield "1"
def bat(x: Iterator[int]): ...

May I pass foo() to bat?

[proven bramble]

Umm, why isn't runtime checking not possible here ?
As in if foo returns a 1 it's all good, if it returns a "i" raise error ?

[quick snow]

When foo() is handed to bar, no API call has been made yet.

[dusk comet]

itertools.repeat(None, n) is 2 times faster than range(n)

[quick snow]

But this is the point when you said you'd check

[proven bramble]

Ohhh yeah

Ummm what about only allowing a subset of type annotations to be allowed in optimised compilation mode ?

[quick snow]

like which ones?

[proven bramble]

As in functions with iter inputs would not be optimised (and in tern no type checking will be required)

[proven bramble]

Ints, floats, arrays (not list)

[quick snow]

__instancecheck__ exists

[proven bramble]

Wdm ?

[quick snow]

class MaybeAnInt:
    def __instancecheck__(self, other):
        return requests.get("...").etc().pp()
maybe_an_int = MaybeAnInt()
def bar(x: int): ...

Python is just too dynamic of a language for this to work in general

wait, I misunderstood things, one sec

[proven bramble]

Take ur time, i am kinda confused atm with the example you gave 😅

[flat gazelle]

what you are talking about is pretty much numba

[proven bramble]

Yes

More or less

[flat gazelle]

it implements a subset of python as a statically typed language producing llvm bitcode.

[proven bramble]

Make numba a core part of python

[quick snow]

Yes, and it can work, because the user manually says "this function is definitely annotated correctly, I promise"

[spark magnet]

RPython is along these lines also

[proven bramble]

Numba throws an error when the types dont match

[flat gazelle]

why? an actually good static compiler is about as complex as python itself

[proven bramble]

Never heard of this, will take a look 👍

[flat gazelle]

aren't RPython, rustpython, graalpy etc. more in that they compile various type configurations of the program, but still have to do typechecks every time. a numba loop adding numbers together never has to check whether it's in fact adding a number, it just knows that it is.

[proven bramble]

Ummm tbh, i just want python to be better, it's really good atm, but kinda slow
Can't really do computationally expensive stuff
Like game programming
I am working on a c++ based "hobby" engine and recently have been trying to get python scripting
Python slows down stuff alot
I have experimented with trying to use numba
But there is a small problem which i never really figured out how to solve (numba doesn't allow c++ implemented functions to be called in njit decorated functions, cause it has no idea about its inputs and outputs types)

[spark magnet]

RPython is a restricted python used in PyPy

[proven bramble]

Cool, i don't really like pypy, I'll read about it before making an opinion on it

[proven bramble]

Complexity aside, adding it to python would be a cool feature right, it would make python even more viable

[spark magnet]

what don't you like about PyPy?

[flat gazelle]

yeah, you definitely want something like lua or tcl there, marshalling data between C++ and python isn't all that ideal for performance.

[proven bramble]

Not as many libraries and less extendable with c/c++

[proven bramble]

Yes! I feel like python is almost there, it will become the perfect language for every thing

[flat gazelle]

this is something graalvm and graalpy should in theory do, but it is definitely way too undocumented and untested to really do much.

[proven bramble]

Yeah

Excited for the future tho
Might be in 4.x update

[rose schooner]

more accurately 3.12 to 3.13
those versions are planned to have a partial JIT

[proven bramble]

Wait really?
Is there a doc or something

[spark magnet]

@proven bramble https://github.com/markshannon/faster-cpython/blob/master/plan.md

[orchid karma]

There likely is, but it was explicitly mentioned during the 3.11 release. The work they did in 3.11 and have planned for 3.12 enables them to prep for some JIT

[proven bramble]

This makes me so excited

:D

[spark magnet]

i'm kind of dreading it 🙂 But only because I maintain coverage.py which gets very involved in details of execution.

[proven bramble]

Ahhahh 😂

All the best

[feral island]

Mark Shannon keeps cautioning though that a JIT isn't magic going faster sauce, there are many things a JIT can't help with and many things we can do to make CPython faster without a JIT

[rose schooner]

would python ever do variable optimizing? like py x = 5 y += x gets converted to the equivalent of y += 5 in bytecode

[feral island]

probably? I guess the risk is that it breaks locals()

and breaking trace functions

though I guess optimizations can just be disabled when a trace function is active

[proven bramble]

CPython is developed using an open-source, collaborative model. That means that any complexity in the implementation should be understandable, and maintainable by someone other than the author(s)```
I see why your dreading it now 😂😂😂

[spark magnet]

usually during the alpha phases, I write 5-10 issues about changes in tracing behavior

[proven bramble]

Again all the best :)

[swift imp]

Doesnt arrow support this?

Like arrow can support processes sharing the same blocks of memory, iirc, might be a future goal

[frigid bison]

Python is going to get harder and harder to optimize because of how flexible it is, let's say you did the optimization that @rose schooner did then it wouldn't be possible to import that module and get x from it, it might not happen but we don't know for sure and there's virtually no way ensure that it doesn't get referenced somewhere else so optimizing something like that is never going to be possible

[swift imp]

@flat gazelle https://arrow.apache.org/use_cases/

https://github.com/apache/arrow-datafusion-python

[flat gazelle]

Oh, that is actually pretty neat, bookmarking that.

[elder blade]

What does ns mean in std-lib code? I see it in both dataclasses and typing then passed as globals

[raven ridge]

"namespace", presumably

At least, everywhere except where it means "nanoseconds" 😄

[boreal umbra]

@rough token kindly remove your question and ask it in #software-architecture

[rough token]

Done :)

[gray galleon]

why does map return map object
instead of list or smth else

[dapper lily]

because it's supposed to be lazy

like a generator

[gray galleon]

is it me or python discourages method chaining in their built in objects

[rich cradle]

i don't think python really has any mechanisms to allow for doing it cleanly

without massive inheritance trees

[gray galleon]

python uses built in functions for things you do with method calling and chaining in other oop languages
which is a shame
bcs method chaining is cool and clean

[rich cradle]

it is

but there's not really a mechanism to make it work on all of the appropriate kinds of objects, to the extent of my knowledge

[rose schooner]

what if there was some sort of mechanism for "lazy loading" of variables

[frigid bison]

Elaborate pls

[main basin]

heelo

[halcyon trail]

Coming back to python after Kotlin hurt for this reason. The method chaining for transforming collections etc

[paper echo]

no please, that's what R has

it's how you get macros and compile-time evaluation

i don't think python needs or wants that

[dusk comet]

what you mean by "method chaining"?
something like that: obj.do_foo().do_bar().do_bar()?

[gray galleon]

yes

[dusk comet]

why does it need "massive inheritance trees"?

[halcyon trail]

I guess especially in the context of working with collections based on the comment about map

Well the only two ways to get method syntax in python are coding it directly on the object, and inheritance

[dusk comet]

i got it

[halcyon trail]

And I suppose monkey patching

[dusk comet]

T = TypeVar('T', bound=Iterable)
G = TypeVar('G')
H = TypeVar('H')

class MapMethodMixin:
    def map(self: T[G], f: Callable[[G], H]) -> T[H]: # i know that `T[G]` is not valid
        return self.__class__(map(f, self)) # assuming `self.__class__` implements constructing from iterable
        return self.__class__.from_iterable(map(f, self))

class maplist(list[G], MapMethodMixin): ...

l = maplist([0,1,2,3])
l.map(str).map('-'.__add__).map(int).map(print)
# output:
# 0
# -1
# -2
# -3

[radiant garden]

Functor 😄

[halcyon trail]

Yeah, the sad thing with that approach is the fact that to make it work you have to create a new type

a lot of newer languages (Rust, Kotlin, Swift) offer ways of doing extensions non-intrusively

(C# too)

[dusk comet]

it is also possible to monkey-patch builtin types, but it is very bad practice

[halcyon trail]

yeah

another issue is that static dispatch is actually a really nice tool for making extensions work

python doesn't have that, so it's more tricky I think to consider how it would work to start with

like, at some point I remember thinking it would be really nice if iterables supported the + syntax for simply concatenating the two iterators. itertools.chain is verbose and not very nice to look at

In languages like Kotlin, C#, Rust, etc you can make this work because you define functions over some constrained static type, and it'll be available on all type smeeting the constraint

in python there's lots and lots of different things that model Iterable and there's no way to inject + into all of them

[dusk comet]

>>> sorted((), a=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'a' is an invalid keyword argument for sort()

Not a bug, but an implementation detail. In this exception traceback shows name sort(), not sorted(), because it is internally calling list.sort()

[halcyon trail]

i guess that for a lazy iterator there isn't a more efficient way of doing it

[grave jolt]

I suppose one reason is that you can't extend an existing class with methods. But you can always make your own function.

[halcyon trail]

the problem is that with free functions you don't get that nice chaining syntax

you get reversed ordering of operations instead

[grave jolt]

yeah, it gets real messy

well, you can make a "pipe"

[halcyon trail]

whenever I try to use itertools it becomes a real mess quickly

i mean it's good for feeding basic stuff into a for loop but chaining stuff not so mujch

[grave jolt]

pipe(thing) >> (map, str) >> (filter, None)

[halcyon trail]

yeah, it's just kinda gross and also it's made more gross by the poor lambdas that python has

that's the other element here

pipe(thing) >> (map, lambda x: x.get_string_rep()[0:4]) >> (filter, lambda x: x.starts_with("he"))

beautiful

[grave jolt]

well, at least there's the job security element

the problem is that some things are methods, and some are functions

For example, functional languages largely don't have this problem. In Elm, there are no methods, only functions. So any nested carnage you come up with can mostly be pipified

[halcyon trail]

i don't see how that is really an issue

[grave jolt]

well, that combined with awkward lambdas

[halcyon trail]

Sure, if you have awkward lambdas though then it's basically game over no matter what, the thing you want to do very often isn't just exactly some existing function or method

when I was coding in Kotlin it was far more common to pass lambdas to these kinds of things, then to have an already-existing function to pass in

[grave jolt]

in ML-like languages that's mostly done with partial application.

thing
|> List.map (getStringRep << String.slice 0 4)
|> List.filter (String.startsWith "he")
``` or ```
thing
|> List.map getStringRep
|> List.map (String.slice 0 4)
|> List.filter (String.startsWith "he")

but yeah, you can't always do that

[halcyon trail]

yeah, i heard about that kind of stuff a lot in ML-likes and personally I'm always very skeptical about currying etc

[grave jolt]

I see, you prefer chilli

[halcyon trail]

I prefer curry over currying, that's for sure

[grave jolt]

idk, I kinda got used to it

maybe it's stockholm syndrome

[halcyon trail]

yeah I mean I'm not one of those folks that's going to try to argue that its fundamentally unnatural or something silly like that

the thing is that as you said, it only solves a subset of problems still.

[grave jolt]

100% natural, almost gluten-free Python lambdas

[halcyon trail]

lambdas in the end solve all of these problems, so if you can just make the lambda syntax really really nice, you have to ask yourself what's the marginal benefit of having a very slightly more concise way to handle specific cases

not only do people need to know about this special syntax, but on top of that currying unfortunately makes a very common class of errors far more obscure to debug

[grave jolt]

you don't need to name the argument 🙂
although in kotlin it's just it

[halcyon trail]

omitting a function argument is terribly common so you want a really good error message for that and currying gets in the way of that

yeah, kotlin and swift both implement this idea

String.startsWith "he" versus { it.startsWith("he") }

even in the best case very little is being saved

that's why IMHO you odn't really see currying even in languages that have borrowed lots and lots of ideas from FP. Rust being probably the best example, though Kotlin and Swift too, to lesser extents.

[grave jolt]

well, Rust is complicated as is

imagine Rust with currying

[halcyon trail]

no thankyou 🙂

[grave jolt]

although there's probably a macro library for that

[halcyon trail]

but eve in Kotlin/Swift it's easy to see that currying is never going to save you more than a few keystrokes

[grave jolt]

of course
https://docs.rs/curry-macro/0.1.0/curry_macro/macro.curry.html

[halcyon trail]

😂

A month or so ago I finally sat down and wrote a small toy project in Rust, I was pretty pleased overall. Although unsafe rust is really scary stuff (much scarier than C++)

[grave jolt]

I probably did too much Python, because I'm kinda scared to do anything in other languages...

how do I unpython myself?

[halcyon trail]

you were just talking getting used to currying though

so clearly you have done a significant amount of non-python

[grave jolt]

well, it's not significant really

[halcyon trail]

well, in that case you're quick at getting used to things 🙂 Either way a good sign!

[grave jolt]

I just did some introductory lambda calculus on my own about 2-3 years ago, that probably helped with the concept

[halcyon trail]

idk, learn C++? It's really hard to find people; people keep liking to make tweets about it being "deprecated" but back in the real world there's mountains of C++ code still being written.

[grave jolt]

nothx

anyway... Python. internals. and some PEPs for good luck

[halcyon trail]

if you're using type annotations throughout then learning any of th emainstream GC languages is going to be pretty easy

like, Kotlin was basically very simple to learn. The thing that gave me the most trouble probably was covariance/contravariance, simply because I hadn't encountered it much at the time in python

[grave jolt]

I am afraid to use Java because of null pointers 😄

[halcyon trail]

Kotlin is null safe 🙂

[grave jolt]

in Python at least you can force yourself to use | None

[halcyon trail]

In Kotlin you have to put a ? after a type to indicate that it's nullable

String is always a string, no null possible. String? is like String | None

[grave jolt]

yeah I vaguely remember going through some Kotlin tutorials

[halcyon trail]

and there's some very nice syntactic sugar around null handling that I really miss in python

[grave jolt]

I'm in infinite tutorial hell, that's what I meant

actually maybe let's move to #career-advice

[median palm]

rust owo ||/s||

[grave jolt]

I don't really have a use case for it tbh

[halcyon trail]

Well, usually companies set the languages, with occasional opportunities to add new things. So often, wanting to use new language is going to involve moving, either ot a different company or a different role.

[dusk comet]

why not:

pipe(some_list).map(f1).map(f2).map(f3).to_list()

[grave jolt]

then you'd have to hardcode all the possible functions

instead of being able to pipe into your own functions

[halcyon trail]

that's kidn of what I like about extensions over pipeline type operators

you get to put map, etc directly on the object. it's a bit more concise and has much better auto completion/discoverability

[dusk comet]

i can hardcode only common ones, and make one method to apply any function:
pipe(some_iterable).map(f1).apply(itertools.something).map(f2).to_list()
pipe(some_iterable).map(f1)(itertools.something).map(f2).to_list()

so, there is no >> << > < ... operators and IMO it looks cleaner than pipe(...) >> (map, f1) >> something >> (map, f2)

[halcyon trail]

that's a reasonable approach

in Kotlin which has extensions, of course sometimes you might want to apply a free (non-extension) function as part of your left-to-right transformation
So you end up needing pretty similar things to apply anyway

someIterable.map { ... }.filter { ... }.let { someFunc(it, 5) }

[grave jolt]

monkeypatch object to have an apply method

[halcyon trail]

🔥

[unkempt rock]

Not sure where best to ask this but does the download button on https://www.python.org/downloads/ always download the most appropriate python version? I tried on win10 64-bit and linux and it seemed to but I don't have a way to test a 32-bit system.

[surreal sun]

tbh Kotlin is so nice in so many ways, but its error messages can be absurdly long sometimes 😭

python's error messages are clean compared to other languages though

[halcyon trail]

I haven't found it's especially bad in that regard tbh

Pythons are good though. Rusts are good too but the concepts involved are just more involved at times

And rust does do some implicit things with lifetimes to make things a lot more pleasant, but a side effect of that can be make error messages a bit less clear

[surreal sun]

yeah it's not the worst and it can be helpful than other languages sometimes but other times it can be a pain to look at

i love extensions & infix functions in kotlin tho, i feel like you have a lot more freedom in kotlin than in python

[halcyon trail]

yeah, I'm a huge fan of extensions

gives you a convenient way to do left-to-right transformations, and it heavily encourages keeping class API's minimal

[surreal sun]

there are so many places i've wanted extensions in python specifically with making libraries but they weren't there

[halcyon trail]

pretty good two for one

right, and third, improves usability of classes you don't control by providing API's adapted to your use cases

[surreal sun]

yeah especially with making wrappers i imagine; being able to extend an existing class without making users use a completely different API

[halcyon trail]

it's especially nice in the context of Kotlin, as the benefit is extra-significant when you have a very powerful IDE

[surreal sun]

intellij is a really powerful IDE for kotlin tbh, surprised how good it is given its support for other languages like python

pycharm is good but intellij with kotlin is amazing

[halcyon trail]

yeah, it's definitely another level

like, auto completing functions/types from the whole project, not just what's locally imported. then if you pick something that's not yet imported, it imports it for you.

or the helping background hints when you use extensions and scope functions and things like that

[gray galleon]

does python have an explicit code object constructor

[gray galleon]

oh i found it

types.CodeType

[elder blade]

With the influx of recent libraries like attrs and dataclasses, Python should just get real compile-time macros instead of using exec all the time

[gray galleon]

the problem with compile time macros
is that its kinda hard to implement
python syntax is not uniform and minimalist like lisp

you can go the c preprocessor route ig but thats not ideal

[gray galleon]

^

[dusk comet]

C preprocessor idea is terrible

[elder blade]

Python's decorators could just be macros

[elder blade]

I don't think we need them? Python doesn't need build-time arguments like C because behaviour between environments should not need to be very different

[dusk comet]

There is interesting idea in rust about macroses.
Macroses are just functions that takes stream of tokens and transtales it to another stream of tokens.

In python it can be implemented in similar way. My idea (it is not ideal):
First, we should make global namespace for all macroses (similar to sys.modules). We can register some function as macros using some decorator.

Then we should change behaviour of compilation. It should take into account some special syntax for function macroses. If there macros is used, sys.macroses[macrosname] is called with some arguments and result is used intead of original function.

What can we change using macroses?

• source code of function (bad idea, too hard and inconvenient)
• ast
• token stream

What syntax?

• @__macros__.macrosname
• @@macrosname

[gray galleon]

hmm
so you are not referring to the syntactic macros of lisp?
just a small subset of it

[gray galleon]

macroses

[dusk comet]

What you mean?

[gray galleon]

is that a plural of a plural

macros-es

[gray galleon]

changing ast using macros is the best option

[dusk comet]

Hmm, nvm
I thought that "macros" is single thing, and for me it is natural to extend it as "macroses" to get plural form

[quick snow]

!pep 638

[fallen slate]

**PEP 638 - Syntactic Macros**

Link

Status

Draft

Created

24-Sep-2020

Type

Standards Track

[elder blade]

Pretty sure it is "a macro" and "multiple macros"

[radiant garden]

Macra

[quick snow]

if macros is singular, then macroi would be plural I guess

[boreal umbra]

In [1]: from collections import UserList

In [2]: class StelerList(UserList):
   ...:     def __format__(self, fmt):
   ...:         print(fmt)
   ...:         return super().__format__(fmt)
   ...:

In [3]: sl = StelerList([1, 2, 3])

In [4]: f'{sl!s}'
Out[4]: '[1, 2, 3]'

In [5]: f'{sl!r}'
Out[5]: '[1, 2, 3]'

In [6]: f'{sl!x}'
  Input In [6]
    f'{sl!x}'
             ^
SyntaxError: f-string: invalid conversion character: expected 's', 'r', or 'a'

It would seem that the /![a-z]/ part of f-strings isn't handled by __format__, which I suppose makes sense. How is it implemented?

I guess it must involve the parser if it's raising a syntax error.

[feral island]

yes, it's in the parser

if you grep for invalid conversion character in CPython you can find it

[quick snow]

https://github.com/python/cpython/blob/main/Objects/stringlib/unicode_format.h#L770-L775

[fallen slate]

Objects/stringlib/unicode_format.h lines 770 to 775

case 'r':
    return PyObject_Repr(obj);
case 's':
    return PyObject_Str(obj);
case 'a':
    return PyObject_ASCII(obj);```

[umbral sphinx]

guys i just need a small help in pandas

[boreal umbra]

You can ask in #data-science-and-ml, but please remember to never "ask to ask". Always ask a complete question.

[rose schooner]

https://github.com/python/cpython/commit/c7f57087149968e9a72f6dee73514ff18fee78e8 this is like the 3rd big change to the interpreter that i've seen since the start of 3.12

[spice pecan]

I don't follow this too closely, what's the rationale for this? Have they been superceded by the new generalized instructions?

[rose schooner]

i think so?

it's probably to make the specializing interface easier

[spice pecan]

Sounds nice

[elder blade]

What even... do { } while(0)?!

[rose schooner]

yeah
you put that in macros to signify that they're multiline

[elder blade]

Ah, it compiles away but still gives a syntax error if used incorrectly?

[radiant garden]

it's required to make multistatement macros behave correctly in all contexts you want it in

there's a bunch of examples you can find where everything else causes unintentionally broken code

[rose schooner]

wdym by "used incorrectly"?

[elder blade]

Used as an expression for example

[rose schooner]

yeah it can't be used like that

why would you even use multiline macros in an expression anyway

[elder blade]

You wouldn't, but my question was whether this loop was a way to enforce that someone doesn't accidentally do it in an expression

[dusk comet]

It is possible to use it inside expression
Statements in C does return values, and you can ignore it

int x = ((do {printf("hello");}while(0)), 42);
// int x = 42;
``` im not sure if it is correct

[rose schooner]

that's a gcc extension i'm pretty sure

[radiant garden]

here's a couple examples of other things that don't work as well as you'd want

#define first(x) do_thing(x); do_other(x);

#define second(x) { \
    do_thing(x); \
    do_other(x); \
  }

int main(void) {
  if (cond)
    first(1) // bad expansion for attempt 1
  
  if (cond) second(2); else something_else(); 
  // can't use the semicolon because it expands to braces, even though it would work fine if second() was a function
}

[rose schooner]

so it isn't standard and may be invalid when other compilers are used

[rose schooner]

the semicolon is unusable on a brace?

[radiant garden]

they expand to the braces of an if statement

[rose schooner]

seems to work fine anyway

[radiant garden]

sometimes

idr exactly the cases where it doesn't

[dusk comet]

no, i can compile that using both clang and gcc:

#include <stdio.h>
int main() {
    int x = (({do{int y = 43; printf("Hello World!\n");}while(0);}), 42);
}

[rose schooner]

yeah clang uses gcc extensions as well

[dusk comet]

:(

[rose schooner]

tested it in tcc

[dusk comet]

there is no good compilers except gcc and clang

[rose schooner]

nvm it's msvc

[rose schooner]

but tcc works fine

[rose schooner]

also you are required to ignore it

it returns void

[dusk comet]

not always iirc

[rose schooner]

gcc (and also probably clang/tcc by extension) errors if you don't ignore it

it's not supposed to be used as a value

[dusk comet]

yes, youre right

[rose schooner]

but you can use the comma operator on it assuming it's the left operand because that's "ignoring" it

[dusk comet]

no, youre wrong

#include <stdio.h>
int main() {
    int x = ({int y = 42; 2;});
}
``` i can compile it using both gcc and clang

you can change all if-statements into expression:

// before
if (cond) {body1;} else {body2;}
// after
cond && ({body1; 1;}) || ({body2; 1;});

[radiant garden]

This is a GCC extension called Statement Expressions. It's not standard C.

[rose schooner]

oh
yeah i forgot the rule that if the last statement is a non-void expression, it returns it

[rose schooner]

can i compile cpython with gcc?
no because i'm on windows

so msvc is used

and msvc does not support statement expressions

and cpython has to be universal so it can't use statement expressions

[rose schooner]

so that means this statistic shouldn't really matter since we're talking about cpython code

[dusk comet]

why cpython can not be built by gcc on windows?

[rose schooner]

because they find it easier to do so using msvc

cpython should not require any sort of extra software in order to build, test, and run it

[dusk comet]

but it requires MSVC to build it

[rose schooner]

yep

by "extra software" i mean stuff like WSL

because i'm pretty sure the makefile that uses gcc/clang needs POSIX-only tools

also the decision has already been made to use msvc

all the effort writing those files in /PCbuild will be wasted if a switch is made

[quick trellis]

more interestingly

did anyone compile cpython with zig yet?

[median palm]

zig can compile C/C++? I remember seeing something on their home page about it

[gray galleon]

https://github.com/lihaoyi/macropy

[rose schooner]

last update is in python 3.7

[gray galleon]

😢

[boreal umbra]

In [1]: class A:
   ...:     def __init__(self):
   ...:         self.__x = 'hi'
   ...:     def foo(self):
   ...:         print(self.__x)
   ...:

In [2]: a = A()

In [3]: a.foo()
hi

In [4]: a.__x
AttributeError: 'A' object has no attribute '__x'

I assume there's descriptor magic at play here? How does this work?

Or is it special parsing where the _A is secretly prepended to the attribute?

I'm thinking it's that. Consider this:

In [9]: def new_func(self):
   ...:     print(self.__x * 2)
   ...:

In [10]: A.bar = new_func

In [11]: a.bar()
AttributeError: 'A' object has no attribute '__x'

[dapper lily]

AFAIK the name mangling happens at class creation time
that's why new_func isn't working in your example

[boreal umbra]

sure, but how is the name actually mangled?

The plot thickens:

In [16]: class A:
    ...:     def __init__(self):
    ...:         self.__x = 'hi'
    ...:     def foo(self):
    ...:         print(self.__x)
    ...:     def __setattr__(self, attr, value):
    ...:         print(f'setattr called {attr = }')
    ...:         super().__setattr__(attr, value)
    ...:

In [17]: a = A()
setattr called attr = '_A__x'

[dapper lily]

https://github.com/python/cpython/blob/main/Python/compile.c#L523

[fallen slate]

Python/compile.c line 523

_Py_Mangle(PyObject *privateobj, PyObject *ident)```

[sturdy timber]

fun name mangling bug https://github.com/python/cpython/issues/96497

[swift imp]

I don't understand at all, how is that possible

I thought name mangling was just a thing when you call help or dir on an object

But if u directly accessed __dict__ it would pop up

[rancid tusk]

Any reason why python cant interpret a variables value in f string specifiers, it always uses the variables name over its value

!e

i = 1
print(f"{1:i}")

[fallen slate]

@rancid tusk :x: Your 3.11 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 2, in <module>
003 | ValueError: Unknown format code 'i' for object of type 'int'

[boreal umbra]

the i is passed as the string 'i' to int.__format__. what was the expected behavior?

[rancid tusk]

I see, just the actual variables value being passed :p

[peak spoke]

you can do {i} to use the variable's value for the format specifier

[rancid tusk]

ah tysm!

[boreal umbra]

how did you find that out?

[dusk comet]

isnt it documented?

https://docs.python.org/3.12/reference/lexical_analysis.html#f-strings

Top-level format specifiers may include nested replacement fields. These nested fields may include their own conversion fields and format specifiers, but may not include more deeply nested replacement fields. The format specifier mini-language is the same as that used by the str.format() method.

[rose schooner]

it's also very weird because they're formatted and then parsed so ```py

f"{2:0<{1}{0}}"
'2000000000'

equivalent to

f"{2:0<10}"
'2000000000'

[rich cradle]

isn't that last part now false? given the f"{expr = }" syntax?

[rose schooner]

what last part?

[rich cradle]

The format specifier mini-language is the same as that used by the str.format() method.

[rose schooner]

oh that

[rose schooner]

it's true

{expr=} isn't part of the fspec mini-language

https://docs.python.org/3.12/library/string.html#formatspec

[rich cradle]

then what's that a part of? is it just its own thing?

[rose schooner]

a more general "replacement field" in f-strings

The grammar for a replacement field is as follows: ...

[rich cradle]

oh i see. it's special cased.

[rose schooner]

wait

i think that's the wrong part of the docs

it actually should be https://docs.python.org/3.12/reference/lexical_analysis.html#formatted-string-literals

different from a "replacement field"

[rich cradle]

oh yeah that's what i was looking at

[rose schooner]

1. offtopic
2. why does it have to pass through a shortlink when you can literally search for it on youtube

[neat delta]

<@&831776746206265384> malicious spam by a superstar in all 4 discussion channels - #career-advice #internals-and-peps #pedagogy and #community-meta

[neat delta]

in all likelihood, it's a ip grabber shortener

[sharp plover]

Dawn got em

[proper cove]

URL redirection most likely. Can be used by hackers to redirect you to a malicious site doing this they can use XSS to run JS in your browser. You must watch out for them

[proper cove]

Doubt it, why would they go through the trouble of sending a rd link to a random discord server just to grab some random peoples IP address?

@rose schooner check your browser search history and see where you got redirected before it took you to yt

[rose schooner]

i didn't click it

[proper cove]

ah then why where you saying it redirected u?

[rose schooner]

huh?
i didn't say that

[proper cove]

Didn't u say the link sent you to another short link?

I can't see the msg because it was deleted but I assume you were replying to something in the chat

unless you were generally speaking

[rose schooner]

i didn't say that

[proper cove]

Ok.

[vital frost]

is my code neat and readable?

[dapper lily]

it's not bad per se, but you might wanna try formatting the code using a formatter like black

[vital frost]

i dont understand, would u explain it more?

[dapper lily]

Python has a style guide called PEP 8 that your code should (not must) adhere to
formatters like black, autopep8, yapf, etc. modify the styling of your code to adhere to that
P.S. this might be better asked in #python-discussion or a help channel

additionally, linters like flake8, pylint, etc. might give you warnings about how your code isn't adhering to PEP 8 (in addition to other lints)

[vital frost]

thank you sir

[vital frost]

what thing i should install to make my code neat, readable, and professional? @dapper lily

i code in vs code btw

[boreal umbra]

Questions about code cleanliness belong in #software-architecture (cc @dapper lily)

Though you shouldn't expect a code review for a screenshot of text. No one can make suggestions without retyping what is in the screenshot, which is an unreasonable request.

[gray galleon]

one thing: Size should be size
capitalized names are only for classes

[boreal umbra]

Remember not to continue discussions that have been redirected to other channels. You could ping them in the appropriate channel and resume the conversation.

[indigo glacier]

BTW, what application are you using to show your code? that looks really interesting

[crude shoal]

u re right it looks quite good

[pale folio]

Not sure if this belongs here, or if I should reach out to the NumPY people. Are there plans for taking an intNumber (1234567890) and converting it to a np.array directly? Currently, to the best of my knowledge, a number has to be in a string format to iterate over the values when they are placed in the array.

>>> strNum1
'1234567890'
>>> array1 = np.array([*strNum1], dtype=int)
>>> array1
array([1, 2, 3, 4, 5, 6, 7, 8, 9, 0])
>>> print(type(array1[0]))
<class 'numpy.int64'>

[feral cedar]

that seems like a pretty rare use case tbh. maybe you can make an issue on their github, but i wouldn't expect anything to come out of it

[pale folio]

Yeah I'm not sure as I'm just starting out with NumPY. Ill wait until I learn more about data structures and usage within np to possibly raise the issue. Thank you for your quick response!

[swift imp]

I'm also seems incredibly inefficient

That's gonna implicitly call int on every single character right?

[quick snow]

What the digits of a number are depend on its base, it's not clear why np.array(1234567890) should use base 10 necessarily, even if it would create an array of digits.
But it returning a 0-dimensional array makes much more sense, so the closest thing I could reasonably see is an alternative constructor like np.array.from_digits(1234567890, base=10).

!e You could do:

import numpy as np
array = np.fromstring(str(1234567890), dtype=np.byte) - 48
print(array)

[fallen slate]

@quick snow :white_check_mark: Your 3.11 eval job has completed with return code 0.

001 | <string>:2: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead
002 | [1 2 3 4 5 6 7 8 9 0]

[quick snow]

(or, I guess, np.frombuffer(str(1234567890).encode()), if you want to get rid of the DeprecationWarning)

[uncut sierra]

py```
sss = f'x:{f"{x:05}":>8} y:{f"{y:05}":>8}'
print(sss)

continuing on a old suject I found no way to go deeper than 2 level of formatting, am I wrong? 🙂

[neat delta]

4 is the limit:

print(f'''1 {f"""2 {f"3 {f'{4}'}"}"""}''')

When f-strings get added to the formal grammar, infinite levels should be possible then, but the date of such an event is unknown

https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit-f.html

[feral island]

fairly likely to happen in 3.12

[uncut sierra]

I thought to use ''' (3 quotes) but it is ugly 😛

[feral island]

https://github.com/we-like-parsers/cpython/tree/fstring-grammar

[uncut sierra]

thanks for the link, yes with a real parser we can go deeper and be more clean.

[rose schooner]

of course it's infinitely nestable when it's an expression

but the format specifier (the part after the colon) cannot go deeper than 2 levels

[grave jolt]

Why would you nest f-strings, especially more than once?

That sounds like a nightmare to read

[radiant garden]

same vibe as allowing expressions in decorators, because it's an arbitrary limitation of the parser

[celest plank]

Yes very easy on the eyes.

[celest plank]

Hey, also, you might want to make use of .lower() to eliminate case sensitivity.

[eternal geyser]

VsCode polacode ext.

[quick trellis]

is there a way in python interactive shell to get the actual current_history_length?

bc readline.get_current_history_length() returns the length of the history file

which is larger than the actual session length

[minor hill]

How did you make this window around it? looks cool asf

[vital frost]

use a vs code extension call code snap

that extension allows you to screenshot your code like mine do. its better than manual screenshot using paint

[proven bramble]

I had a question about upcoming python versions (3.12 and 3.13)
I have read that python 3.12+ is gonna be jitted, will the c embedding and extending stay the same ?
or will the api change a bit

[rose schooner]

i'm pretty sure changing the interpreter would not change any major public interfaces

[grave jolt]

It would probably break all C extensions

Well, depending on the changes

[feral island]

nope, not necessarily. The interpreter is surprisingly self-contained

[halcyon trail]

then also you need to distinguish when you say "break"

ABI break C extensions vs API break extensions

[rose schooner]

the (bytecode) interpreter (AKA the code runner itself) relies on the C API, not the other way around

[gaunt sage]

hmm intersting]

fien

[proven bramble]

Okok thanks for the answer

Looking forward to the next update!!

[south hearth]

did python 3.11 just broke python3.8 ? cant seem to source compile :/

getting this on windows :

W:\Development\Projects\Personal\TimeWaste\cpython-3.8\cpython-3.8\PCbuild\_freeze_importlib.vcxproj(131,5): error MSB3
073: The command ""W:\Development\Projects\Personal\TimeWaste\cpython-3.8\cpython-3.8\PCbuild\amd64\_freeze_importlib.e
xe" "importlib._bootstrap" "W:\Development\Projects\Personal\TimeWaste\cpython-3.8\cpython-3.8\Lib\importlib\_bootstrap
.py" "W:\Development\Projects\Personal\TimeWaste\cpython-3.8\cpython-3.8\PCbuild\obj\38amd64_Release\_freeze_importlib\
importlib.g.h"" exited with code -1073741511.

ok seems pretty weird

x86 builds were successful

only x64 builds failed

[rose schooner]

DLL load error?

what platform are you on

[south hearth]

windows x64

[rose schooner]

did you modify it beforehand

[south hearth]

nope, freshly cloned and builinding

with 3.8 branch

[rose schooner]

did you try doing it again

[south hearth]

yep

[rose schooner]

how about build x64 debug

[south hearth]

build.bat -p x64

and before that i ran get-externals.bat

to get the external libs

like libffi openssl ad stuffs

[rose schooner]

does cpython 3.8 not have the -e option in build.bat

[south hearth]

it does tho but that didnt help much

cuz i like making em seperately

@rose schooner

[rose schooner]

yeah idk how to solve that

[south hearth]

can ya help me with this?
which PyLong_FromLong is used in here? : https://github.com/python/cpython/blob/82ca2839c9ec6bf9a9400e791a52411824df67f3/Python/ceval.c#L2181
the one from longobject.c : https://github.com/python/cpython/blob/82ca2839c9ec6bf9a9400e791a52411824df67f3/Objects/longobject.c#L315
or the one from coverity_model? : https://github.com/python/cpython/blob/82ca2839c9ec6bf9a9400e791a52411824df67f3/Misc/coverity_model.c#L49

[fallen slate]

Python/ceval.c line 2181

PyObject *ret = PyLong_FromLong(INSTR_OFFSET());```
`Objects/longobject.c` line 315
```c
PyObject *```
`Misc/coverity_model.c` line 49
```c
PyObject *PyLong_FromLong(long ival)```

[south hearth]

i fixed that issue btw

[raven ridge]

The one from longobject.c

[south hearth]

what is the one in coverity_model for then?

[raven ridge]

Coverity is a C linter, so my guess is that it's somehow related to some sort of static analysis.

[south hearth]

oh aight thanks

[raven ridge]

https://github.com/python/cpython/blob/82ca2839c9ec6bf9a9400e791a52411824df67f3/Misc/coverity_model.c#L3-L16

[fallen slate]

Misc/coverity_model.c lines 3 to 16

* This is a modeling file for Coverity Scan. Modeling helps to avoid false
* positives.
*
* - A model file can't import any header files.
* - Therefore only some built-in primitives like int, char and void are
*   available but not wchar_t, NULL etc.
* - Modeling doesn't need full structs and typedefs. Rudimentary structs
*   and similar types are sufficient.
* - An uninitialized local pointer is not an error. It signifies that the
*   variable could be either NULL or have some data.
*
* Coverity Scan doesn't pick up modifications automatically. The model file
* must be uploaded by an admin in the analysis settings of
* http://scan.coverity.com/projects/200```

[dusk comet]

>>> dis('f"a"')
  0           0 RESUME                   0

  1           2 LOAD_CONST               0 ('a')
              4 RETURN_VALUE
>>> dis('f""')
  0           0 RESUME                   0

  1           2 BUILD_STRING             0
              4 RETURN_VALUE
>>> dis('"a"')
  0           0 RESUME                   0

  1           2 LOAD_CONST               0 ('a')
              4 RETURN_VALUE

weird, why f'' and f'a' are compiling to different instructions?
i expect them both to compile to LOAD_CONST, because they have no substitution fields

3.11

[grave jolt]

huh, interesting

[dusk comet]

https://github.com/python/cpython/blob/main/Python/compile.c#L4941-L4944

[fallen slate]

Python/compile.c lines 4941 to 4944

VISIT_SEQ(c, expr, e->v.JoinedStr.values);
if (asdl_seq_LEN(e->v.JoinedStr.values) != 1) {
    ADDOP_I(c, loc, BUILD_STRING, asdl_seq_LEN(e->v.JoinedStr.values));
}```

[paper echo]

!e ```python
import dis
dis.dis('""')

[fallen slate]

@paper echo :white_check_mark: Your 3.11 eval job has completed with return code 0.

001 |   0           0 RESUME                   0
002 | 
003 |   1           2 LOAD_CONST               0 ('')
004 |               4 RETURN_VALUE

[paper echo]

@dusk comet it's just a special case for the empty string

[radiant garden]

it's a lack of special case for the empty string

[paper echo]

yeah wait

hang on, BUILD_STRING vs LOAD_CONST ('')

thats odd

[radiant garden]

.

!= 1

[rose schooner]

might make a PR on this one

[paper echo]

hah did they mean to write > 1 instead of != 1?

[rose schooner]

that would make f"" not work
or idk really

[rose schooner]

my last few cpython PRs to optimize are being questioned of their necessity

[feral island]

this one also feels pretty marginal to be honest, seems like it's about optimizing f""?

[rich cradle]

i'm kinda confused on why this can't be optimized

[rose schooner]

this one makes a little more sense though

[rich cradle]

if something is a trivial optimization, why not do it?

it's not like it increases code complexity in any meaningful sense

[rose schooner]

b'', r'', f'stuff' all produce LOAD_CONST

[feral island]

if it's more code, it means more code to maintain

[rose schooner]

f'' is the only odd one out of them all

[feral island]

and why would anyone write f"", so what's the point in optimizing it?

[rich cradle]

if the machinery exists for f'stuff', i'm sure it's trivial to add f''. likely a single line.

and so not doing it seems strange, honestly

although, take that with a grain of salt, given that i don't know how it's implemented

and i'm used to writing compilers that optimize code to the point where it doesn't even resemble the source, so my experience with this is different

[rose schooner]

the PR results in 9 extra characters 1 extra character excluding a comment line

is that too complex?

[feral island]

Might be fine but I'm not the one making decisions on the compiler 🙂

[grave jolt]

I don't think it's worth changing tbh

why would you write f""?

in fact I'd be fine if f-strings with no placeholders were banned altogether

[rich cradle]

a warning would be nice, but banning them sounds extreme

forgetting to remove it makes sense

[feral island]

we internally have a lint rule against f-strings without placeholders

[radiant garden]

auto-generated code

I don't see a reason not to fix this

and bring *checks clipboard* 2ns of speedup

[grave jolt]

hmmm yeah I guess

although here it's an issue of a small speedup

[radiant garden]

an "issue" it would be if it took more effort to code review than it's worth (it doesn't, it's a trivial change) 😄

[sturdy timber]

This is the least illegal use of an empty f-string I can think of

a = '\n'.join((
    f"Something something some {thing}",
    f"Thing some some thingthing some",
    f"",
    f"{some_thing} something some thing.",
))

Gotta keep it lined up :P

[paper echo]

same with implicit string concatenation across lines

[flat gazelle]

I mean, f'' is just a noop there, no?

[native flame]

it makes an empty line

[flat gazelle]

with implicit concat, it isn't

[peak spoke]

it's not concat

[paper echo]

true

[native flame]

theres commas

[sturdy timber]

Someone should make one of those lawful good - chaotic evil grids with all the ways of writing strings over multiple lines.

Chaotic evil would be python def foo(): my_multiline_string = """something something something Something something something Something something something"""

Chaotic good is that but with strip and textwrap.dedent

[grave jolt]

!e

print(fr"{fr'ong'}")

[fallen slate]

@grave jolt :white_check_mark: Your 3.11 eval job has completed with return code 0.

ong

[dusk comet]

!e fbr''

[fallen slate]

@dusk comet :x: Your 3.11 eval job has completed with return code 1.

001 |   File "<string>", line 1
002 |     fbr''
003 |        ^^
004 | SyntaxError: invalid syntax

[quick snow]

no byte-fstrings, sadly

[dusk comet]

ufbr''

[quick snow]

yeah, would have been great for buffers ^ :D

[dusk comet]

bufr''

[quick snow]

Unicode byte-string. However you use it, it will always throw a UnicodeEncode/DecodeError

Or maybe it cancels out and is the default string type of the target Python version (unicode in 3+, bytes in 2-)

[dusk comet]

x = 'hello'
ptr_str = id(str)
ptr_bytes = id(bytes)
ptr_average = (ptr_str + ptr_bytes) // 2
x.__class__ = ctypes.py_object.from_addr(ptr_average)
print(x) # boom!

[elder blade]

Yeah you have to use % 😔

[quick snow]

Thankfully % is fast now, right?

https://github.com/python/cpython/issues/72494

[paper echo]

there's no bytes.format?

!e print( (b'a{}c').format(b'z') )

[fallen slate]

@paper echo :x: Your 3.11 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 1, in <module>
003 | AttributeError: 'bytes' object has no attribute 'format'

[paper echo]

huh

i just kind of assumed that existed

[flat gazelle]

well, the __format__ dunder doesn't exist in a byte varianat

so it can't really exist

[raven ridge]

if the thing you're trying to format is text, you can always use f"...".encode() - but yeah, if it's not text, __format__ really doesn't apply.

[rose schooner]

is there a reason why not

because % works

[raven ridge]

because it would need a whole new dunder, and because it's so limited in utility.

I'm betting % wouldn't work for bytes, either, if it weren't for the fact that it predates Unicode strings entirely.

[rose schooner]

i just discovered something ```py

b'%s' % '\u200b'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: %b requires a bytes-like object, or an object that implements bytes, not 'str'

[raven ridge]

huh, that seems like an implementation detail being leaked in the error message.

[flat gazelle]

what do you mean?

[raven ridge]

"TypeError: %b requires" - there was no "%b" in the code.

[flat gazelle]

ah, fair enough

interestingly, graalpy has the same error, but pypy just has TypeError: requires a bytes-like object, or an object that implements __bytes__, not 'str'

[raven ridge]

raising an error is correct, just the existence of "%b" in the message strikes me as weird.

[flat gazelle]

agreed

[paper echo]

i suppose, but it seems pretty useful for formatting binary protocol messages. although you'd maybe use struct or bytearray for that, but even text-ish protocols like irc could benefit

[raven ridge]

if it's text-ish, then you could always use f"...".encode() for the text parts, and use struct or ctypes.Structure for the not-text parts

[paper echo]

is this kind of thing not common when working with binary data?

magic_number = (1337).to_bytes(16, byteorder='little')
data_type = b'\xf3'
data = b'zxcvasdfqwer'
data_size = len(data).to_bytes(16, byteorder='little')
message = magic_number + data_type + data_size + data

[raven ridge]

sure

[paper echo]

i was thinking it'd be useful to write stuff like that with f-strings or .format

[raven ridge]

I'm not convinced it would be...

I think ctypes.LittleEndianStructure would be much more convenient

[paper echo]

i see, i don't have to work with binary data at all so i don't know what good tools exist for it

[raven ridge]

!e The ctypes.Structure approach would look something like: ```py
import ctypes

class DataHeader(ctypes.LittleEndianStructure):
pack = 1
fields = [
("magic_number", ctypes.c_int16),
("data_type", ctypes.c_int8),
("data_size", ctypes.c_int16),
]

data = b"zxcvasdfqwer"
header = DataHeader(1337, 0xF3, len(data))
message = bytes(header) + data
print(message)

[fallen slate]

@raven ridge :white_check_mark: Your 3.11 eval job has completed with return code 0.

b'9\x05\xf3\x0c\x00zxcvasdfqwer'

[raven ridge]

which is more verbose, but lets you define the layout in one place and reuse it in multiple places.

which is a lifesaver if the format ever changes.

you can also pass the fields as keyword arguments to the Structure constructor: py header = DataHeader(magic_number=1337, data_type=0xF3, data_size=len(data)) or you can call the constructor with no arguments and then fill in the fields by name: ```py
header = DataHeader()
header.magic_number=1337
header.data_type=0xF3
header.data_size=len(data)

[paper echo]

i see, that's essentially the same job as the hypothetical bytes formatting, but also with some type / field width checking at runtime

im glad "sunder" style never caught on. it's so ugly

we are stuck with it in enum and apparently here

or maybe people like it and im the odd one out

[raven ridge]

type checking, width checking, and encapsulating the knowledge of the order of the fields and the amount of space between them. And decoupling the format from the specific places that need to send or receive messages

with the added benefit of being sorta similar to how a C struct is defined, which is helpful if you're interacting with this binary data in multiple languages - often if someone else designed the format, the only reference you might have for it is a C struct, so being able to easily transliterate that into Python is nice.

[radiant garden]

you'd probably just write directly to a buffer

as opposed to a lot of bstring manip

[dusk comet]

magic_number = (1337).to_bytes(16, byteorder='little')
data_type = b'\xf3'
data = b'zxcvasdfqwer'
data_size = len(data).to_bytes(16, byteorder='little')

buf = bytearray()
buf.extend(magic_number)
buf.extend(data_type)
buf.extend(data_size)
buf.extend(data)

message = bytes(buf)

[elder blade]

I use % for binary formatting. Would love to use f-strings
https://github.com/wumpyproject/wumpy-redis/blob/main/wumpy/redis/impl/resp.py#L21-L31

[fallen slate]

wumpy/redis/impl/resp.py lines 21 to 31

for obj in args:
    if obj is None:
        buffer += b'$-1\r\n'
    elif isinstance(obj, str):
        buffer += b'$%d\r\n%s\r\n' % (len(obj), obj.encode('utf-8'))
    elif isinstance(obj, bytes):
        buffer += b'$%d\r\n%s\r\n' % (len(obj), obj)
    elif isinstance(obj, (int, bool)):
        buffer += b':%d\r\n' % obj
    else:
        serialize_data(*obj, buffer=buffer)```

[dusk comet]

Is it possible to insert number into bytestring as byte value?
Something like this:
b'%x' % n -> bytes([n]), not b'123'

[dusk comet]

where would str.maketrans and str.translate be useful?

[flat gazelle]

%c

[flat gazelle]

Transliteration does show up occasionally, for example replacing / with _ in a path, or swapping some characters around.

[dusk comet]

thanks, didnt know that:

>>> b'%c' % 5
b'\x05'

[dusk comet]

s.replace('/', '_')
unused = 'some char that cant be in your string, for example \x00'; s.replace('a', unused).replace('b', 'a').replace(unused, 'b') (yes, it creates 3 new strings instead of 1 and it doesnt work in any case)

[flat gazelle]

Yeah, you can do that, but translate is easier and more readable and doesn't require a placeholder char

If you look through shell scripts, you will see tr every once in a while, which is pretty much the shell equivalent of translate

[dusk comet]

multiple argument in str.replace would be very useful:
s.replace(a, b).replace(c, d) == s.replace(a, b, c, d)
it would not create a lot of temporary strings and it can swap two substrings (if it is implemented like this)

[paper echo]

is this not equivalent to str.translate?

[dusk comet]

.replace also can replace strings, not only chars

[paper echo]

true

[halcyon trail]

FWIW skimming a few other languages, a multi-replace function seems pretty rare

usually there's a basic literal, single replacement function
and then something more advanced in regexp

I'm speculating here but one reason for it may be that there are various semantics that you'd have to agree on that aren't completely obvious

does it do the replacement as multiple passes, as a single pass, how does prioritization work, does it allow subsequent replacements to run on what you already substituted, etc

so, they just provide a very simple function for quick usage that dodges all these questions, and then you have regex, which a) not only handles this use case but obviously much more advanced ones, b) has some kind of quite well defined semantics

[stone field]

re.sub can let you do this very easily already

[vast saffron]

str.translate needs a _TranslateTable which has to be created via str.maketrans first, so not equivalent

[rose schooner]

not very fast and you still have to understand re.Match but yes

[vast saffron]

It is actually slower than just doing multiple chained str.replace

[paper echo]

i imagine that depends on the length of the string and the number of replacements

In [4]: pattern = re.compile(r'abcd')

In [5]: s = 'awnofuhpfhjapefuhsldfubnzsvuna;owefja;pwrughyarughap;weofujhawuhga[orgijapoerighuapiurhgfoaiwyeurfnpw47gnpweb7nv-w49b7unq[349rbvuhnq[3orughjxld;riytgx.'

In [6]: %timeit pattern.sub('x', s)
184 ns ± 0.348 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [7]: %timeit s.replace('a','x').replace('b','x').replace('c','x').replace('d','x')
239 ns ± 0.368 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

i'd be surprised if the regex engine in general was slower than chained replacements. however there might be greater one-time overhead in compiling the regex

of course a more general regex pattern might be slower because of backtracking

oh i messed up the regex... hang on lol

In [9]: pattern = re.compile(r'[abcd]')

In [10]: %timeit pattern.sub('x', s)
2.18 µs ± 17 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

ok, that is surprisingly a lot slower

In [13]: def replace_all(s, r):
    ...:     for c in 'abcd':
    ...:         s = s.replace(c, r)
    ...:     return s
    ...:

In [14]: %timeit replace_all(s, 'x')
332 ns ± 0.723 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

even with the looping and function call overhead its still a lot faster

and of course with a single multi-character .replace it's faster still

is cpython doing in-place replacements or something? i'm more and more finding operations that i expected to be slow turn out to be really fast in cpython

[dusk comet]

>>> x = 'some strange % & * ( ! not interned string '; id(x)
2_588_779_761_552
>>> x = x.replace('a', 'x'); id(x)
2_588_779_010_960
``` i dont think so

>>> x = 'some strange % & * ( ! not interned string ' + str(1231) + '<'; id(x)
2_588_778_912_256
>>> sys.getrefcount((x, globals().__delitem__('x'))[0])
1
>>> x = 'some strange % & * ( ! not interned string ' + str(1231) + '<'; id(x)
2_588_778_916_288
>>> id((x, globals().__delitem__('x'))[0].replace('a', 'x'))
2_588_779_363_808
``` cant reproduce it, `str.replace` always return new object, even if refcount = 1

[vast saffron]

It becomes even slower if you want to replace by a dict of strings

On mobile so writing it out is a pain, but think of it like you want to replace each key with the value.

You would have to OR (|) ,which has bad performance because of backtracking, each key in the regex and probably pass a function into the re.sub for the replacement.

While doing the str.replace you just loop theough the dict and it is still faster.

[paper echo]

does [] get desugared into |? i expected [abcd] to also be a straight-across loop for the regex engine

[radiant garden]

python's builtin regex engine should do linear-time substitutions — as does a finite chain of replace statements — so you'd expect the one with less overhead to perform better 😄

I imagine the extra flexibility of regex hurts you here

[paper echo]

i guess i'm surprised that a single call out to the regex engine is slower than several python method calls

[halcyon trail]

I mean heap allocations and memcpy really aren't that slow

[paper echo]

fair

[halcyon trail]

It depends how long.thr string is and how big your dict is

[paper echo]

i wonder if there would be different performance on a much longer string, like an entire text document

or if you had 100 replacements instead of 4

[halcyon trail]

Obvious repeatedly chaining replace will scale linearly with both of those

But like I said at the start there's actually a huge pile of unclear questions about this kind of dictionary of substitutions

I think that's the main concern, not performance

[radiant garden]

indeed the choice of default semantics raises questions

[halcyon trail]

I reviewed a bunch of other languages and none of them have a subsitutition function like this in the standard library

so I tend to suspect that this is the reason why

[vast saffron]

I implemented the regex replace differently (in a class for stupid reasons) so take that with a grain of salt.

But what I found was that both the normal replace and the regex replace just scale with the string length and the result is the same.

Replace still faster.

Its been a while but ai think I tested it with a str with up to 100.000 chars or even 500.000.
But It was some time ago when I tested it

[flat gazelle]

raku has this feature, and it is indeed fairly complex in behaviour.

> "abcd".trans("ab"=>"cd", "cd"=>"ab")
cdab

[vast saffron]

Would be interesting what the performence difference is when you would do a replace by dict but by looping through the dict and re.sub each pair.

I mean just for fun, as i recon it would be a hell of a lot slower.

[indigo fiber]

@vast saffron it would not yield the same result.

After the first iteration you would have "cdcd".

Then the next iteration would morph it into "abab"

The trans function must build a new string, where each character is built by looking up it's mapping from the original.

[paper echo]

i keep wanting to try raku

[dusk comet]

I think it is doable in 🐍 with 🐟🪝

[paper echo]

you could also just write a function 😆

wouldn't be particularly fast though. does cython have decent support for in-place string operations?

[halcyon trail]

i swear people talk more about performance here than in most C++ slacks/discords I'm in 😂

[gray galleon]

is there a pep for block scoping

[dusk comet]

What's wrong with scoping?

[flat gazelle]

I strongly doubt thats really something that would fit in a PEP, it would be a massive, complex, breaking, change

[gray galleon]

it doesn’t have block scoping

[rose schooner]

"block scoping"?

like adding an extra ```py
if 1:
...

[quick snow]

More like:

x = 23
if True:
    x = 42
print(x)  # prints^Wwould print 23

Not gonna happen

[dusk comet]

i think it will break almost all code:

for i in a:
    if ...:
        x = i
        break
print(x) # NameError
``` ```py
if c:
    x = a
else:
    x = b
print(x) # NameError
``` ```py
x = 1
if a:
    x = x + 1 # `x` is var from current or outer scope?
if b:
    x += 1 # `x` is var from current or outer scope?
print(x) # 1 or something else?
``` ```py
def f(x: list | None):
    if x is None:
        x = [] # current or outer scope?
    print(x) # can it be None?

[rose schooner]

ok i see what you mean now

yeah just put it in a function and call that function

[gray galleon]

keep x = ... function scope and make let x = ... block scope

[dusk comet]

now you can change variable scope in three ways: global x, nonlocal x, x: ...
i dont think adding another way to affect var scope is good idea

[flat gazelle]

honestly, python really doesn't need more weird frame object semantics.

[gray galleon]

perhaps we learn from scheme and use let blocks?

let x = 2, y = 3:
  ...

[rose schooner]

what's "block scope"
does it behave like a function/class one or like any other compound statement scope

[halcyon trail]

Block scope just means that each block has its own scope

If you declare a variable inside an if block for example, you can't use it outside that if block

It's scoped to the if block

Most statically typed languages work like this

[rose schooner]

that sounds exactly like just doing ```py
@lambda f: f()
def _(x=2, y=3):
...

why not just do that

[halcyon trail]

Why not create a lambda and call it each time to get scope?

Because it's ugly as sin I would imagine

The benefits don't really justify the boilerplate and obscurity

[rose schooner]

just assign it to a variable

[halcyon trail]

I'm not sure what you mean

[rose schooner]

call = lambda f: f()
...
@call
def _(x=2, y=3):
    ...

[halcyon trail]

I don't understand what this is supposed to do

[rose schooner]

call the function immediately

[halcyon trail]

Okay, so it's exactly what I already said

And the answer I gave for "why not" still applies

[rose schooner]

wdym by this though

[halcyon trail]

Fwiw I don't think pythons scope is ideal but it's not the end of the world either

You're creating a local function and calling it

So that you can have scope inside that local function

[rose schooner]

yep

i'm pretty sure there was a discussion in this channel before of this same exact thing?

@gray galleon

[halcyon trail]

yeah, probably, since it's the only way to get a new scope inside a function

that's the idea I was responding to. block scope is a good thing, but it's not worth trying to "hack" in with that @call def_ ... thing

[rose schooner]

rip user named call

[halcyon trail]

hah

dammit

[gray galleon]

yes, like that
just that the function is not bound to a variable

[rose schooner]

it should only be binding the result of the function

[rose schooner]

but then again why have that

[halcyon trail]

I don't agree with godly's take either fwiw, I think it's a pretty odd take. Kind of pretending that block scope hasn't been the overwhelming norm.

[rose schooner]

python is not C or a statically typed language

[halcyon trail]

block scope is nice because it's basically just an extension of the principle of least privilege, or similar ideas

you want variables to exist only for as longas you need them

In C++ I've seen functions where people create scopes with no if block or anything, just create a scope to limit how long variables are alive for

[rose schooner]

just del them

[halcyon trail]

not to reuse their names, but just because it makes it extremely clear that you aren't using it again later in the function

del'ing them is also really gross

[gray galleon]

because js has that and scheme/lisp has that

[rose schooner]

are they python? no

this idea will never get past the python-ideas mailing list
they're always gonna suggest things like the decorator+def or the del
or suggest that it shouldn't be added at all

[halcyon trail]

Like I said, the benefits of block scope are nice, but python is what it is. abusing local functions or del to try to get some of those benefits is pointless

No, they're probably not going to suggest either of those things

they're just going to say that if you want to limit the scope of some variables, because they are involved in specific work

then you should just define a new function

[rose schooner]

decorator+def minus the decorator

[halcyon trail]

plus a meaningful name, plus being intuitive to read rather than inventing an idiom that nobody uses in python

[rose schooner]

essentially the same thing

you get a new function scope in the end

[halcyon trail]

so, not at all the same thing; if you doubt it, try submitting both versions in your next code review 😉

FWIW this idiom is used in other languages, so I don't think it's intrinsically a bad idea

C++ and Kotlin both use it

just doesn't fit python

[rose schooner]

literally the latest python-ideas mailing list post gets an alternative as the first comment

[halcyon trail]

?

[rose schooner]

there's more stuff down there cut from the image

[gray galleon]

same situation as multiline lambdas
python doesn’t have it, so use named functions instead

[halcyon trail]

okay

@gray galleon yep, pretty much

[gray galleon]

despite the fact that you use it only one time 😳

[rose schooner]

one phrase in the stuff down there which is mentioned again and again every time a new idea is suggested (and probably what you're trying to say right now to me) is cognitive burden

[halcyon trail]

i mean I don't like the situation that python is in wrt lambdas and scope

but that doesn't mean that given python as a whole it's worth changing

i think python pretty much has to live with it

[halcyon trail]

Was this to me?

[rose schooner]

yes

[halcyon trail]

I mean, I'm not trying to say anything :-). I'm saying it.

In every language it makes sense to follow the idioms and write code that will be easy for others to read.

[rose schooner]

probably everything you said about decorator+def can be summed up to "it's a cognitive burden"

[halcyon trail]

Probably not; i don't think two words sums up a few sentences of explanation

[rose schooner]

ok well it does seem equivalent to me

[halcyon trail]

ok

everything's a cognitive burden. things that are idiomatic are a much smaller cognitive burden than things that aren't. Things that have big benefits are more worth taking on cognitive burdens for. etc.

in the end it boils down to (roughly) quantifying these costs and benefits and making an engineering choice, about how it makes sense to change a language, or write code in a language

[gray galleon]

js introduced let and no one complained about “cognitive burdens”
infact everyone just uses it and forgets about var and function scope

[rose schooner]

i was talking about that alternative i suggested

[gray galleon]

ok

[rose schooner]

let isn't too much of a "cognitive burden" by itself but it seems quite useless that a program does just fine without it

saying "because another language has it" doesn't really make a sensible argument

[gray galleon]

now i think let is mainly there to solve js quirks with var (like hoisting and giving undefined when uninitialized)
and the block scope just comes across as a nice bonus

[gray galleon]

this tbh

[halcyon trail]

Idk much about js but that seems plausible

[quick bison]

need basics in python

for me

[grave jolt]

well, functions are often used in one place only

like some dusty private method

[boreal umbra]

Any reason why the loop variable for comprehensions has its own scope, but not for for loops?

In [19]: [t * 2 for t in range(5)]
Out[19]: [0, 2, 4, 6, 8]

In [20]: t
NameError: name 't' is not defined

In [21]: for t in range(5):
    ...:     t * 2
    ...:

In [22]: t
Out[22]: 4

[radiant garden]

probably legacy baggage

it's too big to break always

[halcyon trail]

sometimes people use the loop variable after the for loop, maybe that was an argument?

if you break from the for loop early, then the loop variable does contain some potentially useful information

it's not the cleanest way to program to my taste, but realistically it does fit in with python's whole lack of block scope

[peak spoke]

isn't it relatively common with some kind of filtering?

I think scoping like that fits more into languages where you can also declare things outside of the loop

[flat gazelle]

I believe it's so that you get predictable behavior when stacking multiple for and if clauses, but I am not sure.

[spark magnet]

i think there was a feeling that a comprehension should be a self-contained functionalish thing, and so they got their own scopes (though listcomps in Python 2 didn't have it)

[gray galleon]

because list comprehensions actually create a new function and evaluate them
hence a new scope

[halcyon trail]

anyone else think this is really sussy

In [6]: json.dumps({"hello": 5, 7: 9})                                                                                       
Out[6]: '{"hello": 5, "7": 9}

why is it implicitly converting my integer into a string? this should just be an error

it is documented explicitly at least

Note Keys in key/value pairs of JSON are always of the type str. When a dictionary is converted into JSON, all the keys of the dictionary are coerced to strings. As a result of this, if a dictionary is converted into JSON and then back into a dictionary, the dictionary may not equal the original one. That is, loads(dumps(x)) != x if x has non-string keys.

however, if you pass sort_keys=True, then you get an error

because it happens to try to sort before it tries to coerce

which seems much worse, actually. Th eformer is an API decision I'm not crazy about but this is plain inconsistent.

[umbral plume]

The decision to coerce all keys to strings was just chosen to be their solution to the problem of JSON only allowing strings as keys, yet python allowing any immutable value, the implicit conversion is just there so you don't have to do it yourself, i suppose. The alternative is json being strict with what sort of dicts can be converted into JSON data, which sorta ruins the entire purpose of the module.
As for sort_keys=True sorting before coercion, i suppose it really is just either an oversight on the end of the maintainers, or it was purposefully left like that to prevent taking a stance on how mixed types should be ordered or something, either are probable imo

[halcyon trail]

The final output only has strings anyway so there's only one reasonable stance for ordering mixed types

Coerce to string, then sort

Nothing else even vaguely makes sense

As to "ruins the purpose of the entire module", that seems like a pretty big exaggeration. If you prefer implicit conversions that's fine but either choice is clearly viable

[inland acorn]

This can produce JSON with duplicated keys, python handles it nicely when loading again (uses the last value defined), but some other langs night throw an exception on that which is worth considering.

This SO post goes into more detail on whether that is bad or not, it is technically allowed by the json standard, but as I said some implementations don't like it: https://stackoverflow.com/questions/21832701/does-json-syntax-allow-duplicate-keys-in-an-object

!e ```py
import json
print(json.dumps({1: 2, "1": 3}))

[fallen slate]

@inland acorn :white_check_mark: Your 3.11 eval job has completed with return code 0.

{"1": 2, "1": 3}

[frigid bison]

!e ```py
import json
print(json.loads(json.dumps({1: 2, "1": 3})))

[fallen slate]

@frigid bison :white_check_mark: Your 3.11 eval job has completed with return code 0.

{'1': 3}

[inland acorn]

I feel like it should at least produce a warning in this case

[quick snow]

At work we once had to use custom hooks of the json library to be able to load duplicate keys (and turn that into {"1": [2, 3]} for the example above), because yes, there is software out there that produces this

[elder blade]

I think it makes sense, the list comprehension is usually to create a list and not much more thereafter. You can only fit expressions into it. The iteration variable is a sort-of throwaway - you can't meaningfully break or nest loops.

That said, I believe that if you use the walrus operator it will leak out into the parent's scope.

[frigid bison]

!e ```py
import json
print({"1": 2, "1": 3})

[fallen slate]

@frigid bison :white_check_mark: Your 3.11 eval job has completed with return code 0.

{'1': 3}

[quick snow]

!e

import json
print(json.dumps({1:1,"1":2, 1.0: 3, 1+0j: 4}))

[fallen slate]

@quick snow :white_check_mark: Your 3.11 eval job has completed with return code 0.

{"1": 4, "1": 2}

[native flame]

that doesnt have to do with json though, the dict created is already {1: 4, "1": 2}

[quick snow]

!e

class i(int):
    def __hash__(self): return 42
    def __eq__(self, other): return False
import json
print(json.dumps({i(1): j for j in range(10)}))

[fallen slate]

@quick snow :white_check_mark: Your 3.11 eval job has completed with return code 0.

{"1": 0, "1": 1, "1": 2, "1": 3, "1": 4, "1": 5, "1": 6, "1": 7, "1": 8, "1": 9}

[native flame]

lol

[dusk comet]

O(N) dicts

[radiant garden]

that's already dicts with unfortunate enough bucketing

[boreal umbra]

should have named the class HorribleInt

[wind shell]

dict sucks

tuples are better in my opinion. but i don't use them because tuple sucks too.

reason: dict is too slow. tuple is too complicated

[boreal umbra]

if you want to engage in a serious discussion, you should probably give a specific critique instead of saying that they "suck". but if you're using tuples instead of dicts, that would indicate that you don't really know what either is for.

[wind shell]

list is the answer

man i'm just joking

[wind shell]

but you can do some eso script:
(('a', 10), ('b', 12))

[boreal umbra]

Please don't troll in our discussion channels

[wind shell]

i know the difference between them since i work with python

What would become of me if I didn't know the difference. I would be fired from my job.

But what I was going to say is that tuples are the fastest primitive collections.

[boreal umbra]

for which operations specifically?

[wind shell]

ah, simple ones. Or even bigger ones

[radiant garden]

speedrunning super mario 64 seemingly

[wind shell]

popping and pushing elements 1000000 times

chad

[boreal umbra]

you can't push or pop from tuples.

[wind shell]

tuples are immutable

i know

but you can do some eso code

[boreal umbra]

so are you still trolling, or what?

[wind shell]

using packing and unpacking

lemme show ya

fast_thing = (1,)
for i in range(1_000_000):
    fast_thing += (i,)
    fast_thing = fast_thing[:-2]

[rose schooner]

want me to send you to implementation hell

[wind shell]

i'm not actually sure if this esoteric "push" and "pop" are gonna affect the performance

[rose schooner]

or critique your ignorance of performance

[wind shell]

maybe

$ time py x.py

real    0m0,609s
user    0m0,015s
sys     0m0,062s

i'll test with lists

[rose schooner]

here's something that works 6681481481.48x faster ```py
fast_thing = ()

[wind shell]

x.py

fast_thing = [1]
for i in range(1_000_000):
    fast_thing.append(i)
    fast_thing.pop()

Result (time py x.py):

$ time py x.py 

real    0m0,739s
user    0m0,000s
sys     0m0,046s

[wind shell]

umm

maybe

[quick snow]

Good luck getting anything out of this dict..

[rose schooner]

also it should be py fast_thing = (1,) for i in range(1_000_000): fast_thing += (i,) fast_thing = fast_thing[:-1] so it's equivalent to ```py
fast_thing = [1]
for i in range(1_000_000):
fast_thing.append(i)
fast_thing.pop()

[wind shell]

yeah

the first is faster

[rose schooner]

but that'll make fast_thing = () 9333333333.33x faster

[rose schooner]

no it's not

[wind shell]

wat

i'll test

[rose schooner]

>>> from timeit import main
>>> main(['-s', "t=(1,)", "for i in range(1000000): t += (i,);t = t[:-1]"])
2 loops, best of 5: 126 msec per loop
>>> 126e+9/13.5
9333333333.333334
>>> main(['-s', "t=[1]", "for i in range(1000000): t.append(i);t.pop()"])
5 loops, best of 5: 55.6 msec per loop

[wind shell]

timeit is actually testing in runtime

prefer using the time cmd

[rose schooner]

yes

testing accurately

[rose schooner]

testing inaccurately

[wind shell]

$ time py tuple.py

real    0m1,030s
user    0m0,015s
sys     0m0,015s

$ time py list.py

real    0m1,244s
user    0m0,015s
sys     0m0,030s

[rose schooner]

what version of python are you on

[wind shell]

actually not, because it shows the actual time from the bios clock

3.11 i guess

$ py
Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

[rose schooner]

ok wait so you're saying a one-time run is better than an averaged multiple run?

that sounds wrong

[wind shell]

timeit is more accurate. but time is shows just the time that passed at all

timeit just counts the time of execution

time counts everything

maybe it's more slow because i'm in windows and my pc is not so fast

[rose schooner]

yeah because the only thing you do in a program is execute

time includes anything else that would not matter in the program

[wind shell]

i got your point

that's true.

but it still faster

i guess