#internals-and-peps

1 messages · Page 143 of 1

surreal sun
#

it seems to have a lot of traction within python-ideas, but that doesn't really represent the Steering Council too well

verbal escarp
#

probably not, for the same reason a lot of other syntactic sugar is being deferred

surreal sun
#

to me, ?= made more sense given how ? is more symbolic as to 'none-aware' and in this case = is like "set this argument if the argument isn't given"

surreal sun
#

I was really looking forward to that

#

A lot less of boilerplate code ngl

paper echo
#

@halcyon trail maybe => is good syntax because the late-bound default argument is an anonymous function that gets called upon use 🙂

halcyon trail
#

the argument isn't given isn't the same as None

#

(or even related to None, tbh)

paper echo
verbal escarp
#

actually.. what if the deferred function returns None?

halcyon trail
#

I'd much rather they keep ?=, in case we actually get non aware operators later

verbal escarp
#

in that case we could combine it

halcyon trail
#

I think none aware operators may have a shot, one day, these operators are actually becoming pretty pervasive

#

Eh, they are two different things

verbal escarp
#

def foo(x => {bar()} ?=24):..

halcyon trail
deft pagoda
#

this syntax is awful

halcyon trail
#

I'm pretty sure that Dante's inferno has a circle for that

paper echo
#

what would ?= be in none-aware world?

halcyon trail
#

And yeah, that syntax is not so great, I don't even understand what you're intending up there

deft pagoda
#

if this is what late-bound defaults look like, i hope they never make it in

verbal escarp
#

x = bar(); if bar() returned None: x = 24

halcyon trail
#

x = bar() ? 24

#

lol

#

a little simpler yeah?

#

or maybe it's ??

#

either way

feral cedar
#

i hope ?= would be an inplace "unwrap" sorta thing

paper echo
#

also first of all Make is great

halcyon trail
#

😕

verbal escarp
paper echo
halcyon trail
#

make was designed to be a language for writing build files, that is so unsuccessful, that people have invested a zillion man hours in writing ways to generate the files automatically

deft pagoda
#

how about using something other than parens after def to signify looking in the parameter space before normal locals -> globals stuff

halcyon trail
#

and it's now being phased out of the C/C++ ecosystem completely

#

(finally)

paper echo
halcyon trail
#

(it will take a very long time, don't misunderstand, but that's the direction)

deft pagoda
#

yes

paper echo
deft pagoda
#

but it will consider a name existing in the parameters as something that will need to be late-bound

paper echo
#

yes, but also expressions

halcyon trail
#

what is it good for? honest question

#

Without fail, every time I've seen make used for literally anything at work, in the last 10 years, it was always a bad choice

paper echo
#

i use it the same way you would use tox or invoke

halcyon trail
#

I have never used tox or invoke either

paper echo
#

eg make run-dev could run your web server in development mode

halcyon trail
#

so could a python script, or many other things

paper echo
#

make test invokes pytest

#

sure, but not if you have intermediate assets that you need to generate

halcyon trail
#

if I did ever want a DSL for specifying dependencies between tasks, why wouldn't I use a library in a real programming language?

#

this is exactly how the actual successful systems for doing similar tasks, that I've seen, work

paper echo
#

counterpoint: make is a perfectly good dsl and shell is a fine enough programming language for basic tasks

#

if it gets complicated or difficult to maintain, use a different tool

halcyon trail
#

there's no reason to use what is now obscure technology, for a poorer tool, because the poorer tool is "good enough"

#

you can use a tool that is both better and more easily familiar

#

if you want an example if dependency specification, albeit with a different focus, you can look at something like snakemake, which is written in python

paper echo
#

i disagree that arbitrary python code + some random library is good enough

#

snakemake sucked

#

i tried to use it for years

halcyon trail
#

this is how literally every programmer I know (barring 1) feels about make

#

i'm sure snakemake isn't perfect, but it's found a fair bit of success, and basing your task specification in an actual language seems by far the more popular way to go these days

paper echo
#

meh, I've seen some pretty horrible code-as-config setups

#

anything turing complete can be turned into a clusterfuck in undisciplined hands

#

and some things that aren't turing complete too

halcyon trail
#

Yes, so you may as well start with an actual languages, so that when people fuck around, it's at least reasoanble 🙂

verbal escarp
#

i like my attrs approach to configuration >.>

paper echo
#

shell is an actual language!

halcyon trail
#

Seriously though, every example I can think of that went the pure DSL route, i.e. not based on an actual language, ended up in disaster

paper echo
#

like i said, i agree that make is awful as a front-line build tool

halcyon trail
#

everybody hates the Cmake language in the C++ world. It's actually the only major complaint about it, that you hear over and over.

#

The actual build system is good, it's pretty fast, it's well maintained, it's supported, etc.

halcyon trail
#

You have to be a masochist to write more than a couple dozen lines of bash these days (usually for things related to environment that just can't be done any other way)

paper echo
#

i agree, I don't

#

but 1-2 lines in a makefile, fine by me

#

yeah i've heard a lot of hate about cmake, i certainly don't understand any cmake files ive read

#

meson seems to be the new cool thing

halcyon trail
#

nah meson isn't going anywhere most likely

#

it would be cool if it did but the amount of inertia behind CMake at this point is crazy

paper echo
#

point being: make is a perfectly good tool for what it is, which is to specify a dependency graph and some shell scripts for generating them

surreal sun
#

it has some correlation imo

paper echo
#

it'd be great to have something like make without all the sharp edges, but snakemake is definitely not it imo

#

it'd be nice if they exposed make functionality as a C api though

halcyon trail
#

But it's literally not a good tool for that. Or at least, very few people seem to think so, because very few people are using it.

#

if someone tells me "yeah we use make and some shell scripts to do XYZ" my first thought is probably just that the code has been around for a long while

#

(I should say, few people are using it as a green-field choice, if they're using it it's because there's existing code, or at best new code that's being integrated with existing systems)

paper echo
#

i use it as a default for ad hoc small things

#

and like i said as a tidy uniform entry point to other scripts

red solar
#

Salt off topic but were u using pypy or graalvm?

paper echo
#

both but only for playing around

red solar
#

Can I ask u some questions about graalvm? (Not sure whether here or another channel)

paper echo
#

yes but i know absolutely nothing about how graalvm works 😆

red solar
#

That’s ok

#

I was just wondering if it has cffi support

paper echo
#

for example @halcyon trail we use makefiles to paper over the differences between our node and python services, and between our old shitty vagrant services and the new shiny docker compose ones. we could use just a shell script or some other tool but literally everyone here has gnu make already installed and knows the syntax

paper echo
red solar
paper echo
#

not at a computer at the moment... maybe something on their website

#

i doubt it does tbh, but it probably has something else via graalvm itself

red solar
#

Hmm ok i’ll try and check

heady mauve
#

Does sys.modules['ctypes'] = None actually prevent further code from importing ctypes?

#

Something tells me it's not that easy.

surreal sun
#

Well I’m gonna guess ctypes gets overwrite when imported

#

iirc you might need an import hook for that

heady mauve
#

Except that import module doesn't reload module

surreal sun
#

Ah true

#

but wouldn’t it break if they did importlib.reload(ctypes)

heady mauve
#

that would, so you'd also have to block importlib

verbal escarp
#

they might del sys.modules["ctypes"] and then import

heady mauve
#

that would do it

flat gazelle
#

Or even just pick an already imported ctypes module from somewhere in the stdlib

heady mauve
#

So yeah an import hook would be necessary, followed by del importlib so that the code that follows wouldn't be able to redefine the hook.

heady mauve
flat gazelle
#

Well, you can access some modules without importing

heady mauve
#

Effectively I want to embed Python as a scripting language and block filesystem access, along with all but a few okayed modules. (No, nothing can change my mind that I want Python for this 😝)

flat gazelle
#

Build a custom embeddable package with as few modules as possible for starters

#

You can compile python without ctypes etc

heady mauve
#

I had forgotten about compiling a different interpreter, not sure if it would be within my skillset...

#

But, that's what project-based learning is for :D

flat gazelle
#

In the end, you aren't getting a useful sandbox by writing python code

heady mauve
#

right

#

unfortunately

spark magnet
#

@heady mauve it is very difficult to secure Python

heady mauve
#

So I think the way to go based on what I've heard here is to compile a python interpreter that cannot run ctypes, os, etc.

heady mauve
verbal escarp
#

well, have a look at a the discord bot

flat gazelle
#

I don't think you can get rid of some of the more code modules

heady mauve
rich cradle
verbal escarp
#

yeah, that

spark magnet
heady mauve
#

I do need Windows support

flat gazelle
#

nsjail works, but if they want to embed it as a scripting Language, that's not a great solution.

#

Windows does have their own sandbox in pro IIRC

spark magnet
heady mauve
#

If I could fire up a subprocess with a modded interpreter to run the embedded scripts, that would work beautifully.

spark magnet
#

@heady mauve needing windows support sounds like you're going to run this on users' machines?

heady mauve
#

yes

#

I'm typing up the whole story atm

verbal escarp
#

okay, so why is accessing ctypes bad in your case?

heady mauve
#

Overall, it's going to be a game with game objects that can execute arbitrary Python code. Unlike projects like Logical (logic gate sim that needs file access) I do NOT want filesystem access for these scripts and I want most libs blocked (socket, ctypes, anything that can act outside the game script). The scripts do need to be able to interact with the main process to control game objects.

heady mauve
spark magnet
#

and you don't trust the game object code? Who writes it?

verbal escarp
#

you fear cheating from players or do you fear players hacking other players?

heady mauve
#

For the game object code, I write it any anybody who contributes to my repo. For the script code, anybody can write it, most often end-users. Scripts will be shared.

heady mauve
verbal escarp
#

then you need proper code reviews

heady mauve
#

yes

spark magnet
#

@heady mauve Python might not be the right tool for this. it will be difficult to prevent access to things

heady mauve
#

That's what everyone has said so far

verbal escarp
#

actually i wouldn't say so

spark magnet
heady mauve
#

I think that if you patch the interpreter itself to permit or ban certain things then you can do pretty much anything

verbal escarp
#

yeah, you could define classes as API and check the AST of the modules before executing them to only access those classes

heady mauve
#

And by doing a subprocessed interpreter, you can cut out bits of the standard library for that specific interpreter to block things like networking.

spark magnet
#

these are interesting theories, but have you tried it?

heady mauve
#

Okay, I need to read up on AST

verbal escarp
#

well, you can filter the AST, right? have a positive list of allowed objects and anything outside of that list raises a red flag

spark magnet
#

@verbal escarp what would your AST filter do with this?

(lambda fc=(lambda n: [c for c in ().__class__.__bases__[0].__subclasses__()
    if c.__name__ == n][0]): fc("function")(fc("code")(0,0,0,0,"KABOOM",(),
    (),(),"","",0,""),{})())()
heady mauve
#

what even is that nebat??

feral cedar
#

hey...i saw that in a blog once :)

heady mauve
#

forkbomb?

spark magnet
verbal escarp
feral cedar
#

it creates a code object with some wacky initial values so it does bad things

heady mauve
#

the heck....?

pliant tusk
verbal escarp
#

but i wouldn't allow access to __class__ so..

pliant tusk
#

you dont need access to __class__

verbal escarp
#

[...] ().__class__[...]

pliant tusk
#
load_addr = type(m:=lambda n,s:lambda v:s(v)or n)(
    (M:=m.__code__).replace(
        co_code=b'\x88'+M.co_code[1:]
    ),{}
)(r:=iter(range(2**(PTR_SIZE*8-1)-1)),r.__setstate__)``` this leverages a different bug to be able to load arbitrary addresses
heady mauve
#

yep, right here

pliant tusk
#

(my point is that there will always be another way to break it)

spark magnet
#

@verbal escarp can i use getattr?

verbal escarp
spark magnet
#

well, yes. sneakiness is the whole point

verbal escarp
#

for a pure API, i would say no getattr needed

spark magnet
#

how small a subset of Python will still be OK for these game objects?

heady mauve
#

Just enough to do basic data manipulation and call some functions

#

ie, I would like full OOP, but the only outside contact the script would have would be through stdin/stout. These would be parsed by the host program and handled from there.

verbal escarp
#

i'd think checking the AST is your safest bet

pliant tusk
#

@heady mauve @verbal escarp without major changes to the interpreter, you will not be able to guarantee safety

heady mauve
verbal escarp
heady mauve
verbal escarp
#

let me look up the talk, maybe that can answer you some questions

pliant tusk
#

attribute access, closures have some memory bugs you can mess with, local variables can be exploited, exception frames can be abused .... (etc)

heady mauve
#

dang

pliant tusk
#

and those are all the ones that are difficult to exploit

#

will you allow importing modules?

heady mauve
#

so long as the code can't break out to C I should be able to put up stopgaps

heady mauve
pliant tusk
#

which ones?

heady mauve
#

re, configparser, things like that

#

And a custom library to simplify interfacing with the host program

pliant tusk
#

what about the modules that re and configparser need?

heady mauve
#

not sure what they need

verbal escarp
#

(Simon Carryer) Abstract Syntax Trees are a tool for analysing and manipulating the structure and function of code. They can be used to do about a million different useful and smart things. This talk is not about those things.

https://pretalx.com/pycon-au-2021/talk/MFUNXT/

python, pycon, australia, programming, conference, technical, pyconline...

▶ Play video
feral cedar
#

if you're letting the players write arbitrary code, you can't let them have re either

heady mauve
#

what's wrong with regex parsing?

pliant tusk
#
from collections.abc import MutableMapping
from collections import ChainMap as _ChainMap
import functools
import io
import itertools
import os
import re
import sys
import warnings``` these are from configparser
heady mauve
#

oh

#

well crud

#

the heck does configparser need io for???

verbal escarp
#

Tim Savage

Allow your users to supply queries or define rules using Python syntax and safely eval them. Processing an AST into safely executable code.

https://2019.pycon-au.org/talks/safely-eval-python-syntax-using-the-ast-module

PyCon AU, the national Python Language conference, is on again this August in Sydney, at the International Convent...

▶ Play video
verbal escarp
#

both talks are cool though

heady mauve
#

DOSing?

feral cedar
#

basically make a regex that is super slow to create and to check

heady mauve
#

I'm not worried unless the script can cause damage on the user's computer. If the script hangs, I'll kill it.

verbal escarp
#

iirc in the talk of tim savage it's about python in a banking environment

#

as in eval() in banking

#

which should give you some idea of how secure it can be

heady mauve
#

banking like handling money banking or something else?

verbal escarp
#

yes

heady mauve
#

ah

#

that's going on the watch later if I don't watch it now

#

thank you!

pliant tusk
#

@verbal escarp that neuters python to the point its almost unusable to get any level of safety

verbal escarp
heady mauve
#

that I am

paper echo
#

and has python bindings

feral cedar
#

!pypi re2

fallen slateBOT
feral cedar
#

~~woah it fuzzy matches too?~~nvm 😔

paper echo
#

don't quote me on the safety thing, check in the docs

pliant tusk
# heady mauve thank you!

i encourage you to keep working on it, but dont be disappointed when you have to sacrifice major usability for safety (or vise versa)

#

theres a reason there are languages designed for this exact use case (lua)

heady mauve
#

right

pliant tusk
#

dont be suprised when people like myself find holes in it then

heady mauve
#

Surprised that you found it, I won't be. Surprised that that's what it was, you can bet I probably will be.

verbal escarp
#

i'd love to see the battle of wits on that 🙂

heady mauve
#

lol

pliant tusk
feral cedar
#

like an arms race lol

heady mauve
#

again, I need windows support and snekbox uses nsjail that can't run on windows

pliant tusk
#

hence the something like

heady mauve
#

to clarify I need windows and linux support

#

I did see cuckoo sandbox but I didn't see much on how to set it up

verbal escarp
pliant tusk
#

you wont be able to allow things like print()

verbal escarp
#

which probably isn't needed anyway

heady mauve
#

print will have to be redone anyway

pliant tusk
#

my point is that if something as simple as a bare function call will have to be disabled, what useful things will it be able to do

heady mauve
#

function calls would be permitted. I don't see why not

#

so long as they can't call malicious code

pliant tusk
#

how will you determine that

naive saddle
#

you can abuse call frames to access some weird internals

pliant tusk
#

^

verbal escarp
#

if you can't access call frames?

pliant tusk
#

how will you block them?

heady mauve
#

can they be accessed without a lib?

pliant tusk
#

yes

heady mauve
#

how?

naive saddle
#

probably via an exception

pliant tusk
#

tracebacks

heady mauve
#

hmm

pliant tusk
#

!e py try:raise except Exception as e:print(e.__traceback__.tb_frame)

fallen slateBOT
#

@pliant tusk :white_check_mark: Your eval job has completed with return code 0.

<frame at 0x7f9707c797d0, file '<string>', line 2, code <module>>
heady mauve
#

ok, so if untrusted code gets access to a frame what can it do?

pliant tusk
#

step up it, modify variables

heady mauve
#

ok

pliant tusk
#

use it to get access to other modules

heady mauve
#

if those modules literally are not here it cant :P

pliant tusk
#

(configparser needs os which can be leveraged to write into memory)

heady mauve
#

it can, so that would not be allowed

#

sad

pliant tusk
#

youre missing the point tho, that is just one example

#

and there are already countless security holes

spark magnet
#

@heady mauve try this: python -v -c "print(1)" it will list the modules Python imports just to get going.

#

(which includes os)

heady mauve
#

well that's a little deeper than I thought it went

#

hmm...

verbal escarp
#

well, i stand by my assumption - it's possible to make it safe by using a positive list of allowed calls/objects, as long as the list isn't too long

naive saddle
#

Just to be clear, I'm 100% all for building a sandbox within Python, just please don't use it in a real scenario. Sorta like amateur crypto: great for learning, but a bad idea for security. You'll probably learn some fun ways Python is highly introspectable 🌺

rich cradle
#

I think what you really need to do is find something like nsjail that works cross-platform, or some alternative specifically for windows.

heady mauve
#

well, os and io were both imported just to run a = 1

verbal escarp
#

watch the talk, if tim savage can make eval safe for banking, you should be able to make it safe for a game

heady mauve
#

I shall

pliant tusk
rich cradle
#

Frankly, I feel like the only way harm can't be caused is if you can close off the interpreter from the system. Anything else is a fools mission.

pliant tusk
verbal escarp
heady mauve
pliant tusk
#

then the checked code will still be able to do dangerous things, provided you dont limit it as much as ast.literal_eval

verbal escarp
#

that's you assuming things, same as me assuming the contrary

#

won't know until @heady mauve gave it a try 😉

heady mauve
#

lol true

spark magnet
#

@verbal escarp i think Tim Savage's system is more limited than desired here, but it's hard to say

pliant tusk
#

im not assuming things tho? i have tried to do this in the past, and to get any degree of safety python becomes significantly less useful

verbal escarp
pliant tusk
#

it probably wont be

#

unless they just want basic math with no variables

heady mauve
#

Well, whatever I do, I need to get rendering working first. I'll probably be in a help channel soon asking about OpenGL/OpenCL interops 😛

verbal escarp
#

well, have fun

heady mauve
#

that I shall

#

finally, pure raymarching is on the horizon

red solar
#

PyCharm says error_msg local variable is not used? (for the one that = None)

#

is this PyCharm being slow, or another of python's weird scoping rules?

#

(ig the if statement is a redundant, but other than that...)

paper echo
#

@red solar confirmed, no cffi

% pyenv shell graalpython-21.2.0
% python -c 'import cffi'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'cffi'
paper echo
#

but again, i believe graalvm has its own ffi stuff

red solar
#

yeah i'm not gonna get into the mess of each implementation's ffi, pypy was enough of a pain :/ spent hours on that before just using cffi

#

maybe if i'm desperate to support graal later

#

but thanks for checking 🙂

paper echo
#

it seems like you can however use graalvm to build libraries that can be called from c, rust, etc. programs

red solar
#

The NFI is an internal language designed for language implementors. It is not considered stable and the interface and behavior may change without warning. It is not intended to be used directly by end-users.
idk, i think technically i'm an end user :/

#
library = Polyglot.eval('nfi', 'load "libSDL2.dylib"')  # load a library
symbol = library['SDL_GetRevisionNumber']               # load a symbol from the lirbary
function = symbol.bind('():UINT32')

that's much nicer than ctypes tho (the .bind())

white nexus
#

update on the lambda stuff:
I didn't do any of it. I made full functions and did it that way, which is sufficient for my usecase and gives me additional options

red solar
#
from atomics._impl.atomic.base import AtomicBase
import sys

a = AtomicBase(8, is_integral=False, is_signed=False)
i = 17
b = int.to_bytes(i, 8, sys.byteorder, signed=False)
a.store(b)
v = a.load()
print(int.from_bytes(v, sys.byteorder, signed=False))

If i run this normally in PyCharm, it will print 17.
If i run it using PyCharm's debugger with no breakpoints set, it will print 17.
If i set a breakpoint on any line from i = 17 onwards, i will get an assertion error from the C library i'm interfacing with (when I say interfacing, it's just ctypes).

#

any thoughts on how tf i would go about debugging this? I'm used to debug mode changing stuff in C/C++ (if there's UB), but i've never seen this happen in Python

#

like i don't see how debug mode in Python would change anything in Python, and it sure as hell shouldn't change anything in C (since that's already compiled)

#

(btw other people can ask questions, don't let this oddity hold up any other convo)

raven ridge
#

what's the assertion error?

red solar
#

the parameter i passed (an int) had the wrong value

#
python3.8: /home/doodspav/CLionProjects/patomic/src/impl/std/std.c:889: patomic_opimpl_load_ulong_explicit: Assertion `patomic_is_valid_store_order(order)' failed.
raven ridge
#

so you put a breakpoint on i = 17, and your library says that the value of i is invalid?

red solar
#

no the assertion is in the C lib

#

i put a breakpoint on i = 17 and somehow it's already run a.load() and crashed before the debugger can pause it

raven ridge
#

well, the first thing to find out is what value order has

#

either add a printf in the C code, or use a C debugger

red solar
#

if i get rid of a.store(), then I still get the assertion error in C

#

the only other function calling C there is a.load()

#

wait

#

tf

#
#define PATOMIC_DEFINE_LOAD_OPS(type, name, order, vis)         \
    static PATOMIC_FORCE_INLINE void                            \
    patomic_opimpl_load_##name(                                 \
        const volatile void *obj                                \
   vis(_,int order)                                             \
        ,void *ret                                              \
    )                                                           \
    {                                                           \
        type val;                                               \
        assert(patomic_is_valid_load_order(order));             \
        val = atomic_load_explicit(                             \
            (const volatile _Atomic(type) *) obj,               \
            order                                               \
        );                                                      \
        PATOMIC_IGNORE_UNUSED(memcpy(ret, &val, sizeof(type))); \
    }

i mean, there's no patomic_is_valid_store_order(order) assertion inside patomic_opimpl_load_ulong_explicit...

#

so wtf

#

but anyway thanks @raven ridge ig this is a C issue now

#

i recompiled and it worked, who knows :/

#

but still really strange that it only broke once i set a breakpoint in python

paper echo
#

@zenith flint a help channel seems appropriate. nobody is ever too advanced for a help channel

zenith flint
gloomy rain
#

You're just going to have to be patient, or find help elsewhere.

zenith flint
gloomy rain
zenith flint
halcyon trail
#

channels don't get nearly that specific

#

help channels are for help "How do I do this, why doesn't this work, etc"

gloomy rain
verbal escarp
#

apropos safe python topic, AST eval with @spark magnet, @pliant tusk, @heady mauve from last night: what about using a python interpreter that doesn't expose as much of its internals as cpython? maybe pypy?

pliant tusk
#

Part of the issue is in order to be a compliant interpreter there has to be a fairly high level of exposing of internals

verbal escarp
#

if i were developing a python interpreter, i'd throw cpython-compliance out the window and focus on PEP-compliance with protocols

pliant tusk
#

What about PEPs that specifically detail introspection of internals?

verbal escarp
#

there are interpreters that don't try too hard to be cpython compliant like Jython

#

although i'm not sure how much those expose their internals

#

i could imagine even Brython could be an option for that

#

since it maps to JS, which doesn't have these kinds of issues

#

in this case, cpython non-compliance could prove to be an advantage

paper echo
# zenith flint Yeah I've tried several times but couldn't find help, hence why I wanted to post...

in general, if you aren't getting answers to your question, at least one of the following is probably true:

  • answering your question requires specific knowledge that the community does not generally have
  • your question is phrased in a way such that it is hard to understand
  • you have not provided enough information for someone to help you
  • your question is understandable, but too complicated, and needs to be broken down and asked in a simpler form; perhaps you have provided too much information, e.g. you have provided code and data, but they are far from "minimal", to the point where it's a lot of work to even try to understand
zenith flint
#

Well most of the time it's more like random people spam something nonsense in the channel and the people who could help won't even see the question anymore

#

(happened 4 times in a row)

paper echo
#

well that too. perhaps stackoverflow is a better forum, although stackoverflow drowns in low quality spam too

zenith flint
#

Yeah gonna try SO now

paper echo
#

note that your particular question i think has the first and last problem:

  • it requires an understanding of rabbitmq and pika, which most people here probably don't have
  • "it hangs silently" is a really nasty and difficult to debug error, and doesn't correspond to any known or obvious pattern of failure
#

does it hang if you swap out logging handlers? maybe you have an unrelated race condition. etc.

zenith flint
#

"it hangs silently" is a really nasty and difficult to debug error, and doesn't correspond to any known or obvious pattern of failure
Yeah I know but that's all I see, not sure how to debug it deeper because there's no exception nothing

#

It works if I replace the the publishing part with something like just printing the log message etc.

#

So the issue is definitely within the publishing but I don't see a reason why/how as it works fine outside of the class

#

I wish coding communities other than this Python Discord weren't complete garbage. There's 10k people on the RabbitMQ Slack and there are 2 messages a day happening on the entire server

halcyon trail
#

I think "garbage" and "inactive" or "quiet" etc are two different things....

paper echo
#

well did you do the obvious basic debugging steps? use a different log handler, try to just connect to rabbitmq without the logging framework, etc.

#

you might even have to drop into a debugger and step through the pika code

#

i've done things like that before

red solar
#

is there a way to specify separate package requirements for installing a package and building it? like installing it needs a single package, but building it needs a fair few

paper echo
red solar
#

oh neat 🙂

#

but also crap, another file i need

paper echo
#

you should have it anyway imo

red solar
#

ok i'm a little overwhelmed

#

i already have a setup.py file

#

can i just have this as my toml file?

[build-system]
requires = ["cmake >= 3.14", "GitPython", "setuptools", "wheel >= 0.25"]
heady mauve
#

I think a version of CPython that locks itself down based on command line args would be nice. Could run with it 100% CPython exact or lock it down whenever.

verbal escarp
#

probably not deemed useful enough to realize

red solar
#

is there any benefit to this over just sandboxing the process using the OS?

verbal escarp
#

"just sandboxing"? heh

red solar
#

when you guys say "safe"... how safe does that mean?

verbal escarp
#

really safe

red solar
#

So no access to any OS provided functionality?

heady mauve
#

Anything that could be used to cause damage to the end user's PC

red solar
#

What would be a use case?

heady mauve
#

embedding python as a scripting language in a video game

#

and please don't say "just use x language" that's all I ever hear

red solar
#

When people embed lua, is it a limited functionality version?

verbal escarp
#

why not use Brython? it looks like the perfect fit to me

#

you could even plug it in a browser directly with zero-install

heady mauve
#

I have never heard of it

verbal escarp
heady mauve
#

well that doesn't really work for me

#

js is a no go for this project for... reasons

verbal escarp
#

you don't code in js

#

it's python which gets mapped to js

#

you don't even see it

grave jolt
#

Just use agda, it compiles to JS 😉 😉

heady mauve
#

js is banned here (sublayers or not)

#

:P

grave jolt
#

Hm... does anyone know why Python thinks it's invalid?

>>> from datetime import datetime
>>> datetime.fromisoformat("2021-10-26T10:35:11")
datetime.datetime(2021, 10, 26, 10, 35, 11)
>>> datetime.fromisoformat("2021-10-26T10:35:11Z")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Invalid isoformat string: '2021-10-26T10:35:11Z'
>>> 

ISO 8601 defines Z as valid, meaning UTC: https://en.wikipedia.org/wiki/ISO_8601#Time_zone_designators

peak spoke
#

python only implements it as a reverse of its isoformat, there's been a lot of discussion on handling the Z but I've no idea why it wasn't added

grave jolt
#

smhmh

#

datetime.fromisoformat(ts.replace("Z", "+00:00")) this should work, right? 🙂

paper echo
#

this to me is intensely unproductive anti-pythonic bikeshedding over what imo is an obviously good change: add Z support, write a big loud warning in the docs that it does not fully implement iso-8601

halcyon trail
#

@paper echo agree

#

this is just silly, IMHO:

That said, in some ways it would violate the spirit of the contract, which is that, as of right now, fromisoformat is intended to be used only on the output of .isoformat, which means that all the people who want it to parse Z are in a sense using it in an unsupported way.

grave jolt
#

I am currently in disappointment of the highest degree

#

!d datetime.datetime.fromisoformat

fallen slateBOT
#

classmethod datetime.fromisoformat(date_string)```
Return a [`datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime "datetime.datetime") corresponding to a *date\_string* in one of the formats emitted by [`date.isoformat()`](https://docs.python.org/3/library/datetime.html#datetime.date.isoformat "datetime.date.isoformat") and [`datetime.isoformat()`](https://docs.python.org/3/library/datetime.html#datetime.datetime.isoformat "datetime.datetime.isoformat").

Specifically, this function supports strings in the format:

```py
YYYY-MM-DD[*HH[:MM[:SS[.fff[fff]]]][+HH:MM[:SS[.ffffff]]]]
```  where `*` can match any single character.

Caution

This does *not* support parsing arbitrary ISO 8601 strings - it is only intended as the inverse operation of [`datetime.isoformat()`](https://docs.python.org/3/library/datetime.html#datetime.datetime.isoformat "datetime.datetime.isoformat"). A more full-featured ISO 8601 parser, `dateutil.parser.isoparse` is available in the third-party package [dateutil](https://dateutil.readthedocs.io/en/stable/parser.html#dateutil.parser.isoparse)...
halcyon trail
#

I can't imagine that anybody using datetime, in practice, is aware that fromisoformat is intended to be used only on the output of .isoformat

#

that's just absurd

paper echo
grave jolt
#

I guess it does says so in the description 🤷

paper echo
#

see also: the thread about json.load_file

grave jolt
#

hm?

halcyon trail
#

I've defined been guilty of ignoring that aspect of the description 🙂 That said, it's really one or the other, if they want to stick to that description, then there needs to be a broader function

#

No matter how you slice it

paper echo
# grave jolt hm?
#

instead of "hey that's interesting! let's work on some proposals to meet this obvious need in the community", it's "i don't personally use this kind of thing and you are a noob for wanting it"

peak spoke
#

I don't agree with that one as much as pathlib's read_text and write_text can be used in a similar fashion, but I guess load/dump are already there offering tiny abstractions

surreal sun
#

Few things are actually considered, a lot of people just like to point out only the flaws and not the good that could come out of it

peak spoke
#

You'll find that most people are resistant to change of what they're used to

grave jolt
halcyon trail
#

my personal and biased opinion is that the reality is that there a lot of languages that are offering "nicer" syntax for these things in a lot of cases

#

people see things and want them in python, at least to the degree practical

#

and people aren't persuaded about the benefits because it's a small, ergonomic thing at best, and it brings up issues that have to be dealt with (e.g. the encoding in the case of json) 🤷‍♂️

paper echo
#

it's funny because in the json case specifically afaik at least one spec requires utf-8, so that should be a non-issue

#

in all other cases... we literally deal with this everywhere else in python now

#

just default to utf-8 and give the user an encoding= kwarg

halcyon trail
#

with open("filename.json") as f: j = json.load(f) looked fantastic 20 years ago when C++ and Java had 20 line equivalents to this

undone hare
#

Only the fact that you'd need to drop 3.10 support to use that function make it not seem that appealing anymore

grave jolt
#

you mean 3.10 support

halcyon trail
#

if such a function is introduced, I do not expect libraries to jump on it immediately?

#

you can say what you're saying about literally any new feature

undone hare
#

I do mean 3.10 now

grave jolt
#

yeah, I guess it's for the long term

halcyon trail
#

it's a non-argument

paper echo
#

it probably won't be used in some libraries ever! but imagine how much easier it would be for the literal thousands of newbies on this server

undone hare
#

But then you have to explain "blah blah blah your book doesn't mention it because it is older than you"

paper echo
#

i'd rather do that

grave jolt
#

you can say this about every new feature

halcyon trail
#

again, all these arguments apply equally to every single feature, function, that could be added to the language

paper echo
#

imagine, you can add one function to the standard library and save probably hundreds or thousands of man-hours across the world and for years to come, making the language friendlier for newbies and safer in general for non-experts

paper echo
#

what was the benefit:cost ratio of pattern matching?

halcyon trail
#

I was talking to Akarys

paper echo
#

oh sorry

halcyon trail
#

I don't think it's massive, fwiw

undone hare
#

It does apply to every single new feature, you are right, but this function is only relevant in case of learning, and I think it is too confusing for that

halcyon trail
#

no, this function is not only relevant for learning...

#

plenty of times when I'm writing code, I want to open a json file as an expression

paper echo
#

learning is one extremely valuable use case, i strongly believe that it is less confusing

halcyon trail
#

i have utility functions exactly like this in my codebase

undone hare
#

Considering it cannot possibly support all the arguments open and loads have, I don't know how useful it would be

paper echo
#

but how often do you or anyone use those arguments

#

0.01% would be optimistically high

#

the point is not to support all needs and use cases

#

it's to make one very very very common use case less verbose, easier to teach, and less error-prone even in "serious" codebases written by serious experienced people

undone hare
#

Hey, you could use big query to check that haha

peak spoke
#

You only need the encoding and path in practically all cases, but I'm not that sold on the utility of this compared to the existing alternatives

flat gazelle
#

I think it's nice just for the convenience when loading sth into a repl

surreal sun
#

In general though, I’ve noticed syntactic sugar PEPs and ideas tend to more often be denied than accepted

paper echo
surreal sun
#

or even just sugar in making things less boilerplate

paper echo
#

i really am not trying to be closed-minded here, but i truly don't understand the resistance to this kind of thing

#

python is a big featureful language, blocking something as small as this will not reverse some perceived tide of concept bloat

#

which i agree exists in python, but not because of useful things

#

you could make the same arguments about str.removeprefix, the only pattern i can decipher is that the accepted changes are the ones that the core maintainers personally find useful

#

which is fine, if you are a minimal language with a standard library that exists only to bootstrap the runtime

#

many such languages exist. python is not currently one of them and probably will never be.

peak spoke
flat gazelle
#

yeah, this seems like a pretty good helper function that would make using python easier.

halcyon trail
#

it's unfortunate that python does not have extension functions; it's so nice to simply add these kinds of functions as extensions to your Path type

flat gazelle
#

not by a lot, but well, it is trivial to implement

inland wadi
#

Uhm, hello

#

May I ask where I could start learning Python?

#

I'm not sure if I'm using the right channel

surreal sun
paper echo
inland wadi
surreal sun
#

!resources

fallen slateBOT
#
Resources

The Resources page on our website contains a list of hand-selected learning resources that we regularly recommend to both beginners and experts.

surreal sun
#

Some really good resources on here, but for future reference, this channel is for discussion about the language itself and what not (channel topic explains it better)

paper echo
#

good design is almost always an accumulation of small good things, and size of change is not generally correlated with size of benefit

#

sometimes the biggest improvements are the smallest, and sometimes the biggest changes are the least cost effective

surreal sun
#

^ a lot of the string methods could be argued with the logic that “this can be done already without it”

surreal sun
halcyon trail
#

when it comes to types, there's an argument about adding more and more member functions, which is that it makes it more and more onerous to replace the type, if you so wanted

#

people more rarely replace modules than types, though in python specifically replacing types is probably also relatively rare.

#

but again, extensions really shine here as they separate the intrinsic, necessary, reaches-into-the-guts of the type member functions, from the helper functions that can be implemented in terms of other stuff

#

and it makes those helpers reusable, squashing one of the main objections to them without having to switch to free-function syntax which is often disliked

#

I almost had PTSD flashbacks today because I was writing some python code and I was expecting startswith ane endswith not to exist, because in C++ they are not added until C++20, which my company isn't even on yet 🤣

verbal escarp
grave jolt
#

this must be my favourite JS function... I think I implemented it more than a dozen times ```ts
const sleep = (ms: number): Promise<void> => new Promise(resolve => setTimeout(resolve, ms))

verbal escarp
#

i'm also a bit pissed by similar cases i encountered previously

paper echo
grave jolt
#

eh

#

where do you put it, in JS?

paper echo
#

right, i'd say that specifically because this is JS it gets a pass

#

browser implementers have enough on their hands as it is

#

but if this were any "standalone" language i'd be pretty annoyed at having to write that. but at least sleeping is not all that common of a need (outside of demos)

verbal escarp
#

and those helpers wouldn't even require a lot of maintenance

paper echo
#

you know what is a really annoying glaring gap in the nodejs api? no built-in functionality to iterate over buffered lines of stdin or another text stream

#

imagine having to implement that yourself, in 2021, for every single application

verbal escarp
#

so the whole "we can't have all the nice things" argument doesn't apply

paper echo
red solar
#

Has no one implemented an std package for js with no dependencies? Surely that’s a thing by now

#

There must be multiple

paper echo
#

fwiw the sterring council and core devs do seem to be taking "dead battery removal" seriously

#

and i really appreciate that they are

verbal escarp
red solar
#

I mean, I figured… but still

verbal escarp
paper echo
verbal escarp
#

just thinking of tuple unpacking for lambdas

paper echo
verbal escarp
#

still hurts

halcyon trail
#

you guys need to program C++ 🙂 You can't even write anything in C++ without thousands of lines of utility code

paper echo
#

now that we have pattern matching, i would love to have full pattern matching in functions and =/:=

halcyon trail
#

want a backtrace on your exception/assert?

#

implement it

paper echo
halcyon trail
#

i might also if it didn't pay well

#

my ambition is for C++ to eventually die as I get older, so I can be like those aging COBOL programmers now

#

and consult at insane hourly rates

verbal escarp
#

heh

paper echo
#

apparently the cobol programmers don't actually make that much compared to the rest of the industry, they just get repeatedly called out of retirement

halcyon trail
#

well, you can't just force people out of retirement

#

you have to pay enough to be worth it

paper echo
#

well no, but even getting the emails/calls would be annoying

verbal escarp
#

"ok, i'm coming, but you pay the lunch"

halcyon trail
#

you can just ignore them.... the company doesn't own you after you retire lol

#

i'm sure it depends in what sector, but at least in NY finance, there are a lot of stories about obscene hourly rates

#

but I admit I don't have any first hand verification, I don't know any programmers that old

paper echo
#

i'd believe it in NY

#

not going into finance while i lived there was a major regret

halcyon trail
#

it's a good scene, at least if you go into fintech

#

working at a straight up bank, I imagine pays fine but can probably get a bit miserable QoL wise, though it really depends what team you are on

#

HFT is nice, you get a lot of the perks of both finance, and tech, IMHO. And the company is generally very technology dependent, so that informs the culture a lot.

paper echo
#

i was in insurance in goddamn new jersey, i was underpaid and everything about it was a miserable slog

#

big major company too

#

this is OT though

halcyon trail
#

OT?

#

hey now, I live in New Jersey 🙂

paper echo
#

(off topic)

halcyon trail
#

oh

#

can continue in another channel if you know which is the right one lol

grave jolt
#

Do you think it's always a bad idea to have a default mutable argument? For example here:

async def fetch_all(
    session: ClientSession,
    url: Union[URL, str],
    field: str,
    params: Mapping[str, Any] = {},
    step: int = 100
) -> AsyncIterator[Any]:
    async with session.get(url, params={**params, "count": 0}) as resp:
        json = await resp.json()
    total = int(json["total"])

    for i in range(0, total, step):
        async with session.get(url, params={**params, "count": step, "offset": i}) as resp:
            json = await resp.json()
            for item in json[field]:
                yield item
verbal escarp
#

#bad-career-choices-discussion 😉

grave jolt
#

so... you propose doing this? ```py
async def fetch_all(
session: ClientSession,
url: Union[URL, str],
field: str,
params: Optional[Mapping[str, Any]] = None,
step: int = 100
) -> AsyncIterator[Any]:
if params is None:
params = {}
...

main ginkgo
#

only thing i can really think of a mutable default being good for is a cache

#

with the option to use a different cache of your own

grave jolt
#

here it's only accidentally mutable -- I'm not mutating it, of course

verbal escarp
#

future you will hate you for that

grave jolt
#

why?

main ginkgo
#

frozendict when 😦

verbal escarp
#

because future you might've forgotten about the context and mutates the thing somewhere else and then you'll end up wasting time debugging the whole thing for hours because there's no obvious exception and then you'll swear to never do it again

grave jolt
#

I guess... but adding None as default and then doing an if seems just ugly

verbal escarp
#

it is ugly

#

that is also why we discussed the => thing earlier today

halcyon trail
#

I think it's always a bad decision to have a mutable default argument, yes

#

and that includes using it as a cache

#

and adding None as a default and doing if doesn't just seem ugly... it is ugly

#

but that is how we do

grave jolt
#

I guess I could use MappingProxyType({}) as the default

halcyon trail
#

You don't need the Optional annotation, FYI

#

that situation is already special-cased into python type checkers

grave jolt
#

depends

#

in Pyright you can configure it to error

halcyon trail
#

If you go out of your way, sure

grave jolt
#

I wouldn't want ```py
def f(x: int = None):
return x + 5

main ginkgo
grave jolt
#

use functools.cache/lru_cache

#

you shouldn't expose the cache as an argument

main ginkgo
#

why add extra unneeded overhead if all i need is just a simple cache?

halcyon trail
#

If you want to have a global default cache, then... have a global

main ginkgo
#

lru_cache has its own specific uses

halcyon trail
#

and use it as the cache

#

and even make it the default argument if you want

#

you can even make the global start with _ if you want it to be an implementation detail

#

there's literally no reason to write it that way; it just makes it harder for people to grok who aren't familiar with this very weird quirk of python

main ginkgo
#

im still not seeing a why

#

you presented an alternative, you didnt explain why that would be better though

halcyon trail
#

because its explicit, instead of depending implicitly on a bizarre behavior of python?

#

It's code that can be read by anyone, easily, with less likelihood of a mistake?

spice pecan
#

It might just be me, but I wouldn't call this behavior bizarre

main ginkgo
#

i dont find it bizarre personally. it makes sense if you understand how python works. and 'implicit' is context-dependent.

halcyon trail
#

you don't find it bizarre personally, but many people will. Nobody will find using an explicit global bizarre, or confusing.

#

And yeah, the behavior is bizarre when you have e.g. a dozen popular programming languages, and python is one of, if not the only, that behaves this way.
mutable default argument is quite literally considered a common "trap" for people learning python because this behavior is so surprising

main ginkgo
#

i guess that's fair enough if youre releasing code that needs to be read by others often

spice pecan
#

I haven't personally seen a lot of complaints about it. AFAIA, it's a mistake you might make a couple times when starting out, but not something you repeatedly forget about

main ginkgo
#

if im just writing something for myself, im not gonna split hairs about it though

halcyon trail
#

well, yeah, I would assume it's a given that your code needs to be read by others, if we even bother discussing which way to do it

#

otherwise can just hack it out however, as long as either way is correct

halcyon trail
main ginkgo
#

if it's a library/framework then i can see the need to keep cognitive load to a minimum. but if it's like, just you or a small team working on it and you all get it then i dont see the harm

halcyon trail
#

there was also a whole convo about this yesterday; because there's a PEP to fix it, so somebody found thsi behavior so irritating that they put in the effort ot write something up

spice pecan
#

That's a fair point, I'm just saying that it's usually something you grasp fairly quickly (albeit after stubbing your toe on it)

halcyon trail
#

yeah, it is definitely not rocket science, and I wouldn't have trouble grokking the code that used this for a mutable cache

grave jolt
#

I am being onboarded on a project of a small team... readability would be great

halcyon trail
#

I would just skim the code, double take a few times, and think that this code was too clever in a bad way, and move on

grave jolt
# grave jolt so... you propose doing this? ```py async def fetch_all( session: ClientSess...

I present to you... the penultimate hack. ```py
from typing import Callable, Any, TypeVar
from itertools import starmap
import inspect

C = TypeVar("C", bound=Callable)

def fix_default_mutable(fn: C) -> C:
sig = inspect.signature(fn)

replace_pos = {}
replace_kw = {}

def replace(param: inspect.Parameter, value: Any, index: int):
    if param.kind != param.KEYWORD_ONLY:
        replace_pos[index] = value
    if param.kind != param.POSITIONAL_ONLY:
        replace_kw[param.name] = value

for i, param in enumerate(sig.parameters.values()):
    if isinstance(param.default, dict):
        replace(param, MappingProxyType(param.default), i)
    elif isinstance(param.default, list):
        replace(param, tuple(param.default), i)
    elif isinstance(param.default, set):
        replace(param, frozenset(param.default), i)

fn.__defaults__ = tuple(starmap(replace_pos.get, enumerate(fn.__defaults__ or ())))
fn.__kwdefaults__ = {**(fn.__kwdefaults__ or {}), **replace_kw}
return fn
#

@verbal escarp I hope this is cursed enough to show you

#

I mean... yeah, that's what it protects against

#

it's for the cases like mine, where I don't intend to change the argument

verbal escarp
#

how does your @fix_default_mutable fix the "it's ugly to write if x is None.." argument - it's just wrapped in a different cloth ;p

grave jolt
#

it's ugly but now it's exciting

verbal escarp
#

lol 😉

#

right.

grave jolt
#

or just object(), yeah

#

or None

verbal escarp
#

@grave jolt while you're at wrapping your head around things, i have a quest for you

#

a while ago i tried to come up with a function that would extract the keyword params from inside the function it's called from

#
def all_kwargs(func, other_locals):
    d = {
        name: other_locals[name]
        for name, param in inspect.signature(func).parameters.items()
        if (param.kind is inspect.Parameter.KEYWORD_ONLY or param.kind is inspect.Parameter.VAR_KEYWORD)
    }
    d.update(d["kwargs"])
    del d["kwargs"]
    return d
#

is the function i got stuck with, which is called as

#
def foobar(*, a=2, b=3):
   print(all_kwargs(foobar, locals()))
#

i tried to simplify the function call to all_kwargs() but i couldn't come up with a solution

grave jolt
#

is it permissible to add a decorator to the function?

verbal escarp
#

sure

grave jolt
#

take out your hack-protection helmet

peak spoke
verbal escarp
elder blade
#

Dear God no

grave jolt
# grave jolt take out your hack-protection helmet

!e

import threading

def kwspy(fn):
    store = threading.local()
    def new_fn(*args, **kwargs):
        store.kwargs = kwargs
        try:
            return fn(*args, **kwargs)
        finally:
            store.kwargs = None

    new_fn.kwargs = lambda: store.kwargs
    return new_fn

@kwspy
def f(a, b, *, c, d):
    print(f.kwargs())

f(1, b=2, c=3, d=3)
fallen slateBOT
#

@grave jolt :white_check_mark: Your eval job has completed with return code 0.

{'b': 2, 'c': 3, 'd': 3}
verbal escarp
#

oh god, the first line says import threading

grave jolt
#

heh

#

doesn't work with generator functions and async, though

#

and will break with recursion

verbal escarp
#

holy crap, wow

#

ok, at the very least it's a creative solution

grave jolt
#

!e
What about this? ```py
def collect(fn):
def new_fn(*args, **kwargs):
return fn(*args, **kwargs, kwargs=kwargs)
return new_fn

@collect
def f(a, b, *, c, d, kwargs):
print(kwargs)

f(1, b=2, c=3, d=3)

fallen slateBOT
#

@grave jolt :white_check_mark: Your eval job has completed with return code 0.

{'b': 2, 'c': 3, 'd': 3}
grave jolt
#

!e
or ```py
def collect(fn):
def new_fn(*args, **kwargs):
return fn(kwargs, *args, **kwargs)
return new_fn

@collect
def f(kwargs, /, a, b, *, c, d):
print(kwargs)

f(1, b=2, c=3, d=3)

fallen slateBOT
#

@grave jolt :white_check_mark: Your eval job has completed with return code 0.

{'b': 2, 'c': 3, 'd': 3}
verbal escarp
#

that would collide with **kwargs

prime estuary
#

Hence why you need the positional arg.

grave jolt
#

yeah, so that's why I suggested the second one 🙂

#

this is 100% thread-safe and will work with generator functions and coroutine

#

s/will/should

verbal escarp
#

hm.. i like it, it's simple

#

not exactly what i asked for, but maybe it'll work well enough to replace the in-function thing

#

thanks

grave jolt
#

why do you want to do this in the first place? @verbal escarp

verbal escarp
#

ah, because of my buffet pattern

grave jolt
#

verbal escarp
grave jolt
#

this must be a Russian buffet because my stomach audibly aches after smelling it

verbal escarp
#

basically i flattened a big function, split it up into many smaller ones which all get the same **locals(), from which each picks the kwarg they want, then pass along the rest

#

it simplified a lot of code and the signatures only take kwargs, which makes working with those functions very nice

#

although it's a bit of a rollercoaster if you try to follow a debugger

paper echo
#

@verbal escarp your buffet pattern makes me think of ocaml 🙂

verbal escarp
#

why's that 😄

paper echo
#

idk about abusing locals like that

#

but the overall appearance resembles pattern matching

verbal escarp
#

yeah, someone proposed a patma function as a replacement, which was nice

#

but i can't use it just yet, supporting py3.8

#

(no pun intended)

grave jolt
#

hey, another hack! asynchronous yield from

#

well... almost

#

doesn't support sending

halcyon trail
#

the people here need to like, NFT, or encrypt, or something, the awful hacks they come up with, so that there's no chance of someone lurking in this channel, and using it in my codebase 🤣

peak spoke
#

!e should work for everything if I copied from inspect correctly

import sys

def all_kwargs():
    kwargs = {}

    upper_frame = sys._getframe().f_back

    locals = upper_frame.f_locals

    code = upper_frame.f_code
    pos_count = code.co_argcount
    arg_names = code.co_varnames
    keyword_only_count = code.co_kwonlyargcount
    kw_only_names = arg_names[pos_count:pos_count + keyword_only_count]

    kwargs.update((name, locals[name]) for name in kw_only_names)

    # **kwargs
    if code.co_flags & 8:
        index = pos_count + keyword_only_count
        if code.co_flags & 4:
            index += 1
        kwargs.update(locals[arg_names[index]])

    return kwargs

def eggs(*, a=2, b=3):
    print(all_kwargs())

def spam(a, /, b, c, *, d=0, e=0, **kwargs):
    print(all_kwargs())

eggs()
spam(0, 1, 2, d=3, e=4, f=5, g=6)
fallen slateBOT
#

@peak spoke :white_check_mark: Your eval job has completed with return code 0.

001 | {'a': 2, 'b': 3}
002 | {'d': 3, 'e': 4, 'f': 5, 'g': 6}
verbal escarp
peak spoke
#

It's all in the inspect source fwiw, just needed the irrelevant parts taken out of it

verbal escarp
#

obviously 😉

surreal sun
#

How does inspect.getsource work? does it just read the module and string match?

#

It seems like it doesn't work in the REPL either, so I'm guessing it does read the module

peak spoke
#

Yes, it goes through the file; the source is not present in any way in the compiled code so that's the only way

surreal sun
#

oof

white nexus
#

So uh.
That lambda problem I had...

#

!e ```py
lambda *args, **kwargs: 5

fallen slateBOT
#

@white nexus :warning: Your eval job has completed with return code 0.

[No output]
white nexus
#

Basically uh, I could've just put that as the lambda signature

#

Not really advanced, but quite the opposite, it was something really simple in solution

hollow aspen
#

do ya'll think you should use Java in vsc?

unkempt rock
snow bronze
#
>>> b=0
>>> exec(a:='[globals().__setitem__("b",b+1),exec(a)]')
  [Previous line repeated 509 more times]
RecursionError: maximum recursion depth exceeded during compilation
>>> b
512

>>> b=0
>>> (a:=lambda:[globals().__setitem__("b",b+1),a()])()
  [Previous line repeated 1022 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object
>>> b
1024

Can someone explain why does only half of the recursion limit occurs before it stops?

raven ridge
#

I believe exec compiles the statement you give to a code object, and then calls that code object as a separate stack frame. So every iteration you have 2 calls, one to exec and one from exec into your code.

snow bronze
#

Oh I see.. that makes sense now

>>> def a():
    global b
    b+=1
    c()
>>> def c():
    a()
>>> b=0
>>> a()
RecursionError: maximum recursion depth exceeded
>>> b
513
snow bronze
raven ridge
#

I'm not sure why those don't match. I'd have expected them to be off by 1 at most.

white nexus
#

Depending on where it's run, the stack might already exist

#

They do match

#

1023 - 1 == 1022
1023 // 2 = 511 - 1 = 510, plus one more for what @raven ridge said with the double stack, it only got through one of them and not both

raven ridge
#

What doesn't match is the "repeated 509 more times" message with the counter showing 512 increments happened

white nexus
#

O

#

Halfway through the line is my guess

raven ridge
#

Yeah - on the last iteration it performs the increment and then isn't able to create a new frame for the exec call - that's a difference of 1. And "Previous line repeated" implies a first line, so that accounts for another 1, so it should be off by 2. And in the 1024 case it was off by 2, but in the half that case it was off by 3 instead

snow bronze
#

like

>>> class a:
    def __del__(self):
        global b
        b+=1
        a()

        
>>> b=0
>>> c=a()
>>> del c
Exception ignored in: <function a.__del__ at 0x000002034617A8B0>
Traceback (most recent call last):
  File "<pyshell#6>", line 4, in __del__
KeyboardInterrupt: 
>>> b
3927640
red solar
#

__del__ is special 🙂

raven ridge
#

That's not recursion, then. There's no way you could be making 3 million recursive calls without the interpreter crashing with a stack overflow.

#

RecursionError is just there for safety and convenience - you can disable it entirely, but if you do, the interpreter will eventually crash instead of raising an exception

red solar
#

lol really?

raven ridge
#

Sure. Stack is a limited resource, and there's no portable way to know how much of it is left. Eventually you run out and get a segmentation violation.

red solar
#

SEH? and maybe linux has an equivalent

#

Python definitely checks stuff with SEH on windows

snow bronze
raven ridge
#

So RecursionError is a safety valve that says "assume something that makes more than N recursive calls is doing something wrong", because the interpreter doesn't know how many more calls it could possibly make successfully.

red solar
#

I always just imagined if you disabled it, the interpreter would still check and raise it if you were about to segfault

#

or maybe raise an OOM exception (idk if python has one)

raven ridge
red solar
#

oh so that 🙂

raven ridge
red solar
#

damn :/

#

ig windows doesn't really have an OOM killer?

#

wym?

#

a prototype of what?

snow bronze
snow bronze
red solar
#

64gb

snow bronze
#

🤯🤯

red solar
#

chill, i couldn't find a gpu so i spent more money on ram lol

snow bronze
#

and here was myself bragging bout my 16GB ram lol

red solar
#

which is perfectly fine for 99% of things

reef spindle
#

@fossil pumice @wide shuttle @granite heath @oblique raven @sturdy timber

red solar
#

the hackathon?

#

oh, well congrats 🙂

#

so you have an idea, and now you need to create a proof of concept?

#

ok - i'm not sure how i'd be able to help though

#

you want to make the proof of concept in python? do you know python?

#

aww @oblique raven you kicked him? 😦

oblique raven
#

No

red solar
#

oh then he left

oblique raven
#

Yup

red solar
#

i'm not sure - i still haven't understood what your issue is - you don't know how to use a specific AI thing? or you have no idea how to get started at all?

#

really? usually these challenges give you some testing environments

#

then what's the issue? why can't you use it?

#

i'm not

#

what's the issue? you have an account, you have free access to the cloud, why can't you deploy?

#
class bdist_wheel(_bdist_wheel):

    def finalize_options(self):
        _bdist_wheel.finalize_options(self)
        self.root_is_pure = False
#

would me doing super().finalize_options() be identical to _bdist_wheel.finalize_options(self)?

red solar
#

damn :/

#

what's the difference?

grave jolt
#

it will be different with multiple inheritance AFAIK

red solar
#

hmmm

#

this is single inheritance tho

peak spoke
#

if there's no mro changing magic in _bdist_wheel they should act the same

red solar
#

hmm... ok fingers crossed there isn't then and i'll go with super() cuz it looks nicer 🙂

elder blade
# red solar this is single inheritance tho

But if someone makes another subclass, like this: ```python
class bdist_car(_bdist_wheel):
def finalize_options(self):
print('Finalized')
super().finalize_options()

class Subclass(bdist_wheel, bdist_car):
pass

red solar
#

bdist_car 😂 😂

elder blade
#

The MRO is now:

  • _bdist_wheel
  • bdist_car
  • bdist_wheel
  • Subclass
red solar
#

it's alphabetical?

elder blade
#

Oh no, that's just a coincidence

#

So the bottom-most inherits in the order I listed

red solar
#

ah i'm blind, i somehow didn't even see Subclass

elder blade
#

Subclass inherits from bdist_wheel which has been changed to inherit from bdist_car which inherits from _bdist_wheel

#

So when you call a method Python will look it up in the reverse order from what I listed. Starting with Subclass and ending with _bdist_wheel

#

But as you notice bdist_car has been injected into the MRO, as bdist_wheel's parent isn't _bdist_wheel (well it's the parent's parent)

red solar
#

hmm... i think i actually want super() then even in that case

#

even though it's not the same

#

not sure why the example i copied didn't use super()

elder blade
#

Yes, good. This is actually why people use super(), even though they know their parent.

red solar
#

how do i figure out which version of stuff like setuptools and wheel i need to require?

#

or do i just not put a version requirement?

red solar
#

eh screw it i guessed

lusty scroll
#

cool hack. seems like a "FrozenList" would work here too as a default argument. thinking why we have frozenset but not frozenlist

unkempt rock
#

I guess there was no use for it, heh

spice pecan
#

frozenlist = tuple
There you go :P

halcyon trail
#

I think I pretty much did that as a type alias, and then never used it 🙂

lusty scroll
#
class frozenlist(tuple):
    pass
#

too bad it says __main__.frozenlist

spice pecan
#

!e ```py
class frozenlist(tuple):
module = frozenset.module

print(f"{frozenlist}, {frozenlist!r}")

fallen slateBOT
#

@spice pecan :white_check_mark: Your eval job has completed with return code 0.

<class 'frozenlist'>, <class 'frozenlist'>
spice pecan
#

You can manually set it to 'builtins' as well

lusty scroll
#

are there other differences between a user-defined type and a builtin type?

spice pecan
#

Depends, most (if not all) built-ins types are implemented in C

#

That'd be the biggest difference ig

lusty scroll
#

if there are no new methods, it could plausibly be a C type?

spice pecan
#

I'm not sure about the details here, frankly. I'd guess that it'd be a Python type that proxies everything to its C parent

#

!e You can probably assign arbitrary attributes after subclassing

class Sub(tuple): pass
Sub().new_attr = 10
fallen slateBOT
#

@spice pecan :warning: Your eval job has completed with return code 0.

[No output]
spice pecan
#

Yeah, it seems to be a regular Python-defined type

#

Just one that happens to rely on it's C-implemented parent

lusty scroll
#

I do see one thing I'm not sure of:

    '__new__':  <built-in method __new__ of type object at 0x7ff7f88950>
#

frozenlist does't have __new__ in its class dict. Just wondering if that's something a builtin type would have

#

the above was in tuple class dict

elder blade
red solar
#

do frozen containers have any optimisations compared to their regular counterparts?

elder blade
#

!e ```python
print(int.slots, int.weakref, int.dict)

fallen slateBOT
#

@elder blade :x: Your eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 1, in <module>
003 | AttributeError: type object 'int' has no attribute '__slots__'. Did you mean: '__class__'?
elder blade
#

On the other hand, all user defined classes either has a __dict__ or __slots__ this is because they're not defined in C of course

elder blade
#

Memory isn't freed perfectly after it's removed

spice pecan
#

I'd say one of the most important properties is hashability

red solar
#

smaller memory in that the PyObject takes up less memory, or it also allocates stuff more efficiently too? (like having FrozenSet be contiguous or something)

elder blade
#

!e ```python
from sys import getsizeof

d = dict()
print(getsizeof(d))

d['a'] = 1
print(getsizeof(d))

del d['a']
print(getsizeof(d))

fallen slateBOT
#

@elder blade :white_check_mark: Your eval job has completed with return code 0.

001 | 64
002 | 232
003 | 232
elder blade
#

See how it stayed the same, even though we removed an item?

red solar
#

yes

#

but frozendict just doesn't let you modify it?

#

(wait does it?)

elder blade
#

If you modify a dictionary it may have some extra overhead even though you don't store any more items in it

spice pecan
#

tuple would be smaller than a list because it allocates exactly enough memory for its elements, no need to overallocate for amortizing appends. Not sure about frozenset, since overallocating is not only done to amortize new additions, but also to reduce hash collisions

elder blade
red solar
#

oh

#

really?

spice pecan
#

Yeah, there's mappingproxy, which gives you an immutable view of a dict, but it's not exactly a frozendict. IIRC it's still unhashable

red solar
#

tbh i thought there was a bunch of frozen containers, not just set :/

spice pecan
#

Well, tuple is effectively a frozenlist

#

But there aren't that many containers to begin with, tbf

#

Yeah, mappingproxy is still unhashable, so it's just an immutable view

elder blade
#

Okay but when would you store a dictionary as a key in another dictionary lmao

spice pecan
#

You may want to store it in a set

#

It's not an entirely unreasonable use case

#

There was a rejected pep that tried to introduce frozendict

#

!pep 416

fallen slateBOT
#
**PEP 416 - Add a frozendict builtin type**
Status

Rejected

Python-Version

3.3

Created

29-Feb-2012

Type

Standards Track

red solar
#

Multiple threads can agree by convention not to mutate a shared dict, there's no great need for enforcement. Multiple processes can't share dicts.

lusty scroll
#

wait, you can look at the class dict for all types, right? they don't have an instance dict though

red solar
#

not sure how i feel about that reasoning

#

sure they can agree, but enforcement is always nice

spice pecan
lusty scroll
#

ah I see

spice pecan
#

!e py print(repr(tuple.__dict__)[:100] + '...')

fallen slateBOT
#

@spice pecan :white_check_mark: Your eval job has completed with return code 0.

mappingproxy({'__new__': <built-in method __new__ of type object at 0x7f51a54eefc0>, '__repr__': <sl...
lusty scroll
#

I thought that was just to make it immutable

spice pecan
#

It is

#

You can't alter it directly. You can get the underlying mutable dict with some gc hacks IIRC, but that's... very dirty, to say the least

#

!e ```py
import gc
d, = gc.get_referents(tuple.dict)
d['index'] = print
().index('Hello, world!')

fallen slateBOT
#

@spice pecan :white_check_mark: Your eval job has completed with return code 0.

Hello, world!
lusty scroll
#

lol 😂

spice pecan
#

This screams "Please don't do this"

lusty scroll
#

the referents are indexed by name?

elder blade
#

No it's an iterable

#

You see how Genshin Khan is unpacking it to d,

lusty scroll
#

ah!

spice pecan
#

Yeah, it's just a list

#

With one item, that I immediately unpack into a tuple of 1 element

lusty scroll
#

I missed the ,

spice pecan
#

It's more readable if you do [item] = collection or (item,) = collection, but I basically copied it from my REPL, so yeah

lusty scroll
#

nice

#

I think frozendict sounds pretty useful, especially if it's hashable

halcyon trail
#

i admit it's not often I've ever wanted to use a dict as a key to another dict

#

on the other hand I'm not sure why they don't have it

lusty scroll
#

I've run into it a few times I think

halcyon trail
#

I can't find an obvious reason why frozenset is there but not frozendict

#

it seems like set was simply standardized later, and perhaps by a different person, and the value of immutability was perhaps better understood/appreciated

#

so frozenset just came in with set

#

it's easier I guess to add something incrementally as part of a PEP than get a whole new PEP accepted

lusty scroll
#

making mappingproxy hashable could work too

#

but I guess that'd be kind of unorthodox

#

A frozendict is hashable if and only if all values are hashable.

#

the trouble often seems to be lists

#

I guess you'd basically need a utility function to recursively copy the dict, changing every list to frozenlist and every dict to frozendict, which doesn't sound great

halcyon trail
#

i don't understand

lusty scroll
#

if you wanted ti make a hashable dict, but there were e.g. list vakues, or other regular dicts

halcyon trail
#

how is a frozendict being hashable iff values are hashable, different from a tuple being hashable iff elements are hashable?

lusty scroll
#

it isn't really I guess

halcyon trail
#

i think this problem is solved by error'ing out, not by recursively copying anything, at present

lusty scroll
#

it would be nice to have an easy way to make an immutable+hashable copy of the data

halcyon trail
#

In [1]: {([1,2,3], [])}                                                                                                                     
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-430e4d202ea5> in <module>
----> 1 {([1,2,3], [])}

TypeError: unhashable type: 'list'
#

it gets tricky once you have class types

lusty scroll
#

yeah, that's what I meant with it not being that practical

halcyon trail
#

you could write a deep copy that specifically converts list to tuple, set to frozen set, etc, while its copying

#

I'd prefer more support for working with immutable types in the day to day, personally, but this comes back a lot to th elack of support for more functional, expression oriented stuff in python

#

python is very imperative so outside of a small amount of sugar it encourages you to create collections and mutate them a lot

#

and while with comprehensions you avoid mutating something, they are unfortunately hard coded to create something mutable... so they are part of the problem, really

pliant tusk
#

there is hamt

#

nevermind that isnt hashable

vapid lotus
#

!e

fallen slateBOT
#
Command Help

!eval [code]
Can also use: e

*Run Python code and get the results.

This command supports multiple lines of code, including code wrapped inside a formatted code block. Code can be re-evaluated by editing the original message within 10 seconds and clicking the reaction that subsequently appears.

We've done our best to make this sandboxed, but do let us know if you manage to find an issue with it!*

vapid lotus
#

oh

#

There is some bot command channel

#

?

#

#bot-commands

#

found

main ginkgo
#

all the times ive needed a frozendict its only been for the hashability, which frozenset(dict.items()) covers well enough. so for me that really only leaves subscriptability (without the memory footprint of a copy) as a selling point. i did see a frozendict module somewhere thats implemented in c.

spice pecan
#

There are several

#

!pypi frozendict

fallen slateBOT
halcyon trail
#

if you just want the hash you may as well just do tuple(dict.items())

#

if you can't meaningfully use the in operator on the set then not much point paying for hashing etc the items inside

main ginkgo
#

imo the hash should be independent of key sequence, but i suppose that's different now we have ordered-by-default dicts

halcyon trail
#

they are ordered by default but I would expect their hash is still independent

#

so your point seems valid to me

lusty scroll
#

but it's not guaranteed, right 🤔

halcyon trail
#

it is guaranteed

#

as of 3.7 iirc

#

but only for dicts

#

not for sets

main ginkgo
#

was implemented in 3.6, made official in 3.7

#

i think?

raven ridge
#

Yep

halcyon trail
#

but the tuple and set are legitimately different in that with the tuple, different order would give a different hash, but not for the frozenset

#

I guess it depends what you're using the hash for

#

or how you're using it

lusty scroll
#

so list(d.items()) should always produce the same list? (if the dict isnt changed)

halcyon trail
#

well, in practice that would always have been true

main ginkgo
#

!e

print({1:2, 2:1} == {2:1, 1:2})

another argument in favour of hash being independent of order, at least in terms of consistency

fallen slateBOT
#

@main ginkgo :white_check_mark: Your eval job has completed with return code 0.

True
main ginkgo
#

dont get the same thing with tuples

halcyon trail
#

Yep, I said that a bit ago. But it really depends, like I said, how you're using it.

main ginkgo
#

eh, i think it's a bit more than that

halcyon trail
#

@lusty scroll actually, I take that back, list(d.items()) will produce the same list for any reasonable implementation... in a given execution. But between different executions it might not, because of salting

#

different executions meaning, runs of the whole program

main ginkgo
#

it would be weird if two dicts with different ordering were considered to have the same content (which i suppose is effectively the meaning of a hash) but then frozendicts didnt

halcyon trail
#

Yeah, no disagreement, my point was only that depending what you're doing, you possibly could just use the tuple version instead, but you're right that the frozenset is more broadly applicable since it preserves that property

lusty scroll
#

PYTHONHASHSEED would take care of that?

halcyon trail
#

what do you mean by "take care of that" ?

lusty scroll
#

eliminate the difference between runs

halcyon trail
#

sure if you fixed that

#

but a) you wouldn't typically, and b) it's still not something to depend on in any way, it's an implementation detail so it can change e.g. when you upgrade python, without warning

lusty scroll
#

fair point

halcyon trail
#

beyond not being guaranteed to change, it's also not any useful order

#

whereas now the order is both guaranteed, and something at least potentially useful

#

though I've never used that property ever, that I can recall

#

I always forget even if changing an existing element changes its order location or not

lusty scroll
#

if you're doing reproducible builds, you would

halcyon trail
#

why would you depend on this for reproducible builds?

lusty scroll
#

on hashes ?

#

it's not about depending on it, but removing uncertainty that could affect the output of your process

raven ridge
halcyon trail
#

right. that, to me seems like a practical compromise for performance reasons, as opposed to ideal behavior for an "insertion order" dict

#

which is totally fine; my understanding is that one of the main reasons why dict got the ordering guarantee was because it was basically free performance-wise anyways

sly jay
#

pretty much yea

#

I don't think Guido would have approved it if it wasn't free

halcyon trail
#

nice username btw, can't stop re-reading it lol

#

but yeah, IME in most cases you want some kind of "insertion order" dict, you usually want modifications to bring it to the top

sly jay
#

I don't keep up with all of his posts or anything but from what I've seen, he really doesn't like making changes which cause the same operation in future python versions to be slower than previous versions

#

same or better is ok, slower is not. That's one of the reasons why cpython still uses the GIL even though basically no computers worth running python on are single core nowadays

halcyon trail
#

yeah, I"ve heard that angle wrt the GIL as well

sly jay
#

He's said he would be open to removing the GIL if a good solution is proposed which is at least as fast in singly threaded code

halcyon trail
#

Yeah, there's a pretty serious proposal for that now

sly jay
#

I think he also doesn't like lazy garbage collecting either

#

which is fair, neither do I.

halcyon trail
#

idk, in practice I haven't found python's eager GC to be useful in any way

sly jay
#

I think we should just eat the singly threaded perf loss but sadly not gonna happen

halcyon trail
#

even in cases where it looks like large allocations should be returned immediately, they haven't, and still caused OOM issues

red solar
#

Eyyyy @sly jay you’re here too 🙂

#

Lol soon this channel’s gonna be all C++ ppl

sly jay
#

lmao

halcyon trail
#

oh what C++ channel are you in?

red solar
#

TPH server

halcyon trail
#

ah don't know that one. damn there's a lot of these eh?

red solar
#

A few lol, tph is more general tho, they do most languages

halcyon trail
#

I think anything wrt performance in python is a rough ride..... I doubt that we will ever again see a new, popular, language, as slow as python

#

or Ruby

sly jay
red solar
#

Ahhh that’s why you’re in llvm, was wondering

#

Unfortunately I don’t :/

sly jay
#

I started writing a function which traces through a macrodef and all its members to dump all their data but I feel like that's a hacky crappy solution

red solar
#

Dm chandler on Twitter, he might reply 🙂

sly jay
#

might try it, thanks

lusty scroll
#

I guess the same complaints would apply about reading the argument names that make up the tuple (even though they'd be positional-only, you still want access to the name for help() and such)?

main ginkgo
#

didnt know you could do that in py2, thats cool

full carbon
#

Anybody with digital twin projects? I have some questions. Thank you kindly

lyric tusk
#

@lyric canopy kn

#

Which is better
Python django VS HTML | CSS | JS

red solar
#

I don’t think that’s an apt comparison? Wouldn’t it be more like django vs spring boot? (Either way idk)

red solar
#

If I create a Python specific wheel for say Python 3.6 (but it’s not abi specific), would I then be able to install that on any Python version from 3.6 to <4?

#

Or do I still need to build each Python version separately even tho it would be the same wheel?

lusty scroll
#

depends

#

you want the interpreter to be py3 I believe

#

like yourpackage-1.0.0-py3-none-any.whl

red solar
#

Ugh :/ then i’d have to get rid of all my type hints tho to make it compatible with all py3 versions

elder blade
#

Or are you using the C API?

red solar
#

Uhhhh

#

Complicated