#internals-and-peps
1 messages · Page 143 of 1
probably not, for the same reason a lot of other syntactic sugar is being deferred
to me, ?= made more sense given how ? is more symbolic as to 'none-aware' and in this case = is like "set this argument if the argument isn't given"
Ah true, None-aware operators got deferred too :(
I was really looking forward to that
A lot less of boilerplate code ngl
@halcyon trail maybe => is good syntax because the late-bound default argument is an anonymous function that gets called upon use 🙂
But this has nothing to do with None
the argument isn't given isn't the same as None
(or even related to None, tbh)
i liked it because it's like ?=in Make
actually.. what if the deferred function returns None?
I'd much rather they keep ?=, in case we actually get non aware operators later
in that case we could combine it
I think none aware operators may have a shot, one day, these operators are actually becoming pretty pervasive
Eh, they are two different things
def foo(x => {bar()} ?=24):..
Liking something because of make...tsk tsk 🙂
this syntax is awful
I'm pretty sure that Dante's inferno has a circle for that
what would ?= be in none-aware world?
And yeah, that syntax is not so great, I don't even understand what you're intending up there
if this is what late-bound defaults look like, i hope they never make it in
no lol
x = bar(); if bar() returned None: x = 24
i hope ?= would be an inplace "unwrap" sorta thing
also first of all Make is great
😕
i could live with that
JS has ??, it's nice
make was designed to be a language for writing build files, that is so unsuccessful, that people have invested a zillion man hours in writing ways to generate the files automatically
how about using something other than parens after def to signify looking in the parameter space before normal locals -> globals stuff
it's not about namespaces, it's about late evaluation
(it will take a very long time, don't misunderstand, but that's the direction)
yes
imo that's not what it's good for, it's no substitute for a real build system
but it will consider a name existing in the parameters as something that will need to be late-bound
yes, but also expressions
what is it good for? honest question
Without fail, every time I've seen make used for literally anything at work, in the last 10 years, it was always a bad choice
i use it the same way you would use tox or invoke
I have never used tox or invoke either
eg make run-dev could run your web server in development mode
so could a python script, or many other things
make test invokes pytest
sure, but not if you have intermediate assets that you need to generate
if I did ever want a DSL for specifying dependencies between tasks, why wouldn't I use a library in a real programming language?
this is exactly how the actual successful systems for doing similar tasks, that I've seen, work
counterpoint: make is a perfectly good dsl and shell is a fine enough programming language for basic tasks
if it gets complicated or difficult to maintain, use a different tool
there's no reason to use what is now obscure technology, for a poorer tool, because the poorer tool is "good enough"
you can use a tool that is both better and more easily familiar
if you want an example if dependency specification, albeit with a different focus, you can look at something like snakemake, which is written in python
i disagree that arbitrary python code + some random library is good enough
snakemake sucked
i tried to use it for years
this is how literally every programmer I know (barring 1) feels about make
i'm sure snakemake isn't perfect, but it's found a fair bit of success, and basing your task specification in an actual language seems by far the more popular way to go these days
meh, I've seen some pretty horrible code-as-config setups
anything turing complete can be turned into a clusterfuck in undisciplined hands
and some things that aren't turing complete too
Yes, so you may as well start with an actual languages, so that when people fuck around, it's at least reasoanble 🙂
i like my attrs approach to configuration >.>
shell is an actual language!
Seriously though, every example I can think of that went the pure DSL route, i.e. not based on an actual language, ended up in disaster
like i said, i agree that make is awful as a front-line build tool
everybody hates the Cmake language in the C++ world. It's actually the only major complaint about it, that you hear over and over.
The actual build system is good, it's pretty fast, it's well maintained, it's supported, etc.
barely 😉
You have to be a masochist to write more than a couple dozen lines of bash these days (usually for things related to environment that just can't be done any other way)
i agree, I don't
but 1-2 lines in a makefile, fine by me
yeah i've heard a lot of hate about cmake, i certainly don't understand any cmake files ive read
meson seems to be the new cool thing
nah meson isn't going anywhere most likely
it would be cool if it did but the amount of inertia behind CMake at this point is crazy
point being: make is a perfectly good tool for what it is, which is to specify a dependency graph and some shell scripts for generating them
But it’s kind of similar to where it’s the absence of the argument
it has some correlation imo
it'd be great to have something like make without all the sharp edges, but snakemake is definitely not it imo
it'd be nice if they exposed make functionality as a C api though
But it's literally not a good tool for that. Or at least, very few people seem to think so, because very few people are using it.
if someone tells me "yeah we use make and some shell scripts to do XYZ" my first thought is probably just that the code has been around for a long while
(I should say, few people are using it as a green-field choice, if they're using it it's because there's existing code, or at best new code that's being integrated with existing systems)
i use it as a default for ad hoc small things
and like i said as a tidy uniform entry point to other scripts
Salt off topic but were u using pypy or graalvm?
both but only for playing around
Can I ask u some questions about graalvm? (Not sure whether here or another channel)
yes but i know absolutely nothing about how graalvm works 😆
And also cmake is dope when u get a project with a properly written cmake file
That’s ok
I was just wondering if it has cffi support
for example @halcyon trail we use makefiles to paper over the differences between our node and python services, and between our old shitty vagrant services and the new shiny docker compose ones. we could use just a shell script or some other tool but literally everyone here has gnu make already installed and knows the syntax
good question but no idea
Can you not check? (Like just import cffi)
not at a computer at the moment... maybe something on their website
i doubt it does tbh, but it probably has something else via graalvm itself
Hmm ok i’ll try and check
Does sys.modules['ctypes'] = None actually prevent further code from importing ctypes?
Something tells me it's not that easy.
Well I’m gonna guess ctypes gets overwrite when imported
iirc you might need an import hook for that
Except that import module doesn't reload module
that would, so you'd also have to block importlib
they might del sys.modules["ctypes"] and then import
that would do it
Or even just pick an already imported ctypes module from somewhere in the stdlib
So yeah an import hook would be necessary, followed by del importlib so that the code that follows wouldn't be able to redefine the hook.
Right. I think the way to go is to block all but a few imports.
Well, you can access some modules without importing
Effectively I want to embed Python as a scripting language and block filesystem access, along with all but a few okayed modules. (No, nothing can change my mind that I want Python for this 😝)
Build a custom embeddable package with as few modules as possible for starters
You can compile python without ctypes etc
I had forgotten about compiling a different interpreter, not sure if it would be within my skillset...
But, that's what project-based learning is for :D
In the end, you aren't getting a useful sandbox by writing python code
@heady mauve it is very difficult to secure Python
So I think the way to go based on what I've heard here is to compile a python interpreter that cannot run ctypes, os, etc.
So I've read. Everywhere.
well, have a look at a the discord bot
I don't think you can get rid of some of the more code modules
Runs on linux and uses OS-level sandboxing
Do you mean secure python using python specially? Because I know snekbox uses nsjail.
yeah, that
is that bad?
I do need Windows support
nsjail works, but if they want to embed it as a scripting Language, that's not a great solution.
Windows does have their own sandbox in pro IIRC
can you tell us more abut the big picture here? What's the overall system?
If I could fire up a subprocess with a modded interpreter to run the embedded scripts, that would work beautifully.
@heady mauve needing windows support sounds like you're going to run this on users' machines?
okay, so why is accessing ctypes bad in your case?
Overall, it's going to be a game with game objects that can execute arbitrary Python code. Unlike projects like Logical (logic gate sim that needs file access) I do NOT want filesystem access for these scripts and I want most libs blocked (socket, ctypes, anything that can act outside the game script). The scripts do need to be able to interact with the main process to control game objects.
With ctypes you can grab pointers to arbitrary memory locations and if there's a python object there you can load it into the current context.
and you don't trust the game object code? Who writes it?
you fear cheating from players or do you fear players hacking other players?
For the game object code, I write it any anybody who contributes to my repo. For the script code, anybody can write it, most often end-users. Scripts will be shared.
I don't want my game to be a malware spreading system.
then you need proper code reviews
yes
@heady mauve Python might not be the right tool for this. it will be difficult to prevent access to things
That's what everyone has said so far
actually i wouldn't say so
do you have a way to secure it?
I think that if you patch the interpreter itself to permit or ban certain things then you can do pretty much anything
yeah, you could define classes as API and check the AST of the modules before executing them to only access those classes
And by doing a subprocessed interpreter, you can cut out bits of the standard library for that specific interpreter to block things like networking.
these are interesting theories, but have you tried it?
Okay, I need to read up on AST
well, you can filter the AST, right? have a positive list of allowed objects and anything outside of that list raises a red flag
@verbal escarp what would your AST filter do with this?
(lambda fc=(lambda n: [c for c in ().__class__.__bases__[0].__subclasses__()
if c.__name__ == n][0]): fc("function")(fc("code")(0,0,0,0,"KABOOM",(),
(),(),"","",0,""),{})())()
what even is that nebat??
hey...i saw that in a blog once :)
forkbomb?
it's the kind of code that gets through defenses
it wouldn't match any objects on my positive list and wouldn't execute :p
it creates a code object with some wacky initial values so it does bad things
the heck....?
the only objects that has are function objects, a tuple, some strings, ints, and a dict
but i wouldn't allow access to __class__ so..
you dont need access to __class__
[...] ().__class__[...]
load_addr = type(m:=lambda n,s:lambda v:s(v)or n)(
(M:=m.__code__).replace(
co_code=b'\x88'+M.co_code[1:]
),{}
)(r:=iter(range(2**(PTR_SIZE*8-1)-1)),r.__setstate__)``` this leverages a different bug to be able to load arbitrary addresses
yep, right here
(my point is that there will always be another way to break it)
@verbal escarp can i use getattr?
sneaky
well, yes. sneakiness is the whole point
for a pure API, i would say no getattr needed
how small a subset of Python will still be OK for these game objects?
Just enough to do basic data manipulation and call some functions
ie, I would like full OOP, but the only outside contact the script would have would be through stdin/stout. These would be parsed by the host program and handled from there.
i'd think checking the AST is your safest bet
@heady mauve @verbal escarp without major changes to the interpreter, you will not be able to guarantee safety
likely
i disagree. hold on, i remember a youtube talk by an australian on that..
what would I need to fix aside from blocking certain modules and builtins?
let me look up the talk, maybe that can answer you some questions
attribute access, closures have some memory bugs you can mess with, local variables can be exploited, exception frames can be abused .... (etc)
dang
and those are all the ones that are difficult to exploit
will you allow importing modules?
so long as the code can't break out to C I should be able to put up stopgaps
only a select few
which ones?
re, configparser, things like that
And a custom library to simplify interfacing with the host program
what about the modules that re and configparser need?
not sure what they need
i think https://www.youtube.com/watch?v=Xd7YnPNZlbk that was the one
(Simon Carryer) Abstract Syntax Trees are a tool for analysing and manipulating the structure and function of code. They can be used to do about a million different useful and smart things. This talk is not about those things.
https://pretalx.com/pycon-au-2021/talk/MFUNXT/
python, pycon, australia, programming, conference, technical, pyconline...
if you're letting the players write arbitrary code, you can't let them have re either
what's wrong with regex parsing?
from collections.abc import MutableMapping
from collections import ChainMap as _ChainMap
import functools
import io
import itertools
import os
import re
import sys
import warnings``` these are from configparser
oh, no! https://www.youtube.com/watch?v=7e-2nUqnNqE&t=3s that was the one
Tim Savage
Allow your users to supply queries or define rules using Python syntax and safely eval them. Processing an AST into safely executable code.
https://2019.pycon-au.org/talks/safely-eval-python-syntax-using-the-ast-module
PyCon AU, the national Python Language conference, is on again this August in Sydney, at the International Convent...
both talks are cool though
DOSing?
basically make a regex that is super slow to create and to check
I'm not worried unless the script can cause damage on the user's computer. If the script hangs, I'll kill it.
iirc in the talk of tim savage it's about python in a banking environment
as in eval() in banking
which should give you some idea of how secure it can be
banking like handling money banking or something else?
yes
@verbal escarp that neuters python to the point its almost unusable to get any level of safety
you're welcome. i don't recall the details though, so enjoy the talk ^^
that I am
if I remember correctly, the google re2 library is specifically designed to be resistant to malicious regex
and has python bindings
!pypi re2
~~woah it fuzzy matches too?~~nvm 😔
don't quote me on the safety thing, check in the docs
i encourage you to keep working on it, but dont be disappointed when you have to sacrifice major usability for safety (or vise versa)
theres a reason there are languages designed for this exact use case (lua)
right
see this 😉
dont be suprised when people like myself find holes in it then
Surprised that you found it, I won't be. Surprised that that's what it was, you can bet I probably will be.
i'd love to see the battle of wits on that 🙂
lol
unless you plan on sandboxing it properly using something like https://github.com/python-discord/snekbox
like an arms race lol
again, I need windows support and snekbox uses nsjail that can't run on windows
hence the something like
to clarify I need windows and linux support
I did see cuckoo sandbox but I didn't see much on how to set it up
which kind of is the point - reduce things to the bare minimum which is still python to basically get a DSL
you wont be able to allow things like print()
which probably isn't needed anyway
print will have to be redone anyway
my point is that if something as simple as a bare function call will have to be disabled, what useful things will it be able to do
function calls would be permitted. I don't see why not
so long as they can't call malicious code
how will you determine that
you can abuse call frames to access some weird internals
^
if you can't access call frames?
how will you block them?
can they be accessed without a lib?
yes
how?
probably via an exception
tracebacks
hmm
!e py try:raise except Exception as e:print(e.__traceback__.tb_frame)
@pliant tusk :white_check_mark: Your eval job has completed with return code 0.
<frame at 0x7f9707c797d0, file '<string>', line 2, code <module>>
ok, so if untrusted code gets access to a frame what can it do?
step up it, modify variables
ok
use it to get access to other modules
if those modules literally are not here it cant :P
(configparser needs os which can be leveraged to write into memory)
youre missing the point tho, that is just one example
and there are already countless security holes
@heady mauve try this: python -v -c "print(1)" it will list the modules Python imports just to get going.
(which includes os)
well, i stand by my assumption - it's possible to make it safe by using a positive list of allowed calls/objects, as long as the list isn't too long
Just to be clear, I'm 100% all for building a sandbox within Python, just please don't use it in a real scenario. Sorta like amateur crypto: great for learning, but a bad idea for security. You'll probably learn some fun ways Python is highly introspectable 🌺
I think what you really need to do is find something like nsjail that works cross-platform, or some alternative specifically for windows.
well, os and io were both imported just to run a = 1
I'm thinking so
watch the talk, if tim savage can make eval safe for banking, you should be able to make it safe for a game
I shall
your list will need to be thousands of items long just to start python (and will have to include "unsafe" modules to do it)
Frankly, I feel like the only way harm can't be caused is if you can close off the interpreter from the system. Anything else is a fools mission.
from what i watched it kinda boiled down to slightly more than ast.literal_eval
not at all. you only need to check the AST over the module provided by the user, not against the whole system

then the checked code will still be able to do dangerous things, provided you dont limit it as much as ast.literal_eval
that's you assuming things, same as me assuming the contrary
won't know until @heady mauve gave it a try 😉
lol true
@verbal escarp i think Tim Savage's system is more limited than desired here, but it's hard to say
im not assuming things tho? i have tried to do this in the past, and to get any degree of safety python becomes significantly less useful
no doubt, question is whether it'll be useful enough to do the job at hand though
Well, whatever I do, I need to get rendering working first. I'll probably be in a help channel soon asking about OpenGL/OpenCL interops 😛
well, have fun
PyCharm says error_msg local variable is not used? (for the one that = None)
is this PyCharm being slow, or another of python's weird scoping rules?
(ig the if statement is a redundant, but other than that...)
@red solar confirmed, no cffi
% pyenv shell graalpython-21.2.0
% python -c 'import cffi'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'cffi'
😦
but again, i believe graalvm has its own ffi stuff
yeah i'm not gonna get into the mess of each implementation's ffi, pypy was enough of a pain :/ spent hours on that before just using cffi
maybe if i'm desperate to support graal later
but thanks for checking 🙂
it seems like you can however use graalvm to build libraries that can be called from c, rust, etc. programs
https://www.graalvm.org/graalvm-as-a-platform/language-implementation-framework/NFI/ and it does seem like graalvm has some kind of built-in ffi
The NFI is an internal language designed for language implementors. It is not considered stable and the interface and behavior may change without warning. It is not intended to be used directly by end-users.
idk, i think technically i'm an end user :/
library = Polyglot.eval('nfi', 'load "libSDL2.dylib"') # load a library
symbol = library['SDL_GetRevisionNumber'] # load a symbol from the lirbary
function = symbol.bind('():UINT32')
that's much nicer than ctypes tho (the .bind())
update on the lambda stuff:
I didn't do any of it. I made full functions and did it that way, which is sufficient for my usecase and gives me additional options
from atomics._impl.atomic.base import AtomicBase
import sys
a = AtomicBase(8, is_integral=False, is_signed=False)
i = 17
b = int.to_bytes(i, 8, sys.byteorder, signed=False)
a.store(b)
v = a.load()
print(int.from_bytes(v, sys.byteorder, signed=False))
If i run this normally in PyCharm, it will print 17.
If i run it using PyCharm's debugger with no breakpoints set, it will print 17.
If i set a breakpoint on any line from i = 17 onwards, i will get an assertion error from the C library i'm interfacing with (when I say interfacing, it's just ctypes).
any thoughts on how tf i would go about debugging this? I'm used to debug mode changing stuff in C/C++ (if there's UB), but i've never seen this happen in Python
like i don't see how debug mode in Python would change anything in Python, and it sure as hell shouldn't change anything in C (since that's already compiled)
(btw other people can ask questions, don't let this oddity hold up any other convo)
what's the assertion error?
the parameter i passed (an int) had the wrong value
python3.8: /home/doodspav/CLionProjects/patomic/src/impl/std/std.c:889: patomic_opimpl_load_ulong_explicit: Assertion `patomic_is_valid_store_order(order)' failed.
so you put a breakpoint on i = 17, and your library says that the value of i is invalid?
no the assertion is in the C lib
i put a breakpoint on i = 17 and somehow it's already run a.load() and crashed before the debugger can pause it
well, the first thing to find out is what value order has
either add a printf in the C code, or use a C debugger
if i get rid of a.store(), then I still get the assertion error in C
the only other function calling C there is a.load()
wait
tf
#define PATOMIC_DEFINE_LOAD_OPS(type, name, order, vis) \
static PATOMIC_FORCE_INLINE void \
patomic_opimpl_load_##name( \
const volatile void *obj \
vis(_,int order) \
,void *ret \
) \
{ \
type val; \
assert(patomic_is_valid_load_order(order)); \
val = atomic_load_explicit( \
(const volatile _Atomic(type) *) obj, \
order \
); \
PATOMIC_IGNORE_UNUSED(memcpy(ret, &val, sizeof(type))); \
}
i mean, there's no patomic_is_valid_store_order(order) assertion inside patomic_opimpl_load_ulong_explicit...
so wtf
but anyway thanks @raven ridge ig this is a C issue now
i recompiled and it worked, who knows :/
but still really strange that it only broke once i set a breakpoint in python
@zenith flint a help channel seems appropriate. nobody is ever too advanced for a help channel
Yeah I've tried several times but couldn't find help, hence why I wanted to post somewhere where it sticks a bit
That's not the purpose of this channel, though.
You're just going to have to be patient, or find help elsewhere.
Which one is?
Are you asking what the purpose of this channel is?
No, I'm asking which channel should I ask my question in, I don't see anything related to neither logging or message queues
channels don't get nearly that specific
help channels are for help "How do I do this, why doesn't this work, etc"
We don't have a topical channel for that, so a regular help channel is your best bet.
Maybe you can try #networks
apropos safe python topic, AST eval with @spark magnet, @pliant tusk, @heady mauve from last night: what about using a python interpreter that doesn't expose as much of its internals as cpython? maybe pypy?
Part of the issue is in order to be a compliant interpreter there has to be a fairly high level of exposing of internals
if i were developing a python interpreter, i'd throw cpython-compliance out the window and focus on PEP-compliance with protocols
What about PEPs that specifically detail introspection of internals?
there are interpreters that don't try too hard to be cpython compliant like Jython
although i'm not sure how much those expose their internals
i could imagine even Brython could be an option for that
since it maps to JS, which doesn't have these kinds of issues
in this case, cpython non-compliance could prove to be an advantage
in general, if you aren't getting answers to your question, at least one of the following is probably true:
- answering your question requires specific knowledge that the community does not generally have
- your question is phrased in a way such that it is hard to understand
- you have not provided enough information for someone to help you
- your question is understandable, but too complicated, and needs to be broken down and asked in a simpler form; perhaps you have provided too much information, e.g. you have provided code and data, but they are far from "minimal", to the point where it's a lot of work to even try to understand
Well most of the time it's more like random people spam something nonsense in the channel and the people who could help won't even see the question anymore
(happened 4 times in a row)
well that too. perhaps stackoverflow is a better forum, although stackoverflow drowns in low quality spam too
Yeah gonna try SO now
note that your particular question i think has the first and last problem:
- it requires an understanding of rabbitmq and pika, which most people here probably don't have
- "it hangs silently" is a really nasty and difficult to debug error, and doesn't correspond to any known or obvious pattern of failure
does it hang if you swap out logging handlers? maybe you have an unrelated race condition. etc.
"it hangs silently" is a really nasty and difficult to debug error, and doesn't correspond to any known or obvious pattern of failure
Yeah I know but that's all I see, not sure how to debug it deeper because there's no exception nothing
It works if I replace the the publishing part with something like just printing the log message etc.
So the issue is definitely within the publishing but I don't see a reason why/how as it works fine outside of the class
I wish coding communities other than this Python Discord weren't complete garbage. There's 10k people on the RabbitMQ Slack and there are 2 messages a day happening on the entire server
I think "garbage" and "inactive" or "quiet" etc are two different things....
well did you do the obvious basic debugging steps? use a different log handler, try to just connect to rabbitmq without the logging framework, etc.
you might even have to drop into a debugger and step through the pika code
i've done things like that before
is there a way to specify separate package requirements for installing a package and building it? like installing it needs a single package, but building it needs a fair few
this is what pep 517 is for! specify them in your build-backend in pyproject.toml
you should have it anyway imo
ok i'm a little overwhelmed
i already have a setup.py file
can i just have this as my toml file?
[build-system]
requires = ["cmake >= 3.14", "GitPython", "setuptools", "wheel >= 0.25"]
I might be able to do that
I think a version of CPython that locks itself down based on command line args would be nice. Could run with it 100% CPython exact or lock it down whenever.
probably not deemed useful enough to realize
is there any benefit to this over just sandboxing the process using the OS?
"just sandboxing"? heh
when you guys say "safe"... how safe does that mean?
really safe
So no access to any OS provided functionality?
No files, no networking, no os settings, nothing of that sort
Anything that could be used to cause damage to the end user's PC
What would be a use case?
embedding python as a scripting language in a video game
and please don't say "just use x language" that's all I ever hear
When people embed lua, is it a limited functionality version?
why not use Brython? it looks like the perfect fit to me
you could even plug it in a browser directly with zero-install
I have never heard of it
Just use agda, it compiles to JS 😉 😉
Hm... does anyone know why Python thinks it's invalid?
>>> from datetime import datetime
>>> datetime.fromisoformat("2021-10-26T10:35:11")
datetime.datetime(2021, 10, 26, 10, 35, 11)
>>> datetime.fromisoformat("2021-10-26T10:35:11Z")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: Invalid isoformat string: '2021-10-26T10:35:11Z'
>>>
ISO 8601 defines Z as valid, meaning UTC: https://en.wikipedia.org/wiki/ISO_8601#Time_zone_designators
python only implements it as a reverse of its isoformat, there's been a lot of discussion on handling the Z but I've no idea why it wasn't added
yes, which is as slow and hacky as you think this is
there is an infuriating thread on discuss.python.org about this
This is already opened as BPO 35829 but I wanted to ask about it over here for discussion. Problem Statement The function datetime.fromisoformat() parses a datetime in ISO-8601, format: >>> datetime.fromisoformat('2019-08-28T14:34:25.518993+00:00') datetime.datetime(2019, 8, 28, 14, 34, 25, 518993, tzinfo=datetime.timezone.utc) The timezone ...
this to me is intensely unproductive anti-pythonic bikeshedding over what imo is an obviously good change: add Z support, write a big loud warning in the docs that it does not fully implement iso-8601
@paper echo agree
this is just silly, IMHO:
That said, in some ways it would violate the spirit of the contract, which is that, as of right now, fromisoformat is intended to be used only on the output of .isoformat, which means that all the people who want it to parse Z are in a sense using it in an unsupported way.
I am currently in disappointment of the highest degree
!d datetime.datetime.fromisoformat
classmethod datetime.fromisoformat(date_string)```
Return a [`datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime "datetime.datetime") corresponding to a *date\_string* in one of the formats emitted by [`date.isoformat()`](https://docs.python.org/3/library/datetime.html#datetime.date.isoformat "datetime.date.isoformat") and [`datetime.isoformat()`](https://docs.python.org/3/library/datetime.html#datetime.datetime.isoformat "datetime.datetime.isoformat").
Specifically, this function supports strings in the format:
```py
YYYY-MM-DD[*HH[:MM[:SS[.fff[fff]]]][+HH:MM[:SS[.ffffff]]]]
``` where `*` can match any single character.
Caution
This does *not* support parsing arbitrary ISO 8601 strings - it is only intended as the inverse operation of [`datetime.isoformat()`](https://docs.python.org/3/library/datetime.html#datetime.datetime.isoformat "datetime.datetime.isoformat"). A more full-featured ISO 8601 parser, `dateutil.parser.isoparse` is available in the third-party package [dateutil](https://dateutil.readthedocs.io/en/stable/parser.html#dateutil.parser.isoparse)...
I can't imagine that anybody using datetime, in practice, is aware that fromisoformat is intended to be used only on the output of .isoformat
that's just absurd
yep, it looks like stubborn rationalization to me
I guess it does says so in the description 🤷
see also: the thread about json.load_file
hm?
I've defined been guilty of ignoring that aspect of the description 🙂 That said, it's really one or the other, if they want to stick to that description, then there needs to be a broader function
No matter how you slice it
We have discussed about new function in the json module, but we didn’t make decision about its naming. Previous discussion: Mailman 3 A shortcut to load a JSON file into a dict : json.loadf - Python-ideas - python.org Issue: Issue 43457: Include simple file loading and saving functions in JSON standard library. - Python tracker poll
instead of "hey that's interesting! let's work on some proposals to meet this obvious need in the community", it's "i don't personally use this kind of thing and you are a noob for wanting it"
I don't agree with that one as much as pathlib's read_text and write_text can be used in a similar fashion, but I guess load/dump are already there offering tiny abstractions
I notice that on a lot of threads/mailing lists
Few things are actually considered, a lot of people just like to point out only the flaws and not the good that could come out of it
You'll find that most people are resistant to change of what they're used to
hmm... why is this an obvious need though?
my personal and biased opinion is that the reality is that there a lot of languages that are offering "nicer" syntax for these things in a lot of cases
people see things and want them in python, at least to the degree practical
and people aren't persuaded about the benefits because it's a small, ergonomic thing at best, and it brings up issues that have to be dealt with (e.g. the encoding in the case of json) 🤷♂️
maybe it's not an "obvious need", but there is an "obvious desire" for it, and i think it has obvious benefits for a variety of reasons
it's funny because in the json case specifically afaik at least one spec requires utf-8, so that should be a non-issue
in all other cases... we literally deal with this everywhere else in python now
just default to utf-8 and give the user an encoding= kwarg
with open("filename.json") as f: j = json.load(f) looked fantastic 20 years ago when C++ and Java had 20 line equivalents to this
Only the fact that you'd need to drop 3.10 support to use that function make it not seem that appealing anymore
wait what
you mean 3.10 support
if such a function is introduced, I do not expect libraries to jump on it immediately?
you can say what you're saying about literally any new feature
I do mean 3.10 now
yeah, I guess it's for the long term
it's a non-argument
it probably won't be used in some libraries ever! but imagine how much easier it would be for the literal thousands of newbies on this server
But then you have to explain "blah blah blah your book doesn't mention it because it is older than you"
i'd rather do that
you can say this about every new feature
again, all these arguments apply equally to every single feature, function, that could be added to the language
imagine, you can add one function to the standard library and save probably hundreds or thousands of man-hours across the world and for years to come, making the language friendlier for newbies and safer in general for non-experts
idk, in this case i think the benefit:cost ratio is massive
what was the benefit:cost ratio of pattern matching?
I was talking to Akarys
oh sorry
I don't think it's massive, fwiw
It does apply to every single new feature, you are right, but this function is only relevant in case of learning, and I think it is too confusing for that
no, this function is not only relevant for learning...
plenty of times when I'm writing code, I want to open a json file as an expression
learning is one extremely valuable use case, i strongly believe that it is less confusing
i have utility functions exactly like this in my codebase
Considering it cannot possibly support all the arguments open and loads have, I don't know how useful it would be
but how often do you or anyone use those arguments
0.01% would be optimistically high
the point is not to support all needs and use cases
it's to make one very very very common use case less verbose, easier to teach, and less error-prone even in "serious" codebases written by serious experienced people
Hey, you could use big query to check that haha
You only need the encoding and path in practically all cases, but I'm not that sold on the utility of this compared to the existing alternatives
I think it's nice just for the convenience when loading sth into a repl
In general though, I’ve noticed syntactic sugar PEPs and ideas tend to more often be denied than accepted
python already defaults to utf-8 literally everywhere else as of 3.8, why is it so bad to add a few kwargs for the most popular cases
or even just sugar in making things less boilerplate
i really am not trying to be closed-minded here, but i truly don't understand the resistance to this kind of thing
python is a big featureful language, blocking something as small as this will not reverse some perceived tide of concept bloat
which i agree exists in python, but not because of useful things
you could make the same arguments about str.removeprefix, the only pattern i can decipher is that the accepted changes are the ones that the core maintainers personally find useful
which is fine, if you are a minimal language with a standard library that exists only to bootstrap the runtime
many such languages exist. python is not currently one of them and probably will never be.
I'm not sure I understand, I was just thinking about such a small abstraction being worth being added when it can already be done succinctly, I have a similar opinion on the existing file versions of the json functions that can lead to misconceptions on it doing incremental parsing of the file instead of just being a wrapper to loads
yeah, this seems like a pretty good helper function that would make using python easier.
it's unfortunate that python does not have extension functions; it's so nice to simply add these kinds of functions as extensions to your Path type
not by a lot, but well, it is trivial to implement
Uhm, hello
May I ask where I could start learning Python?
I'm not sure if I'm using the right channel
iirc that was mainly to end confusion because people kept using strip and were wondering why it wouldn’t work
i am generally opposed to making 1000s of people rewrite the same helper util (and associated tests!!) over and over and over, when it could just be part of the stdlib instead. every time you reimplement something, you risk messing it up. and every time you avoid implementing a useful abstraction to avoid adopting a nonstandard helper function, you risk messing it up.
Oh seems that I'm using the wrong channel, my bad
You could ask in #python-discussion usually, but for now check out the resources below!
!resources
The Resources page on our website contains a list of hand-selected learning resources that we regularly recommend to both beginners and experts.
Thanks
Some really good resources on here, but for future reference, this channel is for discussion about the language itself and what not (channel topic explains it better)
good design is almost always an accumulation of small good things, and size of change is not generally correlated with size of benefit
sometimes the biggest improvements are the smallest, and sometimes the biggest changes are the least cost effective
^ a lot of the string methods could be argued with the logic that “this can be done already without it”
But it still exists because it’s just nice to have
when it comes to types, there's an argument about adding more and more member functions, which is that it makes it more and more onerous to replace the type, if you so wanted
people more rarely replace modules than types, though in python specifically replacing types is probably also relatively rare.
but again, extensions really shine here as they separate the intrinsic, necessary, reaches-into-the-guts of the type member functions, from the helper functions that can be implemented in terms of other stuff
and it makes those helpers reusable, squashing one of the main objections to them without having to switch to free-function syntax which is often disliked
I almost had PTSD flashbacks today because I was writing some python code and I was expecting startswith ane endswith not to exist, because in C++ they are not added until C++20, which my company isn't even on yet 🤣
i think it's down to a few like rhettinger who resist that sentiment because it's easy to just write the helper function when you need it, no need to pollute the stdlib with usefulness
this must be my favourite JS function... I think I implemented it more than a dozen times ```ts
const sleep = (ms: number): Promise<void> => new Promise(resolve => setTimeout(resolve, ms))
i'm also a bit pissed by similar cases i encountered previously
classic example of boilerplate that should be a stdlib function
right, i'd say that specifically because this is JS it gets a pass
browser implementers have enough on their hands as it is
but if this were any "standalone" language i'd be pretty annoyed at having to write that. but at least sleeping is not all that common of a need (outside of demos)
and those helpers wouldn't even require a lot of maintenance
you know what is a really annoying glaring gap in the nodejs api? no built-in functionality to iterate over buffered lines of stdin or another text stream
imagine having to implement that yourself, in 2021, for every single application
so the whole "we can't have all the nice things" argument doesn't apply
big
on this, too. again, i think of all the crap in the stdlib that can surely be deprecated with sufficient coordination from linux distro maintainers (e.g. wav, aiff, asyncore)
Has no one implemented an std package for js with no dependencies? Surely that’s a thing by now
There must be multiple
fwiw the sterring council and core devs do seem to be taking "dead battery removal" seriously
and i really appreciate that they are
there only can be one standard!
I mean, I figured… but still
maybe a bit too overzealous
how so?
just thinking of tuple unpacking for lambdas
that was years ago, and not the same thing as "removing dead batteries" as per https://discuss.python.org/t/pep-594-removing-dead-batteries-from-the-standard-library/1704/2
still hurts
you guys need to program C++ 🙂 You can't even write anything in C++ without thousands of lines of utility code
now that we have pattern matching, i would love to have full pattern matching in functions and =/:=
i avoid C++ on purpose and intend to write as little of it as possible until forever
i might also if it didn't pay well
my ambition is for C++ to eventually die as I get older, so I can be like those aging COBOL programmers now
and consult at insane hourly rates
heh
apparently the cobol programmers don't actually make that much compared to the rest of the industry, they just get repeatedly called out of retirement
well, you can't just force people out of retirement
you have to pay enough to be worth it
well no, but even getting the emails/calls would be annoying
"ok, i'm coming, but you pay the lunch"
you can just ignore them.... the company doesn't own you after you retire lol
i'm sure it depends in what sector, but at least in NY finance, there are a lot of stories about obscene hourly rates
but I admit I don't have any first hand verification, I don't know any programmers that old
it's a good scene, at least if you go into fintech
working at a straight up bank, I imagine pays fine but can probably get a bit miserable QoL wise, though it really depends what team you are on
HFT is nice, you get a lot of the perks of both finance, and tech, IMHO. And the company is generally very technology dependent, so that informs the culture a lot.
i was in insurance in goddamn new jersey, i was underpaid and everything about it was a miserable slog
big major company too
this is OT though
(off topic)
Do you think it's always a bad idea to have a default mutable argument? For example here:
async def fetch_all(
session: ClientSession,
url: Union[URL, str],
field: str,
params: Mapping[str, Any] = {},
step: int = 100
) -> AsyncIterator[Any]:
async with session.get(url, params={**params, "count": 0}) as resp:
json = await resp.json()
total = int(json["total"])
for i in range(0, total, step):
async with session.get(url, params={**params, "count": step, "offset": i}) as resp:
json = await resp.json()
for item in json[field]:
yield item
#bad-career-choices-discussion 😉
yes.
so... you propose doing this? ```py
async def fetch_all(
session: ClientSession,
url: Union[URL, str],
field: str,
params: Optional[Mapping[str, Any]] = None,
step: int = 100
) -> AsyncIterator[Any]:
if params is None:
params = {}
...
only thing i can really think of a mutable default being good for is a cache
with the option to use a different cache of your own
here it's only accidentally mutable -- I'm not mutating it, of course
future you will hate you for that
why?
frozendict when 😦
because future you might've forgotten about the context and mutates the thing somewhere else and then you'll end up wasting time debugging the whole thing for hours because there's no obvious exception and then you'll swear to never do it again
I guess... but adding None as default and then doing an if seems just ugly
resource of future bugs
I think it's always a bad decision to have a mutable default argument, yes
and that includes using it as a cache
and adding None as a default and doing if doesn't just seem ugly... it is ugly
but that is how we do
I guess I could use MappingProxyType({}) as the default
You don't need the Optional annotation, FYI
that situation is already special-cased into python type checkers
If you go out of your way, sure
I wouldn't want ```py
def f(x: int = None):
return x + 5
why
why add extra unneeded overhead if all i need is just a simple cache?
If you want to have a global default cache, then... have a global
lru_cache has its own specific uses
and use it as the cache
and even make it the default argument if you want
you can even make the global start with _ if you want it to be an implementation detail
there's literally no reason to write it that way; it just makes it harder for people to grok who aren't familiar with this very weird quirk of python
im still not seeing a why
you presented an alternative, you didnt explain why that would be better though
because its explicit, instead of depending implicitly on a bizarre behavior of python?
It's code that can be read by anyone, easily, with less likelihood of a mistake?
It might just be me, but I wouldn't call this behavior bizarre
i dont find it bizarre personally. it makes sense if you understand how python works. and 'implicit' is context-dependent.
you don't find it bizarre personally, but many people will. Nobody will find using an explicit global bizarre, or confusing.
And yeah, the behavior is bizarre when you have e.g. a dozen popular programming languages, and python is one of, if not the only, that behaves this way.
mutable default argument is quite literally considered a common "trap" for people learning python because this behavior is so surprising
i guess that's fair enough if youre releasing code that needs to be read by others often
I haven't personally seen a lot of complaints about it. AFAIA, it's a mistake you might make a couple times when starting out, but not something you repeatedly forget about
if im just writing something for myself, im not gonna split hairs about it though
well, yeah, I would assume it's a given that your code needs to be read by others, if we even bother discussing which way to do it
otherwise can just hack it out however, as long as either way is correct
I mean, the convo you just joined started with someone (not me) complaining about it 🙂
if it's a library/framework then i can see the need to keep cognitive load to a minimum. but if it's like, just you or a small team working on it and you all get it then i dont see the harm
there was also a whole convo about this yesterday; because there's a PEP to fix it, so somebody found thsi behavior so irritating that they put in the effort ot write something up
That's a fair point, I'm just saying that it's usually something you grasp fairly quickly (albeit after stubbing your toe on it)
yeah, it is definitely not rocket science, and I wouldn't have trouble grokking the code that used this for a mutable cache
I am being onboarded on a project of a small team... readability would be great
I would just skim the code, double take a few times, and think that this code was too clever in a bad way, and move on
I present to you... the penultimate hack. ```py
from typing import Callable, Any, TypeVar
from itertools import starmap
import inspect
C = TypeVar("C", bound=Callable)
def fix_default_mutable(fn: C) -> C:
sig = inspect.signature(fn)
replace_pos = {}
replace_kw = {}
def replace(param: inspect.Parameter, value: Any, index: int):
if param.kind != param.KEYWORD_ONLY:
replace_pos[index] = value
if param.kind != param.POSITIONAL_ONLY:
replace_kw[param.name] = value
for i, param in enumerate(sig.parameters.values()):
if isinstance(param.default, dict):
replace(param, MappingProxyType(param.default), i)
elif isinstance(param.default, list):
replace(param, tuple(param.default), i)
elif isinstance(param.default, set):
replace(param, frozenset(param.default), i)
fn.__defaults__ = tuple(starmap(replace_pos.get, enumerate(fn.__defaults__ or ())))
fn.__kwdefaults__ = {**(fn.__kwdefaults__ or {}), **replace_kw}
return fn
@verbal escarp I hope this is cursed enough to show you
I mean... yeah, that's what it protects against
it's for the cases like mine, where I don't intend to change the argument
how does your @fix_default_mutable fix the "it's ugly to write if x is None.." argument - it's just wrapped in a different cloth ;p
it's ugly but now it's exciting
@grave jolt while you're at wrapping your head around things, i have a quest for you
a while ago i tried to come up with a function that would extract the keyword params from inside the function it's called from
def all_kwargs(func, other_locals):
d = {
name: other_locals[name]
for name, param in inspect.signature(func).parameters.items()
if (param.kind is inspect.Parameter.KEYWORD_ONLY or param.kind is inspect.Parameter.VAR_KEYWORD)
}
d.update(d["kwargs"])
del d["kwargs"]
return d
is the function i got stuck with, which is called as
def foobar(*, a=2, b=3):
print(all_kwargs(foobar, locals()))
i tried to simplify the function call to all_kwargs() but i couldn't come up with a solution
is it permissible to add a decorator to the function?
sure
take out your hack-protection helmet
getting the param types and names from the code obj didn't pan out?
i did some experiments, but i gave up halfway
Dear God no
!e
import threading
def kwspy(fn):
store = threading.local()
def new_fn(*args, **kwargs):
store.kwargs = kwargs
try:
return fn(*args, **kwargs)
finally:
store.kwargs = None
new_fn.kwargs = lambda: store.kwargs
return new_fn
@kwspy
def f(a, b, *, c, d):
print(f.kwargs())
f(1, b=2, c=3, d=3)
@grave jolt :white_check_mark: Your eval job has completed with return code 0.
{'b': 2, 'c': 3, 'd': 3}
oh god, the first line says import threading
heh
doesn't work with generator functions and async, though
and will break with recursion
!e
What about this? ```py
def collect(fn):
def new_fn(*args, **kwargs):
return fn(*args, **kwargs, kwargs=kwargs)
return new_fn
@collect
def f(a, b, *, c, d, kwargs):
print(kwargs)
f(1, b=2, c=3, d=3)
@grave jolt :white_check_mark: Your eval job has completed with return code 0.
{'b': 2, 'c': 3, 'd': 3}
!e
or ```py
def collect(fn):
def new_fn(*args, **kwargs):
return fn(kwargs, *args, **kwargs)
return new_fn
@collect
def f(kwargs, /, a, b, *, c, d):
print(kwargs)
f(1, b=2, c=3, d=3)
@grave jolt :white_check_mark: Your eval job has completed with return code 0.
{'b': 2, 'c': 3, 'd': 3}
that would collide with **kwargs
Hence why you need the positional arg.
yeah, so that's why I suggested the second one 🙂
this is 100% thread-safe and will work with generator functions and coroutine
s/will/should
hm.. i like it, it's simple
not exactly what i asked for, but maybe it'll work well enough to replace the in-function thing
thanks
why do you want to do this in the first place? @verbal escarp
ah, because of my buffet pattern
❓
this must be a Russian buffet because my stomach audibly aches after smelling it
basically i flattened a big function, split it up into many smaller ones which all get the same **locals(), from which each picks the kwarg they want, then pass along the rest
it simplified a lot of code and the signatures only take kwargs, which makes working with those functions very nice
although it's a bit of a rollercoaster if you try to follow a debugger
@verbal escarp your buffet pattern makes me think of ocaml 🙂
why's that 😄
idk about abusing locals like that
but the overall appearance resembles pattern matching
yeah, someone proposed a patma function as a replacement, which was nice
but i can't use it just yet, supporting py3.8
(no pun intended)
the people here need to like, NFT, or encrypt, or something, the awful hacks they come up with, so that there's no chance of someone lurking in this channel, and using it in my codebase 🤣
!e should work for everything if I copied from inspect correctly
import sys
def all_kwargs():
kwargs = {}
upper_frame = sys._getframe().f_back
locals = upper_frame.f_locals
code = upper_frame.f_code
pos_count = code.co_argcount
arg_names = code.co_varnames
keyword_only_count = code.co_kwonlyargcount
kw_only_names = arg_names[pos_count:pos_count + keyword_only_count]
kwargs.update((name, locals[name]) for name in kw_only_names)
# **kwargs
if code.co_flags & 8:
index = pos_count + keyword_only_count
if code.co_flags & 4:
index += 1
kwargs.update(locals[arg_names[index]])
return kwargs
def eggs(*, a=2, b=3):
print(all_kwargs())
def spam(a, /, b, c, *, d=0, e=0, **kwargs):
print(all_kwargs())
eggs()
spam(0, 1, 2, d=3, e=4, f=5, g=6)
@peak spoke :white_check_mark: Your eval job has completed with return code 0.
001 | {'a': 2, 'b': 3}
002 | {'d': 3, 'e': 4, 'f': 5, 'g': 6}
king of the lab eh.. channel!
It's all in the inspect source fwiw, just needed the irrelevant parts taken out of it
obviously 😉
How does inspect.getsource work? does it just read the module and string match?
It seems like it doesn't work in the REPL either, so I'm guessing it does read the module
Yes, it goes through the file; the source is not present in any way in the compiled code so that's the only way
Ah alright.
On another note,
I've seen some people make disassemblers for Python before via reading opcodes/opargs and it was pretty interesting but it wouldn't work for sometimes i.e. and and or since those don't exactly have an opcode for itself, etc.
oof
@white nexus :warning: Your eval job has completed with return code 0.
[No output]
Basically uh, I could've just put that as the lambda signature
Not really advanced, but quite the opposite, it was something really simple in solution
do ya'll think you should use Java in vsc?
That question belongs in an off-topic channel, this channel is for Python discussion
I am very sorry.
>>> b=0
>>> exec(a:='[globals().__setitem__("b",b+1),exec(a)]')
[Previous line repeated 509 more times]
RecursionError: maximum recursion depth exceeded during compilation
>>> b
512
>>> b=0
>>> (a:=lambda:[globals().__setitem__("b",b+1),a()])()
[Previous line repeated 1022 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object
>>> b
1024
Can someone explain why does only half of the recursion limit occurs before it stops?
I believe exec compiles the statement you give to a code object, and then calls that code object as a separate stack frame. So every iteration you have 2 calls, one to exec and one from exec into your code.
Oh I see.. that makes sense now
>>> def a():
global b
b+=1
c()
>>> def c():
a()
>>> b=0
>>> a()
RecursionError: maximum recursion depth exceeded
>>> b
513
and the number in the error [Previous line repeated **1022 **more times] it doesn't include the first and last call?
I'm not sure why those don't match. I'd have expected them to be off by 1 at most.
Depending on where it's run, the stack might already exist
They do match
1023 - 1 == 1022
1023 // 2 = 511 - 1 = 510, plus one more for what @raven ridge said with the double stack, it only got through one of them and not both
What doesn't match is the "repeated 509 more times" message with the counter showing 512 increments happened
Yeah - on the last iteration it performs the increment and then isn't able to create a new frame for the exec call - that's a difference of 1. And "Previous line repeated" implies a first line, so that accounts for another 1, so it should be off by 2. And in the 1024 case it was off by 2, but in the half that case it was off by 3 instead
And also why is __del__ allowed to bypass the recursion limit, is it a bug or another reason or its not actually in recursion? and also why does KeyboardInterupt stop it?
like
>>> class a:
def __del__(self):
global b
b+=1
a()
>>> b=0
>>> c=a()
>>> del c
Exception ignored in: <function a.__del__ at 0x000002034617A8B0>
Traceback (most recent call last):
File "<pyshell#6>", line 4, in __del__
KeyboardInterrupt:
>>> b
3927640
__del__ is special 🙂
That's not recursion, then. There's no way you could be making 3 million recursive calls without the interpreter crashing with a stack overflow.
RecursionError is just there for safety and convenience - you can disable it entirely, but if you do, the interpreter will eventually crash instead of raising an exception
lol really?
Sure. Stack is a limited resource, and there's no portable way to know how much of it is left. Eventually you run out and get a segmentation violation.
SEH? and maybe linux has an equivalent
Python definitely checks stuff with SEH on windows
ah I see...
I just thought it was recursion for some reason because this creates millions of objects
WARNING KILLS PC
>>> class a:
def __del__(self):
global b
b+=1
a()
a()
>>> c=a()
>>> b=0
>>> del c
So RecursionError is a safety valve that says "assume something that makes more than N recursive calls is doing something wrong", because the interpreter doesn't know how many more calls it could possibly make successfully.
I always just imagined if you disabled it, the interpreter would still check and raise it if you were about to segfault
or maybe raise an OOM exception (idk if python has one)
kills pc how?
It does, MemoryError
oh so that 🙂
Perhaps on Windows it does. On Linux it definitely just segfaults.
Oh just segfaults?
64gb
🤯🤯
chill, i couldn't find a gpu so i spent more money on ram lol
and here was myself bragging bout my 16GB ram lol
which is perfectly fine for 99% of things
@fossil pumice @wide shuttle @granite heath @oblique raven @sturdy timber
the hackathon?
oh, well congrats 🙂
so you have an idea, and now you need to create a proof of concept?
ok - i'm not sure how i'd be able to help though
you want to make the proof of concept in python? do you know python?
aww @oblique raven you kicked him? 😦
No
oh then he left
Yup
i mean, #data-science-and-ml might help? other than that not sure
i'm not sure - i still haven't understood what your issue is - you don't know how to use a specific AI thing? or you have no idea how to get started at all?
really? usually these challenges give you some testing environments
https://www.hackerearth.com/challenges/hackathon/reimaginefuture-hackathon/custom-tab/resource-centre/#Resource Centre i think you're supposed to set up a free account, and then use their APIs which are presumably free for some low limit?
then what's the issue? why can't you use it?
i'm not
what's the issue? you have an account, you have free access to the cloud, why can't you deploy?
class bdist_wheel(_bdist_wheel):
def finalize_options(self):
_bdist_wheel.finalize_options(self)
self.root_is_pure = False
would me doing super().finalize_options() be identical to _bdist_wheel.finalize_options(self)?
not fullly
it will be different with multiple inheritance AFAIK
if there's no mro changing magic in _bdist_wheel they should act the same
hmm... ok fingers crossed there isn't then and i'll go with super() cuz it looks nicer 🙂
But if someone makes another subclass, like this: ```python
class bdist_car(_bdist_wheel):
def finalize_options(self):
print('Finalized')
super().finalize_options()
class Subclass(bdist_wheel, bdist_car):
pass
bdist_car 😂 😂
The MRO is now:
_bdist_wheelbdist_carbdist_wheelSubclass
it's alphabetical?
ah i'm blind, i somehow didn't even see Subclass
Subclass inherits from bdist_wheel which has been changed to inherit from bdist_car which inherits from _bdist_wheel
So when you call a method Python will look it up in the reverse order from what I listed. Starting with Subclass and ending with _bdist_wheel
But as you notice bdist_car has been injected into the MRO, as bdist_wheel's parent isn't _bdist_wheel (well it's the parent's parent)
hmm... i think i actually want super() then even in that case
even though it's not the same
not sure why the example i copied didn't use super()
Yes, good. This is actually why people use super(), even though they know their parent.
how do i figure out which version of stuff like setuptools and wheel i need to require?
or do i just not put a version requirement?
eh screw it i guessed
cool hack. seems like a "FrozenList" would work here too as a default argument. thinking why we have frozenset but not frozenlist
I guess there was no use for it, heh
frozenlist = tuple
There you go :P
I think I pretty much did that as a type alias, and then never used it 🙂
lol, that's not bad
class frozenlist(tuple):
pass
too bad it says __main__.frozenlist
!e ```py
class frozenlist(tuple):
module = frozenset.module
print(f"{frozenlist}, {frozenlist!r}")
@spice pecan :white_check_mark: Your eval job has completed with return code 0.
<class 'frozenlist'>, <class 'frozenlist'>
You can manually set it to 'builtins' as well
are there other differences between a user-defined type and a builtin type?
Depends, most (if not all) built-ins types are implemented in C
That'd be the biggest difference ig
if there are no new methods, it could plausibly be a C type?
I'm not sure about the details here, frankly. I'd guess that it'd be a Python type that proxies everything to its C parent
!e You can probably assign arbitrary attributes after subclassing
class Sub(tuple): pass
Sub().new_attr = 10
@spice pecan :warning: Your eval job has completed with return code 0.
[No output]
Yeah, it seems to be a regular Python-defined type
Just one that happens to rely on it's C-implemented parent
I do see one thing I'm not sure of:
'__new__': <built-in method __new__ of type object at 0x7ff7f88950>
frozenlist does't have __new__ in its class dict. Just wondering if that's something a builtin type would have
the above was in tuple class dict
They don't have slots or a dict
do frozen containers have any optimisations compared to their regular counterparts?
!e ```python
print(int.slots, int.weakref, int.dict)
@elder blade :x: Your eval job has completed with return code 1.
001 | Traceback (most recent call last):
002 | File "<string>", line 1, in <module>
003 | AttributeError: type object 'int' has no attribute '__slots__'. Did you mean: '__class__'?
On the other hand, all user defined classes either has a __dict__ or __slots__ this is because they're not defined in C of course
Smaller memory
Memory isn't freed perfectly after it's removed
I'd say one of the most important properties is hashability
smaller memory in that the PyObject takes up less memory, or it also allocates stuff more efficiently too? (like having FrozenSet be contiguous or something)
!e ```python
from sys import getsizeof
d = dict()
print(getsizeof(d))
d['a'] = 1
print(getsizeof(d))
del d['a']
print(getsizeof(d))
@elder blade :white_check_mark: Your eval job has completed with return code 0.
001 | 64
002 | 232
003 | 232
See how it stayed the same, even though we removed an item?
If you modify a dictionary it may have some extra overhead even though you don't store any more items in it
tuple would be smaller than a list because it allocates exactly enough memory for its elements, no need to overallocate for amortizing appends. Not sure about frozenset, since overallocating is not only done to amortize new additions, but also to reduce hash collisions
There is no frozendict
Yeah, there's mappingproxy, which gives you an immutable view of a dict, but it's not exactly a frozendict. IIRC it's still unhashable
tbh i thought there was a bunch of frozen containers, not just set :/
Well, tuple is effectively a frozenlist
But there aren't that many containers to begin with, tbf
Yeah, mappingproxy is still unhashable, so it's just an immutable view
Okay but when would you store a dictionary as a key in another dictionary lmao
You may want to store it in a set
It's not an entirely unreasonable use case
There was a rejected pep that tried to introduce frozendict
!pep 416
Multiple threads can agree by convention not to mutate a shared dict, there's no great need for enforcement. Multiple processes can't share dicts.
wait, you can look at the class dict for all types, right? they don't have an instance dict though
not sure how i feel about that reasoning
sure they can agree, but enforcement is always nice
Yeah, but you won't get a dictionary, you'll get a mappingproxy
ah I see
!e py print(repr(tuple.__dict__)[:100] + '...')
@spice pecan :white_check_mark: Your eval job has completed with return code 0.
mappingproxy({'__new__': <built-in method __new__ of type object at 0x7f51a54eefc0>, '__repr__': <sl...
I thought that was just to make it immutable
It is
You can't alter it directly. You can get the underlying mutable dict with some gc hacks IIRC, but that's... very dirty, to say the least
!e ```py
import gc
d, = gc.get_referents(tuple.dict)
d['index'] = print
().index('Hello, world!')
@spice pecan :white_check_mark: Your eval job has completed with return code 0.
Hello, world!
lol 😂
This screams "Please don't do this"
the referents are indexed by name?
ah!
Yeah, it's just a list
With one item, that I immediately unpack into a tuple of 1 element
I missed the ,
It's more readable if you do [item] = collection or (item,) = collection, but I basically copied it from my REPL, so yeah
i admit it's not often I've ever wanted to use a dict as a key to another dict
on the other hand I'm not sure why they don't have it
I've run into it a few times I think
I can't find an obvious reason why frozenset is there but not frozendict
it seems like set was simply standardized later, and perhaps by a different person, and the value of immutability was perhaps better understood/appreciated
so frozenset just came in with set
it's easier I guess to add something incrementally as part of a PEP than get a whole new PEP accepted
making mappingproxy hashable could work too
but I guess that'd be kind of unorthodox
A frozendict is hashable if and only if all values are hashable.
the trouble often seems to be lists
I guess you'd basically need a utility function to recursively copy the dict, changing every list to frozenlist and every dict to frozendict, which doesn't sound great
i don't understand
if you wanted ti make a hashable dict, but there were e.g. list vakues, or other regular dicts
how is a frozendict being hashable iff values are hashable, different from a tuple being hashable iff elements are hashable?
it isn't really I guess
i think this problem is solved by error'ing out, not by recursively copying anything, at present
it would be nice to have an easy way to make an immutable+hashable copy of the data
In [1]: {([1,2,3], [])}
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-1-430e4d202ea5> in <module>
----> 1 {([1,2,3], [])}
TypeError: unhashable type: 'list'
it gets tricky once you have class types
yeah, that's what I meant with it not being that practical
you could write a deep copy that specifically converts list to tuple, set to frozen set, etc, while its copying
I'd prefer more support for working with immutable types in the day to day, personally, but this comes back a lot to th elack of support for more functional, expression oriented stuff in python
python is very imperative so outside of a small amount of sugar it encourages you to create collections and mutate them a lot
and while with comprehensions you avoid mutating something, they are unfortunately hard coded to create something mutable... so they are part of the problem, really
yeah, exactly
there is hamt
nevermind that isnt hashable
https://github.com/python/cpython/blob/main/Python/hamt.c it is immutable tho
!e
!eval [code]
Can also use: e
*Run Python code and get the results.
This command supports multiple lines of code, including code wrapped inside a formatted code block. Code can be re-evaluated by editing the original message within 10 seconds and clicking the reaction that subsequently appears.
We've done our best to make this sandboxed, but do let us know if you manage to find an issue with it!*
all the times ive needed a frozendict its only been for the hashability, which frozenset(dict.items()) covers well enough. so for me that really only leaves subscriptability (without the memory footprint of a copy) as a selling point. i did see a frozendict module somewhere thats implemented in c.
https://github.com/Marco-Sulla/python-frozendict this one i think
if you just want the hash you may as well just do tuple(dict.items())
if you can't meaningfully use the in operator on the set then not much point paying for hashing etc the items inside
imo the hash should be independent of key sequence, but i suppose that's different now we have ordered-by-default dicts
they are ordered by default but I would expect their hash is still independent
so your point seems valid to me
but it's not guaranteed, right 🤔
Yep
but the tuple and set are legitimately different in that with the tuple, different order would give a different hash, but not for the frozenset
I guess it depends what you're using the hash for
or how you're using it
so list(d.items()) should always produce the same list? (if the dict isnt changed)
well, in practice that would always have been true
!e
print({1:2, 2:1} == {2:1, 1:2})
another argument in favour of hash being independent of order, at least in terms of consistency
@main ginkgo :white_check_mark: Your eval job has completed with return code 0.
True
dont get the same thing with tuples
Yep, I said that a bit ago. But it really depends, like I said, how you're using it.
eh, i think it's a bit more than that
@lusty scroll actually, I take that back, list(d.items()) will produce the same list for any reasonable implementation... in a given execution. But between different executions it might not, because of salting
different executions meaning, runs of the whole program
it would be weird if two dicts with different ordering were considered to have the same content (which i suppose is effectively the meaning of a hash) but then frozendicts didnt
Yeah, no disagreement, my point was only that depending what you're doing, you possibly could just use the tuple version instead, but you're right that the frozenset is more broadly applicable since it preserves that property
PYTHONHASHSEED would take care of that?
what do you mean by "take care of that" ?
eliminate the difference between runs
sure if you fixed that
but a) you wouldn't typically, and b) it's still not something to depend on in any way, it's an implementation detail so it can change e.g. when you upgrade python, without warning
fair point
beyond not being guaranteed to change, it's also not any useful order
whereas now the order is both guaranteed, and something at least potentially useful
though I've never used that property ever, that I can recall
I always forget even if changing an existing element changes its order location or not
if you're doing reproducible builds, you would
why would you depend on this for reproducible builds?
on hashes ?
it's not about depending on it, but removing uncertainty that could affect the output of your process
Changing the value for a key doesn't reorder it, but deleting and then re-inserting with a new value does
right. that, to me seems like a practical compromise for performance reasons, as opposed to ideal behavior for an "insertion order" dict
which is totally fine; my understanding is that one of the main reasons why dict got the ordering guarantee was because it was basically free performance-wise anyways
nice username btw, can't stop re-reading it lol
but yeah, IME in most cases you want some kind of "insertion order" dict, you usually want modifications to bring it to the top
I don't keep up with all of his posts or anything but from what I've seen, he really doesn't like making changes which cause the same operation in future python versions to be slower than previous versions
same or better is ok, slower is not. That's one of the reasons why cpython still uses the GIL even though basically no computers worth running python on are single core nowadays
yeah, I"ve heard that angle wrt the GIL as well
He's said he would be open to removing the GIL if a good solution is proposed which is at least as fast in singly threaded code
Yeah, there's a pretty serious proposal for that now
I think he also doesn't like lazy garbage collecting either
which is fair, neither do I.
idk, in practice I haven't found python's eager GC to be useful in any way
I think we should just eat the singly threaded perf loss but sadly not gonna happen
even in cases where it looks like large allocations should be returned immediately, they haven't, and still caused OOM issues
lmao
oh what C++ channel are you in?
TPH server
ah don't know that one. damn there's a lot of these eh?
A few lol, tph is more general tho, they do most languages
I think anything wrt performance in python is a rough ride..... I doubt that we will ever again see a new, popular, language, as slow as python
or Ruby
ey you wouldn't happen to know how to construct a Clang::MacroDefinition and the objects it depends on from scratch, would you? The libclang doc is only really clear on how to use these objects but not structure new ones
I started writing a function which traces through a macrodef and all its members to dump all their data but I feel like that's a hacky crappy solution
Dm chandler on Twitter, he might reply 🙂
might try it, thanks
I guess the same complaints would apply about reading the argument names that make up the tuple (even though they'd be positional-only, you still want access to the name for help() and such)?
didnt know you could do that in py2, thats cool
Anybody with digital twin projects? I have some questions. Thank you kindly
I don’t think that’s an apt comparison? Wouldn’t it be more like django vs spring boot? (Either way idk)
I’d ask in #web-development
If I create a Python specific wheel for say Python 3.6 (but it’s not abi specific), would I then be able to install that on any Python version from 3.6 to <4?
Or do I still need to build each Python version separately even tho it would be the same wheel?
depends
you want the interpreter to be py3 I believe
like yourpackage-1.0.0-py3-none-any.whl
Ugh :/ then i’d have to get rid of all my type hints tho to make it compatible with all py3 versions
Is it just Python?
Or are you using the C API?