#internals-and-peps
1 messages Β· Page 100 of 1
Where do I see the rejection?
Uuh it should have been somewhere in the PEP
Yeah i guess this isn't updated yet? Other rejected peps had it on top
I thought It wasn't added but I was curious about the explanation in the rejection
#mailing-lists last link in there
Awesome, thanks!
where are those 10 parahraphs on hooks? can't seem to find
my old friend
settrace will be difficult to support in a language agnostic way.
(assuming anything truly can be)
"for life"!!!!
No takesy backsies.
I will not be implementing settrace; there will be no Python debugger; there will only be a language agnostic neos debugger.
settrace is part of the Python standard library. Will you be keeping a list of what parts you won't be supporting?
to answer your question, PyPy and Jython support settrace
settrace is an implementation detail.
i don't know how you decided that.
i see π
probably in your environment people wouldn't need it, so it will be fine.
indeed. the neos debugger will be wicked.
first language agnostic debugger perhaps and first debugger rendered in OpenGL perhaps.
the neos AST can be visualised as nodes and there will be a display option for the deubgger to render in node form as well as text form
why nodes? hipsters like nodes. π
i really think it's better to avoid "there will be", and stick to "there is"
that wouldn't be wise, avoid "there will be" means you have no plan, no vision of what you want to achieve; the now is an artefact of the future.
oh, i definitely think you should plan, and have a vision, etc. But advertising it here as "there will be", when you have a long way to go, just starts tricky debates.
it doesn't serve your purpose.
"there will be" is no different to stating a function requirement which is standard software engineering practice. define requirements first.
functional*
i'm building a pogo stick that will go to the moon. it's one of its functional requirements.
sounds fun. good luck with that. π
exactly π
however you have regressed to claiming that my solution is impossible again which is a shame.
no, i'm saying we should skip the entire question, by talking about what is, or what you are working on right now, and how we can help.
okey dokey. π
the goal for the next 4 weeks is to get "hello world" parsed, byte code emitted, libffi integrated with stdout being redirected to an OpenGL window. I will probably have to take a few days off work to achive all that though.
that includes the bytecode execution component also, right? Did I see somewhere that you have your bytecode defined?
yes, it is a lot of work and unfortunately (or fortunately) I have a day job.
the only hurdle for my project is time.
do you have the bytecodes defined?
I think I defined a subset when I last worked on neos several years ago
I need to create a linker too of course
lot of work but luckily a lot of it is fun rather than tedious
if that wasn't the case I wouldn't be doing it
these bytecodes look very low-level compared to (for example) the JVM or the Python bytescodes.
that is deliberate
makes translation to machine code (JIT) less work
but that was my first stab at it; I may fettle it when I revisit it soon
but translation of python will be harder, and the jitting won't have access to the higher-level concepts.
python doesn't go anywhere near the bytecode
the fact the source code was python is lost during compilation
right, i understand
yeah I will definitely have to take some time off work to do it in 4 weeks :/
less talking about it in here I guess π
I did actually create a statement of intent for the VM 2 years ago: https://github.com/i42output/neos/blob/master/src/bytecode/vm.cpp
ADD and B(ranch) defined!
π
I think Brainfuck only has add and branch so it might be turing complete π
as always, we are interested to hear about progress, and can help with Python details
do you get paid by the hour for writing that? π
i'll send you an invoice π
@weary garden so the Python code is lost when it's compiled, but you'll have a debugger. Does that imply the debugger will only be able to debug at the level of the bytecode, not the Python code?
if so, that's going to make Python code very verbose, and very difficult to debug. For a simple example of why, x + y in Python is defined to behave as essentially this series of calls:
def add(x, y):
res = type(x).__add__(x, y)
if res is not NotImplemented:
return res
res = type(y).__radd__(y, x)
if res is NotImplemented:
raise TypeError("Unsupported operands for +")
return res
If the debugger forces you to step through that whole series of byte code every time it hits a +, it's going to get old very fast.
to say nothing of the fact that resolving type(x).__add__ or type(y).__radd__ may require hash table lookups, or even themselves execute arbitrary Python code if the type has a dynamic __getattr__ defined.
@raven ridge no, when semantic concepts are folded the source code elements they refer to are registered and will be used by the debugger to display code, variables etc
the exact from of the debug information is TBD
But... Does that mean that "Python add" will need to be its own semantic concept?
add looks like a function to me
well, yes and no - it's a function in the sense that any operator call is a function, I suppose
there is a definition in the interpreter for what + means, and it evaluates to a series of dynamic type lookups, attribute lookups, and method calls
well like any debugger you can step into or step over code
sure. I guess I'm more asking about the level of abstraction you'd choose to expose to the user here. There's Python-specific concepts at play here for how + is defined to behave, and I suppose I'm wondering whether you plan to unroll/inline them by default, or expose them as though they're a function call, or what.
though as long as the user won't be seeing all of the byte code for a hash table lookup by default, it probably doesn't matter much.
the debugger will be aware of language agnostic semantic concepts and I could add support to tag them in the schema so they are not decomposed by the debugger; not really thought about debugging in any detail yet.
A bit of a relief
https://twitter.com/raymondh/status/1371606366331150336?s=19
π
I'm surprised opinions are this strong about this one
i guess the uses for it outside of pandas weren't very motivating
yes. you don't need them
What was the use in pandas π€
The getitem and setitem dunders are for indexing
ah df[x=1] instead of df[df['x'].eq(1)], yeah that could make sense
Personally I didn't care for it
I felt the idea of acting as a logical mask to be weird
And it wasn't doing enough to serve as a .loc replacement
Re: PEP 637 (and other divisive PEPs), I think that the deliberation process would be helped (to what degree, I'm uncertain) if users didn't get the impression that a proposal is inevitable prior to a SC approval. It's hard to imagine anything other than oppositional mindsets coming from that situation.
There are implicit signals in mailing list discussions (most subtle, some less so), which seem to accumulate by a weighting on each person's reputation. But where someone of good standing in the community takes a declarative approach to a proposal - does it not directly foster a sense of inevitability?
A well-meaning example is Raymond Hettinger's tweets on recent PEPs, of the form "In Python's near future, ..."
e.g. for 637:
https://mobile.twitter.com/raymondh/status/1364794069222707201
I'm quite surprised 637 was rejected after 634 was accepted, frankly. The syntactic cost of pattern matching is clearly much higher, the ability to ignore it much lower, and the benefits are at least equally nebulous
Agreed. Particularly given the emphasis on syntactic cost in SC's response.
The concern around performance for a very common process is more persuasive, personally. Edge cases may be as well, but I have not looked at those in depth.
I posted the other day about attempting to get type info for code objects processed with dis. Ended up rewriting what I was doing to use ast, and got tons of other benefits on top of it.
Im just wondering is this the right channel to speak about mathematics?
Generally, no
what about @property?
that is a "Pythonic" way of doing getter/setter, isn't it ?
Yes, if you needed specific behavior that would usually warrant a getter/setter, you'd use a property instead.
@oblique crystal the classic "trivial getters" you see in Java are unnecessary in Python.
The main thing to note is that property allows your end users to keep using the class object as if it was still giving an attribute, while allowing you to freely add the benefits of custom logic later if need be. Thus it essentially solves the problem of breaking an interface that would have happened if you hadn't defined getters and setters upfront in a different language.
yeah I get that. For me I view the properties in Python as improved way of doing getter/setter π
so here when you said "you don't need them" you meant not a python way with properties but Java-ish geo.getCoord() and geo.setCoord()
class Student:
def getGrade():
return self.grade
def setGrade(grade):
self.grade = grade
def getName():
return self.name
...
``` yeah, don't do this
yeah, if you need a public property, use a public property
I have questions regarding the certificate details (those you can check near the URL) but I have no idea to ask such questions because it isn't exactly python... or coding for that matter. Where could I go?
Where is the Url ??
the one you get from the lock symbol ya know?
Off-topic channels
There are three off-topic channels:
β’ #ot0-psvmβs-eternal-disapproval
β’ #ot1-perplexing-regexing
β’ #ot2-never-nesterβs-nightmare
Their names change randomly every 24 hours, but you can always find them under the OFF-TOPIC/GENERAL category in the channel list.
Please read our off-topic etiquette before participating in conversations.
@ivory lagoon This might be a good place to go.
How are they related to my unrelated topic tho?
@ivory lagoon questions unrelated to Python go to off-topic channels
The names are random.
They change every day, and do not pertain to the conversations.
the page for pep 637 doesn't say it was rejected? when was it rejected?
The reason is right here sir https://mail.python.org/archives/list/python-dev@python.org/thread/6TAQ2BEVSJNV4JM2RJYSSYFJUT3INGZD/
Hello @shell yoke , please see #βο½how-to-get-help
still find it weird that they reject 637 but pattern match got in despite the divide in the community about if it should be different or not be added at all
Hello i am already there but no one was helping so thought would ask in here
Well, in this case have you seen our asking good questions guide?
yup i did
this is so sad.
I wanted kwargs for __class_getitem__ to make a typing system for pandas
What does keyword indexing have to do with this? 
BinClassFrame = pd.DataFrame[columns=('tp', 'fp', 'tn', 'fn'), columns_ordered=False]
in other news, PEP 597 got accepted as the first step for getting rid of default locale dependent encodings which is nice https://mail.python.org/archives/list/python-dev@python.org/thread/YIZGJBMDQAOZHD6MXQCLEXIAKGUQFMQM/
You can do
BinClassFrame = pd.DataFrame[{"columns": ('tp', 'fp', 'tn', 'fn'), "columns_ordered": False}]
``` or ```py
BinClassFrame = pd.dataframe(columns=('tp', 'fp', 'tn', 'fn'), columns_ordered=False)
besides, type checkers probably won't work well with anything like that
you'll need type-level lists
but I can dream
I mean, you can make a linked list
tuple[Literal['tp'], tuple[Literal['fp'], tuple[Literal['tn'], tuple[Literal['fn'], None]]]]
my mortal enemy
linked lists
and there might be a way to make one type checker to work with it
It was a pipe dream for sure. A pandas type compatibility checker would probably have to be its own tool.
I don't think mypy lets you get that granular
With the keyboard lib, it's possible to hook on the windows key system and, for example, create new shortcuts. But is it possible to disable some ? For example, detect that a specific shortcut is use and prevent its action ?
This is a question about a specific lib, while this channel is for discussing Python language itself.
I don't have experience with keyboard lib but I sometimes map my keys with AutoHotkey and some windows key functions stayed the same even after remapping... So I don't really know how much can be done in any code when it comes to win key.
thank you for the clarification. I didn't really know where to ask, as this is not really a "help" thing, and the python-general is overloaded
I already created my own shortcuts with this lib, and using Qt I know it's possible (I would have to double check tho) to disgard events
I guess I'll have to see for myself
You could try looking for contact info on the lib's website and ask people responsible for it
no you can write custom plugins that operate on the mypy ast
Take a look at our topical channels if you have a general question that isn't on-topic for the description of this channel. Let us know in #community-meta if you have a question about that.
Right, that's what I thought
(because of the attrs example)
Honestly, I feel that choosing indexing for generics is bound to clash with indexing in general
Our keyboards have 4 types of brackets and we use the most popular ones
Wait, 4?
I only thought there was (), [], and {}.
ig if you include <>?
Oh, yeah ig.
Yeah, triangular brackets. C# uses them for generics, for example, if my memory serves right.
Only the square brackets and parenthesis would work without introducing even more new syntax.
I wonder if parenthesis are feasible. Why weren't they chosen? Passing arguments to functions has the most powerful syntax.
Maybe it'd be too confusing to have them look like functions/classes
I don't think that new syntax is bad thing for new concept (generics). But again, I don't mind verbose langs.
Well, you already pass arguments to types, it's constuctors, and types are already valid arguments. At least you usually index instances, not types themselves.
Yeah, they did have to introduce new syntax anyway to support annotations. However, I don't know about all the implications of adding new syntax. There are probably considerations beyond my understanding.
Have a design dilemna
For any who are curious, this is the project I've been working on the last day or two. https://github.com/syegulalp/myjit
https://github.com/AirikWarren/Pyguin
Working on my static site generator to make it less specific to the websites that I hacked together at first and having a hard time deciding what a good and versatile / robust way to deal with allowing ease of extendability when it comes to extending the rendering function
env = Environment(loader=FileSystemLoader(f'{template_dir}/html_templates'))
filenames = [x for x in env.list_templates() if "base.html" not in x and not x.startswith("_")]
nav_links = create_title_extension_dict(env)
stylesheet_path = f'{template_dir}/css_templates'
for i in zip(filenames, nav_links):
f = open(f'./output/{i[0]}', "+w")
f.write(env.get_template(i[0]).render(title=i[1], nav_links=nav_links))
This is what needs to go
Not every simple static site template is going to just need a single dictonary of page titles and relative links
Looking at the code its similar concept to numba?
Or actually more numba witb explicit type declarations from the type hints vs numba's manic signature spec
@radiant fulcrum partly! But also I wanted to start with something that wasn't dependent on numpy, and maybe add that in later - start with something that could generate very simple binaries
Good idea
I also liked the idea of having something that could generate "sidecar binaries" the way Cython does -- you run it, get an AOT-compiled version of your function, and you can distribute that instead of the whole of LLVM
The exact details of how that works are TBD but I'd like to make it as transparent as possible for the developer - they don't have to undecorate functions or anything like that
And if it eventually turns into its own language...
My first goal would be to get enough functionality to be able to have it run Conway's Life -- basically, loops, if, arrays, a couple of other things
andC FFI of some kind.
I'm still trying to figure out what would be a good way to express C calls in this style
perhaps
from myjit.ffi import malloc
...
x:Ptr[Byte] = malloc(32)
@grave jolt slow your roll, son :D I'm still just getting my footing here. But nothing says in time I couldn't, I suppose
the idea would be to use Python's syntax and type decos, but not use Python's extreme dynamism
ah
this is also an arena for me to gradually learn all kinds of other things!
Is there a way so affect how a classβs methods are run? I was thinking of somehow dynamically applying a decorator to all functions in the class but Iβm wondering if thatβs even possible/if thereβs another way
@signal tide Cant you add a normal decorator on all classes and then have logic so that it runs it either normally or modifies the decorator as you want it to?
Yes but I donβt want to add a decorator to all methods manually
decorate the class then
How would that work?
iterate over the methods in the class, replacing them with versions wrapped in your behaviour
I recently read an article on RedHat about why Ruby avoided using LLVM for their JIT due to it being slow and incredibly large. Not that this should deter you I just thought it was interesting, but in case you wanted to also read it https://developers.redhat.com/blog/2020/01/20/mir-a-lightweight-jit-compiler-project/
I've played around with keystone engine and PeachPy in a similar way as well and it's pretty awesome and very cool what you're doing!
thanks! Yeah, I'm open to alternatives, but so far LLVM has been okay, since llvmlite uses a scaled-down build of it that doesn't contain anything they don't need -- for instance, it doesn't have any of the IR generation components since they do that themselves in Python
If MIR comes along with a Python wrapper, I might play with that
Yeah, I've been meaning to try llvmlite too, after seeing how numba worked.
@limpid marten Been playing with it for a while now, so I can offer pointers. I'm actually wondering if there's another channel better suited to this particular discussion but I can't think of one - #c-extensions doesn't really fit, does it?
Is there? JITs are an advanced-discussion at least I hope so, and it's related to Python. 
Seems like there could be a performance/speed related channel eventually but this will do for now
I'm not too familiar with LLVM IR, but is there a concern with register allocation?
in what sense?
Like if you were instead creating your JIT using x86 assembly, you'd have to be mindful of how many locals you have, and arguments passed to the function and if you need to either have a variable on the stack or in a register for your JIT'd function.
But it looks like LLVM IR is a bit higher level than that.
Oh, LLVM abstracts all that
here's some code I created
def test() -> Byte:
x: Byte = 8
y: Byte = 16
return x + y + 32
Ah, do you read annotations?
here's what LLVM turns that into with my JIT
define i8 @"main"()
{
entry:
%"x" = alloca i8
store i8 8, i8* %"x"
%"y" = alloca i8
store i8 16, i8* %"y"
%".4" = load i8, i8* %"x"
%".5" = load i8, i8* %"y"
%".6" = add i8 %".4", %".5"
%".7" = add i8 %".6", 32
ret i8 %".7"
}
yes, the whole idea is to have static typing with Python's annotation syntax, or somethign close to it
ahh, nice, so you don't have to call Py_Add
or, whatever the function is.
That's really awesome though!
right, I'm not doing anything with the Python runtime yet
it's all resolving to native machine types
which is probably what you want anyway
exactly. This is meant for specific functions that need de-bottlenecking, mainly numeric functions but I do want to make it possible to do things like string manipulation eventually
so in time it might grow to become its own AOT/JIT language
Man is searching the c pythonapi a pain.
My next step is to add support for arrays, as one of the "milestone 1" projects I have in mind is to implement my Conway's Life demo using this as an accelerator
Ahh, that'd be awesome.
Have you tested how long it takes for Python to construct ctypes types?
I know there is some overhead with that.
well, with the decorator, there should ideally only be a one-time cost to wrap the function.
basically, we compile the function, then get in and out types ahead of time, then generate the function wrapper with the ctypes objects already constructed.
Ideally, you'd use this for a function that you'd call and would have a lot of internal overhead. In time I'd like to generate something where you can have a native Python extension module
I meant moreso the function arguments, converting from pyobjects to ctypes ones, but yeah I understand, not necessarily the function pointer returned by the JIT compiler.
right. Again, the cost for that should be minimal. It's stil ideal to take whatever loops you might be doing and push it into this function instead of calling this function repeatedly
Yeah of course! π
One thing that scares me is trying to debug a JIT compiler, but I guess since you can just look at the IR that was generated it would make it a bit easier.
yup, the IR is dumped with each run for now as a debug gesture
Please keep me updated on your progress, things like this are so interesting!
What are you making?
here's the project https://github.com/syegulalp/myjit
Interesting. I'll take a look at it,
loved your work on aki with llvm, looking forward to this π
ctypes? Seems like it would be both easier and faster to take the route that Cython cpdef functions take, and have a Python callable trampoline that uses, e.g., PyLong_AsLong to convert a Python int it recieves to a native int suitable for passing to the compiled function
Likewise you'd handle strings by calling PyUnicode_AsStringAndSize and passing a char* and a size_t down
ah, thank you! Shame I didn't get very far with that, language design is a headache :D
yes, I think that's the best long-term path, but I'm not quite comfortable working with the cpython API yet
I can start with using ctypes, and then migrate that out as I get more familiar with how the C API works
right now I just want to get the basics running, even if not optimally implemented
Fair enough, but that seems like it might wind up being a lot of throwaway work if you invest too much into ctypes.
This all definitely feels on topic for #c-extensions to me, FWIW. Adding JIT compiled functions is definitely "extending Python using compiled extensions"
well, I'm trying not to make too much work in that department upfront, most of my work is on the guts of the thing for now
also, yes, I'll probably make more discussion about this in #c-extensions later. Got to get to work now :D
I can't imagine there is much work using ctypes? Maybe I'm missing something though.
I find ctypes much harder to use than the C API.
Boilerplate of creating a new extension module aside, at least.
right, I'm just using it to pipe things into and out of the module for now
::drags self bak to work::
Hi
@median palm Scroll up for a discussion of a project that might be of innarest to you
Hey
Alright
Ooh cool, I'll check it out, thanks :D
Starred the repo :D
@median palm thanks! It's still in the early stages, but it builds on many of the things I was toying with in my earlier compiler project
Just don't let us down, no pressure. π
Be a while before it's useful, but one thing at a time
Get it to work first, right?
Yes, get a proof of concept with a demo (Conway's Life is my favorite demo)
can someone tell e a way to convert pinescript to py
this is strictly a discussion channel. See #βο½how-to-get-help
hi, does anyone know about naka rushton's equation for a neural network project
@obsidian sundial wrong channel try #data-science-and-ml
Ty
I can't wait for ParamSpec to finally be a thing
I've been wondering how many libraries on PYPI that are explicitly about working with data (i.e. not doing Network or Disk I/O) actually mention time-space complexity of their public functions/methods?
Does everyone just code like YOLO, "that's premature optimization, dawg", we'll profile that shit later and optimize it, so just ship itβ’
Probably not a lot, time-space complexity really isn't the concern for most users, especially given how high-level Python already is
I'd assume that if it really required benchmarks, some library could use regression tests as part of their CI
https://pypi.org/project/immutables/ mentions complexities, but most things don't
if you are operating at a scale where complexity matters, your data structures likely aren't written in python
Hello @unkempt rock, this channel is for Python related talks, can you ask in off-topic please?
Should I switch to python 3
You should, Python 2 reached its EOL at the beginning of this year
Is there any major difference?
!ot You can send it in off-topic
Off-topic channels
There are three off-topic channels:
β’ #ot0-psvmβs-eternal-disapproval
β’ #ot1-perplexing-regexing
β’ #ot2-never-nesterβs-nightmare
Their names change randomly every 24 hours, but you can always find them under the OFF-TOPIC/GENERAL category in the channel list.
Please read our off-topic etiquette before participating in conversations.
Well, python 3 has a slightly differential syntax, as well as other structural differences (i think), and python 2 isnt supported at all anymore, while python 3 has loads of support, as well as new features coming to it
I donβt know a lot about Python 2, but I think the main features are that you donβt need all the iterable version of builtins, it is now the default, division of two ints will give you a float, and print is a function
If I learned python 2 how long do u think it would take for python 3
Ok Thanks!
The biggest difference is that strings are sequences of Unicode codepoints rather than sequences of byte values.
That's the one that made porting from Python 2 to 3 difficult. Everything else is trivial in comparison
Right
so if we have something like
x = (1, 2, 3, 4, 5)
if len(x) == 1:
do_something()
will it iterate through x to count the number of items, then compare that number to 1, or will it only iterate until the number of elements counted is greater than one, in which case the condition can't be True?
or is there another way that it gets the number of items in x
does it even need to iterate?
Β―_(γ)_/Β―
It doesn't iterate at all.
I don't think so, tbh
A tuple knows its size.
ok what about a list
Same.
Same, a list knows its size.
oh
Yep, a list is just a dynamic array in memory. It's a single, contigous piece of memory
so if i do
if len(list(mydict.keys())) == 0:
then it's constant time?
sorry i meant assuming that we ignore the time taken to turn mydict.keys() into a list
or... mydict == {}
Or if not mydict:
oooooh
well, I'm not particularly fond of bool coercion, but if you like that, do that
!e
If you had a linked list, you'd have to traverse it, yes.
xs = (1, (2, (3, (4, (5, None)))))
def llen(xs):
if xs is None:
return None
_head, rest = xs
return 1 + len(rest)
print(llen(xs))
@grave jolt :white_check_mark: Your eval job has completed with return code 0.
3
A linked list that didn't store its size. But even for linked lists, that's a common optimization.
Yep, yep,
At least, for general purpose linked lists.
another example is a null-terminated string
doesn't C use null-terminated strings
it does
That's arguably the single biggest shortcoming of the C language.
I'm thinking about writing something that'll call a function based on whether the output of the function ( some file ) is older or newer in time (mtime) than some dependency.
Kinda like makefiles
I thought this would be a thing, but I can't seem to find it, so I'm not sure if it's an awful idea
Hm, seems there's doit, wish I found that earlier
Kinda like a build system? Sounds like fun.
one of the biggest mistakes in Pythonβs design IMO
being a mistake myself, I know what things are mistakes, and bool coercion is not a mistake
why do you like bool coercion? I think it goes against Python's principles, it's very implicit. x == {} or x.empty() would be more explicit and clear
It can also cause silent bugs, e.g.:
if order.is_shipped:
``` this will always succeed, because it's a method, and should've been written as: ```py
if order.is_shipped():
i personally like the fact that it's implicit, it allows for more compactness
bugs are fun
rofl
if not thing:
if thing == []:
Yep. Those precious two characters!
i personally like the fact that it's implicit
think of the zen π
Well, actually, implicit coercion is recommended by PEP 8
which I don't understand
implicit coercion lets us do things like
return primary or get_fallback()
which I always think is neat, but is it better?
probably not, I dont know
it can be a trap with values that can be falsey
but maybe thats the intended behaviour
if primary can be both None and an empty set, maybe you want exactly what you're getting there
well, in that case or is really inflexible because it assumes that there's only one way to inrerpret a particular type as empty
by type I mean a more extended understanding... like, the set of values
the type decides whether it needs to be replaced by the fallback
in the sense that Optional[list[int]] is a "type"
If you go with that interpretation, the semantics for Optional being empty is that it's None.
maybe it could be resolved by introducing some type like Truthy, which is a proxy that's always truthy. So Optional[Truthy[list[int]]]
what do I read to expand my knowledge on types and the surrounding theory? because you seem to be able to think about them in a way that fascinates me a little bit
π© I know less
Well, in general, it's sometimes helpful to think of types as sets of possible values instead of whatever your language calls a type.
@sacred tinsel are you familiar with variance? (invariance/covariance/contravariance)
So function are contravariant in the parameter types. It means that if X is a subtype of Y, then Y -> Z is a subtype of X -> Z, for example:
A = Union[int, str] -> bool
B = int -> bool
int <= Union[int, str]
A <= B
``` but the same can be said of any other contract. For example, suppose that `f` takes a function that expects a positive integer and returns a string. It follows then that `f` can also accept a function that expects a non-negative integer or any integer at all; but it wouldn't necessarily work with a function that expects even integers.
My point is that it can be useful to think of different contracts as of types. Python already does that -- there are things like Union and Callable and Hashable and Protocol which describe contracts, but aren't really types, i.e. classes.
It's not just two characters. They do different things. Did you mean to reject it if someone passed a tuple in? Cause now you do!
but it wouldn't necessarily work with a function that expects even integers.
why not?
Because it expects a function that works with positive integers, so it can call it with 5 or 3.
Well, if you expect any sequence, you should use len or something like that.
okay, so it follows that f can accept a function that expects a non-negative integer because non-negative integers are a superset of positive integers
so we're not violating anything
!e
You could also argue that it's a bug and not a feature -- it will silently ignore empty values like 0 and None instead of throwing a type error:
def multiply_by_two(numbers):
if not numbers:
return []
return [numbers[0] * 2, *multiply_by_two(numbers[1:])]
print(multiply_by_two(0))
print(multiply_by_two(None))
βexplicit is better than implicitβ
@grave jolt :white_check_mark: Your eval job has completed with return code 0.
001 | []
002 | []
also telling that numpy explicitly doesnβt let you do that
you can argue that's a feature and not a bug, too.
hiding errors is a feature?
you're defining it to be an error, not the language
well, yes
the programmer chose a construct that produces some particular behavior when None is given as an input. That may be desired behavior, in which case they leveraged a feature, or it may be undesirable behavior, in which case they have a bug.
I generally like bool coercion, but I have accidentally checked for the truthiness of a function, instead of the result of calling the function. I can see how making some things fail to implicitly convert to bool would be a win, instead of allowing it for all objects.
I think my criteria would be that as long as there's some instance of a given type that could convert to False, I'd allow other instances to convert to True. If every instance of that type will always convert to True, I'd prevent coercion to bool.
yeah, maybe not all objects being coercible is a good idea
how would you determine that
well, let's see. That's statically determinable for every builtin type. There exists some value for int and list and float and dict for which it's falsey, etc. There exists no such value for function or method, etc. For user-defined types, by default every instance of every user defined class is truthy, so user defined types would not be coercible to bool by default. If you add a __len__ or a __bool__ to them, they would be.
same should be true for __hash__, perhaps?..
something similar is done for __hash__
but you wouldnβt be able to determine whether an instance that can convert to False can exist
unless youβre saying
!e
class A:
pass
print(hash(A()))
@grave jolt :white_check_mark: Your eval job has completed with return code 0.
8774510714766
assume that one exists if you canβt prove that it doesnβt
right - we can know the reverse. We can't know for certain whether a falsey value exists, but we can know for certain when no falsey value could exist.
if the object has a __bool__ method, there must be some instance which is falsey (that's how I understand godlygeek's proposal)
why
not proven
@raven ridge :x: Your eval job has completed with return code 1.
001 | Traceback (most recent call last):
002 | File "<string>", line 5, in <module>
003 | TypeError: unhashable type: 'A'
we do something like this for hash, already. User-defined types are hashable by default, unless they define a __eq__ in which case they're not, unless they define a __hash__
it wouldn't be ridiculous to say that instances of user-defined types are not coercible to bool by default, unless they define __bool__ (or __len__ perhaps).
I donβt like boolean coercion in general, but it does kind of fit the Python model
more specifically Iβm against empty containers being falsey
Hm, I sort of disagree with that now.
It would be a contract that if an object has a __bool__ method, it makes sense for it to participate in a boolean comparison, not necessarily that there's an empty element.
An example would be a wrapper like Truthy (<#internals-and-peps message>), which is a proxy type (i.e. allows all other operations) but is always truthy. It is never 'empty', but it makes sense to use it in a boolean context.
right. In the same way as we say that the existence of __eq__ is enough to make us uncertain whether hashing is safe, and therefore prevent it, it's reasonable to say that the existence of __bool__ is enough to make us certain that coercion to bool is reasonable, and therefore allow it.
fair enough
yes exactly
in the hash case, we don't look at what's in the __eq__ - it doesn't necessarily make hashing unsafe, it could be def __eq__(self, other): return self is other
zen is okay in some parts but it's not always best advice
yup
but specifically here, wtf is if []
well, guess what,
if[]==thing:
if not thing:
if thing:
if[]!=thing:
in that case yes, it's shorter
indeed
I don't see this as a question of conciseness, mostly. I see it as a preference for duck typing. Knowing whether a container contains elements in a generic way makes it easier to write duck-type functions.
though I do suppose len can accomplish the same thing. But if len(foo) == 0 makes me feel sad, for a reason I can't put my finger on π
My go-tos for being explicit with collection predicates are if len(obj) and if not len(obj), since people often expect coercion and for integers it is unambiguous.
Although to be specific, this applies to any instance of collections.abc.Sized (i.e. implements __len__)
I find it annoying that dictionaries passed to functions are by reference rather than value, and often use deepcopy at the start of a function to avoid this
It's that common?
I don't want the function to have side effects I guess
I think Ned Batchelder called it passing by assignment, as the parameters are assigned directly to the objects just like a regular assignment would.
I rarely use a deep-copy, although I mostly create new objects if I want to return something instead of mutating the old. Obviously, if you're creating a new outer dict with the elements of the old, you're still creating a situation of potential mutable aliasing (depending on whether or not the elements are mutable).
I think Fluent Python has a few lines on explicitly doing something like:
class SomeClass:
def __init__(self, some_finite_iterable_data):
self.data = list(some_finite_iterable_data)
but that's still only the outer container object.
I donβt think I ever used deepcopy 
Hmm, so it's not a common annoyance?
Can't remember the last time I had to explicitly copy somewhere; in most cases the result is something completely new where it doesn't make sense to pass the old data through
The only copy I used recently was to iterate on a list while editing it
maybe, this is a core python thing which kinda bugs me though, but it's probably just me
i often have
def f(*, _dict):
_dict = copy.deepcopy(_dict)
<do stuff with _dict>
return _dict
I find it a lot more common to build up the new object (eg with a dict comprehension) than to first copy the input and then mutate it
I dont find myself needing deepcopy
This is more often what you want, and using deepcopy in your example is fine.
And as @grave jolt suggested, Python has no value types, so the closest you can get is with immutable.
you mean - more often implicitly altering is wanted?
I've used deepcopy to guarantee that any data passed to certain functions can't be unexpectedly mutated
I've been in the same boat as rie, but I prefer the python approach to variable passing still. I don't want to take a performance hit by unnecessarily having python copy everything on my behalf when I don't need to in most cases
And once you're used to how variable passing works (I support "pass by assignment") it's fairly easy to understand the behaviour
@visual shadow yeah - i mean, i get how it works, i'm just not sure i like it?
tho it does make me feel i'm missing a point
Is it wrong to just say everything is pass by value in Python? Although I don't really think it's all that useful to name it as long as you understand the semantics.
Object references are passed by value right? So everything is pass by value?
yeah, thats exactly what i use it for... tho it doesn't work with matplotlib objects for some reason
It might not specify it's own __deepcopy__ implementation, given most of matplotlib is written in C++.
yeah it just failed : (
also, how realistic is this performance hit? I'm pretty ignorant of optimisation
i know that R copies lists rather than references them though, for example
Mutable objects passed around have caused more bugs than I can count. We (work) started writing extension code for immutable objects, not just containers, and not for performance reasons. Larger the code base, the more likely an accidental mistake with it happens, especially with any amount of concurrency added in.
@urban sandal so this is anti by reference? would you prefer the deep copy behaviour as default too? or not
Still passed the same way, just objects defined in native extensions can be immutable. This is acceptable to avoid issues, and a new object with transforms on it can be created if needed.
apart from the fact that many objects can't be copied, recreating containers for non trivial sizes is usually a considerable amount of work for example
And to check that it can be copied you'd need to check the object on every call which would also be a performance hit
Persistent data structure (like HAMT (https://github.com/MagicStack/immutables )) exist specifically to use immutable data with as little copying as possible. It's not just a dict that can't be mutated.
fair, maybe i should consider, i thought maybe it was something kinda esoteric
AFAIK Cpython uses HAMTs somewhere internally
well, if you have a dict of 10 values, it's probably overkill
i usually have nested dictionaries of survey data
The main problem we had with immutables (and why we cant just switch to it) is there isn't a way to efficiently create nested structures directly (such as during deserialization), also it didn't exist when we started doing this.
Contextvars
This is technically correct, though it's one level below the abstraction level for python. Since python doesn't have the concept of pointers and memory addressing, I find that bringing pass by value/reference just creates confusion for most people
Those terms fundamentally sit at the wrong level of abstraction in the context of Python imo
I disagree with it being technically correct, depending on how technical you want to be. Doesn't pass by value imply the value itself is copied? In Python, passing a class instance doesn't copy the whole object; it passes a reference to it.
It's technically correct if you say the reference is passed by value, just like you could say passing a pointer is passing by value because the pointer is copied.
Yeah, passing a Python object works the same as passing a Java object: it passes references by value
The terms "pass by value" and "pass by reference" are misleading when applied to modern languages. It's a false dichotomy.
C++ is an example of a language that has both pass by value and pass by reference. If you pass a vector<int> by value, it gets copied, creating a new heap allocated dynamic array. If you pass it by reference, then assigning to the parameter inside the function modifies the object. Python doesn't match either of those behaviors.
Python is not pass by value, because if you pass a list to a function, the function doesn't get it's own private copy.
It's also not pass by reference, because if you assign to that parameter in the function, the caller of the function does not see that change.
I have seen the term pass by value reference
Imagine that a user runs the below simple code.
with open(my_file, 'r+') as obj:
res = obj.readlines()
obj.write('bla bla')
Now, I want to be able to write to a different path.
Meaning allow the user to do all the reading they want from my_file but when they
do obj.write('bla bla') I want it to go to my_custom_out_file
How can I **intercept **the write() function to change the out file path?
No, this is for discussion about the python language. #βο½how-to-get-help may give you some more information.
this is a discussion channel about the Python language itself.
Is making them use a custom open function an option?
Otherwise you can look into trying to using mock.patch to wrap the return value of open.
I think a custom open is the approach I will have to go with and update the implementation of write
What exactly does __instancecheck__ do? According to python docs: https://docs.python.org/3/reference/datamodel.html#class.__instancecheck__
it handles calling isinstance(instance, class), but I've tried to implement it and I wasn't able to get it working:
Mind the note,
these methods are looked up on the type (metaclass) of a class. They cannot be defined as class methods in the actual class. This is consistent with the lookup of special methods that are called on instances, only in this case the instance is itself a class.
same reason
def fun():()
fun.__add__ = lambda s, o: 54
fun + 5
``` doesn't work essentially
oh I see, I haven't noticed the note below, metaclass usage makes sense, thanks!
I'd never seen def f():() before, instead of pass or ...
That's some lovely code golf right there π
I see Python almost has universal numbers:
and by that I mean the floating point literal has a precision exceeding double precision before converting to double precision.
kind of
pretty sure they're just the standard f64 types
1e-95 is in range for a double
but isn't double in 2.3E-308 to 1.7E+308 ?
(though I got python to do 1e-323, which must be denormalized?)
interesting
i think it's just cutting the overflow off implicitly
if you do 1e324 you'll get 0.0 so 
!e ```py
import math
print(math.ulp(0.0))
print(math.ulp(0.0).hex())
@raven ridge :white_check_mark: Your eval job has completed with return code 0.
001 | 5e-324
002 | 0x0.0000000000001p-1022
That's the smallest representable float.
cool
And yeah, that's denormal.
!e And the highest, for the sake of completeness, is
import sys
print(sys.float_info.max)
print(sys.float_info.max.hex())
@raven ridge :white_check_mark: Your eval job has completed with return code 0.
001 | 1.7976931348623157e+308
002 | 0x1.fffffffffffffp+1023
I guess I meant it accepts literals with more decimal digits than the decimal precision of double but I guess that is fairly standard thinking about it, perhaps.
confirmation:
in Ada such literals are "Universal Numbers", i.e. infinite precision so would not give same result as above
isnt that basically just Python's Decimal types
@weary garden Your decimal literal only has 1 significant digit, the one at the end, right?
yeah
0.10000000000000000000000000000000000000000000000000000000000000001 obviosuly has more than 1 and more than double
maybe i'm confused about float terminology. it's not surprising that these float literals are supported, is it? they are within range
0.10000000000000000000000000000000000000000000000000000000000000001 is not representable as a double
right
but would be representable as a universal number in Ada
It's within range of a double, but not exactly representable, so you need to round it
double only has 16 decimal digits of precision
how does ada represent it?
it doesn't matter: it is a literal
infinite precision
so it is held as infinite precision by the compiler and then rounded to whatever precision the variable being assigned to is
i see, so the net effect is the same, it ends up rounded
what would the result be?
because I have an expression (subtraction) involving literals not variables
i see
in Ada the result would be 0.00000000000000000000000000000000000000000001 not 0.0
so I will have to DEGRADE my universal number semantic concept so it behaves as expected for Python which is a shame
Where is that behavior of ada documented
google ada universal number
ada universal numbers are numeric literals with infinite precision
@pliant tusk can't find what?
documentation detailing ada having infinite precision float types
i'm experimenting here, but i don't know ada π https://www.tutorialspoint.com/compile_ada_online.php
neither do i, itll be a learning experience
https://docs.python.org/3/reference/lexical_analysis.html#floating-point-literals says "The allowed range of floating point literals is implementation-dependent."
great
sounds like arbitrary precision float literals would be compliant, as long as they coerce down to double precision when needed.
in which case I will use my universal number semantic concept for Python as-is \o/
Python doesn't specify the behavior for signalling nans, but most (all?) implementations treat them as quiet nan
drinking my last bottle of Shiraz: need to buy a new case.
it does distinguish +0.0 and -0.0
I don't think it gives you the ability to distinguish different NaNs from one another.
IEEE 754 says the bit pattern of a NaN is allowed to convey information, but I don't believe Python floats give you any way to inspect it.
I have a broad question (with very low stakes for my current purposes): At what point do you have to start worrying about memory consumption by a dict of dicts? Was kicking around the idea of implementing a simple cache in a class to improve init speeds very slightly. Top-level dicts would be keyed by ~10-character strings, and each internal dict holds about a dozen values--all short strings, ints, or None.
at the point where you can't afford the amount of memory it takes, I guess
if it takes 15 GB but your machine has 20 GB, who cares?
Haha, fair.
Precisely this. Python would, technically speaking, be a pass by value, with just the caveat that all python variables are pointers themselves.
Thus giving you exactly the behaviour you observe. At that stage though, we've introduced two concepts that python doesn't have, and imo should not need to be introduced when talking about just python
Re the cache question and the memory concern for dicts, both sound like premature optimization @hexed aspen
Don't worry about the speed until it actually shows itself to be an issue.
Don't worry about the memory consumption until it actually shows itself to be an issue.
(re memory consumption i may be the devils advocate and say that sometimes you have a decent grasp on a cache that would become too big in production, and there it's fine to think about it.)
I'm still conflicted on when that is the right thing to think about, but its definitely after you've built a working code and seen the timings, you may not need to cache in the first place.
For some reason, we are constantly asked to squeeze Python into the false "by-value or by-reference?" dichotomy.
yeah, agreed. I stick to just saying "pass by assignment" when explaining it, it's been the easiest one to convey by far in my experience
I liked the "assignment" way of phrasing it in your talk, @spark magnet
I've basically just copied it and I'm using it myself a lot now
And also the "why do you worry about it?"
thanks π
Do you have some spite-driven development for this year's PyCon?
π funny, i was just doing some machete-mode debugging that I think will be a blog post today
(that's this one if you haven't seen it: https://nedbatchelder.com/text/machete.html)
That will be an entertaining read.
Ah, yeah, another good talk. I've watched most of them, I think. I do love to watch Python talks.
i should watch more of them
I don't really like the terms, as for a person that doesn't already know how the system works it's indescriptive, and if they do know it it doesn't really need a special term imo
which terms?
I've compiled a list of good talks for the juniors I'm coaching at work. I think a lot of those talks contribute something that books and most courses don't.
pass by value, assignment, reference and the others discussed above
yes, it's good to hear the conversational voice. books can be very "handed down from the mountain"
How would you explain it?
I like the assignment explanation, as it neatly ties in with so many other things within Python that "secretly" assign names to values
Defining a function looks nothing like a simple assignment statement (like it does do in a language like R)
I'd probably explain it whole like any of you would - in a few sentences. I just don't like trying to fit it into a single term like pass by assignments as it really doesn't do much for someone that doesn't already know what's going on
true. a full explanation is warranted.
@spark magnet only a false dichotomy in Python?
@weary garden no, it's a false dichotomy in many other languages.
many?
more than two π
it certainlly isn't in C++
ok
it is in Python, JavaScript, Java, Ruby, ...
perhaps one of the reasons people believe there are only two ways to pass arguments to functions is because they are stuck in C-thinking.
(or their professors are)
passing scalar types by reference has a performance hit though: one reason why C and C++ are classed as "high performance" languages.
is there a way to connect the users spotify with my python app?
i don't see why it would be slower to pass a scalar by reference. You copy the pointer instead of copying the scalar. it's the same amount of work.
because of the potential overhead of dereferencing the pointer which is more likely to be a CPU cache miss
i see. certainly the "everything is an object" model imposes many small costs
will python ever get static typing?
It's a false dichotomy because it suggests that every way of passing parameters is either pass by reference or pass by value. This is not true. Argument passing in Python works neither like pass by value in C++, nor like pass by reference in C++. Same with lots of other languages: Java, C#, JavaScipt, Ruby, etc.
In C++, if you've got
void f(int &x) {
x = 5;
}
int main() {
int a = 10;
f(a);
std::cout << a << std::endl;
}
``` it'll print out 5. You can't do that in Java or Python or C#. They don't pass by reference, you cannot rebind a variable in the calling scope to a new value
Even C only has pass by value. There's no way to rebind a variable in a calling scope with a function in C, you need to use a macro for that, or change the way to call the function (by passing a pointer to the variable instead)
mypy is a separate tool to run over type-annotated Python code. It's not a Python execution engine.
ok
@wide shuttle this one might be too long... https://nedbatchelder.com/blog/202103/machete_mode_tagging_frames.html
I had a puzzle about Python execution today, and used a machete-mode debugging trick to figure it out.
@raven ridge I am sure C++ isn't the only language to support pass by reference and pass by value.
I think it's a good read. It feels long because of the stack traces, maybe, but it doesn't feel like a much longer read than your other blog posts that go into a topic.
Sure, there are probably others. Our point is simply that many people mistakenly believe that those are the only two possibilities.
It's a very creative way to debug the situation that I probably wouldn't have thought of
thanks for reading it. it's hard to step back and see it with fresh eyes π
@raven ridge Java has pass-by-value: the value being passed is the address of an object
that is one way people try to fit the language to the mold, but i don't find it a useful way to talk about it. Python also passes the address by value, but it doesn't give you "pass by value", since now the function can mutate the value and the caller sees the effects.
I think the most important aspect of argument passing is: what do the caller and callee share, and how can each affect the other's view of the data?
I will decide on terminology cromulency after doing a bit of research: the most pervasive view will win out.
In C++, if I pass an array by value, and the called function changes the array, the caller doesn't see the change. In Python, a called function changing a list, the caller does see the change.
the most pervasive, or the most accurate? and whose views are you tallying up?
such cromulency will dictate the names I give to semantic concepts
@weary garden you've never seemed swayed by what the crowd does before, why "pervasiveness" now? π
I didn't say it was.
as neos allows any programmaing language to be specified in terms of semantic concepts the names I choose for such concepts has to be along the lines of the majority view in order to be the most inclusive for the understanding of a broad audience.
That doesn't have the same semantics as pass-by-value in C++. If you pass a vector by value in C++, it's copied. If you pass an ArrayList in Java, it isn't. You can call the Java behavior "pass by value" if you want, but at that point the term is essentially worthless in terms of helping someone understand the behavior
Wikipedia will be my primary source as its peer review effectively meets my requirements
@weary garden Wikipedia calls Java's object technique "Call by Sharing"
I've never heard that one before π
(as well as Python, JavaScript, Ruby, etc)
it goes on to say, "However, the term "call by sharing" is not in common use; the terminology is inconsistent across different sources. For example, in the Java community, they say that Java is call by value."
Python's model is the same as Java's, except without the exception for primitives that Java has, because Python doesn't have primitives.
@weary garden in my experience, people will never agree on the labels, so it sort of doesn't matter what you call it. people will argue with you anyway.
@weary garden for when you get around to it: https://en.wikipedia.org/wiki/Evaluation_strategy
Evaluation strategies are used by programming languages to determine two thingsβwhen to evaluate the arguments of a function call and what kind of value to pass to the function.
To illustrate, a function application may evaluate the argument before evaluating the function's body and pass the ability to look up the argument's current value and mo...
I already differentiate between an argument and a parameter
right: parameter is in the function definition, argument is in the call.
@fiery path what is your talk for PyCon?
as far as Java is concerned it is often claimed that Java doesn't have pointers when the reality is that everything is a pointer in Java as evidenced by the existence of NullPointerException.
npe is kind of a bad name
but just because it's implemented with pointers doesn't mean it is a pointer
semantics.
This is another long-standing debate. I say Python doesn't have pointers because in Python code you cannot manipulate pointers. Same for Java. But both languages rely heavily on references, which C people recognize as pointers under the hood.
It's about the magic of self and gives an informal introduction to descriptors for people who are just getting to know the language a little bit better. I hope to intrigue them and show them some new paths to explore.
C++ references are different from c++ pointers
you don't call a "reference" a "pointer" in c++
cool. descriptors are one of those things i still have to look up the few times i think i need them.
same concept
indeed, my first instinct is to base the naming of my semantic concepts on the C++ model even though neos itself is supposed to be language agnostic.
but c++ refs are different from other langs
the problem is the same names being used for different things at different levels of abstraction
you will face the constant tension between naming things for their mechanism, or their effect.
multiple levels of abstraction is always a central challenge.
I can't think of any other language with pass-by-reference that matches how C++ does it. I'm sure there are some, I doubt C++ invented it, but it's certainly not common.
I might have langauge.static.reference and language.dynamic.reference
true dat
where language.static.reference is effectively an immutable pointer
how many different definitions can we find for "dynamic" π
well maybe language.reference.dynamic given that static languages can have non C++esque references
I'm using the def of "each variable has a specified type" here for dynamic, idk how youre interpreting it
*static
possibly language.gc.reference
there's definitely a non-gc lang that follows pyyhon-style
yes, naming things is a hard problem
OK, I think I will settle on language.pointer language.mutable_reference and language.immutable_reference
personally I would do something like
"Pass by copying" for c-style value copying
"Pass-by-assignment" for python style
"Pass-by-reference" for c++ refs
wb c primitives
well these concepts aren't just involved in argument passing
what's the difference between "pass by copying" and "pass by assignment"?
What's the difference between "mutable reference" and "immutable reference"?
So that's the one that Python has, not the one C++ has?
Assignments to references in C++ trigger operator= and mutate the object. So in that sense, the reference type that you're calling "immutable reference" has the behavioral difference of being usable to mutate the object by assigning to them
Meanwhile, the references in Python, which you're calling mutable, can be rebound, and can mutate the passed object, but cannot rebind the passed object
yeah I am happy with that. pointer and reference_pointer.
Ada with out type qualifier and I believe C# have it with the ref keyword I believe are some examples.
It would be interesting if you could override the assignment operator in Python, you might be able to "mimic" reference types in some horrible way.
Except that would break the entire language π
Hahaha certainly.
language.gc.reference is language.reference_pointer
language.reference is akin to a C++ reference
AKA an alias
language.reference_pointer is a pointer you can change the address of but cannot manipulate like a C pointer
Is there a way people annotate function arguments in Python to say that they treat it as const?
I've looked through typing and can't seem to find anything, but feel it would be useful at least for documentation's sake?
Am I perhaps missing something?
When you say "const", do you mean that the argument won't be reassigned, or that the object referred to by the argument won't be modified through that reference?
No, doesn't exist.
laughs in Object π
as long as you put things in the right box, you've got dynamicism!
I forgot--how are chained comparison operations evaluated again? Given a, b, c, where those are each numpy arrays of the same shape, a == b == c has the intended semantics if it's evaluated as (a == b) == (b == c) (I think), but I don't think that holds up for lt, gt, le, or ge.
I don't think that will work because I'm pretty sure it's evaluated as a == b and b == c at which point it will try to case a == b to a bool which numpy doesn't like
>>> import numpy as np
>>> a = np.array([1, 2, 3])
>>> b = np.array([1, 2, 3])
>>> c = np.array([1, 2, 3])
>>> a == b == c
Traceback (most recent call last):
File "<pyshell#4>", line 1, in <module>
a == b == c
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()```
!e
import numpy as np
val = np.array([1, 2, 3]) == np.array([1, 2, 3]) == np.array([1, 2, 3])
print(val)
@boreal umbra :x: Your eval job has completed with return code 1.
001 | Traceback (most recent call last):
002 | File "<string>", line 2, in <module>
003 | ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
I see. For some reason I thought chained comparisons were possible with numpy
Though I do realize you could do it manually with bitwise operators.
a == b == c is the same as a == b and b == c
(a == b) == (b == c) doesn't make much sense
not for numpy arrays, I don't think
because those return boolean arrays
a == b == c is equivalent to a == b and b == c in any case.
What experience do you have with numpy?
@grave jolt :white_check_mark: Your eval job has completed with return code 0.
001 | 4 0 LOAD_GLOBAL 0 (a)
002 | 2 LOAD_GLOBAL 1 (b)
003 | 4 DUP_TOP
004 | 6 ROT_THREE
005 | 8 COMPARE_OP 2 (==)
006 | 10 JUMP_IF_FALSE_OR_POP 18
007 | 12 LOAD_GLOBAL 2 (c)
008 | 14 COMPARE_OP 2 (==)
009 | 16 RETURN_VALUE
010 | >> 18 ROT_TWO
011 | 20 POP_TOP
... (truncated - too many lines)
Full output: https://paste.pythondiscord.com/jireqogaro.txt
It literally doesn't matter what objects those are.
But numpy doesn't use those operators to return bools
and?
if I did a == b == c with numpy arrays, I wouldn't want an instance of bool at the end. I would want an array of bools.
but if internally it's doing what the and and or operators do, that won't work
and doesn't make a bool
!e
import numpy as np
val = np.array([1, 2, 3]) and np.array([1, 2, 3])
print(val)
@boreal umbra :x: Your eval job has completed with return code 1.
001 | Traceback (most recent call last):
002 | File "<string>", line 2, in <module>
003 | ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
!e
class Foo:
def __init__(self, label):
self.label = label
def __eq__(self, other):
rv = Foo(self.label + other.label)
print(f"== is called on {self} and {other}, returning {rv}")
return rv
def __bool__(self):
print(f"__bool__ is called on {self}")
return True
def __repr__(self):
return f"Foo({self.label!r})"
Foo("a") == Foo("b") == Foo("c")
@grave jolt :white_check_mark: Your eval job has completed with return code 0.
001 | == is called on Foo('a') and Foo('b'), returning Foo('ab')
002 | __bool__ is called on Foo('ab')
003 | == is called on Foo('b') and Foo('c'), returning Foo('bc')
What exactly do you want to do?
Compare two arrays elementwise?
That's https://numpy.org/doc/stable/reference/generated/numpy.equal.html, it seems
My original point was that I wasn't sure how the expression a == b == c could return a single boolean array of elementwise comparisons, and it apparently does not.
!e
import numpy as np
val = np.array([1, 2, 3])
print(val == np.array([1, 2, 3]))
print(val == np.array([2, 3, 4]))
@grave jolt :white_check_mark: Your eval job has completed with return code 0.
001 | [ True True True]
002 | [False False False]
because if it internally breaks that statement down into (a == b) and (b == c) (which I learned here just now), I don't believe that would work because of what it appears np.array.__bool__ is doing
a == b returns a numpy array, and when Python tries to do and, it needs to find the truth value of the left operand first. So it tries to do bool(a == b), which fails, because of the reason in the exception message
Right
So your question is how to get an array of truth values where all three arrays are equal to each other?
so I'm not sure what you meant by this message. is there some reason why a == b and b == c would evaluate to the desired boolean array, or did I misunderstand what you were trying to say?
I'm not sure what your question is about now
My original question was "how can a == b == c evaluate to a boolean array?" and the answer is "it can't"
right
sorry, I misunderstood you then
(but a == b == c really does mean a == b and b ==c)
when comparison operators uphold the expectation that they return a single instance of bool, yes
(or they have implicit truthy/falsey values that "make sense")
I'm not sure what you're saying
it's not really that important
!e
class Foo:
def __init__(self, label):
self.label = label
def __eq__(self, other):
rv = Foo(self.label + other.label)
print(f"== is called on {self} and {other}, returning {rv}")
return rv
def __bool__(self):
print(f"__bool__ is called on {self}")
return self.label != "cd"
def __repr__(self):
return f"Foo({self.label!r})"
Foo("a") == Foo("b") == Foo("c") == Foo("d") == Foo("e")
@grave jolt :white_check_mark: Your eval job has completed with return code 0.
001 | == is called on Foo('a') and Foo('b'), returning Foo('ab')
002 | __bool__ is called on Foo('ab')
003 | == is called on Foo('b') and Foo('c'), returning Foo('bc')
004 | __bool__ is called on Foo('bc')
005 | == is called on Foo('c') and Foo('d'), returning Foo('cd')
006 | __bool__ is called on Foo('cd')
As you can see, first it takes Foo("a") == Foo("b"), which returns Foo("ab"), which is truthy.
Then it takes Foo("b") == Foo("c"), which returns Foo("bc"), which is truthy.
Then it takes Foo("c") == Foo("d"), which returns Foo("cd"), which is falsey, so evaluation is stopped.
!e
import numpy as np
bool(np.array([1, 2, 3]))
@boreal umbra :x: Your eval job has completed with return code 1.
001 | Traceback (most recent call last):
002 | File "<string>", line 2, in <module>
003 | ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
Numpy deliberately sabotages efforts to get the implicit boolean value of arrays.
I know
well, when bool(a == b) raises an exception, a == b == c is the same as a == b and b == c as well π
I dislike that numpy deliberately breaks bool(array) is there any reason that they didn't want it to behave like lists?
There are two possible cases: at least one element of the array has a truthy value, or every element of the array does. And instead of giving preference to one, they made it so that you have to pick.
If they wanted to make the arrays behave as lists, that wouldn't be a problem.
That is, having a non-zero number of elements?
Yes
What about np.array([[], []])?
What would you get if you ask for the size of that array?
(2,0), I think?
Those are the dimensions
Numpy will say the size is 0
!e
import numpy
a = numpy.array([[], []])
print(a.size)
@wide shuttle :white_check_mark: Your eval job has completed with return code 0.
0
Since that's the definition of size in numpy arrays, I'd say the analogy with a list would be to be falsy. But it does get muddy.
The exception above hints at another interpretation though
Whether or not any or all elements are truthy, which is different from the interpretation of the truthiness of a list.
I think the numpy way is fine. It makes sense in the context.
!e
import numpy
print(len(numpy.array([[], []])))
@boreal umbra :white_check_mark: Your eval job has completed with return code 0.
2
Alright
I think typically, the contract is that objects that implement __len__ should be truthy if the length is > 0
though numpy arrays (and objects that use the data model similarly) have their reasons for being an exception
That means that we can have a consistent bool for numpy arrays as well
For the record, I'm not necessarily arguing that the designers of numpy were correct to make the design choice that they made. Or that some other design would be better.
I think this brings back to the discussion we had earlier, about conversion to bool and that not everything should be allowed to be treated as a boolean
@gleaming rover @raven ridge
The numpy behaviour seems correct to me, whether a numpy array is empty is slightly ambiguous
but isn't it also kinda ambiguous whether [[], [], []] is 'empty'?
why should that be empty
its a list filled with three empty lists
just because the glasses are empty doesnt mean that the cabinet is empty
Lists are mutable - if we accepted a definition where [] was empty but [[]] was not empty, wouldn't it be odd that putting an empty list into an empty list resulted in an empty list of length 1?
did you make a typo?
didnt we accept that [[]] isnt empty
er, yes, I did make a typo
if we accept that [] and [[]] are both empty, then there would be empty lists of length 1
or 2 or 3 or whatever. Or arbitrarily nested, for that matter. [[[]]] would still be empty.
It depends on what the list represents. If a list represents e.g. a game board, [[], [], []] might semantically be empty.
but I understand that a list being empty has a clear meaning in Python
or to take the numpy example -- [False, False, False] might mean an empty/falsey result, so you have to use all or any in that case
Yeah. I can see how it could have been defined that way. There's a very good reason not to define it that way, though - that cannot be implemented efficiently
god all this theoretical programming talk
not really my cuppa tea but interesting nonetheless
is the literal point of this channel π
well im more practically oriented
Same. I just watch this channel and hope to get smarter.
if it works dont touch it, thats my philosophy
This is more of a "it works, but how"
My general issue with implicit conversion to bool is that it assumes that there's only one way in which something, e.g. a list, can be empty
Well, not really - it just means that they have to pick one thing for bool(x) to mean
right
in numpy they decided not to pick, which is in itself a thing they could pick
they picked "it's meaningless" - or ambiguous, I guess.
yeah, IIRC we agreed that objects shouldn't define __bool__ by default
yeah.
remember, though, that back in the Python 2 days, there was no __bool__, there was only __nonzero__
and from that point of view, it's clearer how we got where we are
"a list is non-zero if it contains more than 0 elements" is pretty clear.
yeah, that's actually probably more accurate.
not a way I would have thought of it, but it makes sense as well.
well, it sort of makes sense on values that define +
0, "", [], ()
also False with respect to |/or
and {} with respect to |
and {*()} with respect to |
and bytearray() and bytes() with respect to +
yeah
Can you compile python, to native performance?
Not all of python, but numba can compile some of python to native performance
Native performance being as fast as cpython?
native performance being as fast as the functionally equivalent code in C
heyy can anyone say how to make a modmail bot from discord.py?
numba doesn't compile python, but a specific subset of python it can compile effectively
hello, this question would be better suited to #discord-bots
ohh sorry, actually i am new so got a little confused
What can't it compile?
https://numba.pydata.org/numba-doc/dev/reference/pysupported.html anything but the things listed here
notably heterogenous lists and dicts, as well as non-list comprehensions
... you can use pypy it is much fastesr than py ...
I use pyinstaller to compiler little scripts, beware of your antivirus it often flags some of those .exe has bad virus, at least my company antivirus ...
PyInstaller doesnt compile so much as wrap
It wraps the interpreter and code together which is what sets the whole AV off because it looks and behaves like alot of viruses that spawn internal processes
yeah and that is the source of false virus detection too, dumb antirus detect the packager and trig a virus alert, ... you sould see all those IT guys running around like crazies ...we got a virus ... we got a virus lol
I think if you recompile pyinstaller's bootloader yourself you can avoid that
i mean i dont blame them
an awful lot script kiddy viruses are made with Pyinstaller which doesnt help
but equally the whole system isnt something you'd massively distribute
the only distributed Python app ive seen that's wrapped with PyInstaller is Docker's CLI and compose
yeah I guess
AFAIK, the PyInstaller trojan positive is only triggered by onefile bundles. The strong point of PyInstaller is the ease of use, things like packaging PyQt into an executable are usually not so easy with other libraries. But there are way more performant alternatives like PyOxidizer, which is also a packager that embeds the Python interpreter, like PyInstaller; or Nuitka, a Python compiler that can build onefile executables.
@grizzled grove #data-science-and-ml might be the best place to ask
Oops, sorry for that. π
!e
print(*b'a')
You are not allowed to use that command here. Please use the #bot-commands channel instead.
Cool I will look at Nuitka π
!e ```py
print(*b"abc")
@raven ridge :white_check_mark: Your eval job has completed with return code 0.
97 98 99
That's because *obj in a function call iterates over the object and passes each element as a seperate argument, and iterating over a byte string gives you the value of each byte as number between 0 and 255.
They have a repo with utilities that you might also want to take a look. I think it's not linked in docs.
https://github.com/Nuitka/NUITKA-Utilities
Is there a way you could reverse that? Such as the ascii values into string
reversing it means getting back to a bytes object, rather than a str object
bytes([ord('a'), ord('b')])
!e ```py
print(bytes([int(num) for num in "97 98 99".split()]))
@raven ridge :white_check_mark: Your eval job has completed with return code 0.
b'abc'
quick question that doesn't deserve a help channel:
does c++ have a structure similar to python's with open() that doesn't need a .close()?
c++ version:
void readfun() {
string nyNum;
ifstream MyReadFile("filename.txt");
cout << "\nDisplaying lines from a text file. \n";
while (getline(MyReadFile, myNum)) {cout << myNum;}
MyReadFile.close()
}
python version:
def readfun():
print("\nDisplaying lines from a text file. \n")
with open("textfile.txt","r") as f:
for line in f: print(line)
Any help would be appreciated, as well as an @ if you could. Thanks!
@devout kelp this would be better suited to an off topic channel. C++ uses destructors for resource management, since they are predictable, unlike python destructors, which can run sometime after the file goes out of scope (though in cpython they are also predictable).
thanks, and will do! @flat gazelle
In addition for things like file objects instead of wrapping it in a class, you can use std::unique_ptr's deleter template argument. Just check it out on cppreference.com they have an example.
xored = tuple(int(x) ^ _XOR_MODE_ for y in _bins for x in y if x.isdigit() else x)
``` I have syntax error on that else statement, but I can recall that something similar have worked for me in the past
So what is wrong with that else statement, how to fix that?
You should ask in a help channel as this channel is not suited for questions like that, but you're trying to use an incomplete conditional expression as python doesn't allow an "else" for the if part of a comprehension
why is list.copy a thing?
the language has a dunder for copy.copy
just make users use copy.copy instead of having both the dunder and the method?
Wouldn't be surprised if the method predates the module
it would be weird if they didn't deprecate it
i have never used the method
type and .__class__ are both not deprecated π
well, .next got removed.
I think deprecations on builtins are a bit more conservative. Was a bit surprised recently when I found out inspect has the getmro function
i doubt any of the (list | dict | set).copy will be removed
it's hugely different, .class is still quite common to see
but they do the same thing, right?
list, set, dict .copy are much less common to see
yes
what is the recommended thing for user defined mutables?
implementing the dunder right?
obj.__class__ gives the objects class. type can take 2 different sets of args, a single object, or (str, tuple, mapping) which creates a new class
the fact that type(obj) gives the objects class is a special case
oh, is it an implementation detail too though?
no thats a detailed part of the language afaik
i use these often for functions that take in a mutable that i dont want to modify
Yeah I use dict.copy a lot when iterating over a dict that I'm going to change
Well, .next probably should have been a dunder all along. It was always meant to be called automatically by the interpreter, not manually by the programmer. And backwards compatibility for that during the 2 to 3 switch was just ```py
next = next
@unkempt rock this channel is for discussion. Take a look at #βο½how-to-get-help
is there a good way to communicate unused parameters via * & **? _ is applicable for an argument, but obviously can't be used for both
tho I still get whining from IDEs about unused parms with __
Good question. I don't know of anything nor do I recall seeing any attempt at that in the wild. I would approach it by either leaving them named as args and kwargs or maybe use a double underscore for one of them.
And just ignore the lint error or whatever you're getting.
isn't _ special cased in the python environment? it seems weird I can't use it for both args
I can always throw # noqa at it, but I don't like swallowing errors if I can fix them π’
No it isn't special cased (unlike e.g. C#). It's just a naming convention.
it usually holds last-evaluation, which means it's somewhat special at least
unless that's a REPL trick
It's special cased in the repl only.
ahh
And in pattern matching in 3.10, where it will become a special case in the context of a match statement.
as a catch-all, ye?
Yeah.
Other than those 2, it's just a regular variable
And even in the repl, if you assign to it, it stops being special until you del _
huh, didn't know that TIL
It's because the repl _ special case puts the last expression into __builtins__._
So if there's a global or local variable called _, it shadows the built-in.
I'm at least happy that the _ special case in pattern matching doesn't rebind _. That would have been messy with those gettext aliases.
There are a few rebinding pitfalls, such as rebinding names of partially matching (non-matching) patterns
Which is also implementation dependent (as it's deliberately undefined behavior)
why deliberately?
that doesn't make it acceptable, however
"the way it's always been" is not great thinking π€·
It's not "the way it's always been"
"C is full of it" implies that to me
It has a specific reason: It allows implementations to optimize how they handle it
C is full of it because such optimizations are historically important in C
but that's kind of an important behavior to have a distinction around
whether a variable is bound or not
Something as simple as evaluation order of the operands is undefined in C
Have i introduced you to our lord and savour Rust lang?

Haha
I'm already there
Something as simple as this contains undefined behavior in C:
a = 3;
b = a * a++;
I'd write Rust if I got paid to do so but my boss worries my coworkers wouldn't be able to learn it (webdev python team)
postfix doesn't have priority?
It's because the order of subexpressions isn't defined
Reading the same variable twice without a sequence point in between is undefined behavior.
And only a specific, small, set of things are sequence points
So, if you have (a + b) * (c - d) either subexpression may be evaluated first
So, in the case above, even if we were to write parentheses, (a) * (a++), it matters whether we do the righthand subexpression first or the lefthand one
In any case, I was going to share this interesting example of structural patma:
a = "some value"
name = (1, 2)
match name:
case (a, 3):
print("This does not print, as it doesn't fully match")
case _:
print("This does, catch all")
print(a)
The first case, (a, 3) is a partial match, with the assigned before the mismatch, which, in the current CPython implementation, will lead to a being reassigned
So the last print will print 1 and not "some value"
If we were to do this:
a = "some value"
name = (1, 2)
match name:
case (2, a):
print("This does not print, as it doesn't fully match")
case _:
print("This does, catch all")
print(a)
We'd still print "some value"
Since our matching steps failed before we reached the a rebinding
The specification leaves open that other implementations solve this in a different way, with different rebinding results
Honestly i still prefer python web dev to rust
Just one of the things where i feel the dynamic nature of python makes it so much nicer to work with
quick question -- is it uncommon for Python programs to actually use structs to represent data structures? Where somewhere like Rust or Go I'd use a struct, in Python I end up creating a dict with string keys
Python is, after all, strongly typed
and it would make sense for me to be able to say "here's a type, it composes X and Y and Z fields"
Seems like you're describing a class, which are very common.
If you mean a plain old data, no extra methods class, there dataclasses
the problem with classes in Python is that they are very loose and end up becoming too verbose for the purpose of simply defining a structure
i.e. let's imagine a hypothetical
struct Coord {
x: f64
y: f64
}
In thay case a named tuple is probably what you want
Or a dataclass.
Either or yeah
a "dataclass" sounds semantically relevant
I don't remember that being a thing in Python
I'd usually prefer dataclass over namedtuple unless you specifically need unpacking/indexing
Dataclasses be easily frozen aswell making immutable for the most part
And same
I use dataclasses way more than named tuples
No?
Dataclasses basically just remove all of the boiler plate and give you the option to add and remove certain features
New in version 3.7.
Think of it like a basic derive macro in rust
I'm just shocked at the variable type specifier
last time I touched Python was 2.7 iirc
Thats just typehints
...and this was irc.freenode.org/#python π
Do mind that it's just a type hint
