#internals-and-peps

1 messages ยท Page 108 of 1

grave jolt
#

Well, I guess it's a matter of taste. I find it weird that it's not the same as +, and I've seen many beginners here fall into the trap of += on lists.

halcyon trail
#

it is mostly the same as + though

#

the result is the same, it just mutates in place

grave jolt
#

yes, it's subtly different

halcyon trail
#

realistically, if you have a list, and you give someone else a reference to that list

#

you should not be hoping that they don't mutate it

#

it's a mutable type

#

so, to look at x += y and say that can mutate someone else's copy, I think is very definitely fixing the problem here at the wrong location

grave jolt
#

I'm not saying that it's bad because it's mutating someone else's copy

halcyon trail
#

but that's the only difference, no?

#

aside from those really weird cases where the assignment fails

#

but the mutation has already occurred

grave jolt
#

I can't provide specific cases, but newcomers are already pretty confused about mutating an object vs reassignment

halcyon trail
#

I found it very confusing, but in the other direction

#

C++ and C don't have notions reassignment

grave jolt
#

yeah, they have different semantics altogether

halcyon trail
#

well, their "reassignment" is a mutation (although, in C it's hard to observe the difference)

#

So stuff like x = y, and then mutating y changing x, was mind blowing

#

it was an actual moment of "oh, this is what those FP immutable people are always complaining about... I can actually respect that"

grave jolt
#

in FP language, there's usually reassignment but no mutation

halcyon trail
#

I've never been burned by x += y specifically. But I've been burned many times by just this general web of mutable state.

#

Yeah, if it's actually pure, which most aren't, they tend to default to immutable data structures

#

though

#

But yeah, it is unfortunate that it hasn't caught on more IMHO

#

at least, immutable by default, is pretty easy to work with still, whip out an immutable structure that one time you need it

#

*a mutable structure

grave jolt
#

I don't program in Clojure, but I like the way Rich Hickey describes it: there are 'values' and 'places'. A value:

  • is immutable
  • doesn't have identity
  • contains data
    And a place:
  • is mutable
  • has identity
  • stores a value (which is immutable)
halcyon trail
#

I guess I'd have to watch the talk

grave jolt
#

Basically, he suggests to use immutable values, and to use reassignment (not mutaiton) if you have to.

halcyon trail
#

I agree that should be the default

#

there are cases where that's just difficult though

#

but, definitely the right default

grave jolt
#

yeah

halcyon trail
#

definitely the mutability situation in python is IMHO pretty bad even for a mainstream language and it's one of my bigger beefs with it I think

grave jolt
#

it's also made more difficult by the fact that Python has no built-in persistent data structures

halcyon trail
#

yes

#

and even if there were nobody would use them

#

because the comprehensions are first class magic

#

and always return, and probably always will, the mutable data structures

#

this is actually what convinced me that comprehensions are just a bad idea in general

grave jolt
#

actually, maybe you're right. += mutating is not that bad, given it's consistent across list, deque and dict & set (with |)

halcyon trail
#

There are persistent data structure libraries for python but it's pretty awful because now you can't use [x for x in ...]

#

everything has to be wrapped with the name of the persistent data structure

#

and if you have a team of people programming python for a while remembering that on every single line you write is just a pain and almost impossible really

swift imp
#

what is a persistent data structure?

halcyon trail
#

More or less another word for an immutable data structure

grave jolt
halcyon trail
#

although, persistent data structures usually fake mutation in a specific way

#

you can "mutate", but it doesn't really mutate, it gives you a new "copy" of the data structure, but the new copy reuses bits of the old version

grave jolt
# swift imp what is a persistent data structure?

It's an immutable data structure that avoids copying data. For example, you can make a linked list like this:

x = (5, (4, (3, (2, (1, None)))))
y = (6, (7, x))

here you've made two different linked lists, but they share most of the data in memory

swift imp
#

I understand, thanks.

halcyon trail
#

the problem tends to be is that real immutable data structures tend to get pretty complicated and have to make more performance trade-offs

#

a mutable array for example is insanely simple, fast, and useful data structure

#

comparatively, IIRC there is no persistent list with O(1) random access

static bluff
#

You guys are still arguing about this?

#

XD

dawn arch
#

"discussing"

static bluff
#

Ahh yes

halcyon trail
#

lol why is it in quotes? Felt like a discussion to me ๐Ÿคทโ€โ™‚๏ธ

grave jolt
#

with GOTOs and such

dawn arch
#

i was just teasing

unkempt rock
#

I need help urgently

#

anyone please help me

dawn arch
#

there are help channels

static bluff
#

I was wondering if I could ask you guys an OSX related question. I've tried asking on the off topic but its pretty obscure

radiant fulcrum
#

you could try #unix @static bluff

static bluff
peak spoke
#

It would break all the existing code if you did a rename.
I'm not sure why snake case aliases weren't added to some of the modules with outdated naming but it would still be something that'd need to be discussed beforehand

charred wagon
#

It's a backwards incompatible change

#

They wouldn't accept it

#

Sounds more reasonable. You still should propose it rather than jumping the gun with a PR

#

I believe you just open an issue in their tracker. Or you could discuss it on discuss.python.org

rich cradle
#

Ah, Mark got it

peak spoke
#

I'm sure something like this was already discussed before on either the tracker or mailing lists

#

There are some other problems if you just bind an another name to the methods when you take subclasses into account

#

if the subclass overrides the method, the other name won't be updated to the new reference

visual shadow
#

possibly, though i'd argue that snake_case is the obvious way to do things in python-speak

#

ofcourse, i use pytest and call it good. This isn't a hill worth dying on.

peak spoke
#

iirc pyside6 includes snake case names as one of its new features, could look at how that was done as I think it was done through an import but the source was not very accessible last time I checked

halcyon trail
#

I think the value is just very low relative to the effort

#

it's a guarantee that it's been discussed before, and almost a guarantee that someone has offered to do the work. programmers like painting the bikeshed.

blissful comet
halcyon trail
#

I still can't really wrap my mind around the premise here. Like, the idea of using this to generate some C++, and then what?
If you never look at the C++, then just use some other language that compiles to naive
If you want to maintain the C++ and modify it later, you'll need to know C++, and you'll need to think about many things that this thing cannot magically get right

#

I actually am really curious to pump in certain python code and see what C++ comes out but I'm too lazy to install it

raven ridge
#

That's a bad idea for a function - it means you redefine the inner function once per call to the outer function, which is wasted effort if the inner function isn't a closure. But with a class inside a class, it's an even worse idea - because you're defining a new inner class each time the outer class is constructed, those inner class instances will all be of different types.

#

Given your example, if you do

c1 = C()
c2 = C()
assert type(c1.x) == type(c2.x)
``` that'll print `False`, because each call to `C` created a new, different class.
halcyon trail
#

yeah.... I've seen patterns like this before, in fact someone gave an example of how to decorators that involved creating nested classes

#

and the nested class even captured state

#

so, instead of just writing a class, that explicitly maintains some state used for the decorated function, and defines call

#

he wrote a function that basically spit out a class that was really a singleton, no instance fields but just captured global state

raven ridge
#

For decorators, where your goal is to create a distinct class each time the thing is called, that's fine. But if creating distinct classes is an unintended side effect, that's a bad idea

halcyon trail
#

it was very very strange. And this person was giving a talk.

magic python
halcyon trail
#

No, I mean, this was a decorator for a function

magic python
#

I mean, if it's a few lines and passed to an apply call in pandas it seems fine, and the readability makes up for it? Maybe but ๐Ÿค”

raven ridge
#

But in the case of an inner class, that changes the contract, and is likely to violate user expectations

#

Like "if I create two instances of this class, their attributes have the same type"

magic python
#

Interesting, I really need to go through some of the examples people are referring to here re types and stuff, as I haven't considered it before

#

Probably doesn't help that I'm currently using classes as data storage ๐Ÿค”

halcyon trail
#

I'd also suggest steering clear of nested classes. Nested functions are ok, they can help with readability. But classes are more complex creatures to start with, you'r edefining types now in a weird way which may confuse type checkers, and now you have both explicit state of the class and possible implicit captures

#

a nested function only has the implicit capture, a normal class has only the explicit state. A nested class potentially has both.

blissful comet
#

Clang-format or rustfmt

halcyon trail
#

So, clang + clang-format?

#

gotcha

#

I'll give it a go. I have the following python in mind, I want to see how it translates it to C++:

def getOrDefault(d: Dict[str, str], key: str, default: str) -> str:
    if str in d:
        return d[key]
    return default
#

My guess is that it will just take the easy way and return by value

blissful comet
#

A python programmer would probably write

d.get(key, default)

halcyon trail
#

Sure, I would just write this explicitly to force it to generate it

blissful comet
#

There is a quick map of how to translate such idioms to target language. Not sure if I added this one.

blissful comet
halcyon trail
#

the thing is that C++ programmers when they try to write a function like that, will return by reference, which actually is a bug

#

the actual C++ answer is you just don't have a function like that because there's no way to write it that's both safe and efficient

blissful comet
#

Yeah the type inference code can be taught about life times

halcyon trail
#

yeah, you just can't get around the fact that different approaches are idiomatic in different languages, things don't translate, etc

raven ridge
#

C++ doesn't even have a standard library type analogous to Python's str type.

halcyon trail
#

i don't think anything like that has ever caught on, the closest I guess is something like protobuf where you have a very limited DSL for data exchange that pumps out code in multiple languages

#

well, depends I guess on how tightly you define "analogous"

blissful comet
raven ridge
#

There's no standard library type in C++ that is a sequence of Unicode codepoints.

halcyon trail
#

std::string is definitely the analogous types

visual shadow
#

more like C-- am i right? ๐Ÿ˜› (i'll see myself out of here)

halcyon trail
#

but it has a lot of mismatches

#

and yeah, unicode

raven ridge
halcyon trail
#

std::wstring then, I suppose

raven ridge
#

Closer, but still no

#

An array of wchar_t is still not an array of codepoints.

halcyon trail
#

analogous is not identical to identical ๐Ÿ™‚

blissful comet
halcyon trail
#

is analogous analogous to identical.... ๐Ÿค”

raven ridge
grave jolt
visual shadow
#

oh my word..that's a thing?

halcyon trail
#

I don't really agree, but that's ok

blissful comet
#

Would std::u8string be a better option? Don't mind requiring C++20

visual shadow
#

and this is why we have so many encoding standards. get it together people!

raven ridge
#

A Python str is a dynamic array of Unicode codepoints, which are 21 bit integers.

blissful comet
#

I don't aim for binary compatibility with that. Only source level compat.

flat gazelle
#

it has different time complexity for element access

#

and more importantly, slices

raven ridge
flat gazelle
#

honestly, if unicode didn't assume they could fit everything neatly into 65k codepoints, things would be a lot better with regards to this

grave jolt
# visual shadow oh my word..that's a thing?

yeah https://en.wikipedia.org/wiki/C--

C--

C-- (pronounced see minus minus) is a C-like programming language. Its creators, functional programming researchers Simon Peyton Jones and Norman Ramsey, designed it to be generated mainly by compilers for very high-level languages rather than written by human programmers. Unlike many other intermediate languages, its representation is plain ASC...

raven ridge
#

Those 64k are the basic multilingual plane. We're well past that. Have been for longer than emojis have existed, since I believe all emojis are outside the BMP

flat gazelle
#

they initially did and C# and java and windows and to an extent C and C++ went with UTF-16 as a result. But turns out it isn't enough and now you have wchar_t which is 16 bits

blissful comet
raven ridge
halcyon trail
#

I am pretty thankful on a regular basis that my job requires me to literally know nothing about unicode

raven ridge
#

That's not perfect, but at least analogous.

halcyon trail
#

it really depends in what sense you are defining "closest". If in terms of functionality, sure. If in terms of idiomatic usage, there's realistically a mountain of C++ code there that uses std::string, as, well, a string

#

note as bytes

raven ridge
#

Sure. And you could use UTF-8 byte strings because of that. But then you can't support constant time subscripting.

halcyon trail
#

Sure, my point is just that it depends in what sense you consider the word "analogous"

raven ridge
#

Yeah. I meant "supporting the same features"

halcyon trail
#

in many (most?) of the domains where C++ is used, if you wrote teh same program in python and C++, you would substitute std::string for str

#

(or, particularly on windows, perhaps wstring)

blissful comet
halcyon trail
#

that to me is "analog". The same way that the analog of say, a list in python, is Vector in scala

#

But Vector is an immutable list that doesn't really provide O(1) access. But, i fyou were translating python code, that's how you would be encouraged to do it.

#

What's important is more if they are used in similar situations rather than having identical capabilities. But, I guess it's all the semantics at this point

raven ridge
# halcyon trail in many (most?) of the domains where C++ is used, if you wrote teh same program ...

I agree, and what you're saying is true, but that's not enough for something that wants to translate code in one language to equivalent code in another language. If you translate str blindly to std::string, then the behavior will not be equivalent unless the characters are all ASCII. The fact that this isn't much of a problem for C++ programmers doesn't mean it's not a problem for a program generating C++ code.

#

At least, it is if you want it to work

halcyon trail
#

Sure, that's a fair point in a code generation context

#

I can't imagine this actually working anyway on that level though, tbh

#

I initially thought it was purely for pedagogical purposes

raven ridge
#

That would make sense. The point I was driving at was that, if you take idiomatic Python code and try to generate C++ code from it, the C++ code will either not be equivalent, or will be very far from idiomatic, needing to go as far as creating its own string type

visual shadow
#

re Vector vs list, i wouldn't call Vector analogous for the very same reasons you mentioned

#

to me, they're too different.

#

If a language doesn't have classes for example, and you could achieve the same "work" in a different language with just functions.. well you wouldn't be calling that language X's classes as analogous to language Y's functions would you

raven ridge
visual shadow
halcyon trail
#

"just use int" hah, you can get a 45 minute convo going in the cpp slack

#

by saying that

#

I default to int64_t for example

#

@visual shadow I would say they are analogous because they are both the default choice, ordered, random access supporting, containers, for their respective language

#

list is going to be your default sequential container in python and Vector is goign to be your default sequential container in Scala

#

their API's look a big different and their big O is a bit different, but they are used for the same set of purposes most of the time, and they are both the most used choice in that role

#

to me analogous means if i was translating python into Scala, then typically, each varaible that was a list in python would become a Vector in scala

peak spoke
flat gazelle
#

oh, that is a very cool idea

#

there are some really smart people writing these peps, huh

dawn arch
#

hopefully lol

blissful comet
#

This is kinda a first order use case for py2many. To avoid typically programming errors due to integer overflows and underflows.

We do some type widening (see infer_ops.py and generated code in the repo), but more work needs to be done.

blissful comet
halcyon trail
#

neat

sacred yew
#

@blissful comet I haven't had the chance to try it yet, but how does your transposer deal with "python-specific" features like dunders?

blissful comet
iron perch
#

I think I should ask this here...
where can I find in-depth resources for python? Like how Python works how it stores variables in lower level

spark magnet
#

@iron perch there are lots of blog posts and pycon talks, etc, about it also

grave jolt
#

I wish there were more open-source books lemon_pensive

boreal umbra
#

Is a function (an instance of types.FunctionType, to say nothing of other callables for the moment) mutable? Does the concept of mutability even apply?

spark magnet
grave jolt
fallen slateBOT
#

@grave jolt :white_check_mark: Your eval job has completed with return code 0.

001 | <function f at 0x7f57ac106040>
002 | <function totallyNotF at 0x7f57ac106040>
003 | quack
gleaming rover
#

did I see camel case in my Python there

#

๐Ÿ˜ก

grave jolt
boreal umbra
gleaming rover
finite sparrow
spark magnet
grave jolt
#

the code objects themselves are immutable, like tuples

gleaming rover
#

that wouldnโ€™t be mutation

#

how fast would a Python interpreter written for CPython be

#

I would guess still usable for small things?

spark magnet
gleaming rover
spark magnet
gleaming rover
#

because PyPy compiles

#

I mean, like

finite sparrow
#

i should have said that the actual __code__ is writeable. sorry.

gleaming rover
#

a Python interpreter run fully as a Python program on CPython

spark magnet
grave jolt
gleaming rover
#

admins

#

this is off topic

grave jolt
#

yeah, not sure why Stelercus started bringing up camels

#

smh

gleaming rover
finite sparrow
#

is there a way to make it seem towards static analyser tools like linters and type checkers as if a function has the exact same arguments as another function, even if it takes *args, **kwargs (which is only passed on to the other function)?

grave jolt
#

ah, i c what you mean

halcyon trail
#

yeah I don't think that's what he wants

#

you probably have some kind of wrapper function for another specific function and you're trying to just avoid rewriting out the entire signature

#

it is really repetitive and annoying

#

and I don't know a way to avoid it, offhand

finite sparrow
#

i was hoping for maybe something like this

def onefunc(a: int, b:int) -> int:
    ...

def other(*args: argsof[onefunc], **kwargs: kwargsof[onefunc]) -> returnof[onefunc]:
    ...
grave jolt
#

@finite sparrow If they have the same return type, you can trick the linter with a trick like that this
If they have different return types, you'll need ParamSpec from Python 3.10.
If you don't want to repeat yourself, instead of passing kwargs you could pass a "parameter object"/"options object".

halcyon trail
#

like "perfect forwarding of the types"

#

afaik not possible

grave jolt
#

ParamSpec makes it possible

finite sparrow
#

lets assume the simplest case, it entirely matches the other function in all types

grave jolt
#

but it doesn't actually check anything, so it's your responsibility to make sure it's right

halcyon trail
#

i don't follow, is mypy going to statically call out other("hello", "there")

#

based on the code above

grave jolt
#

yes

finite sparrow
#

hm, and i assume this is mypy specific, wont work with other linters and intellisense tools?

grave jolt
grave jolt
#

well, you can never know, there's no specification anywhere

finite sparrow
#

i mean, the dream would be that intellisense picks up on it and shows me the arguments for onefunc

#

wait no, my bad i forgot to import Any

#

it does work

#

huh, thats nice

grave jolt
#

works with mypy, same with pyright

unknown.png
#

but it is a hack

finite sparrow
#

pylance after fixing the forgotten import

unknown.png
#

i mean, its only a few lines. may be hacky but at least its conscise

grave jolt
#

the dream

unknown.png
#

well, not the dream, you can use ParamSpec in Python

finite sparrow
#

is there a term for a callable having the exact same signature and the same return for the same arguments than another

grave jolt
#

I don't think so

#

It's just a special case of the Decorator Pattern โ„ข๏ธ

finite sparrow
#

wait hm

#

i just had a idea that might be so much easier in my case

grave jolt
#

...or maybe you just have a common interface, like

TimingFunction = Callable[[float], float]
halcyon trail
#

I don't follow, the example with assert_same_as doesn't use paramspec

grave jolt
#

it doesn't

finite sparrow
#

in my case i can actually make it a decorator which is called manually in the init of my class

grave jolt
#

what are you even doing @finite sparrow?

#

I though you were making an ebook converter, suddenly metaprogramming

finite sparrow
#

lol no, this isn't for the ebook converter

grave jolt
#

ah

finite sparrow
#

i mainly asked out of curiosity, the actual usecase is really better solved with a decorator

#

i just forgot for a second decorators dont need to assign to the same name and dont need to use the @ syntax

#

lol

#

im basically trying to expose the same API as a underlying class but with slight modifications before and after the underlying method is called

#

in my case this ended up being way easier

class A:
    def __init__(self):
        self.underlying_instance = UnderlyingClass()
        self.other: UnderlyingClass.somefunc  = self.mydecorator(self.underlying_instance.somefunc)
#

and now the word underlying feels wrong, great

halcyon trail
#

pretty fascinating bit of code

#

the assert_same_as I mean

#

the use case I've hit for this is for example, writing a wrapper for things like subprocess.run

#

which has a million arguments

#

having to repeat them all is fairly awful

#

but if you just do *args, **kwargs you throw away type safety, IDE help

finite sparrow
#

one sec lemme test sth

halcyon trail
#

yeah, so mypy is smart enough to understand assert_same_as but pycharm is not

finite sparrow
#

interesting, pylance understood it

#

i dont use mypy, would be curious to see whether it takes that

grave jolt
#

discord.py seems to not have any actual type annotations

#

as in, they're in the docstrings

finite sparrow
signal tide
#

are there any plans to integrate in any more typing implementations?

covert cape
#

Saw some discussion about typing being a pain in their server

flat gazelle
#

I think functools.wraps should also at least help

static bluff
#

So I decided to stop thinking of my language as an extension of Python, and work on it as my own language

#

And its really nice having a blueprint of what I want more or less but also the freedom to make judgement calls or go different directions if I want

#

I'm considering making all strings fstrings by default, unless they are raw

lament sinew
#

so like ruby

static bluff
#

I suppose

#

What is ruby about?

lament sinew
#

it's just japanese python

static bluff
#

Actually?

flat gazelle
#

ruby has very different priorities from python

#

it was designed specifically to make programmers writing it happy

#

it is heavily inspired by perl, and as such often has multiple ways to do one thing and the programmer can pick which is more convenient

static bluff
flat gazelle
#

it is a very neat idea, but it can often lead to really annoying outcomes and programmers getting trapped in an endless cycle of what works best. Any given program in ruby or raku or perl can be nearly infinitely iterated upon, when the more sensible thing to do would be to just move on and write more code.
But it does help quite often, as it lets you keep code very close to how you imagined it

visual shadow
flat gazelle
#

for anything small to medium, it is really nice. But I wouldn't want it for a codebase too large to comprehend entirely

#

but I am more familiar with the idea in raku than in ruby. Last I used ruby was a few years ago

paper echo
flat gazelle
#

yup, that too

paper echo
#

I agree, I think I would rather write a big application in python than ruby

#

Ruby syntax is a bit "loose" , and documentation is harder to search through

#

However I do quite like Crystal

flat gazelle
#

common lisp is a very extreme example of this, learning all of the common lisp standard is reasonable, but you have to learn the specific way in which a given codebase decided to apply those features, from macros to APIs (i.e. learning a new language per codebase you want to use understand). In python, there is a lot more in common even between 2 wildly different codebases. Ruby may not be as extreme, but from the very little I saw of ruby, it was something of an issue

paper echo
#

fwiw common lisp has a lot of dark corners of the hyperspec that you don't need to know in order to be productive at a basic level

#

and common lisp programmers tend to be unusually disciplined and use similar idioms, and most lisp programs aren't that hard to read. although most lisp libraries are severely underdocumented and it tends to be a lot harder to read the library source code to figure out how to use it compared to a python library

#

i have heard that complaint about ruby as well

#

and seen it first hand with perl

flat gazelle
#

most libraries are documented through their tests. And yeah, lisp programmers actually update when new things come around. quicklisp actually has good adoption, even for libraries that are older than it.

paper echo
#

quicklisp is also satan spawn

#

but true re: tests

#

i dont think its a good replacement for docs

swift imp
#

Anyone think type annotations will become more common place and introduce new mechanics, similar to how pydantic is using them?

peak spoke
#

It is now compatible with type hinting through Annotated so I'd expect a few more uses for the cases where that was the deterrent

paper echo
#

imo the whole point of type annotations is that you can statically verify the types, eliminating the need for runtime validation

#

i don't believe pydantic allows you to "turn off" runtime validation, or does it?

#

one reason why i prefer attrs+marshmallow+desert is that i actually understand how it works...

winged minnow
#

no

swift imp
#

I mean even like

peak spoke
#

I haven't used pydantic but I think there are some nice use cases for it like discordpy's converters if Annotated is used so typing can be used properly along with it.
I recall needing something like it for a config class where I went with assigning values instead of the annotation holding the value as I wanted the typing to work

swift imp
#

So the dispatching modules use annotations to dispatch

#

Based on type

#

I've played around with doing value based dispatching

#

Using type annotations of Literal

flat gazelle
#

the issue is a lot of existing type annotations cannot be checked at runtime

swift imp
#

I feel like they did that on purpose

#

And it totally could be

flat gazelle
#

well, no, there are just things that you can't check at runtime

#

does a function return a specific type is entirely non-deterministic

swift imp
#

Ok but for example they block isinstance on List

#

And it's like

#

Whyyyy

#

It's like, well we can't do it for everything, so we are blocking for everything

peak spoke
#

I think doing it yourself is the better alternative compared to it possibly taking unexpectedly long

flat gazelle
# 
In [2]: isinstance([], List)
Out[2]: True
``` works fine. I would not want an instance check to be potentially infinite
swift imp
#

If the list is parametrized you can't

flat gazelle
#

or at least unreasonably long

swift imp
#

List[int]

#

Or you can't in 3.7 at least, which is what work uses

flat gazelle
#

I can make a sparse list that is 10million items long with minimal memory footprint and make this new fancy isinstance take an hour

swift imp
#

Ok, I'm not saying verify content

#

But if I type annotate for List[int] then I can't do isinstance([], List[int])

flat gazelle
#

the only time that check makes sense is for empty lists

#

you can't return a correct result for a full list in reasonable time

swift imp
#

U r missing my point

#

I don't want it to check content

halcyon trail
#

what else can it check?

swift imp
#

I'm saying if I annotate an argument as a list containing ints, I then can't use the annotations to check that the argument is at least a list

halcyon trail
#

gotcha

swift imp
#

It just blocks it completely bc List is now paramterized

halcyon trail
#

There's quite a bit of funkiness around this stuff in python, IME, compared to a real statically typed language

#

what you could do potentially is use get_origin which gives you List from List[int]

#

It is very strange in a way that you even can do isinstance([], List) if you think about it

flat gazelle
#

so you want isinstance(['a'], List[int]) to be True?

halcyon trail
#

If you think about traditional generics that question doesn't even make sense

#

In a traditional generics system, List just isn't a type

#

So you can't ask if something is an instance of List

swift imp
flat gazelle
#

an error is better than a wrong result

halcyon trail
#

yeah, I'd agree with that

#

it's definitely better to error

#

the problem with using isinstance with things like List is that it's basically the intersection of the static and dynamic type systems

#

And things pretty much stop being sensible all around

flat gazelle
#

java does also implicitly fill Object in places where you omit a generic

#

but java also only somewhat has generics and you cannot check at runtime at all

sacred yew
#

impling java generics are good

halcyon trail
#

Sure, I was about to add an adendum that in type erased generics systems it's a bit different

#

but in C++, Haskell, etc, something like List isn't a type, it's effectively a function mapping types to type, e.g. int to List<int>

#

so, you can't ask questions like "is x an instance of List", it just makes no sense

flat gazelle
#

C++ doesn't have generics, but IG it does work anyway

halcyon trail
#

You can ask "is x an instance of List<T> for some T"

#

idk templates are pretty much unconstrained generics, I'm not sure what distinction you're drawing

flat gazelle
#

generic blocks are typechecked at definition, block templates at instantiation. (block being class, function, etc)

halcyon trail
#

those are constrained generics

#

C++ has unconstrained generics

swift imp
#

Matlab has run time validation

#

But they also lack data structures

#

So I guess they win there

#

And is how they can justify it

#

You can validate size, shape, type and content

flat gazelle
#

does matlab support heterogenous arrays at all?

swift imp
#

Cell array is the closest

#

You can shove whatever u want in it

halcyon trail
#

you can actually just pull in Java data structures into matlab

#

whenever you want to

#

I remember doing this a few times. since the matlab IDE is all written in Java

#

it's kind of weird and obscure but it works

swift imp
halcyon trail
#

oh? what do you mean

swift imp
#

Java is being killed off at The Math works

#

No more JARs

flat gazelle
#

as far as I can tell, cell arrays do not support runtime type checks across all of the contents

swift imp
#

U can specify any arbitrary validation function

paper echo
#

sometimes you need to do the latter as a "guard" in order to achieve the former... but i want control over that process as the programmer

#

i don't like excessive magic

#

(that said, i am hoping that python and mypy can iterate towards 1) proper refinement types, and 2) static analysis on "regex literals", which would be a real achievement imo)

mild flax
paper echo
#

being able to statically determine the number and names of match groups

#

its not that important, just a wishlist item ๐Ÿ˜„

halcyon trail
#

C++ actually has this as of the last few years

flat gazelle
#

there is also a haskell library that does this

paper echo
#

refinement types would be way more interesting anyway

def f(x: float, y: float) -> PositiveFloat:
    return x - y
           ^^^^^

main.py:2: error: Incompatible return value type (got "float", expected "PositiveFloat")
swift imp
#

Wait is that exception pep already in place

paper echo
#

which one?

#

unfortunately without refinement types, the Neg will "leak" out of g, and you won't know that you made a mistake until you try to use the return value from g

swift imp
#

There's a pep for more detailed exception responses

paper echo
#

im not actually sure how this actually works in haskell, i should probably read their docs

swift imp
#

And that snippet you posted,looks like it

paper echo
#

oh cool, i had no idea

#

i would love this

#

would be great for other frameworks like pytest to report interesting stuff

blissful comet
#

I have a dict that's keyed by type classes as in {str : 42, ...} and I need to look it by type strings such as "str". Looking to avoid eval(). What's the recommended practice here?

unkempt rock
blissful comet
#

Needs to handle datetime.date also, which is not a builtin.

#

The strings are coming by parsing annotation nodes in the AST.

halcyon trail
#

You don't need eval, but you do need getattr

#

or maybe, globals()

#

If you're handling datetime.date, does that mean the plce where you hadnle it is going to have from datetime import date

#

or import datetime

#

or neither?

blissful comet
#

It's similar to resolving ForwardRef

halcyon trail
#

sure, but my point is that if you want to get to types from strings without building an explicit dict then you need to have rules for doing so

#

I'm not even sure what the rules are

#

If all the types you need are going to be in scope, then you can look them up in globals

#

I guess, globals + builtins

#

Otherwise you could just have an explicit dictionary

#

{"str": str, "date": datetime.date}

#

etc

blissful comet
#

That's what I do for normal python -> rust type mapping.

fallen slateBOT
#

src/typing.py lines 238 to 250

def _eval_type(self, globalns, localns):
    if not self.__forward_evaluated__ or localns is not globalns:
        if globalns is None and localns is None:
            globalns = localns = {}
        elif globalns is None:
            globalns = localns
        elif localns is None:
            localns = globalns
        self.__forward_value__ = _type_check(
            eval(self.__forward_code__, globalns, localns),
            "Forward references must evaluate to types.")
        self.__forward_evaluated__ = True
    return self.__forward_value__```
halcyon trail
#

I don't really understand why you'd need eval to convert a string into a type

blissful comet
#

because if you write:

import datetime

today: datetime.date = ...

and you parse the code, you get a string, not a type class. Then I need to map the string to the corresponding rust type.

halcyon trail
#

I mean, I guess it's a nice shortcut if you need to deal with the strings also having "." in them, which you didn't really make clear if you did

#

Right, so like I said, it's a shortcut, IMHO the proper way to do it is to split the string by .

blissful comet
#

Or in the general case, eval().

halcyon trail
#

And now you go hunting in globals/locals/builtin

blissful comet
#

The problem with strings is that if someone does import as "foobar", it breaks

halcyon trail
#

it will not break, no

#

because then foobar will become an entry in globals

blissful comet
#

I hear you - if we look at builtins/globals etc - it works.

#

But that's eval() imo

halcyon trail
#

what do you mean "that's eval"

#

you're feeding globals/locals into it either way

#

All I'm really saying is that you can do this with getattr/dictionary lookup, afaics

blissful comet
#

I think I got my answer - I was concerned about using some type of an expression evaluation, but since typing is doing it, I'm not that concerned.

paper echo
blissful comet
#

I think the idea behind APIs like this is to discourage people from using AST and doing their own (potentially broken) type annotation

#

AST also tends to be unstable. There were backward incompatible changes between 3.8 and 3.9

#

But for a transpiler, dealing with AST is kinda inevitable

paper echo
#

interesting that there's no mention of nuitka and only 1 passing mention of shedskin in the repo

blissful comet
#

Yes - the stuff I'm building is similar to cython and nutika

paper echo
blissful comet
#

*nuitka

#

But the implementation strategy is different. Python -> rust instead of python -> C

paper echo
#

interesting idea

#

whats the point of doing that vs just using a different language? so you can use existing libraries?

#

and i guess distributing standalone binaries

#

i always found it interesting how scheme and lisp implementations can be super fast while python isn't

halcyon trail
#

idk about super, but lisps do seem impressively fast relative to a) being dynamic and b) the number of man hours that have gone into their implementations (relatively low)

paper echo
#

im not sure about (b), but all the man hours were put in 20 years ago

#

and some were developed for commercial purposes and only open sourced later, such as chez

halcyon trail
#

I'm only saying compared to insane heavyweights like the JVM, gcc, clang, MSVC

#

which very obviously have far, far, far more man hours than any lisp implementation has had, probably by orders of magnitude

paper echo
#

ah yeah

halcyon trail
#

.net

paper echo
#

ive heard good things about the jvm for server applications

halcyon trail
#

Basically, the entire tier of fastest languages is built on those technologies though

paper echo
#

ive been really wanting to try out clojure and kotlin

#

https://www.notamonadtutorial.com/rebuilding-the-racket-compiler-with-chez-scheme/

Chez Schemeโ€™s biggest claim to fame is its performance. It has always been among the best-performing Scheme implementations. Its object-tagging and allocation regime, its hybrid stackโ€“heap implementations of continuations, and its compiler structure all remain state-of-the-art, even in 2020.
idk if this means anything to you

halcyon trail
#

racket is a gradually typed lisp isn't it

paper echo
#

its dynamic but there's a gradually typed version of it called typed racket

halcyon trail
#

When I google it the description of racket on the main page says:

Racket is the first language to support higher-order software contracts and safe gradual typing. Programmers can easily deploy these tools to harden their software ...

paper echo
halcyon trail
#

I see what you mean

#

The macro system in Racket has been used to construct entire language dialects. This includes Typed Racket, which is a gradually typed dialect of Racket that eases the migration from untyped to typed code,[49] Lazy Racketโ€”a dialect with lazy evaluation,[50] and Hackett, which combines Haskell and Racket.[51]

#

and.... this is why nobody uses lisps ๐Ÿ™‚

#

As amazing as they are

blissful comet
paper echo
#

racket is a teaching and research language

blissful comet
#

The thing closest to this haxe - but it's an entirely new language.

paper echo
#

the fact that it also is a highly usable app development language is secondary and also somewhat accidental

#

in some respects racket is a better common lisp than common lisp, in that it has lots of batteries included versus the minimalism of scheme

halcyon trail
#

the thing is that in a weird way, the huge advantage of lisp, has sort of ended up being a disadvantage in practice, the language doesn't need to be so opinionated and join in the rat race for features because programmers/libraries can simply add those things themselves

#

but then the language being unopinionated tends to lead to fragmentation, lack of consensus, which seem to be ultimately more harmful than the damage of constantly standardizing new features, the limitations of those features, etc

#

I do think a real world targeted, gradually/optionally typed lisp would be a pretty amazing thing. It's also rather hard for lisps to gain traction nowadays because everything is really shifting towards static typing, IMHO

paper echo
#

as a hobbyist lisp developer, imo the #1 impediment to adoption is lack of tooling. you need something like parinfer to not go crazy writing lisp. you also need what every other language needs: docs, test suite, package manager, text editor integration, a standard web framework, etc.

racket has all that except the good text editor plugins.

other lisps which are even more lacking in tooling (e.g. dependent on slime which doesnt really support anything but emacs, and vim as a second class citizen) basically only attract archetypal wizard people, which leads to an ecosystem of esoteric underdocumented libraries rather than things that normal people can pick up and use, which exacerbates the cycle of no-tooling, no-docs, etc. and being generally unfriendly for non-wizards.

halcyon trail
#

hasn't clojure made progress on a lot of these things

#

also the ability to quite easily wrap a java library in a pinch

paper echo
#

i was about to say, clojure is doing the best job

#

and clojurescript

halcyon trail
#

it's just been such a long time since a dynamic language really gained major traction

paper echo
#

frankly i dont think any "true" (((lisp))) will ever get as popular as python

halcyon trail
#

i remember a year or two ago a blog post argued that it just won't happen again, I think it made pretty good points

paper echo
#

its like haskell, it will always have its niche and its niche will probably keep growing, but people will learn python first

#

thats an interesting idea. there is definitely a pendulum swing towards static

halcyon trail
#

the closest thing to a dynamic language gaining major traction is julia, and I wouldn't really call it a general purpose language

paper echo
#

i like it. less shit to test, tooling can provide more assistance (e.g. refactoring)

#

fwiw theres no reason julia can't or shouldn't be general purpose

halcyon trail
#

Yeah. I mean, dynamic is a harder sell in the age of much better IDE's, much better type inference (so static languages have waaaaaay less boilerplate than they used to)

paper echo
#

im not sure what its concurrency story is though. so not sure julia is a good language to write a web server in

halcyon trail
#

I mean Julia is targetting something specific, it's spending a lot of language design real estate on that specific thing

#

nobody is going to use it for writing a random server that doesn't involve math, or unless that's the main language they know. Not know, and probably, not ever.

#

Any more than people have written random servers in matlab

paper echo
#

thats true, but the more i use julia the less i feel like it needs to be that way

#

the syntax is pretty generic lua/ruby-flavored with some haskell-ish elements

#

jit startup time is irrelevant in a long-running webserver

#

what ends up in a language ecosystem seems to be more about culture (in many cases driven by the core devs) than about technical merit

halcyon trail
#

idk, even if Julia is a nice language, jus tthe fact that it's a) dynamic, b) relatively obscure, means it' sprobably almost always going to be a bad pick for random stuff

#

for doing math, it has the big selling point of super nice syntax for math, which almost no languages have, and better performance than its competitors

paper echo
#

sure, but like i said theres nothing inherent in the language that makes it bad for other purposes. it's just not what the community is focused on.

#

like attracts like and rejects unlike

halcyon trail
#

Well, language design real estate is finite, or rather, it has cost, and when it's all spent targeting things you don't care about it mostly guarantees it won't be a great choice for your domain

#

Just having "decent syntax" is a really weak selling point these days. You can get decent syntax from almost any language. And Ruby, Python, Kotlin, Java, Go, will all give you far more libraries, and depending which one of those you choose it will give you other benefits as well

paper echo
#

sure. i guess my point is that theres nothing in the julia language feature set that makes it especially bad for web stuff, and in fact it has some really good features (like great unicode support)

swift imp
#

Julia is trying to compete with Matlab simulink, which is probably never going to happen

halcyon trail
#

you're saying, that's all that julia is, or that julia is trying to do that in addition to other things

#

i've never really gotten that impression, simulink is a specific tool within matlab, the julia I've seen looks to pretty much be starting out by trying to compete head on with "normal" matlab

swift imp
#

No one buys matlab for matlab, they buy it for simulink

halcyon trail
#

said like someone who's trying to sound really knowledgeable

#

universities buy tons of copies of matlab, for all kinds of purposes

#

many of the people who use matlab are not using simulink

#

source: I have a PhD in physics, all my papers were done with lots of work in matlab for data analysis, fitting, visualization purposes, and I've never used simulink

swift imp
#

Academia doesn't count

#

Most universities get the whole thing with all but like 3 toolboxes which are purely production related, for free

#

Then you have large engineering entities that use it for modeling in Simulink, those are the people that Julia show cases against, they even have videos of it

gleaming rover
#

...or a better Java

halcyon trail
#

it can only be more practical and worse if your idea of "good" is something other than a practical programming language

#

@swift imp where did you hear it's free?

#

I googled, I don't find any direct numbers but I did find this link "request a quote"

gleaming rover
#

usability by the masses is not the only thing a language should have

halcyon trail
#

it's not about usability by the masses. that's a narrative that was big 30 years ago in PG essays about lisp that most people don't take seriously any more

#

Even smart people make more mistakes of understanding with complex tools than with simple tools, you have to judge whether that's a good trade-off for the mistakes that the abstractions save you.

hollow hill
#

I saw many people recommend
Introduction to Algorithms, 3rd Edition (The MIT Press)
To learn algorithm and data structure so should i get it ?

spark magnet
feral cedar
#

it's incredibly rigorous

hollow hill
#

Thank you

iron perch
#

If variables points to an address then where does points get saved

#

I think there is some kind of paradox here

grave jolt
iron perch
#

๐Ÿ“

grave jolt
#

what do you mean by "points"?

iron perch
#

Pointer

grave jolt
# iron perch Pointer
  • if it's a global variable, the pointer is stored in a dictionary for global names (which you can access with globals())
  • if it's a local variable, the pointer is stored in a frame. A frame is a structure that lives for the duration of the function call.
#

So when you call a function, a frame is created. It makes some space for local variables (including arguments). Each variable stores a pointer to an object.

iron perch
#

Another question
Where did you learn that

grave jolt
#

๐Ÿคท

#

Well, I did a bit of C.

#

And I played around with Python

iron perch
#

I'm trying make a dictionary using only lists with 0(1) lookup

raven ridge
#

All variables are stored in memory. A pointer is just a variable whose value is the address of another value.

static bluff
#

So I wanna ask you guys which language I should learn next- but not in the beginner kind of way. Which language do you guys think would be most fascinating? (though not so different from python that it's huge overhead to learn)

visual shadow
#

do you have a sense of how lower level languages work? if not, i'd say C is a good candidate. do you simply want to be amazed? if so, i'd say Rust might be a good candidate. Different paradigms? they might be a bit different from python though, though really good for opening your mind

#

theres ofcourse so many more languages out there, but i dont know what else to suggest.

static bluff
#

I know I've got to get to C eventually- but honestly I just don't feel like it right now

#

But I'm teaching myself compiler design, so, I'm wanting to broaden my horizons

flat gazelle
#

raku may be interesting. Unlike python, it is very eager to add new features, is gradually typed and is really useful for writing parsers. However, learning resources are few.

static bluff
#

I just heard the name Raku just today!

visual shadow
#

i'd say if you wanna pick up compiler design, start with something extremely barebones to implement. Semi serious perhaps implement a brainf*ck compiler

#

you'd probably finish that in half an hour tops, and it would be fun to do

#

well..compiler/interpreter or perhaps go the whole 9 yards, lexer etc

static bluff
#

My lexer is almost done actually

#

I get the impression the parser is going to much more of a pain

visual shadow
#

as long as you take it one step at a time, you'll be fine ๐Ÿ™‚

static bluff
#

What are you working on these days?

placid coral
#

i have a question about python

visual shadow
visual shadow
static bluff
#

๐Ÿ˜ข

#

I get that though. One project just keeps leading me further down the rabbit hole to the next

acoustic crater
nimble dome
#

Hey guys

#

I have a question

halcyon trail
#

If you are interested in compiler design and want to learn a new language, learn lisp

nimble dome
#

Is there a way to search for the values in a http.cookiejar object?

halcyon trail
#

You can write a list interpreter in like 100 lines of python

boreal umbra
static bluff
#

Morning all! I know this isn't a social channel and all but nontheless, g'day!

unkempt rock
#

Hi guys. I don't know how relevant this is for Python but I am using Python to do this so I might as well just ask it here. My plan is to use images of a well-known object (a painting in a museum for example) and then use those to query some online database to retrieve the correct name for that painting.

I know this will involve AI but I was hoping someone here can point me in the right direction. Is there already some Python library to do this?

#

that online database could be a wikidatabase

visual shadow
static bluff
#

So about type annotation: its not the prettiest thing in the world. It know that is a bit of a neccessary evil, but if you could implement a whole new syntax for it, how'd you do it?

Here's what I've got:

@annotate( line=str, value=str, line=str )
@annotate( start=tuple[ int, int, int ] )
@annotate( end=tuple[ int, int, int ] )
def __init__( self, tokentype, start, end, value, line, *positional, **encyclopedic ):
deft pagoda
#

i think i'd just repeat the current syntax, mostly because it looks similar to other languages and so should be more familiar

static bluff
#

But its so uglyyyyyyyyyyy

#

๐Ÿ˜ญ

static bluff
# 
    """The 'Regex' class is a utility class designed to make less hideous the process of writing
    (and reading) a large and multifaceted regular expression.

    The structure of the expression, namely its various capture groups, is defined within the class'
    'expression' property, while all of the actual patterns are stored and documented as 
    individual properties unto themselves.

    The sole function of the Regex class, aside from the expression itself, is to compare the
    lexer's source buffer against the expression, throw an error if no matches are found,
    and to return each match group (mapped to its name) as a dictionary"""

Too wordy?

brave badger
#

Seems standard

static bluff
#

And it makes sense, more or less?

swift imp
#

Are the microsoft council members going to be able to speed up python ny 3.11?

#

Seems really optimistic

quaint willow
#

Do you guys prefer to work on pc or laptop ?

raven ridge
static bluff
#

Can someone explain to me how a language can be coded in itself?

radiant fulcrum
#

the process is called bootstrapping

#

most compiled languages use this principle

acoustic crater
#

if it's interpreted it can't rly but PyPy uses R with Python syntax

#

if it's compiled you compile a compiler for the language written in itself with a different compiler

raven ridge
#

A turing complete programming language can run any program, and an interpreter for a programming language is a program, therefore an interpreter for a language can be written in that language.

radiant fulcrum
#

You write the first compiler in a existing language which compiles the basic subset of the language, then you write the main compiler in the language

acoustic crater
#

but that interpreter needs to continue to be interpreted by the initial interpreter

raven ridge
#

right.

radiant fulcrum
#

yeah, interpreters are a bit harder to bootstrap

acoustic crater
#

so it can't rly cuz the base interpreter is still in whatever the original language was

raven ridge
#

that's still a Python interpreter written in Python, even if it's run in a Python interpreter that's written in C.

acoustic crater
#

I'd say at root it's still coded in C in that case though

raven ridge
#

Why C? Why not assembly, or machine code?

acoustic crater
#

because the C is what was compiled and the base of any running process

raven ridge
#

seems rather arbitrary to discount that the code that the user wrote for their interpreter was written in Python, consider that the code some other interpreter author wrote was written in C, and discount that the C was compiled to machine code via assembly.

cedar glen
raven ridge
#

you've picked a random hop along the chain to call the "true" language.

acoustic crater
#

okay so the name "CPython" is arbitrary as well?

raven ridge
#

yep.

acoustic crater
#

lol so there's no reason to try to make a judgement on this from your perspective

cedar glen
#

it's not in c after you compiled it

raven ridge
#

right.

acoustic crater
#

but it was "coded" in C

raven ridge
#

sure. And in my hypothetical, there would be an interpreter coded in Python, which you could then run with CPython.

cedar glen
#

something needs to go seriously wrong if after compilation your code is still in C

static bluff
#

He's totally right, Spoony. It doesn't matter what language is processing another. So long as the input can be properly understood, the data properly managed and dispatched, you can run anything in anything

#

MTG is, apparently, turing-complete

acoustic crater
#

I still think an interpreter chain is "coded" in the base language

raven ridge
#

that's completely arbitrary and not based in any reality.

acoustic crater
#

so is writing a pointless self-interpreter

raven ridge
#

the reality is that there would be an interpreter written in Python, which could be run using any other Python interpreter - including CPython, but also including pypy, or whatever.

#

that interpreter was coded in Python, regardless of how you run it.

static bluff
#

I suppose you could even run C in python if you wanted XD

#

That would be fascinating experiment

raven ridge
#

there are C interpreters, and even C++ interpreters, that exist.

static bluff
#

So lemme just make sure I understand

#

I'd compile my first compiler in pure python. Then I'd rewrite the compiler in my new language and compile that, using the previous one. Once that is done, I can distribute my compiler compiled in itself

raven ridge
#

yep.

#

for a compiled language, that's it.

static bluff
#

๐Ÿคฏ

raven ridge
#

The earliest C compilers weren't written in C, but all modern C compilers are written in C or C++. For instance.

#

at some point along the chain, someone used a C compiler written in B or in assembly or something to compile a C compiler written in C. And from then on, they kept writing the new compilers in C.

static bluff
#

Maaaaaaaaan

#

Humans are clever bastards

raven ridge
raven ridge
sacred yew
#

rpython is not R?

raven ridge
#

no...

acoustic crater
#

it's R with python syntax

sacred yew
#

tf

#

r stands for restricted iirc

#

as in

#

its python

#

but only some parts

raven ridge
#

right.

radiant fulcrum
# 
RPython is a restricted subset of the Python language. It is used for implementing dynamic language interpreters within the PyPy toolchain. The restrictions ensure that type inference (and so, ultimately, translation to other languages) of RPython programs is possible.```
sacred yew
#

pypy is written in rpython, which is compiled to c iirc

#

its not an "interpreter chain"

#

there's only 1 interpreter, which is written in (r)python, used to interpret python

static bluff
#

I feel ya godly

#

I'm pickin' up what yer puttin' down

terse marten
#

How long since first line of code is average if constant study to get an entry job

#

In Canada

static bluff
#

Two years

#

Assuming you have some sort of certificate

#

Or rather, you earn some sort of certificate within those two years

#

Web development, web design (similar, but different from the former) are hugely in demand, as are C/C#/C++

#

Python is a great skill to have, but most employers think of it as an auxiliary skill. The real work for application design is done in the C languages, and the native mobile languages of iOS and android

#

Web development is, of course, written in Javascript with SQL and possibly python for the back end

#

Javascript is something of a microcosm though, as you'll be expected to know at least jQuery and React, possibly Angular as well

#

That said - it's theoretically possible to just code your ass off- learn the ropes and build a few neat projects just to show that you can code and then coldcall the hell out of the software companies in your area

#

Or else network and make a few friends in the industry. When you feel you're ready, you can just start applying for jobs

rich cradle
static bluff
#

I suppose, yeah

radiant fulcrum
#

still the most used framework in modern web apps

static bluff
#

So I think I mostly understand this bootstrapping idea for compiled languages

#

But I gather it can't be done or is atleast much trickier with interpreted ones

fringe cargo
#

I hope one day Iโ€™ll be able to understand the stuff you guys are talking about

static bluff
#

And its exactly that hope that makes me sure you'll get there one day

spark magnet
grave jolt
#

every new version will get gradually slower, because you're running a VM on top of a VM on top of a VM, right?

#

(assuming "interpreted" means "running in a VM")

spark magnet
#

Perhaps we need to distinguish not between languages but between implementations. "Python is an interpreted language" doesn't mean much when you consider PyPy.

static bluff
#

Fix's question seems important

#

But beyond that, if Python is not slow because its interpreted, why is it slow? Duck typing?

spark magnet
static bluff
#

Vaguely

spark magnet
#

It's Python, fast.

static bluff
#

I can't say I understand it

spark magnet
#

I don;t entirely understand it either.

#

the point is, languages aren't fast or slow. Implementations of languages can be.

static bluff
#

This lexer is really coming along

grave jolt
spark magnet
static bluff
#

!paste

fallen slateBOT
#

Pasting large amounts of code

If your code is too long to fit in a codeblock in discord, you can paste your code here:
https://paste.pydis.com/

After pasting your code, save it by clicking the floppy disk icon in the top right, or by typing ctrl + S. After doing that, the URL should change. Copy the URL and post it here so others can see it.

static bluff
#

I've written the most of the actual token generation logic in previous versions, but I'm just chugging along working on the comments, and readability

#

OH!

#

And before you dive in

#

I've tried to keep my otherwise abhorrent coding style to a minimum, but there might be a few idiosyncrasies that raise your eyebrows. Before you say anything- I knowwwwwwwwwww

grave jolt
#

ah yes, REDACTED++

static bluff
#

That why I'm building my own language to start with XD

spark magnet
#

won't say anything about the indentation and white space.....

static bluff
#

Mostly what I'd love to get your opinions on are the comments, and any opportunities for optimization

spark magnet
#

to tell you the truth, i find it hard to read this, so it's hard to comment.

grave jolt
static bluff
#

SorryguysIhavetroublereadingyourcodetoo

spark magnet
#

@static bluff why do you have all those imports at the top? Many are unused.

static bluff
#

Just a (bad) habit I've developed. Same as including *args, **kwargs in every method

spark magnet
#

ok, don't do that ๐Ÿ™‚

quiet crane
#

Yeha I was just going to ask about *args/**kwargs ๐Ÿ˜›

grave jolt
#

the code is... how I feel at a social event ๐Ÿ˜ฐ too many things seem to be going on

spark magnet
grave jolt
#

encyclopedic?

static bluff
#

Before you guys get too carried away, I find most normal python completely incomprehensible. I tried for years to bridge the gap but gave it up as a bad job

quiet crane
#

dictionary is encyclopedic, right? ๐Ÿ˜‰

grave jolt
static bluff
#

I think my code is flat, clean, and efficient

quiet crane
#

Never seen foxtools, is that just a way to keep the type annotation, but shove it away into some "hidden" place (decorator)?

static bluff
rich cradle
grave jolt
#

does the lexer work?

static bluff
#

Well that's a you problem, isn't it

quiet crane
#

The type of the source parameter and source attribute is different. I would define that type somehwere, and then use it inline with the foxtools stuff.

grave jolt
#

maybe you can show an example of how to use it?

static bluff
#

๐Ÿ˜„

rich cradle
#

It's not a me problem if most people can't read it, is it?

static bluff
#

Sorry, lemme rephrase, that sounds like a problem for everyone who isn't me

#

(I say this, mostly tongue in cheek :P)

spark magnet
#

@static bluff it limits opportunities for collaboration

static bluff
#

I know ๐Ÿ˜ฆ

spark magnet
#

you might be fine with that

grave jolt
#

!e
well, there are different styles of formatting, it's very personal

for i in range       (
    1                ,
    101              ):
    b                =\
    ""
    if               (
      i              %
      3              ==
      0              ):
      b              +=\
      "Fizz"
    if               (
      i              %
      5              ==
      0              ):
      b              +=\
      "Buzz"
    print            (
    b or i           ,
    sep              =
    " "              )
static bluff
#

But as I've said, I find most python completely incomprehensible. Its hard to put my foot on specifics

grave jolt
#

What languages' idiomatic styles do you like?

spark magnet
#

@static bluff importing things you don't need, and having parameters you don't need: those are good habits to break

#

This is like a YouTube how-to video where the first shot is of a tool bench covered in tools, and the video on;y uses a few of them

#

The imports and the signatures help people understand what's coming next.

static bluff
grave jolt
#

what was it a year ago?

static bluff
#

I like my code flat. Longer lines that do more, list comprehension/maps versus indentation

spark magnet
#

But this code doesn't seem flat.

static bluff
#

I usually break every major component into multiple smaller classes, all in their own modules

spark magnet
#

for example, your regex uses lots of pieces from elsewhere

static bluff
#

You know guys

grave jolt
#

I think this is way too many comments:

    def compute( self, lexer: 'Lexer', *positional, **encyclopedic ):
        """Match the regular expression against the lexer's source buffer. If a match found,
        map each capture group within the match to its name and return the mapping as
        a dictionary to the lexer.

        If a match is *NOT* found, a syntax error is raised."""

        # A slightly more readable implementation of a try-except block.
        with contextlib.suppress( AttributeError ):

            # Perform a standard regex match of the expression against the lexer's source buffer
            match = regex.match( self.expression, lexer.sourcebuffer );

            # Map each of the match's capture groups to the group's name, and return the result
            return dict( zip( *self, match.groups() ) );

        # Not match was found: the only reason for this is the detection of an
        # unrecognized character. A syntax error must be raised.
        raise SyntaxError( 'unrecognized character', lexer.context );

This looks just as well:

    def compute( self, lexer: 'Lexer', *positional, **encyclopedic ):
        """Match the regular expression against the lexer's source buffer. If a match found,
        map each capture group within the match to its name and return the mapping as
        a dictionary to the lexer.

        If a match is *NOT* found, a syntax error is raised."""
        with contextlib.suppress( AttributeError ):
            match = regex.match( self.expression, lexer.sourcebuffer )
            return dict( zip( *self, match.groups() ) )
        raise SyntaxError( 'unrecognized character', lexer.context )

You also didn't explain why AttributeError would ever be raised, which is very surprising to use for control flow.

static bluff
#

I didn't post it so that I could get ripped to shreds over it

#

I like my coding style, I've thought long and hard about everything you guys are saying, I've decided to say fuck it

#

So I guess that's the last time I post my projects

spark magnet
#

@static bluff sorry, I didn't mean to rip you to shreds.

quiet crane
#

@static bluff Keep posting, interesting to see other's projects! ๐Ÿ™‚

static bluff
#

I'm just gonna make my own way. Thanks though

unkempt rock
#

@static bluff theres no reason to feel bad about what has been said. your code is very different from most python code which is out there. A standard exists for a reason: it usually lays the ground for good code and going against it usually isn't a good idea. people only gave their inputs on what would be good things to change about it: feel free to take it with a grain of salt if you want to

rich cradle
#

@static bluff May I ask, what languages's syntax do you like?

static bluff
#

There's probably a language out there that will speak to me more, but so far Python is the only language I've ever really enjoyed

#

I don't really think it's a matter of syntax so much as dialect. The alphabet and the words are the same, but how it's spoken is different. I very much understand the importance writing code in a way that other people understand, but writing code the way most people write their code makes my skin crawl

visual shadow
#

Hm.. Weird. So you understand the importance of communication, but then don't want to participate anyways because reasons?

spark magnet
#

@visual shadow there's a tradeoff between what feel right to yourself, and what works for other people

static bluff
#

_> I personally find my code very clean. I'm looking over it now and I don't see anything that shouldn't make sense, even to a beginner

#

Odd, sure, but not confusing

spark magnet
#

@static bluff i'm curious what you think about our comments about *positional, **encyclopedic

visual shadow
unkempt rock
visual shadow
static bluff
#

Heres the rub though my friends

visual shadow
#

But you keep posting here, then rejecting any comments that would naturally follow, which then throws me off.

static bluff
visual shadow
#

I absolutely love your questions

#

I absolutely detest your reaction to others opinion of it.

#

It's as if you've decided you're above the answers.

spark magnet
#

@visual shadow you are coming on kind of strong

static bluff
#

Noted. Though, getting a bit salty is what happens when a pack of wild coders descend on you like wolves

#

Esspecially if you preface your code with: I know perfectly well its odd, can we focus on the function

visual shadow
#

Apologies. I'll go cool off. Also, hope you realise we're not a pack of wild coders waiting to descend on people like wolves here.

#

That again, is so dismissive.

static bluff
#

That, my friend, is perspective

visual shadow
#

Aye. I hope you start valuing perspective of others.

static bluff
#

_>

#

<_<

#

Anyway, the whole reason I'm writing this language is because I'm tired of trying to make my coding style agree with everyone elses. I'd quite honestly just use my own scripting language

visual shadow
#

That's actually not a bad idea at all btw.

spark magnet
#

@static bluff why not just use Python the way you like? A whole new language is a really long way to go

static bluff
#

To paraphrase Bender, I'll just write my own programming language! With hookers! And flapjacks!

static bluff
#

The reason is, it's endlessly fascinating

spark magnet
#

oh, that I understand ๐Ÿ™‚

static bluff
#

I don't think I've ever had so much fun as diving head first into this project

#

I think language design might be my calling

grave jolt
#

well, many languages came about as a result of people not liking some other language

#

even Python, as I understand, came out as a reaction to Perl and other scripting languages

static bluff
#

Code is just something that I get. I think my style is the result of experimenting with abuse of Python's syntax so extensively, and without input from others, that I eventually developed my own accent/dialect

#

๐Ÿ™‚ My hope would be that a single really good idea comes from this language and maybe gets coopted into the language ecosystem at large. I don't expect anyone else to ever use mine though- its really just for me

#

Sorry for the saltiness though guys. Programmers should have thicker skin

#

How about all of you? How goes the fight?

spark magnet
#

I'm spending way too long writing code to make custom git branching diagrams for a blog post ๐Ÿ™‚

static bluff
#

Oh fun!

#

Is blogging as fun as it sounds?

feral cedar
#

what are you using for it? i've always wanted to make one of those too

spark magnet
#

@static bluff blogging is.... work but fun work?

static bluff
#

How long have you been coding nedbat?

spark magnet
#

It depends what you mean, but maybe 45 years? ๐Ÿ™‚

grave jolt
#

at this point I usually insert a number base joke, but here I can't (I'd have to code for at least 29 years)

#

also @static bluff sorry if my comments seemed aggressive

static bluff
#

I was just feeling a bit overwhelmed

grave jolt
#

yeah I understand you ๐Ÿ™‚

static bluff
#

One thing I'll say though- I wish you guys would try to have some fun with me on this

#

I know building my own language is a fools errand and total overkill, but its fun

#

Its not every day you can throw caution to the wind and suggest whatever syntax/functionality at all ๐Ÿ˜›

undone hare
#

I wouldn't call that a fools errand honestly, there is a lot of stuff from language design and programming you can apply elsewhere

static bluff
#

๐Ÿ˜„

#

Talking with you guys is always so much fun

spark magnet
#

@static bluff i enjoy making languages, etc, and wanted to see your approach. i honestly had a hard time seeing it.

static bluff
#

Seeing which? The code itself? I guess the whitespace might be a bit much if you're not used to it

spark magnet
#

"seeing it" meaning, "understanding your approach by reading your code"

static bluff
#

My understanding is that this lexer is no different from the standard regex based lexer

#

The docstring for the Lexer class and for its 'populate' method sums it up though

#

Functionally speaking, I'd be happy to know your thoughts

quiet crane
#

@static bluff some constructive feedback (I hope). The regexp you have is a monster, meaning it's really big and hard to grasp. It also has a numbering system such that if you ever have to add something earlier inte the regexp you would have to update all the numbers.

Idea 1: use named capture groups.
Idea 2: use some shorter intermediate representation that you then translate into the actual regexp.

static bluff
quiet crane
#

@static bluff whatever you come up with I think. I don't know if there is anything existing suitable.

gleaming rover
#

๐Ÿฅด

spark magnet
swift imp
#

I am trying to dispatch based on state of an instance and it's not working. Its like my registered methods aren't being recognized as methods, and no self is being passed.

class ValueDispatch:
    def __init__(self, method, doc=None):
        self.dispatch_table = {}
        self.method = method
        if doc is None:
            doc = method.__doc__
        self.__doc__ = doc

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        else:
            try:
                return self.dispatch_table[(obj.foo, obj.bar)]
            except KeyError:
                return self.method

    def register(self, foo, bar):
        def register_closure(func):
            self.dispatch_table[(foo, bar)] = func
            return func
        return register_closure

class Blah:
    def __init__(self, foo, bar):
        self.foo = foo
        self.bar = bar
    @ValueDispatch
    def method(self, a, b):
        print('default method')
    @method.register(1,2)
    def special_method(self, a,b,c):
        print('registered method')
# 
In [48]: test = Blah(1,2)

In [49]: test.method
Out[49]: <function __main__.Blah.special_method(self, a, b, c)>

In [50]: test.method(1,2,3)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-50-06bb6e68a079> in <module>
----> 1 test.method(1,2,3)

TypeError: special_method() missing 1 required positional argument: 'c'
#

I mean its apparent that I am doing the registration incorrectly, I just don't know why its wrong

#

dude what the heck is __get__ on a function...thats the issue

# 
def __get__(self, instance, owner):
        if instance is None:
            return self

        method = self.registry[getattr(instance, self._state_attr)]
        return method.__get__(instance, owner)

like what

static bluff
#

NEOS?

raven ridge
swift imp
#

but what do I have to do __get__ from within my descriptor to make the registered function be a bound method?

#

boggles my god dang mind

#

I mine it just seems like thats how it is but man was that freaking weird

raven ridge
#

the way that methods bind to self in the first place is through the descriptor protocol - you may need to reimplement part of what MethodType does, though I've never tried anything like this myself.

swift imp
#

yeah by all account it seems like I do have. I suppose I can blame no one but myself for not doing my research well enough. That's what I get for yolo'ing

static bluff
#

So about tokenizing/parsing fstrings- I'm a bit confused. The docs say that python tokenizes fstrings completely normally and any fstring magic that happens is done in the parsing stage. I'm only just starting to move from lexing but my understanding is that to recognize the contents of an fstring replacement field as being a valid expression, a token stream representing the string and its contents needs to be compared against the grammer.

The only thing I can think of is that fstrings are lexed in some sort of post-processing step either just after the document at large is lexed, or else 'just in time'

#

I know that fstrings are parsed as 'JoinStr' nodes and contain string literals and replacement fields- I don't think the python syntax analyzer breaks what's inside the replacement fields into its components though

#

I think I'm going take a different route, and create a new lexer instance to parse the internals of the fstring each time one is encountered

raven ridge
spark magnet
#

@static bluff i thought this was for your own language? Why take on all of Python?

static bluff
#

I don't have the context to build these tools without a roadmap. For the lexer and parser at least I'm starting by building ones that can do python code, and then I'll have the infrastructure I need to branch out however I like

spark magnet
static bluff
#

But fstrings are great!

#

My language is going to start out as syntactically quite similar to python. A lot of the same keywords and operators, whitespace based codeblocks. Its the implementation that will differ- access modifiers, method overloading, stuff like that

#

I mean, I love python โค๏ธ There are a few things that drive me mental about it but I'm not ashamed to be using it as a roadmap

spark magnet
#

@static bluff I would recommend doing a smaller subset all the way through rather than lexing everything first.

static bluff
#

I'm told I need to get better at taking peoples advice. What I want to say is that the path I'm on feels very right- but I also understand you're the expert. If I start having trouble I'll do as you say and minimize

raven ridge
#

the trick to f-strings is that they're parsed in two phases - from the lexer's point of view they're a single string, the parser re-lexes them into a stream of tokens and re-parses them using its own mini-grammar for the inside of the string.

static bluff
static bluff
raven ridge
#

There's no reason why you couldn't, I suppose - I suspect it's just more complex to do it that way, which is why they don't.

static bluff
# 
 def lexstring(**matchedpatterns):

   if matchedpatterns[ 'string' ]:
      
      if matchedpatterns['prefix'] is None:
          createToken( 'STRING', *otherdata )
      if 'u' in matchedpatterns['prefix'] or 'U' in matchedpatterns['prefix']:
          createToken( 'UNICODE-STRING', *otherdata )
      if 'f' in matchedpatterns['prefix'] or 'F' in matchedpatterns['prefix']:
          
          flexer = Lexer(matchedpatterns[ 'string' ])
          tokens.extend( flexer.tokens )
#

Something like that

raven ridge
#

you've gotta deal with string escapes as well.

static bluff
#

^^^

#

And unicode escapes also

raven ridge
#

until you're able to match \" or \' you don't even know where the string ends.

static bluff
#

The whole prefix mixing thing is gonna be a bit of a mess, but not the worst I don't think

#

I think actually I've got that covered

raven ridge
#

when there's an "r" prefix you don't have to check for \" or \'

#

so raw strings need to be handled differently anyway

static bluff
#

The regex I've got catches escaped characters between quotes

raven ridge
#

then it's wrong for raw strings, right? ๐Ÿ˜„

static bluff
#

Probably XD

raven ridge
#

assuming you're matching both with the same regex, anyway.

static bluff
#

Good catch, I wouldn't have thought of that

raven ridge
#

raw strings, f strings, unicode strings, and byte strings all have different internal grammars - the things allowed inside them all are different.

static bluff
#

Oh, so, actual entirely different grammar specs?

raven ridge
#

inside a raw string, \" is two characters, inside a regular Unicode string it's one.
Inside a Unicode string you can have a \u1234 escape, inside of a byte string you can't.
Inside of an f string, you can have expressions inside {...}

#

etc

static bluff
#

Of course

raven ridge
#

so yeah - the actual tokens allowed in each of them are different.

static bluff
#

I'll have to take a look at tokenize.py again- I know it does a bit of processing on strings before the tokens are created- replacing escaped characters with unescaped equivalents in non-raw strings, etc

#

Thanks for the help

paper echo
gleaming rover
paper echo
#

i tried for a little bit with apache spark stuff, seemed like a decent "modern" programming language but nothing was or is compelling me to try it for server app dev. maybe i should give it another look though now that scala 3 is out

gleaming rover
#

sbt can be problematic

#

but you can use Maven/Gradle

paper echo
#

ive heard even worse things about gradle ๐Ÿ˜†

gleaming rover
#

Scala has some REALLY nice features though

paper echo
#

i remember i couldnt for the life of me wrap my head around implicits

gleaming rover
#

implicits come to mind (which I wish Python had)

#

oh.

#

๐Ÿ˜”

paper echo
#

maybe because i hadnt written enough scala and hadnt encountered the use case yet

gleaming rover
#

typeclasses are one big one

paper echo
#

also i found it interesting/weird that case classes were not classes, and that they were called "case classes" as opposed to "struct" or something

gleaming rover
#

why do you say theyโ€™re not classes

paper echo
#

i thought they weren't interoperable with other classes

gleaming rover
#

case class is a weird name though

paper echo
#

they're basically python dataclasses right?

gleaming rover
#

I suppose the idea is pattern matching cases

#

ye

gleaming rover
paper echo
paper echo
gleaming rover
#

stupid name if you ask me

#

but well

paper echo
#

maybe things were different years ago? but if that's all it is then it makes more sense

gleaming rover
#

as far as I know (2.5 years back or so) theyโ€™ve always been this way

#

there are certain restrictions on inheritance

#

because of said autogeneration

paper echo
#

makes sense

#

scala syntax does seem pretty nice

#

nice flow, not too verbose

#

i guess interop with java stuff is cool

wary briar
#

Hello Guys, I have an interest in Data Science and I want to find out if anyone is currently learning data science with python? I would love to have a partner in my journey.

visual shadow
unkempt rock
#

am i advanced if i have created an api

#

on my own

lament sinew
#

It's not a necessary nor sufficient condition.

unkempt rock
#

ะŸั€ะธะฒะตั‚!

#

My name is nik

boreal umbra
#

@unkempt rock Hello! This is strictly a discussion channel. Try chatting in #python-discussion.

static bluff
unkempt rock
#

idk

#

ive been learning py for 3 years

#

so i believe an API is pretty much feasible at this level

#

anyone here has experience with pytest?

rich cradle
fallen slateBOT
#
Resources

The Resources page on our website contains a list of hand-selected learning resources that we regularly recommend to both beginners and experts.

https://pythondiscord.com/pages/resources
eager trail
unkempt rock
#

yea

#

im making an api

#

and i wanna add tests

#

but

#

i need to stop the websockets

#

and whenever i try it throws a ConnectionClosedError

#

tldr how to use pytest with async functions

sacred yew
#

this is not a general help channel

#

@unkempt rock

#

it would also help them if you provided code

tacit hawk
vale nebula
#

I was playing with recursion and run into RecursionError: maximum recursion depth exceeded. So i just set a higher value but python now just dies silently, eg:

import sys

if len(sys.argv) > 1:
    sys.setrecursionlimit(int(sys.argv[1]))
print(f"{sys.getrecursionlimit()=}")

def recursion():
    return recursion()

recursion()```

```bash
$ python -VV
Python 3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)]

$ python recursion.py
sys.getrecursionlimit()=1000
Traceback (most recent call last):
  File "D:\OpenSource\python-discord\recursion.py", line 10, in <module>
    recursion()
  File "D:\OpenSource\python-discord\recursion.py", line 8, in recursion
    return recursion()
  File "D:\OpenSource\python-discord\recursion.py", line 8, in recursion
    return recursion()
  File "D:\OpenSource\python-discord\recursion.py", line 8, in recursion
    return recursion()
  [Previous line repeated 996 more times]
RecursionError: maximum recursion depth exceeded

$ python recursion.py 3000
sys.getrecursionlimit()=3000

$ ```
No exception. no nothing. Is this known? I search a bit but found nothing.
grave jolt
#

@vale nebula Works for me, not sure what's up.

unknown.png
#

Maybe on windows it doesn't show when you have a segmentation violation?

#

Try this:

import ctypes
ctypes.string_at(0)
vale nebula
#

OSError: exception: access violation reading 0x0000000000000000

grave jolt
#

well, you shouldn't change the recursion limit in real code ๐Ÿ˜›

vale nebula
#

I guess it's like the segmentation fault for windows but the limit is very low, it's 2565 for me.

#

Not that I need it.

lament sinew
#

i assume stack size is very os dependent.

#

but i think you can edit on linux

vale nebula
#

I'm on win10 64bit (20H2).

lament sinew
#

should work for windows i think

vale nebula
#

It starts with 3.8. Version 3.6.8 and 3.7.6 don't seems to behave the same way.

whole grail
#

mmm advanced

spark magnet
paper echo
#

does pypy have tail call elimination?

#

that'd be pretty cool

#

actually i wonder if hy does... should be easy to check

#

aw nope

(defn foo [x]
  (yield x)
  (foo (+ 1 x)))

(foo 3)

compiles to

def foo(x):
    yield x
    return foo(1 + x)


foo(3)
#

that is a tail call, right?

raven ridge
#

Hm, it's not really a recursive call at all, really... The "tail call" is just constructing a generator object

halcyon trail
#

python doesn't have guaranteed TCO, not sure if it has TCO at all

#

TCO is pretty overrated tbh, I remember seeing some threads at various points where people complained about python's lack of TCO

grave jolt
halcyon trail
#

maybe overrated is the wrong word because most people realize it's not that critical. Some people value the idea a lot

#

TCO is primarily important in situations where you are using recursion the way that most languages would use looping

grave jolt
#

ye

halcyon trail
#

if you have a super duper pure language that doesn't want to mutate local variables, but call a new function instead. or a very simple language that wants to minimize primitives (like scheme)

grave jolt
#

I think lisps use a special construct instead of having TCO TCO

halcyon trail
#

the average language which has plenty of mutation and a built in for loop has no, no need for TCO

#

that'sclojure

static bluff
#

So, pattern matching looks helluh cool