internals-and-peps

[iron perch]

I died reading that ^
but basically because it does look up first then assign which is kinda expensive or something?

how about using try and except rather than checking? how is it better?

[pliant tusk]

if you are accessing the dictionary for the same element more than once then it is marginally faster to store to a variable

try/except will most of the time be slower due to how exception handling needs to traverse up the stack

[short salmon]

whats the thresehold of advanced

[undone hare]

I think the channel description defines it pretty well:

Discussion on the use cases, implementation and future of the Python programming language including PEPs, advanced language concepts, new releases, the standard library, and the overall design of the language.
Basically this is channel is more about an overview of the language itself and its ecosystem

[novel vector]

What is the most efficient way to make a bytearray from int? Cause things like bytes(14) will print some crappy \x00 x14 only

[boreal umbra]

did you try bytearray(14)?

[novel vector]

Yeah its pretty much the same:

>>> bytearray(14)
bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')

[boreal umbra]

what was the desired output?

[novel vector]

Something like a byte representation of int 14, like '\xSOMETHING' in one byte

[boreal umbra]

you can't represent 14 with one byte

[novel vector]

Tried to look into a structs but seems like an overkill

I mean byte object

You can repr lots of bytes with one byte object I guess

[boreal umbra]

>>> bytes('14', encoding='ascii')
b'14'

You could do that, I guess

[flat gazelle]

do you just mean b'\x0e'?

not that, just a single byte object whose sole byte is 14

[novel vector]

I guess this is why some ppl still tend to use py2, low level seems so messy with py3, ie :

>>> i = bytes('14', encoding='ascii')
>>> i.hex()
'3134'

Like, why the hell hex of 14 is 3134 somehow?

[flat gazelle]

because it's the ascii code of 1, which is 49, followed by the ascii code of 4

[novel vector]

Yeah, so its not the actual byte value that we wanted then

[flat gazelle]

do you just mean ```py
bytes([14])

[novel vector]

Oh, this works, thanks

[void vault]

hey guys! i recently learned OOP and i am learning more of Python. I wanted to ask that which standard libraries should i learn to create a GUI like KivyMD since i plan to make a GUI from scratch

[boreal umbra]

@void vault this would be a topic for #user-interfaces.

[void vault]

oh okie

[unkempt rock]

pls answer

[charred wagon]

py. For future reference, this isn't the right channel for such question.

[unkempt rock]

sorry

[brave badger]

@torpid plinth Not really appropriate for this channel nor is it something that we want being shared in the server in general. Let's try to avoid sharing malicious scripts here.

[torpid plinth]

btw, this isnt atteneded as malware, its suppose to be a matrix rain program

yeah ik thats what a mod would be writing about.

me and my friend were making palaendrome code and seeing if we could write it on paper. but then we wanted to do more than just desk check, then i told him about the chr function in python

and we got into creating matrix rain, we had a simple for loop which was long and we just printed out the ascii which gets generated automatically using chr.

i added the threads so it can run faster and longer

where can i share it btw

@brave badger sorry for the ping but i need a answer, why is my code considered malware

i made it as a matrix rain code and another server said its malware.

[brave badger]

We simply don't want users sharing scripts that can crash computers; you also claimed that was the case.

If you have further concerns, feel free to contact @summer lichen.

[torpid plinth]

like its actually upsetting me, i made it with good intentions, its to show chr function.

[uncut sierra]

This should be allow 😛

a = [2,1,1,3,4,1]
b = []
for x in a if x == 1:
    b.append(x)

for x in list if condition :

well it is true I can use filter tho ...

[oblique crystal]

why not comprehension?

[grave jolt]

I don't think this is going to be accepted given that

for x in a:
    if x == 1:
        b.append(a)

already exists

or, well, yes, ```py
b = [x in a if x == 1]

[oblique crystal]

oh

[shrewd topaz]

u should use list comprehension

there u go

[uncut sierra]

well if syntax for list compresion exist why not extend the for loop syntax to accept a simple condition .... instead of going throught filter ...

[grave jolt]

You can already use an if inside a for loop

if the body is long and you don't want the extra indentation, you can use continue:

for x in a:
    if x != 1: continue
    ...
    ...
    ...

[uncut sierra]

yes I know ...

well if nobody see the point forget about it 🙂

[grave jolt]

You can also use a generator expression instead of filter:

for x in (x for x in a if x == 1):
    ...

[uncut sierra]

it is syntax sugar, but sometime sugar might be good

[oblique crystal]

generator is not a syntax sugar actually

[uncut sierra]

genrator, not but it could be more simple

in case it is used as filter, not transformer...

[grave jolt]

well, gen = (x for x in a if x == 1) is the same as

def _gen(a):
    for x in a:
        if x == 1:
            yield x
gen = _gen(a)

[oblique crystal]

yeah

[uncut sierra]

gen can be used to transform ... for x in (x+2 for x in a if x == 1):

I thhink

but a quick syntax for filter could be nice for x in a if x == 1:

... back to coding my little thing , break if over 😕

[peak spoke]

I don't think it really improves on the readability of using a separate statement right under it, and adding an another expression to the statement will just make it harder to read for non trivial uses

[uncut sierra]

we are used to read it in list comprehension 🙂

[peak spoke]

Well, those have to be expressions so it fits there more, and are commonly misused which leads to an incomprehensible mess

[uncut sierra]

🙂

[oblique crystal]

list ||in||comprehension

[gusty marsh]

what do you guys think of os.path and pathlib? Which would you choose to go with and why?

[visual shadow]

Pathlib because it's nicer

Especially the path joins are so convenient

And it just overall offers more sensible api. Except I wish it allowed deleting directories recursively ugh

[gusty marsh]

what do you mean by "more sensible api"?

[visual shadow]

It provides every operation you could want on a path, on the path object itself. Say if I wanted to know the parent folders of a path. If I wanted to get the extension. Or the base name. Etc.

Those are things that shouldn't force you to memorize and import external function calls to access, pathlib just provides them all on the object itself.

[gusty marsh]

os.scandir gives you that tho

[visual shadow]

So now we've added os.path and os.scandir when pathlib could have handled it both. And then some.

[gusty marsh]

for the above comment of mine

[visual shadow]

Are you trying to say pathlib is inconvenient for trying to provide a single interface? Because otherwise I don't even see the point of this. Yes pathlib won't do something that you couldn't already do. It just tries to provide a single interface for path operations

[gusty marsh]

not really, i am just wanted to know your thoughts on it, i use pathlib btw 😄

[peak spoke]

I don't think os.path is all that bad, but it's really inconvenient and not as readable compared to pathlib, even if you ignore things like using proper path seps

[visual shadow]

Honestly it was primarily the path joins that got me to switch

If there's one thing I wish pathlib did (okay fine two things, I want that folder delete!) it's that I wish pathlib subclasses strings

But honestly for everything else, including iterdir, path manipulation operations, pathlib just does it nicer.

[peak spoke]

You can mostly ignore it as / usable almost for everything, but working with strings isn't very convenient either and you need to account for trailing slashes etc.

Or, well, at least should as I believe at least windows can handle things like duplicated slashes and things like that

[gusty marsh]

right, i agree with that, and use pathlib myself and it is more readable. But whatever pathlib provides is already supported by os, other then the path joins.

[visual shadow]

We've had code break in the past because people were trying to collaborate on code from different os. The verbosity of os.path.join made people want to take shortcuts. Pathlib made for a more compelling case to get people to finally start using better path controls

Absolutely, I think pathlib offers just about 0 new functionality in the sense that people used to be able to use paths and related operations without pathlib. It's essentially a big api unification

[gusty marsh]

right, at the end, it just goes to user's preference on which way they would like to use path joins

[gusty marsh]

is this what you mean by trailing whitespaces ?

[sacred tinsel]

I started learning Python on 3.6 so pathlib was already available and I never really used os.path, though when I come across legacy code that extensively uses it for non-trivial operations I have a tendency to migrate it and it usually ends up feeling a lot better

I don't really like using the / operator on the objects though, and I generally construct paths as Path("a", "b", "c") since that looks the cleanest to me

[gusty marsh]

os does that too btw

[boreal umbra]

Why don't you like it?

[sacred tinsel]

I think it's just one of the cases where some people will argue it's a neat use of the operator and some will argue it's unnecessary operator abuse, and it probably feels a bit more like the second to me, but it's not a hill that I'd die on

I wanted to say that if the lhs ends up being some wrong type you'll get a more cryptic exception than necessary but looking at it it's probably not even the case 😄

If you have, e.g.:

>>> def extend(path):
...     return path / "file.ext"

As opposed to:

>>> def extend(path):
...     return Path(path, "file.ext")

The second will work with path: str, but maybe you don't need to worry about it

[boreal umbra]

It satisfies the property that the value returned by one call is a valid left operand for a subsequent call

[sacred tinsel]

The exception you get from a wrong type being passed is also probably going to be a little better too

Yeah, I don't mind if you use it, I just choose not to

[boreal umbra]

Though I agree that it's probably the most... of beat (?) use of an infix operator in the stdlib

Also, obligatory "I still want function composition with matmul".

[swift imp]

Me too

I also want coalescing operators

[bronze anchor]

Does anyone know how I could share the "profile" for a logger I create in main with all sub modules I have? In main, after cli args are parsed, I create a FileHandler using the file destination. This works fine for main(), but when I create loggers in new modules, they are using the default values.

I've sort of been bashing my head against the table with this logging module and multiple streams for hours 🤦‍♂️

[sacred tinsel]

I think you would want to attach the handler to the root logger rather than the local logger for your main module, e.g. as @fallen slate does here
https://github.com/python-discord/bot/blob/9e59a1aa4d14481411abb931731fa2c807a96ae8/bot/log.py#L31-L32

[fallen slate]

bot/log.py lines 31 to 32

root_log = logging.getLogger()
root_log.addHandler(file_handler)```

[sacred tinsel]

Log records from local module loggers will propagate to the root logger unless told not to, so unless you need a more sophisticated setup, this is how you can attach the handlers in one place only

If you need more help it'd probably be better to open a help channel

[swift imp]

Figures u start typing once I delete it lol

I thought it was a bad question lol

[sacred tinsel]

No it was a good question

[swift imp]

Does logging module add a lot of overhead to code?

[sacred tinsel]

I'll answer anyway 😄 It probably depends on what you'd consider a lot, usually it will be negligible. There was only 1 time that we had ran into performance issues caused by logging and profiling revealed that the true bottleneck was constructing the message strings over and over again rather than the logging mechanism itself

The culprit code was also using f-strings which is convenient to work with, but it means that the whole string has to be constructed and formatted even if the log doesn't get emitted (so there's no way to "disable" it)

The logging module allows you to pass the base strings with formatting placeholders and the args separately, which brings the advantage that the string doesn't actually get formatted if you don't emit the message, so you can save some of the overhead by disabling the local logger or bumping its level, etc

I don't know how much it would have helped though, since we abandoned the approach entirely and instead tracked state changes in a custom object that would produce summary logs after each run / session

[eager trail]

Yay for caching

[subtle citrus]

hi guys i have a project and am gonna talk about python
i wanna know pls what can we do with python

[meager nest]

@subtle citrus a lot of things

the three main things python is used for which i can think of is

Web Development Stuff with Flask, Django

Machine Learning and AI

Data Science and Data Visualization

other than this you can make games, desktop GUIs, web scraping

and a lot more

https://www.python.org/about/apps/

[mortal stag]

Hi! i am on linux and i want to start programming on python

what app

can i use

[meager nest]

@mortal stag You first need to install Python on your machine if you are on a linux based distro you probably have it, you can check the version using python3 --version

and another thing you need is a code editor

I recommend using Visual Studio Code if you are a beginner

[cosmic berry]

if u r comfo with vim use it

it depends on which u r comfortable with

[forest kite]

Hey

How do you guys upgrade python?

I have a pretty messy system rn with 4 python versions installed and i need only the latest one

[meager nest]

I am on ubuntu so sudo apt-get update does the job

[meager nest]

I would recommend to remove all the python versions delete all the files associated with it and then install the latest version

[forest kite]

Hmm

I tried doing that

But it broke my system somehow

[meager nest]

which OS?

[forest kite]

Had to use timeshift

Ubuntu

[meager nest]

you should never remove the stock python version that comes with ubuntu

there are a lot of system programs that depend on it

[forest kite]

Oh

[meager nest]

on windows its not a problem

[radiant fulcrum]

btw you might wanna move to #unix or a help channel

[forest kite]

So which version is that? I have 2.7 3.6 3.8 and 3.9

[radiant fulcrum]

#internals-and-peps isn't really the right place

[meager nest]

Yea sure

[forest kite]

Oh ok

[oblique crystal]

i did once.....

[thin wyvern]

yeah, i found that out the hard way... lol

[grave jolt]

yay https://github.com/microsoft/pyright/discussions/1782

[flint basin]

uhm?

[grave jolt]

it seems that Pyright/Pylance will be able to make friends with pydantic

[flint basin]

i saw it instantly

[flint basin]

woah cool ❤️

[halcyon trail]

Something like that would be really welcome

One pain point even in mypy is that you cannot just define your own decorator even if it is say a trivial pass through to dataclass, for the purpose of changing defaults, and still have things work

[magic python]

do you have a rule of thumb for the different logging levels? There are 6 levels, i'm not sure how people typically determine what defines a particular level though

oh - there's an outline here: https://docs.python.org/3/howto/logging.html

[peak spoke]

info/debug can be harder to get right, but with warning it's a heads up that something may have gone wrong, errors for things like unhandled exceptions which prevent something from completing and critical if the whole program is affected by the error (and possibly stops)

[magic python]

i currently have a lot of stuff that generates analysis, and there's a lot of printing out to stdout as it is produced, thinking of putting that over to logs - but not too sure how others manage log files and stuff 🤔

[grave jolt]

well, with mypy you can always write a plugin

but many other type checkers (like pyright) don't have a plugin system

[gusty marsh]

i mostly use - info, debug and error
-> error: when something unexpected happened, and it will cuz some trouble in your code/core application
-> info: useful information to log, like your application starts up
-> debug: say i am making a discord cog, and i get stuck as the ctx.send is sending a empty list, so i would use debug statemetns to "print" the list out at various places, so i can know where it is causing the problem

sometimes, i mix debug statements with print statements

[azure fiber]

I want to use Dgraph with Django, I have searched for it's support but not able to find anything except pydgraph, but I am not able to use it with django. Can anyone suggest me a way?

[unkempt rock]

I need a suggestion for
I want to be a machine learning engineer but after 10 grade which subject I have to choose.? And what to do.?
Please help me

[sacred wave]

where are you from?

[visual shadow]

any track that gets you into computer science would be fine really. so, pick a track with maths and computer science in it, if you have both

[flat gazelle]

https://github.com/python/cpython/blob/master/Include/internal/pycore_long.h#L15-L25

[fallen slate]

Include/internal/pycore_long.h lines 15 to 25

static inline PyObject* __PyLong_GetSmallInt_internal(int value)
{
    PyInterpreterState *interp = _PyInterpreterState_GET();
    assert(-_PY_NSMALLNEGINTS <= value && value < _PY_NSMALLPOSINTS);
    size_t index = _PY_NSMALLNEGINTS + value;
    PyObject *obj = (PyObject*)interp->small_ints[index];
    // _PyLong_GetZero(), _PyLong_GetOne() and get_small_int() must not be
    // called before _PyLong_Init() nor after _PyLong_Fini().
    assert(obj != NULL);
    return obj;
}```

[swift imp]

class ParentRegistry:
   registry = {}
   def __init_subclass__(cls, *, foo, bar):
      ParentRegistry.registry[(foo, bar)] = cls
   def __new__(cls, foo, bar):
      selected_child = cls.registry[(foo, bar)]
      instance = super(cls, selected_child).__new__(selected_child, foo, bar)
      return instance
   def __init__(self, foo, bar):
      super().__init__(foo, bar)
    
class child(ParentRegistry, foo="foo", bar="bar"):
   def __init__(self, foo, bar):
      super().__init__(foo, bar)

Children never get instantiated directly, all access is through parent registry .

What's the thoughts on this?

[lament sinew]

what's the use case?

[fallen slate]

Hey @unkempt rock!

It looks like you tried to attach a Python file - please use a code-pasting service such as https://paste.pythondiscord.com

[undone hare]

Hey @unkempt rock! This channel is for more in-depth discussions around the Python language. Feel free to ask your question in off-topic!

[spark magnet]

why?

[swift imp]

Each subclass is a custom implementation of parent that is dependent on the information hashing the child

And this creates a common entry point to each implementation, which will return the correct one

[spark magnet]

Why not just use a function to return the correct one?

@swift imp you could do the same thing with less code by just writing a function.

maybe?

[swift imp]

Well then you have manually add the function to the registry

By subclassing you get the benefit of inheritance

And then __init_subclass__ handles the registration

This seems intuitive to me but I'm a little over a year into python and any kind of software engineering in general

Hence I'm looking for opinions

[swift imp]

?

[spark magnet]

i meant, "maybe a function could serve this purpose"

@swift imp it seems to me that you don't mind using the least-used features of the language, in esoteric ways. (three-space indents!?)

[swift imp]

Typed on my phone lol, sorry

[spark magnet]

@swift imp i guess it would help to see a real use of this

[swift imp]

I'm gonna try to reply later when I'm home

[halcyon trail]

@swift imp i think this is fine, I do stuff like this not exactly often, but it's not rare either

It's a pretty straightforward to handle creation of a polymorphic factory without boilerplate or risk of forgetting to register children, which is a pretty standard use case

[halcyon trail]

I didn't quite do it the way though that this code works though. I used init subclass and a registry, but I just had an ordinary function use the registry, not immediately sure why you decided to implement new instead

[forest birch]

A resettable counting closure

!e

def make_counter():
    running_count = 0
    reset_count = 0

    def counter():
        nonlocal running_count
        running_count += 1
        return f'Counter: {running_count}'

    def resetter():
        nonlocal running_count
        nonlocal reset_count
        reset_count += 1
        running_count = 0
        return f'Counter: {running_count}, Resetted: {reset_count}'

    return counter, resetter


counter, resetter = make_counter()

print(counter())
print(resetter())
print(counter())
print(counter())
print(resetter())
print(counter())
print(resetter())
print(counter())

[fallen slate]

@forest birch :white_check_mark: Your eval job has completed with return code 0.

001 | Counter: 1
002 | Counter: 0, Resetted: 1
003 | Counter: 1
004 | Counter: 2
005 | Counter: 0, Resetted: 2
006 | Counter: 1
007 | Counter: 0, Resetted: 3
008 | Counter: 1

[forest birch]

Question: are closures like the above seen much "in the wild"?

[flat gazelle]

most people just use classes

I have seen it a ton in JS code though

[peak spoke]

Don't think I ever saw nonlocal used in normal code I went through

[forest birch]

I'm going to profile the closure vs a class.
I'm guessing the closures are much lighter weight.

[flat gazelle]

they probably are, considering they use locals instead of dict elements, though slots should help with that. But that isn't all that important, a class can handle this in a neater way.

[peak spoke]

All of the functions have to keep their own dict so not faring much better there

[flat gazelle]

but the variable accesses aren't into dicts, but into frame locals and closure cells, whereas self. is always a dict lookup

[peak spoke]

Ah, I was just thinking of its footprint

[forest birch]

I'm thinking of using this in a decorator for use when tracing / debugging my code.
I think I can use it to keep a running count of how many times the decorated function has been called.

I don't think that's possible with classes?

[grave jolt]

I did this in JS 👀

it's actually pretty cool there

[radiant fulcrum]

JS has some interesting event loop semantics with closures

[pliant tusk]

I find closures neater when 2 functions need to refer to the same object but I don't want to use a global

[swift imp]

Single entry point and it makes more sense imo because you are instantating a ParentRegistry technically

Plus I can abstract the logic out by making a metaclass

[raven ridge]

The functools.lru_cache decorator is an example. It adds a cache_clear() method as an attribute of the decorated function to drop the cache.

https://github.com/python/cpython/blob/3.9/Lib/functools.py#L622-L637

[mild flax]

!e

class C:
    def __init__(self, f):
        self.f = f
        self.count = 0

    def __call__(self):
        self.count += 1
        return self.f()

@C
def p():
    print("hello world")

p()
print(p.count)
p()
p()
print(p.count)

[fallen slate]

@mild flax :white_check_mark: Your eval job has completed with return code 0.

001 | hello world
002 | 1
003 | hello world
004 | hello world
005 | 3

[mild flax]

maybe im not totally understanding the constraints, but i think something like that would work

[spark magnet]

Instead of making and holding onto two functions, you can make and hold onto one object with two methods. Why wouldn't an object be able to do this?

[deft pagoda]

you don't really need a class for this either though

i guess, unless you just really wanted a class

[swift imp]

@spark magnet @halcyon trail

from abc import ABC, abstractmethod

class SpecialCallableABC(ABC):
    """A callable class ABC that houses the implementation of [x].
    The implementation for [x] depends on specific values for
    foo and bar. Pass values for foo and bar to register the
    implementation method for that combination, and that concrete
    implementation get stored in the registry under key (foo, bar).
    """

    registry = {}

    def __init_subclass__(cls, *, foo, bar):
        print(f"registering implementation of [x] for foo={foo} and bar={bar}")
        SpecialCallableABC.registry[(foo, bar)] = cls

    def __new__(cls, foo, bar):
        print("Selecting correct implementation of [x].")
        selected_child = cls.registry[(foo, bar)]
        instance = super(cls, selected_child).__new__(selected_child)
        print(f"created instance of {type(instance)}.")
        instance.__init__(foo, bar)
        return instance

    def __init__(self, foo, bar):
        print("Initializing implementaiton of [x] base class, the registry.")
        self.foo = foo
        self.bar = bar
        super().__init__()

    def __call__(self, *args, **kwargs):
        return self.implementation(*args, **kwargs)

    @abstractmethod
    def implementation(self):
        """The abstract method which is the implementation for [x]
        when foo and bar are certain values."""
        ...

class child(SpecialCallableABC, foo="foo", bar="bar"):

    def __init__(self, foo, bar):
        print(f"Initializing the concrete implementation of [x] for foo={foo} and bar={bar}")
        super().__init__(foo, bar)

    def implementation(self):
        print(f"Calling the implementation for [x] when foo={self.foo} and bar={self.bar}")

x_implementation = SpecialCallableABC("foo", "bar")

Selecting correct implementation of [x].
created instance of <class 'main.child'>.
Initializing the concrete implementation of [x] for foo=foo and bar=bar
Initializing implementaiton of [x] base class, the registry.
Initializing the concrete implementation of [x] for foo=foo and bar=bar
Initializing implementaiton of [x] base class, the registry.
x_implementation()
Calling the implementation for [x] when foo=foo and bar=bar

[spark magnet]

@swift imp i was hoping to see real examples. What are these children, what are foo and bar?

[swift imp]

Ah, sorry. So unfortunately the place I work, I cannot show the code. However, to expand a little. foo and bar are settings for a simulation, and each child is the implementation on how to process the simulation output. There's more metadata than foo and bar but I truncated for brevity. The ABC is also incredibly larger with more stuff to fill out.

There's about 30 different simulations and each simulation has like a dozen or so different scoring implementations depending on the metadata combinations. For what its worth, at work, each child is indeed a callable.

[spark magnet]

I would probably have made a class decorator, and a function to find the right class.

[deft pagoda]

in the curio package, i know beazley keeps the event loop inside a closure so nothing will mess with it -- i thought it was really strange when i saw it

[swift imp]

That is something we may still do, but this method seemed appealing to me because we could metaprogram the ABC for each simulation, making the burden a lot less.

[deft pagoda]

starts here: https://github.com/dabeaz/curio/blob/master/curio/kernel.py#L188

[spark magnet]

I like the idea of separating the registration from the ABC-ness of the classes

[novel vector]

In [94]: ord('灩')
Out[94]: 28777

In [95]: ff[0]
Out[95]: 231

In [96]: ff[1]
Out[96]: 129

In [97]: ff.decode()[0]
Out[97]: '灩'

In [98]: ord(ff.decode()[0])
Out[98]: 28777

In [99]: ff
Out[99]: b'\xe7\x81\xa9\xe6\x8d\xaf\xe4\x8d\x94\xe4\x99\xbb\xe3\x84\xb6\xe5\xbd\xa2\xe6\xa5\xb4\xe7\x8d\x9f\xe6\xa5\xae\xe7\x8d\xb4\xe3\x8c\xb4\xe6\x91\x9f\xe6\xbd\xa6\xe5\xbc\xb8\xe5\xbc\xb0\xe6\x91\xa4\xe6\x8d\xa4\xe3\xa4\xb7\xe6\x85\xbd'

In [100]: ff[0]*ff[1]
Out[100]: 29799

Can some1 explain me how the value of this char is being calculated from the first bytes? My guesses are not adding up here

[spark magnet]

this is better asked in #python-discussion

[halcyon trail]

@swift imp i would just find yoru code very surprising because when you call a constructor directly in python you generally expect to get exactly that type out of it

If you do f = Foo() then I expect f to be exactly a Foo, not a child of Foo, in general

but that dcould just be my own biases

I don't consider "extract out the behavior into a metaclass" compelling because the whole idea of init_subclass is to avoid metaclasses to beginwith

[swift imp]

Well I was thinking if I have to write a bunch of registries I could use __prepare__

[halcyon trail]

metaclasses can rapidly get tricky to work with, if you have multiple inheritance and such the metaclasses have to be compatible, so you may need to create new metaclasses, adn so on

[swift imp]

That's very true

[halcyon trail]

Not quite sure what you mean by a bunch of registries tbh

but prepare already seems like a lot of black magic relative to the scope of your problem

[swift imp]

Like if I wanted to make a SpecialCallableACB1, SpecialCallableABC2, SpecialCallableACB3 I could use metaprogramming to automatically add the custom __new__, __init__, __init_subclass__ instead of repeating it each time

[halcyon trail]

you could, that's probably in the zone of diminishing returns though, tbh. Also, you don't really need the 'new'

__new__

I'm actually not sure why you need the init either, but probably just need to look at it more

[spark magnet]

@swift imp that seems overcomplicated. Make a Registry class, and give it a method that is the class decorator. then you can do callable1reg = Registry(); ... @callable1reg.register class MyCallable1; etc

[halcyon trail]

I dunno, having a separate registry class seems more error prone, if I undersatnd you correctly, now people who write new derived classes have to inherit from the correct base and also decorate from the correct registry

I have mixed feelings about the rest of it but using an init_subclass hook is actually less complicated than a decorator in many ways

[swift imp]

__new__ is more so that we have a common access point. An alternative to just do SpecialCallableRegistry.registry[] to get the correct child.

Its probably more clear just to access the dictionary directly tbh

[halcyon trail]

Well, usually in these patterns the "common access point" is actually a function

not calling the constructor, and not accessing the dictionary either

you would have a function somewhere called make_callable(....) that takes all the arguments

[swift imp]

I just feel like, if I am going to be instantiating the class anyway, why not just select it in __new__ and save the details of the function. Like __new__ is getting called no matter what

[halcyon trail]

some of those arguments handle selecting which class is called whereas others might be forwarded to the constructor

not sure what you mean by "save the details of the function"

[swift imp]

So if I dont go the __new__ route and use the function, __new__ is still getting called. So why not just utilize __new__ and save myself from writing the function.

[halcyon trail]

the function IMHO is a lot clearer because... it's just a function. You know exactly how it's going to be called, you know how it's using the registry, there's no "implied" API that there is with all dunder methods

I would happily use a dunder method if it offered a concrete advantage but here it doesn't really

I mean if you use the function you wouldn't need to write new, you'd just be using the default new

[swift imp]

Yeah I mean, its totally not normal for a class constructor to return a subclass lol

[halcyon trail]

Yeah

[swift imp]

So I totally get your point there

[halcyon trail]

what I would usually expect is that you can't construct an abstract base class directly, at all

so it's doubly surprising

[swift imp]

Well __new__ makes the instance. Simply accessing __new__ of an ABC wont raise an AbstractError

[halcyon trail]

I understand that, but in terms of typical expectations

at any rate, even leaving that aside, I still think there's no benefit to overloading new here

[swift imp]

Ehh, I guess it depends on how well you know abc.ABC and __new__ (no offense), I was pretty positive it would be doable but I am not calling __init__ on an ABC.

[halcyon trail]

Err, I'm not offended but if you think that my viewpoint is what it is because of lesser understanding of python then I guess why bother asking for help in this channel to start with 🙂

[swift imp]

No no, your points are amazing. It just about the expection of ABC and __new__.

[halcyon trail]

I mean abstract classes generally

they're not a python invention

You don't expect to be able to call the constructor for an abstract class

(as an end user)

[swift imp]

oh you mean __call__ on the metaclass? Thats the constructor, in my mind

[halcyon trail]

Well, maybe I misunderstood but yo uoverloaded new so that you can do SpecialCallableABC(....)

[swift imp]

like SpecialCallableABC() should throw an error before it even gets to __new__ you mean? As in within type.__call__?

[halcyon trail]

I mean people generally don't expect to invoke constructors of ABC's. So generally I wouldn't write, or expect code, like x = SomeABC()

[swift imp]

GOTCHA

Yeah that's another issue I thought about.

[halcyon trail]

Even putting that aside, I do still think a function is simpler than a dunder method

[swift imp]

It is, without a doubt 🙂

[halcyon trail]

with a function there's no unstated assumptions, or implicit calls

you use dunder methods only when you want things to specifically fit in with expectations

like if you want a normal for loop to work, or an if, etc etc

So, all 'm really saying really is that overloading new here instead of just using a function to construct derived instances doesn't really ipmrove your solution

and it is IMHO more complex

whereas init_subclass does improve the solution a great deal, because manual registration would be error prone, etc

[swift imp]

https://stackoverflow.com/a/18219145/9063378

The second answer lol

justuse a function

[halcyon trail]

heh yeah

[swift imp]

fwiw too I was inspired heavily by this post https://stackoverflow.com/a/28076300/9063378

It also references the factory function pattern

[halcyon trail]

Gotcha

I will give that a read tomorrow

I actually did this exact pattern for the first time and was really happy to discover how simple init subclass made it

Only a few weeks ago

My first solution was using metaclasses and I believe new

[swift imp]

Yeah, the pep for it makes it very clear the registry was the intent of the dunder.

[halcyon trail]

Yeah. It's a very well explored problem in software engineering. I wrote a blog post about how to do it in C++ although the solution is a bit fragile because C++

When in python though you do have to make sure the module where the classes live get inported

That's easy to miss

[swift imp]

For sure

Good night!

[novel vector]

In py local / nonlocal / global concept is so rtarded, god have mercy

[pliant tusk]

why do you think that?

[novel vector]

In the docs its described as :

def foo():
   x = 10
   def bar():
       nonlocal x
       print(x)
       x += 1
   bar()
   print(x)
foo()
10
11

But I do the same construction in my code:

            for line in fl:
                cap_line = ''

                if args.ignore_case:
                    try:
                        nonlocal cap_line
                        cap_line = line

And its SyntaxError: name 'cap_line' is assigned to before nonlocal declaration

[pliant tusk]

nonlocal {name} needs to occur in a new function before the variable {name} is used in any way. That is because the nonlocal statement tells the python compiler how to handle that variable

in your example, nonlocal occurs inside the function, before x is used

in your code, you are not entering a new scope, so you do not need nonlocal

[novel vector]

Yet, without a nonlocal its always empty variable somehow

[pliant tusk]

are you certain that args.ignore_case is truthy?

[novel vector]

oh it wasnt truthy, fml, im stupid and sleepy, pardon me wasting your time, sir

[pliant tusk]

all good, happy to help

[wispy junco]

Hi, would this be the right place to ask about how to get started with machine learning python? My goal is quite specific is to find a combination of numbers for the best result

apologies if i dropped my question to the wrong channel

[native flame]

#data-science-and-ml might be better suited

[wispy junco]

thank you!

[acoustic crater]

are there any messy aspects to subclassing built in sequence types other than slicing, multiplying, and concatenating returning the builtin type instead of subclass?

[unkempt rock]

Plenty, In CPython at least, there's no documented behaviour of when overriden methods get implicitly called. It usually doesn't happen:

In [2]: class Dict(dict):
   ...:     def __getitem__(self, other):
   ...:         print(f'fetching {other}')
   ...: 

In [3]: a = Dict(a=1)

In [4]: a['a']
fetching a

In [5]: a.get('a')
Out[5]: 1

The simple answer is to subclass collections.UserDict

In [6]: class Dict(c.UserDict):
   ...:     def __getitem__(self, other):
   ...:         print(f'fetching {other}')
   ...: 

In [7]: a = Dict(a=1)

In [8]: a['a']
fetching a

In [9]: a.get('a')
fetching a

[fallen slate]

nonlocal

7.13. The nonlocal statement

nonlocal_stmt ::=  "nonlocal" identifier ("," identifier)*
``` The [`nonlocal`](https://docs.python.org/3/reference/simple_stmts.html#nonlocal) statement causes the listed identifiers to refer to previously bound variables in the nearest enclosing scope excluding globals. This is important because the default behavior for binding is to search the local namespace first. The statement allows encapsulated code to rebind variables outside of the local scope besides the global (module) scope.

Names listed in a [`nonlocal`](https://docs.python.org/3/reference/simple_stmts.html#nonlocal) statement, unlike those listed in a [`global`](https://docs.python.org/3/reference/simple_stmts.html#global) statement, must refer to pre-existing bindings in an enclosing scope (the scope in which a new binding should be created cannot be determined unambiguously).

[unkempt rock]

That question would fit better into a help channel

[acoustic crater]

oh wow how did I never see the user sequences in collections, thank you

but get is different from __getitem__

I wouldn't call a dunder that implies index operator access an implicit call from a non dunder method

and docs for UserDict do not imply that behavior either it seems

UserList docs do explicitly say

List operations which return a new sequence attempt to create an instance of the actual implementation class.

among other things

if anything, to me, it seems like an undocumented implementation hiccup that UserDict uses __getitem__ for get

[eager trail]

Does setting a method string variable to be equal to a class string variable:
1: create a memory address for the class string variable, resulting in dynamic updating of the method variable if the class variable changes
2: create a simple string that is a copy of the class variable
3: create a complex copy of the class variable that is more memory intensive than a simple string

Referring to python 3.7 if it matters

[acoustic crater]

both names point to the immutable string

if you change the immutable a name points to it points to a different immutable

in general anyway, there are implementation details, but changing the immutable one name points to won't change the immutable another name points to

python doesn't have variables, everything like a variable is something like a pointer called a "name", if that helps

[eager trail]

Ahhh gotcha

Thanks!

[acoustic crater]

np

[acoustic crater]

UserList does exactly what i need though and turns out deque had this functionality all along

[unkempt rock]

dict.get

Return the value for key if key is in the dictionary, else default. If default is not given, it defaults to None, so that this method never raises a KeyError.

That sounds like dict.get should just be

get(self, other=None):
    try:
        return self[other]
    except KeyError:
        return other
``` since its purpose is trying to get the key or returning the default if not found?

In which cases would you want the get method of your dict to not basically be a getitem in the background? If you were making a dict that converts the values to a string before returning it, it would be odd that calling the get method loses that behaviour

[acoustic crater]

...though I wrote what I think functions exactly like UserList* in 1/4 the code as a list subclass instead of a wrapper

if I was making one, sure, but I expect built in methods to be defined in c and not to call each other unless they are directly linked like __add__ and __iadd__

[unkempt rock]

That's exactly why UserDict is written in Python, the purpose is to make sure your subclassed behaviour is used where expected

[acoustic crater]

wasn't it written before you could just subclass dict?

[unkempt rock]

I'm not sure about that

[acoustic crater]

but yeah if I want to know exactly what changing the methods will do it's nice to be able to just go look at it in collection's __init__.py

...and it's nice that deque works as if it's "UserDeque" if subclassed, too, as I just found out with my tests I was using for my own list subclass

even though it's defined in c

[visual shadow]

I can't think of a practical use right now, but one example where this behaviour would differ is with, say, default dicts.

There the get, as implemented, is careful enough to not insert a key on a get operation, while a key lookup on the dict would do it.

[unkempt rock]

That's certainly one use case, it's more so about the expectation - while implementing your own dicts, you'd probably think that get would use dunder getitem and in the case of defaultdicts you would want to override the get method to make your own default key logic

[acoustic crater]

I like my list better than UserList haha

UserList certainly acts more like a user defined list object

but I like the magic of objects defined in c

deque also behaves like my list I think, need slightly different tests

yep, returns subclass types from transformative methods and isn't stingy about construction and initialization

[acoustic crater]

also my list is 25 lines of code with more doc strings than UserList which is 126 lines

[strong lantern]

hello

[winged tundra]

!ressources

!resources

[fallen slate]

Resources

The Resources page on our website contains a list of hand-selected learning resources that we regularly recommend to both beginners and experts.

https://pythondiscord.com/pages/resources

[grave jolt]

So perhaps this hack is what's responsible?.. somehow?..
https://github.com/python/cpython/blob/3.9/Lib/typing.py#L1997-L1998

[fallen slate]

Lib/typing.py lines 1997 to 1998

_TypedDict = type.__new__(_TypedDictMeta, 'TypedDict', (), {})
TypedDict.__mro_entries__ = lambda bases: (_TypedDict,)```

[paper echo]

https://docs.python.org/3.9/library/typing.html#typing.TypedDict
https://github.com/python/cpython/blob/master/Lib/typing.py#L2241-L2299

_TypedDict = type.__new__(_TypedDictMeta, 'TypedDict', (), {})
TypedDict.__mro_entries__ = lambda bases: (_TypedDict,)

this is cursed

bottom-of-the-iceberg cursed

TIL you can actually torture python into thinking you can inherit from a function

or apparently from anything?

[grave jolt]

!e

def f(*args):
    print("f", args)

f.__mro_entries__ = lambda bases: (str,)

class foo(f):
    pass

print(foo, foo.mro())

omg this is cursed

[fallen slate]

@grave jolt :white_check_mark: Your eval job has completed with return code 0.

<class '__main__.foo'> [<class '__main__.foo'>, <class 'str'>, <class 'object'>]

[paper echo]

!e ```python
class ThanksIHateIt:
pass

class ThisReallySucks:
pass

ouch = ThisReallySucks()
ouch.mro_entries = lambda bases: (ThanksIHateIt,)

class Something(ouch):
pass

print(Something)

[fallen slate]

@paper echo :white_check_mark: Your eval job has completed with return code 0.

<class '__main__.Something'>

[grave jolt]

F

[mild flax]

jfc

[grave jolt]

this is cursed on so many levels

[paper echo]

https://www.python.org/dev/peps/pep-0560/#mro-entries

https://docs.python.org/3/library/types.html?highlight=__mro_entries__#types.resolve_bases

python 3 is common lisp 2

(except not at all because everything is a special case)

poor pypy devs

keeping up with all this shit

[grave jolt]

!e
@paper echo

class Dog:
    pass

class Bird:
    __mro_entries__ = lambda whatever, man: (Dog,)

class Duck(Bird()):
    pass

print(Duck.mro())

[fallen slate]

@grave jolt :white_check_mark: Your eval job has completed with return code 0.

[<class '__main__.Duck'>, <class '__main__.Dog'>, <class 'object'>]

[paper echo]

bottom of the iceberg

im telling you

im afraid to read the pep

[grim flax]

what does __mro_entries__ means?

[forest birch]

It holds the method resolution order for multiple parent inheritance

[grim flax]

Thanks!

[grave jolt]

apparently, it's a hack to allow inheriting from objects which are not types

[grave jolt]

that's just MRO (accessed via the mro method on a type)

so normal classes don't have __mro_entries__

[coarse cargo]

Is reading through the documentation the best way to get a deeper understanding of python?

[blissful comet]

Personal experience: transpiling python to another language is a great way to get a deeper understanding of python as well as the target language.

If you're looking for how to use python to solve X, the other suggestion sounds more interesting.

[coarse cargo]

Would learning c++ be beneficial too?

[halcyon trail]

I wouldn't learn C++ to gain a deeper understanding of python

[blissful comet]

Some of the python libraries are written in C. If your goal is to understand how the C API works etc, suggest C. Really depends on what you're trying to do.

[halcyon trail]

I mean, it's a bit hard to say, directly learning another language to try to understand a first language more deeply, it doesn't really pay off that quickly. In the broader scheme of things, understanding more about languages generally will help you understand python more, particularly in terms of the design trade-offs and such

[blissful comet]

py2many - the transpiler I'm working on helps you translate python to C++, apart from 6 other languages. You can use it as a "google translate" for python to learn other languages based on your understanding of python.

[halcyon trail]

I think that can be useful for gaining a beginning understanding but, kind of like with real languages, it's going to not work well for anything more advanced

in the sense of, you can translate some python step by step into C++, but pretty soon you come to issues where the C++ approach would just be quite different

etc

[blissful comet]

I was able to stretch it quite far:

https://github.com/adsharma/py2many/tree/main/tests/cases

All of these cases translate to C++, generated code compiles and produces the same output as python.

Ditto in the other 6 languages.

Looking for feedback on what python features to prioritize

[halcyon trail]

Sure, these are relatively basic examples though, right? Can I see the translated C++ btw?

nm I found it. So even in these basic examples right, it's not so idiomatic in C++ to take a vector for a sorting algorithm, you'd typically take an iterator pair

[blissful comet]

Yes - long way to go. But lot of progress in the last few weeks.

C++ and other code here:

https://github.com/adsharma/py2many/tree/main/tests/expected

[halcyon trail]

And even more basically, you would never take a vector<int> by value in that example

[blissful comet]

For rust, I have enough smarts to pass everything by ref

[halcyon trail]

in general the deeper you go though the more fundamental the difference in approach is going to be between the languages

and will get to the poitn where direct translation is not really possible because the preferred API to expose to clients is different, etc

[blissful comet]

The argument is that C++/rust people spend way too much time thinking about copy vs move and other memory management issues. Using python as a source language allows you to focus on the problem and then have a tool figure out memory management for you.

If you look at binit.rs, the return value of bin_it() is a Vec<i32> not a reference. It was inferred based on mutability of the return value.

[halcyon trail]

the thing is that argument is basically equivalent to saying you shouldn't use C++/Rust in the first place

if your codegen can figure out the right copy/move etc, then so can an optimizing compiler

in practice though this doesn't happened

If you're writing high performance code then you need to think about memory layout, ownership, etc up front, because it affects the solution, design, and so on of the code

[blissful comet]

Agreed - for the general case. Languages such as nim are trying to make the case that reference counting + tweaks can take you pretty far. My value add is that - I can do pass by value for some common access patterns and perhaps remove reference counting overhead via static analysis.

Yes, optimizing compilers can do it too, but for C++, they have a long history of past behaviors they need to stay compatible with. They fundamentally operate at a much lower level of abstraction compared to python source.

[halcyon trail]

the "perhaps removing reference counting via static analysis" is a common feature in most GC languages

swift, go, etc. So it's not really a value add on its own (unless its done substantially better) than languages already in the wild.

but at any rate I think we are oscillating between this transpiler as a learning tool and this transpiler as effectively a fast implementation of python

[junior surge]

What kind of a Python environment would say "Process returned -11 (0x-b)" without mentioning the signal name SIGSEGV, "segmentation violation" or "segmentation fault"? Or is this a Microsoft Powershell problem? https://www.cyberciti.biz/tips/segmentation-fault-on-linux-unix.html I asked the person who asked me, but they haven't replied yet. If it is a Microsoft problem, then is it appropriate to tweet Guido vR. or is there a better contact there? Does anyone even know how to file bugs against Powershell? Do you have to be in a dev partner program? Is there a way to fix this by wrapping the Python executable in a parent process that only checks for nonzero process exit status, and if connected to a tty, prints something more informative and then exits with the same signed byte?

[acoustic crater]

happens a lot when using ctypes the wrong way, it really depends on what you're doing

[charred wagon]

Python doesn't print its exit code before it exits. That message you are seeing is generated by your shell.

PowerShell is open source on GitHub, so you can look and file an issue there.

[blissful comet]

True - none of this is very new. But it's my (subjective) belief that python being a popular language allows you to express ideas faster and more succinctly than swift/go or other alternatives. This work provides an alternative to those looking for a faster runtime.

But it could have a secondary use case as an educational tool. rosettacode.org for python.

[paper echo]

https://www.python.org/dev/peps/pep-0647/ 👀 👀 👀

[acoustic crater]

look at me, I am the compiler now

[velvet torrent]

hi

can someone please help me ?

[grave jolt]

yep, it's accepted

I actually like that feature in TypeScript

helps with type narrowing in many places

[paper echo]

does this mean we get refinement types

[grave jolt]

basically, it's a bridge between the type land and the value land

[charred wagon]

I wonder if they'll retroactively apply that to existing type guards in the standard library

Like stuff in the inspect module

[grave jolt]

don't know what that is 👀

[paper echo]

@grave jolt an integer greater than 5, but a type

more generally, it's a type and a predicate that "refines" the type to only be valid for a subset of values

[grave jolt]

nah
well, if you make a user-defined type like PositiveInteger with NewType, you could make something similar

It's just a feature for control flow

[paper echo]

yep, but now imagine you can refine that type statically

rather than just indicating it with words and/or checking at runtime only

[velvet torrent]

hi mr salt rock lamp

can i ask you a question please

[paper echo]

if it's a general help question, i'd prefer if you refer to #❓｜how-to-get-help . otherwise i am happy to talk about my experiences as an ionic compound in an off-topic channel.

[velvet torrent]

can we speak in a off topic channel ?

[paper echo]

you can tag me there

[rotund lintel]

Has anyone here ever used Google Cloud Code https://cloud.google.com/code for Cloud Run knative? https://cloud.google.com/code/docs/shell/cloud-run-overview It's supposed to be like VS Code but for the web, and since I have to make a Flask Cloud Run thing I am about to take the plunge. But to be honest I've never even been an Eclipse user, let alone VS Code and I feel queasy. I'm mostly a vi and occasionally emacs person. Any words of soothing reassurance would be highly appreciated.

[blissful comet]

That'll be a very powerful tool when combined with dataclasses. Type checkers can be very good data validators too.

[paper echo]

yep! if i can offload more work to mypy and write smaller tests, im very happy

[rotund lintel]

False alarm, and because I can't delete those icon preview cards on the Android Discord interface, I'm reverting to gcloud in the non-cloudshell shell

https://cloud.google.com/run/docs/quickstarts/build-and-deploy/python

oh goodness it made another preview card out of an icon again. I swear I would delete all three if this Android Discord interface would let me. I need to start using less apps and more web

I expect to look back on all this and laugh in less than seven years

[cyan river]

pls beg

pls search

[rotund lintel]

please pray for me, in whatever athiest, agnostic, or theistic sense you pray.

[eager trail]

PRAY TO THE OMNISSIAH

Do you have time to talk about our Lord and savior, the Machine God? Join now and we'll lobotomize you and turn you into a servitor free of charge

[gritty sequoia]

LOL

[shrewd cipher]

Good day all! I have a data frame in Pandas that I would like to add a dynamic variable to every row of one of the columns. Please, could someone advise how this can be achieved?

[feral cedar]

check the #data-science-and-ml , those guys are good with pandas

also, this is not a help channel, but a discussion channel

[shrewd cipher]

thanks, just realised that, sorry

[tidal cave]

Does anyone knows what websites that we can use for practice coding for beginners?

[feral cedar]

codingbat and codewars

[tidal cave]

Thank you

[sour rose]

ı love skrt

[errant apex]

i just had my lifes biggest accomplishment

i posted a assembly question on stackoverflow and got 2 positive votes after a few days

[blissful comet]

I find pyright pretty good too and written by Eric, who also wrote this pep. Catches bugs in the IDE as you type code.

But syntactically what does a dataclass with a type guard look like?

[paper echo]

if i understand the pep, it's something like this:

from dataclasses import dataclass
from typing import Any, Optional, TypeGuard

@dataclass
class Thing:
    x: float
    y: float

BigThing = Thing

def is_big_thing(obj: Any) -> TypeGuard[BigThing]:
    return isinstance(obj, Thing) and obj.x + obj.y > 1.0

def do_stuff_big(obj: BigThing) -> float:
    return obj.x * obj.y

def do_stuff(obj: Thing) -> Optional[float]:
    if is_big_thing(obj):
        return do_stuff_big(obj)
    else:
        return None

so statically python doesn't know which branch the code will take, but it knows that if the type guard function returns True, then the type of the guarded binding must be a certain thing

the actual contents of the function are still only executed at runtime, but the TypeGuard[Foo] annotation gives the type checker more information to help refine the type statically

[gleaming rover]

oooh

type guards

[paper echo]

In all other respects, TypeGuard is a distinct type from bool. It is not a subtype of bool. Therefore, Callable[..., TypeGuard[int]] is not assignable to Callable[..., bool].
But, this is weird.

[gleaming rover]

syntax though sighs

[paper echo]

Really really weird.

will isinstance return bool or TypeGuard[_A]?

[fallen slate]

Hey @earnest mist!

Uh-oh! It looks like your message got zapped by our spam filter. We currently don't allow .txt attachments, so here are some tips to help you travel safely:

• If you attempted to send a message longer than 2000 characters, try shortening your message to fit within the character limit or use a pasting service (see below)

• If you tried to show someone your code, you can use codeblocks
(run !code-blocks in #bot-commands for more information) or use a pasting service like:

https://paste.pythondiscord.com

[paper echo]

https://mail.python.org/archives/list/[e-mail]/thread/NOLCFYLYAQQHXISNMPYCEOAZ7ZPFCGUW/#NOLCFYLYAQQHXISNMPYCEOAZ7ZPFCGUW

went nowehere, conversations apparently happened behind closed doors 😐

or perhaps not https://mail.python.org/archives/list/[e-mail]/thread/I3BPPNR53VK7X7B2HOI6VX3JLGA73DPK/#WOBIVLY22ROGITAQ2K5MIKQREDKBWIUZ

lots to read here

[blissful comet]

What if I want a type guard that says x < 10?

[paper echo]

i am hoping that they have their heads screwed on straight and that i will be allowed to pass a ... -> TypeGuard[_A] callable where a ... -> bool callable is expected

[paper echo]

class LessThan10(float):
    pass

def is_lt_10(x: float) -> TypeGuard[LessThan10]:
    return x < 10.0

special_value: Optional[LessThan10]

value = 100.0
if is_lt_10(value):
    special_value = value
else:
    special_value = None

[steady timber]

yo guys, i have a question, can i ask here?

[paper echo]

refer to #❓｜how-to-get-help

unless it's a question related to the topic of the channel

[steady timber]

its related to facebook bots

[paper echo]

that's going to be against our server rules entirely unless you're doing something with an official facebook API

!rules 5

[fallen slate]

Rules

5. Do not provide or request help on projects that may break laws, breach terms of services, be considered malicious or inappropriate. Do not help with ongoing exams. Do not provide or request solutions for graded assignments, although general guidance is okay.

[blissful comet]

I meant combining with dataclasses (your earlier example) with acceptable syntax. How do we apply this to Thing.x?

[steady timber]

oh ohk

[steady timber]

are facebook bots illegal?

[paper echo]

from dataclasses import dataclass
from typing import Any, Optional, TypeGuard

class LessThan10(float):
    pass

def is_LessThan10(x: float) -> TypeGuard[LessThan10]:
    return x < 10.0

@dataclass
class Thing:
    x: LessThan10
    y: float

i guess like this

[paper echo]

they might be, depending on your legal jurisdiction and what exactly you are doing with it. however they are blatantly against the facebook terms of use, and our rules forbid discussion of such things no matter the specific legal details wherever you live.

[blissful comet]

A bit more verbose than I like, but usable. Thanks!

Wonder why the pep didn't have this example

[steady timber]

Hey guys, I am new to development. Though I have been solving comptetive programming questions but i kinda wanted to create a Facebook bot.

The thing is i have to moderate a live session on Facebook in which people ask questions and i have to copy these questions and paste them onto Google docs so that the host can read the questions.

Can this be automated by a Facebook bot made in Python? If so, could you guys please provide me an outline to how should i start studying in order to create this bot.

[paper echo]

class Person(TypedDict):
    name: str
    age: int

def is_person(val: dict) -> "TypeGuard[Person]":
    try:
        return isinstance(val["name"], str) and isinstance(val["age"], int)
    except KeyError:
        return False

def print_age(val: dict):
    if is_person(val):
        print(f"Age: {val['age']}")
    else:
        print("Not a person!")

yeah this is a bit over-complicated, they're reinventing runtime-checkable Protocol with this example rather than showing something interesting/useful

[blissful comet]

Thing(20, 1) will raise error? How?

[steady timber]

Kindly tell me if this is legal

[paper echo]

i don't know, but it's off topic in this channel. i think you should see #❓｜how-to-get-help and i also think you message the mod mail bot if you need to clarify our rules.

it does seem like your usage is not malicious. but i don't want to accidentally tell you something is acceptable, and be incorrect.

[paper echo]

not at runtime unless you add some validation logic to __init__

ah that's a good question.. i don't think it will fail if you type it literally either

type checking the literal 20 i think would require mypy to actually run the type guard at static checking time

[blissful comet]

Exactly

[paper echo]

unless they added some special case for refinements on certain "base" data types, but you can specify arbitrarily complicated constraints at which point you need a full refinement type solver, which i think uses SMT solving

if is_LessThan10(val := 20.0):
    thing = Thing(x=val, y=0.0)

😆

[steady timber]

Oh ohk thanks

[halcyon trail]

I feel like the discussion about re types is what dependent typing is all about

You tend to only see it in relatively out there languages like Idris though

[grave jolt]

Greenspun's tenth rule -- Python has almost implemented a broken version of lisp in the type annotation syntax 🙂

[deep crow]

https://dpaste.org/s0Dq how do args work?

[halcyon trail]

that link is broken for me

[fallen slate]

Hey @deep crow!

Uh-oh! It looks like your message got zapped by our spam filter. We currently don't allow .txt attachments, so here are some tips to help you travel safely:

• If you attempted to send a message longer than 2000 characters, try shortening your message to fit within the character limit or use a pasting service (see below)

• If you tried to show someone your code, you can use codeblocks
(run !code-blocks in #bot-commands for more information) or use a pasting service like:

https://paste.pythondiscord.com

[deep crow]

https://bpa.st/FWHQ

[halcyon trail]

okay, so you have a function signature like this:

      def __init__(self,set_point, *actions):

[deep crow]

is set_point arg or kwargs?

[halcyon trail]

set_point is a a regular argument

[deep crow]

oh so there is a third thing

[halcyon trail]

not sure what you mean by that

[deep crow]

.

[halcyon trail]

--> 152 level_controller = Controller( PAction(prop_const=2500000),IAction(prop_const=0.5), DAction(prop_const=.8),set_point=0.4, )

[deep crow]

arg, kwarg and regullar argument

3 things

[halcyon trail]

there's *args, **kwargs, and other arguments, but even other arguments there are various categories

[deep crow]

where are the docs for this?

[halcyon trail]

have you looked on the official docs?

[deep crow]

what's the name of the page?I could only get some Realpython and geeks link on google search

[deep crow]

can you share the link?

[halcyon trail]

https://docs.python.org/3/tutorial/controlflow.html#more-on-defining-functions

most questions like this you have about python, the official documentation should be your first stop

also, I don't think any of this convo qualifies as "advanced-discussion" but I don't really know

[deep crow]

sorry I asked it on a help channel and no one replied and other discussions started

so I came here, maybe due to the broken link

so the syntax should be level_controller = Controller( 0.4,PAction(prop_const=2500000),IAction(prop_const=0.5), DAction(prop_const=.8) ) , right? But I want that 0.4 to be possibly labelled.It's not very clear what it means here if I just see that line

[halcyon trail]

you can label it, it just has to go first still

level_controller = Controller(set_point=0.4,PAction(prop_const=2500000),IAction(prop_const=0.5), DAction(prop_const=.8))

[deep crow]

This was my first try and it failed and that's why I tried putting it to the end

[halcyon trail]

why does that fail?

[deep crow]

                                                ^
SyntaxError: positional argument follows keyword argument



---------------Another Attempt failed-------------


AssertionError                            Traceback (most recent call last)

<ipython-input-26-2c90327cdd89> in <module>()
    150 
    151 #controllers
--> 152 level_controller = Controller(0.4, PAction(prop_const=2500000),IAction(prop_const=0.5), DAction(prop_const=.8) )
    153 pressure_controller = Controller(1000 * 1000,PAction(prop_const=20),IAction(prop_const=4), DAction(prop_const=8) )
    154 

<ipython-input-26-2c90327cdd89> in __init__(self, set_point, *actions)
     43 
     44     self.set_point = set_point
---> 45     assert actions is True, "Atleast one action is required"
     46     self.actions = action_types
     47     self.prop_constant = prop_constant

[halcyon trail]

hmm, indeed, wasn't expecting that one

FWIW, what I actually recommend you do is take an action_list

Controller(set_point=0.4, [PAction(prop_const=25000000), IAction(....), DAction(....)])

[deep crow]

                                                ^
SyntaxError: positional argument follows keyword argument```

oh thats a diff position it seems

[halcyon trail]

You would need to change the function definition for that

[deep crow]

I can obviously make it easy but I was considerning it as an oppurtunity to learn args

[halcyon trail]

if you want your arguments to always be labelled for example you can do:

def __init__(self, *, set_point, action_list):
    ....

[deep crow]

no, I don't, i remember this thing

[halcyon trail]

So, as you can see, the rules can be a bit tricky

?

[deep crow]

too much tricky and hard to remember

[halcyon trail]

ok....

I don't understand, first you say you want to learn as opposed to just doing it a simpler way, and then you say it's too tricky/hard to learn?

[deep crow]

hard is not a problem but it's too much confusing.

[halcyon trail]

okay

then just do the simplest thing

[deep crow]

yes did it

def (self, set_point, actions)

[halcyon trail]

def __init__(self, set_point, action_list):
    ...

yep

that's what most people would write anyway

the main purpose, fwiw, of *args and *kwargs, is mostly to transparently forward things to other functions without repeating all their arguments

rarely it is used to provide a more convenient syntax for passing a list, as you have done, but in most cases it's better to just take a list if that's what you mean

[deep crow]

yes, I agree

[halcyon trail]

basically if you don't use *args / **kwargs somewhere in the body of your function there's a good chance you shouldn't be using args/kwargs

I have personally violated that rule a couple of times, but usually for kwargs, simply because the foo=bar syntax can be so much nicer than {"foo": bar} sometimes 🤷‍♂️

but not often

[deep crow]

ya, the readability is the most important thing, I will try to find a better usecase

[halcyon trail]

for list it's a really small win anyway, just saves you []

[deep crow]

yep

thanks for the tips

also isn't assert len(actions)>0 , "Atleast one action is required" equivalent to assert actions is True , "Atleast one action is required" ? actions is the same list .It isn't due to some reason

[tropic fulcrum]

actions is True only works if actions is actually the bool value True. You could do assert bool(actions) in place of assert len(actions) > 0.

[deep crow]

oh yes, I forgot, == and is difference

[deep crow]

thanks for reminding

[tropic fulcrum]

Not quite. Even assert actions == True will fail. You are testing the "truthiness" of the list actions, which evals to True if the list is non-empty.

So you could also just do assert actions, but I think assert bool(actions) might make your intention clearer

[deep crow]

yes it failed, and list has Action objects still it was evaluatetd to be false

[deep crow]

yes I ll use bool but I still don't know why assert actions == True fails if actions has 3 objects of type Action

[tropic fulcrum]

Because actions is a list, not the bool value True.

[deep crow]

ok, I ll use bool()

[halcyon trail]

I actually kinda wish python lists had an empty method like most languages

[tropic fulcrum]

list.__bool__()

[halcyon trail]

That said I actually would not use an assertion for something like that

probably better to use an if and throw a proper exception

[flat gazelle]

I find == [] to be adequate for cases where you want to be explicit

[deep crow]

the difference between "is" and "==" was that one checks object comparison and another the truth value, I don't remember where I read this but this was the picture inside my head

[halcyon trail]

is checks object identity

== checks value equality

[deep crow]

yes the "hash"

[halcyon trail]

it's not really the hash, though there is some relationship

[tropic fulcrum]

bool() checks truth value, not the same as equality with True

[deep crow]

ok

[flat gazelle]

and yes, assertions only make sense when you can assume they hold.

[halcyon trail]

assertions are a bit murky in python tbh because it's not very common to actually elide them, unlike most languages

[deep crow]

in Math code I think these are usual

[halcyon trail]

math code?

[flat gazelle]

essentially, your code shouldn't break if the assertions stop being run

such as when -O is passed to the python interpreter

[deep crow]

ya, if you see some tensorflow code or numpy code asserts are very common

[halcyon trail]

yes, so those are in C, where eliding asserts is very common and standard in release builds

and also where performance is in general very critical

e.g. in C you will not bounds check arrays by default

so the notion of assertions is important

in python given all these differences it's much more common to simply be nice and definitively give the user an error

[deep crow]

even if you see tensorflow (python) codes on github , the people use lots of asserts

[flat gazelle]

honestly, I wouldn't use asserts in python except for tests. There are some valid cases for them, such as convincing the type checker, but you can generally do something else as well.

[halcyon trail]

assert frameworks pretty much always have their own asserts though

sorry, test frameworks i mean

so using the built in assert would still be discouraged

[flat gazelle]

pytest is the best test framework exactly because it uses the assert keyword

[halcyon trail]

But yeah I agree with the above

[deep crow]

don't worry about that, it's just a 1 page script, nothing like a big project

[halcyon trail]

Interesting

but the conceptual difference is basically this

[deep crow]

yes, in my big code on a kivy app, I don't use asserts at all, only in tests

[flat gazelle]

you can justify asserts for parameters that are generally passed as literals, but even then the cause is generally the programmer not wanting to think about what is an actually sensible exception to raise is.

honestly, in a quick script I wouldn't bother checking ahead of time at all and just let it error later once the actual reason you want the assert happens

[halcyon trail]

if you write "assert len(actions) != 0" what you are saying to your users is: the length of the actions list must not be zero, I'm going to take this as a precondition of my function. If you violate this precondition I promise nothing.
If you write " if len(actions) != 0: raise RuntimeError(...) " what you are saying to your users is: if the length of the actions list is zero, I'm going to throw RuntimeError.

but yeah, in a quick script I wouldn't bother much with these

[deep crow]

yes thats the point, just a simple jupyter notebook program

[acoustic crater]

also in a quick script you can just check falsiness of an empty sequence and imo if you wanna be clear something is a sequence then use type hints not verbose comparisons

[deep crow]

yes, I ll add docstrings and annotations

[acoustic crater]

I just wrote a battery of tests longer than the small module they test so I would say annotations might be overkill but I also might be a hypocrite

[static bluff]

Hello advanced python people! I'm hoping someone can give me some insight into a project I'm working on.

I'm wanting to implement some tweaks to Python's syntax and base functionality. I am wanting to implement:

• actually private members (functions/methods and data attributes)
• type enforcement through python's annotation syntax
• multiline anonymous functions via arrow notation
• minor syntax changes including a few additional operators

I want to do this by modifying source code on import using an import hook and an ast node transformer- both custom built of course. I want to do it this way because I can then distribute this new syntax as a normal python package with no additional dependancies.

I personally see it progressing something like this:

1. an import hook intercepts the import and, if a package is found to carry the appropriate custom file extension, compiles the module using the modified process
2. the source code is tokenized using a modified grammar which:
   a) recognizes arrow notation
   b) recognizes custom operators
3. the tokens are converted to an ast
4. a node transformer translates nodes in the ast to a naturalized python equivalent:
   a) class definitions are modified to implicitly include a cythonic base class which facilities privacy
   b) function definitions are replaced with modified function objects which facilitate some implementation of type enforcemnet
   c) custom operators are replaced with infix operators
5. the code is compiled to bytecode

A package called 'protected-class' already exists which facilities privacy, and I believe I can modify it to suite my purposes. By implicitly injecting the cython built class that supports privacy into the bases of any class definition, I think I can achieve privacy

Various packages for type enforcement exist. By replacing function definitions with calls to a custom built function-like class which validates arguments prior to executing the original code block, argument validation can be achieved.

I have not considered type enforced data members as of yet, but the infrastructure involved in the above will be a first step

The first two of the three above will rely heavily on a node transformers. type enforced data members as well as modified syntax will likely rely on both node transformation and custom grammar and tokenization.

What I really need to get started, I suppose, is an in depth tutorial on pythons grammar and how to modify it. Being able to tokenize the modified code into something that can then be dealt with with a node transformer strikes me as a first step

If I'm not mistaken, turning a stream of tokens into an ast is a matter of recognizing patterns in token types. That would be a second step

Thirdly, a node transformer would translate the source to it's python equivalent

And finally, the naturalized ast would be compiled as normal

Does anyone have any thoughts on the matter, or resources they could point me to

[acoustic crater]

I'd say do something simpler with tokens while you're still theorizing

and also I'd say you're looking for a solution where there isn't a problem, why not just use mypy, and assuming something is possible without a loooot more work, actually private enclosures

also multiline anonymous functions are already possible with exec and is adding more operators gonna solve a problem any more than it's just more cruft to learn?

like follow your heart but this seems overly ambitious for someone just figuring out how to get started lol

[static bluff]

I'm looking for runtime type checking. I'm pretty sure mypy is compile time

[viscid lintel]

how would you sum a large set of numbers? say 100mm sum over a genexp takes ages 😦

[acoustic crater]

type guards are being added too for runtime

[static bluff]

Oh cool. Well, mypy may be a component I'll use then

[acoustic crater]

I mean to the language itself

[static bluff]

And I'm going for power- not speed

I have my reasons, the greater project at large is rather complicated and I'm tired of explaining it

[acoustic crater]

maybe spend this time learning a different language because you're removing the features that make Python powerful

of course you're gonna do you this is just the feedback you asked for lol

[static bluff]

What makes python powerful is insanely effective inheritance and attribute lookup procedures, iterators, dunder methods corresponding to top level syntax, and high readability

[static bluff]

😄 yes and I appreciate it

But I don't think any of what I'm proposing detracts from python. It's just a slightly more rigid, law-imposing implementation of it

[acoustic crater]

it's a complete paradigm shift and languages with these features already exist

[static bluff]

😛 Like I said, I have my reasons. Not least of which is- I like the challenge

[acoustic crater]

and were designed for it instead of being monkey patched lol

[static bluff]

😄 😄 😄

Though I'm curious which languages you have in mind

[acoustic crater]

well start small remains my primary advice, just add one new operator or something

well if you want to parse code you should learn haskell because it's great for that

and this project boils down to parsing code

[static bluff]

Just fyi, I'm not a beginner. Been at it a while, and this project has been stuck in my brain like a splinter for years. Term just ended and I want to take a crack at it

As I said, step one is tokenization- which means a custom grammar

So I guess, what, learn what type of grammar and tokenizer python uses, read a crap tonne about them, familiarize myself with what is already used, and then take a stab at having it properly tokenize a simple test case?

[halcyon trail]

there's a page on the python docs that fully specifies python's grammer I believe

so i guess your very first step is probably to find some existing code, or write your own, that contructs an AST for vanilla python, correctly

once you have that, you would then modify it I suppose

[static bluff]

nice

[halcyon trail]

I actually have never heard of a language that enforces access control at runtime, btw

[tropic fulcrum]

For the import hooks part, there is a hacky example in the pyparsing examples directory that adds a "statemachine" construct, and hooks in as a ".pystate" file, that is Python with state machine definition added, and the import hook then converts to a Python class and adds as a program module.

https://github.com/pyparsing/pyparsing/tree/master/examples/statemachine

[static bluff]

Ooooooo

[tropic fulcrum]

Not sure if you want to use any of it, but might give you some ideas on what you can do.

[static bluff]

Parso has a custom tokenizer but it uses some incomprehensible regex

[halcyon trail]

yuk

regex for tokenization

[static bluff]

I've had a hard time wrapping my mind around it. I'd much prefer some sort of object oriented approach. But quicknir, your suggestion of building a tokenizer/ast builder for vanilla python is fantastic

[halcyon trail]

mandatory SO link

if you can't parse HTML with regex then you definitely cannot parse python with regex

[static bluff]

I mean, it works. But it's ugly-nasty

[halcyon trail]

i think it's literally impossible for it to work

[acoustic crater]

yeah I've tried

it's impossible unless you avoid edge cases

[static bluff]

https://github.com/davidhalter/parso/blob/master/parso/python/tokenize.py

Nasty jazz

[halcyon trail]

i think at a more basic level it's just impossible because regex, certainly, regular regex (not all regex is regular) can't keep nesting indefinitely

[static bluff]

It DOES support tokenizing without crashing on error though. I've tried modifying it in the past but its not the best

[halcyon trail]

python does

[acoustic crater]

i just tried to construct a regex that could detect valid semicolon locations and there was no way to make it work for several reasons especially brackets in strings

[halcyon trail]

you can keep nesting list comprehensions or function definitions infinitely

[static bluff]

Hey, I'll take you guys' word for it

[halcyon trail]

maybe if it's just tokenization and not parsing, it could work

hmm, I still think it can't, but perhaps there are certain caveats, maybe it uses some kidn of mix of strategies, not sure

[static bluff]

As for 'paradigm shift', I really don't think that a statically typed python flies too much in the face of python's core motivations. Yes, python does use duck typing but duck typing isn't exactly a main driver for python

You could remove it and the language would still work quite well

[halcyon trail]

out of curiosity what do you think the main driver is for the language at this point

other than everything related to network effects

[acoustic crater]

imo ease of use and readability, both of which come partially from duck typing

[raven ridge]

You mean if you redesigned the entire standard library and all of the primitive types?

[static bluff]

Everything is an object, unrivalled inheritance algorithms including metaclassing, highly effective attribute lookup systems relating to inheritance, dunder methods which correspond to top level syntax, iterators, an effective import system, high readability, duck typing (yes, I do like duck typing, for what its worth), and a host of other minor facets which make life easier (like *args, **kwargs)

It's a truly remarkable machine. I want to get this project done as much out of fascination as anything else. I like the challenge

Python's base nature is to make things easier. To do things the effective, elegant way. All I'm proposing is a smidge less flexibility, in favor of eplicitness

What really bugs me is that python went to great pains to expose it's import system, but not its compilation process. I've always felt python is a language designed to expose its own innards as much as possible, to allow people to bend the rules as they see fit

[acoustic crater]

you can go read the C

once you start digging in python you hit the C wall and the ways it's obfuscated pretty quick

[static bluff]

Lemme frame it like this I suppose- I can't be the only one whose found needs to make tweaks to python. My own intentions aside, a pure python exposure of the compilation process can only be a value-add

[acoustic crater]

then maybe you should focus on PyPy and RPython cuz that's kinda the point of those

[halcyon trail]

learning stuff is always a value add

that said, most companies, people doing python profesionally, etc, would not allow your custom forked version of python within a mile of their projects

[static bluff]

I know

[halcyon trail]

that's not to discourage doing whatever you feel like for educational reasons, you should indeed do whatever you feel

[acoustic crater]

idk if you can truly pure python cpython

without reimplementing hundreds of data structures and functions to run off themselves lol

[static bluff]

😄

[acoustic crater]

and you'd have to reimplement garbage collection and memory addresses but spoof them and the overhead would be enormous

[static bluff]

This project is one part experiment, one part portfolio piece, and one part actual project with a final result

[acoustic crater]

lol easy to talk big when you haven't even made an operator yet

gl

[static bluff]

😛 alright, debbie-downer

Thanks for the direction everyone

[radiant fulcrum]

PyPy tries it's best to

They do generally a very good job doing it but it's no small feat

Though structures i would say are much easier to replicate

it's the semantics that are probably the hardest

[acoustic crater]

yeah exactly and they use an rpython backend designed for the task rather than the cpython backend

[paper echo]

https://news.ycombinator.com/item?id=27043217 apropos to the discussion

yagizdegirmenci

Cinder: Instagram's performance oriented fork of CPython

Hacker News · May 4, 2021, 9:38 PM

[halcyon trail]

I feel like drop box tried this for ages with pyston

and then eventually gave up

And rewrote chunks of their software in, IIRC, Go and Rust

[blissful comet]

https://github.com/strawberry-graphql/strawberry-graphql-django/issues/20

is discussing refinement types and your example from earlier today.

[blissful comet]

All said and done, python's market share has increased. Go and Rust have a lot of catching up to do, although Rust gets a lot of love.

Facebook initially transpiled all of PHP into C++ and ran a giant single binary in production for some time:

https://en.wikipedia.org/wiki/HipHop_for_PHP

It was eventually replaced with a JIT based runtime.

Between rewriting all of your code in Rust/Go and gradual typing of existing code base with some dynamic features (which is the Cinder approach) sits py2many.

You give up on the dynamic features of the language, make your code base fully statically typed (modulo type inference) and avoid a rewrite.

[halcyon trail]

You're saying this like this is a production ready tool

"all said and done" - yes, because a lot of the market share that's around is smaller companies with few language related performance issues. It doesn't really relate directly to what I said.

[blissful comet]

Where did I say py2many is production ready? 😀

pytorch and tensorflow (ML workloads) and data scientists are the ones driving the python market share up. In all of these cases, there is an acceleration engine doing the real work. I saw a nvidia engineer on twitter claim (tongue-in-cheek) that C++ does all the heavy work and python gets the credit.

[halcyon trail]

You're saying that it sits between two different production ready options

[blissful comet]

Cinder claims its not production ready

[halcyon trail]

As opposed to "it will, when completed" or "it would sit" etc

not sure where it says that

but realistically, ok, even if it's not, I suspect it's closer than the 10 hello world samples 🙂

[blissful comet]

On their github: Is this supported? Short answer: no.

[halcyon trail]

Supported for external users

That doesn't indicate whether they are using it internally

[blissful comet]

It's a fair claim that Cinder and Pyston are much farther along than py2many. I'm not comparing production readiness. I'm comparing potential approach

[halcyon trail]

Most likely they are already using it internally to some degree, or will be shortly

[blissful comet]

They do say they're using it.

[halcyon trail]

I'm not sure offhand what the benefit is compiling to lots of different, high level languages

Nim compiles to C, which ok, at least gives you some benefits, and it's definitely less work up front then compiling to LLVM IR

and C has some status realistically as a universal language

But it seems like a large dispersion of effort to try to target multiple different languages. Especially when many of these languages are realistically about as easy as python to write.

[grave jolt]

about as easy as python to write.
except C

[blissful comet]

Rust has been the development focus. But each language has it's own strengths:

Kotlin - opens up the possibility of using python as a language to write code once and use it on both iOS and Android

Dart - opens up access to web app development

I'm aware of the argument that Dropbox makes about just writing the app twice. But there are lots of counter arguments to that.

Even if this approach is successful, some vendor controlled UI code will be written N times because of the nature of the mobile market.

C has some status realistically as a universal language

C also has a pretty bad reputation as a language to write security sensitive code in.

[halcyon trail]

that doesn't really matter, C ABI's are the still the standard in many worlds

it's just a practical thing

anyhow

[feral cedar]

why even kotlin though, it's basically just as easy to write in as python

[blissful comet]

Matter of opinion. Google/wuffs is a counter examples

[halcyon trail]

what's a matter of opinion?

[blissful comet]

python works everywhere. Kotlin native recently nixed linux as a supported platform.

[halcyon trail]

That C ABI's are the standard in many worlds? That's not a matter of opinion.

[feral cedar]

what about the jvm? 3 billion devices remember

[blissful comet]

The world has moved to mobile apps. No one is writing apps in C.

[blissful comet]

Not a fan

[halcyon trail]

lol what

"the world"

meanwhile millions of people are employed still writing straight up C

[blissful comet]

embedded apps for raspberry pi? or sensor devices? Most kernels are written in C as well.

[feral cedar]

let's move this to off topic, we're not talking about python anymore

[blissful comet]

But I'm talking user facing apps

[halcyon trail]

fair enough

[paper echo]

has it ever been proposed that logging.* functions default to the getLogger(__name__) logger instead of the root logger?

and that getLogger() with no arguments should default to getLogger(__name__) and not getLogger('')

if __name__ does not exist for whatever reason, create a random uuid

[raven ridge]

how would that work? Inspecting the call stack?

[paper echo]

idk, cpython runtime magic?

[raven ridge]

I'm not sure if it's ever been proposed, but it'd certainly never happen. That's a really big backwards compatibility break for very little gain.

[paper echo]

it doesn't have to be __name__, anything but the root logger

does anyone depend on that functionality?

[raven ridge]

yes, absolutely

[paper echo]

true, i sometimes get lazy and use getLogger() with no args to configure application-wide logging

logging.info though

[raven ridge]

if you decide to propose it, I wouldn't bother proposing a change for getLogger or the default behavior of logging.info - there's no chance that'll get accepted. You might be able to convince people that a logging.getModuleLogger would be a good idea, though.

[paper echo]

eh, that's a different proposal

mostly i'm frustrated with libraries that use logging.* internally instead of logging.getLogger(__name__).* or something else that would be more appropriate

[raven ridge]

so your goal is to break backwards compatibility, then

you want to take existing libraries that do one thing, and make them do something different in a future version of Python

that's the sort of change that has the highest bar.

it's really hard to get buy in on a change that could break applications.

[halcyon trail]

the worst thing about logging really is that we're stuck with % formatting

forever

logging is probably my favorite example of why "well just write a named function, why do you need nice lambdas" falls apart
If you have really nice lambdas, you can just utilize the lambda itself to make your logging evaluate lazily. Python could theretically have an API like logger.info(lambda: f"stuff") but it's ugly, so it doesn't happen.

Instead they went with making it lazy by utilizing % formatting, which a) isn't as lazy, and b) coupled logging to % formatting, and here we are 2 major overhauls of string formatting later, still stuck with %

[eager trail]

I've been wrestling with the way my place is doing logging and passing it to cloudwatch... Preach!

[gleaming rover]

I don’t get why

could you elaborate?

like AFAIK there is nothing wrong with passing the equivalent of a template string?

also it’s been a few weeks so I’ll just say the worst thing about logging is camelCase

[gleaming rover]

this is a really nice idea that I had not thought of actually

[raven ridge]

The worst thing about logging is that it's implemented in Python, and thus adds a lot of overhead that makes it impractical to add tons of logging statements that are disabled by default

I've used C++ logging frameworks where the overhead of a log statement that isn't executed is measured in the tens of nanoseconds, which means you can add logging statements into inner loops just in case they may be helpful in the future. In Python you might wind up spending half of your program's time in logging if you did the same.

[tropic fulcrum]

If you have log messages that are complicated or otherwise expensive to construct, you can conditionalize with if logger.isEnabledFor(logging.DEBUG). I rarely see this, but it can short-circuit some of the expensive log message construction.

[raven ridge]

yes, but even that is expensive.

[halcyon trail]

@gleaming rover what's wrong with it is that even though % formatting is doubly deprecated, we still need to remember it for logging

[raven ridge]

~>python -m timeit -s 'import logging; logger = logging.getLogger("foo.bar")' 'logger.isEnabledFor("DEBUG")'
100000 loops, best of 3: 2.19 usec per loop

100x slower than the C++ versions I've seen, give or take.

[gleaming rover]

no, I mean

[halcyon trail]

All of python is ballpark 100x slower so I'm having trouble understanding what makes this unique

[gleaming rover]

why are we stuck with it forever

as in, you mean like unless a deprecation + migration happen?

[halcyon trail]

Right

[gleaming rover]

is there no C library?

[halcyon trail]

Whereas if you used a lambda approach then logging would know nothing about the string formatting approach

And this issue wouldn't even exist

[raven ridge]

nope, it's implemented entirely in Python, and there's no C library behind it.

[gleaming rover]

ye I get this

[gleaming rover]

I mean, an alternative

I guess not?

[raven ridge]

ah, that I don't know. I've never actually gone looking outside the stdlib; I just don't log much in libraries.

[gleaming rover]

I wonder why

this seems like a pretty important thing

[halcyon trail]

Probably because in most cases it's not nearly a big enough deal, you can find logical places to do your logging so it doesn't hammer performance, or encourage users to enable it selectively

And logging is nice in many other ways

It's really flexible, hierarchical logging is a fantastic approach

[unkempt rock]

FWIW loguru plans on being implemented in C

[raven ridge]

they're not mutually exclusive; it's possible to get hierarchical logging with minimal performance overhead in the disabled case.

[halcyon trail]

Sure, never said they were

Just in practice it's not usually critical

Logging performance is easier to work around than the performance of the things you actually need to do

And python is slow at those too

[raven ridge]

that's the wrong way to look at it, I'd argue. If logging was free or nearly free, you could use it everywhere. If you could use it everywhere, it would be more valuable than it is now, where you're forced to consider where to use it.

[halcyon trail]

I can't say that logging costs have ever been a huge consideration for me in python

In C++ logging costs are a much bigger concern, even with an insanely optimized logger

[raven ridge]

Yes, it's true that CPU bound stuff is slower in Python than in other languages, but logging is disproportionately slower, even compared to other CPU bound stuff.

[gleaming rover]

how come

[charred wagon]

Not true. There is support for the .format syntax, but it has to be configured manually.

[raven ridge]

well, it's practically true.

There's support for the .format syntax, but if you enable it, you must ensure that that you don't use any libraries that expect to use % syntax - which is all of them.

[charred wagon]

Isn't it configurable on a per-logger basis

[raven ridge]

it is

[charred wagon]

So why would other libraries be a problem

[raven ridge]

you can't, in practice, configure the root logger to use anything other than % formatting

and most people only configure the root logger.

[charred wagon]

I wouldn't call that "stuck" then, even practically. But the interpretation of what that means is subjective.

[halcyon trail]

@charred wagon good to know, but there's still issues with it obviously

And even then, ok , great, now we're only one full string formatting overhaul behind

Instead of two

[charred wagon]

I don't see how it'd ever be possible to make it work with f-strings, if that is what you are getting at.

[halcyon trail]

I showed above

[charred wagon]

Because f-strings get evaluated unconditionally

[halcyon trail]

Wrap them in a lambda

[charred wagon]

Well, that's arguably not much better, as you admitted yourself.

[halcyon trail]

In python specifically, sure

Because python lambda syntax is ugly

[raven ridge]

So IIUC, the Logger passes the log record to a Handler, and the Handler passes it to a Formatter, and the Formatter handles the interpolation, but the user provides the format string. Which means that if a library uses % format strings then the application must ensure that any Logger used by that library is configured to use a Handler that uses a Formatter that uses % for interpolation, and the application must ensure that any Logger that is used by that library is configured to not propagate log records upwards to another Logger that uses a Handler that uses a Formatter that uses .format() interpolation.

[halcyon trail]

My point wasn't that logging is bad, given pythons features

My point is that the typical python arguments against lambdas don't have legs

[raven ridge]

Which does, in practice, make it an impossibility. It's possible with the caveat that you're essentially setting up two independent logging systems, rather than one, and log records can't move between the two, and you no longer benefit from the hierarchical logging structure that logging is designed for.

[halcyon trail]

Lambdas let you do things that just aren't nice any other way and that influences api design

[charred wagon]

It doesn't seem impractical to configure a root logger with % and then namespace your own lib's root along with a {} formatter.

[raven ridge]

Sure, as long as you have two separate log files

but then you might as well just be using two different logging libraries.

not being able to have one FileHandler handle some logging statements from modules that want to use % and some from modules that want to use .format() is a big limitation, in practice.

[charred wagon]

Okay, good point. I recalled the relationship was the other way around, where it goes through a formatter first and then to handlers.

[raven ridge]

I'm pretty sure I have the relationship right, based on https://docs.python.org/3/howto/logging.html#logging-flow

and what I recall from the last time I looked at whether this was useful 🙂

[charred wagon]

Probably, I was just going off vague memory

[gleaming rover]

lazy evaluation 😔

[normal helm]

Anyone know scikit learn?

[limpid plaza]

Hello, i have to ask you something ...
I woked up a 6h thi morning thinking about a thing ... Do u know if we could learn Code to a Neuronal system ... i developp.
I would like use Deep learning to learn to a neuronal systeme python code...
How could i apply that ? 😮

sry im a french student my english is not really good

[grand dock]

oh u like Python. ok, Name all command

[acoustic crater]

!e print([x for x,y in __import__('builtins').__dict__.items() if type(y) == type(print)])

[fallen slate]

@acoustic crater :white_check_mark: Your eval job has completed with return code 0.

['__build_class__', '__import__', 'abs', 'all', 'any', 'ascii', 'bin', 'breakpoint', 'callable', 'chr', 'compile', 'delattr', 'dir', 'divmod', 'eval', 'exec', 'format', 'getattr', 'globals', 'hasattr', 'hash', 'hex', 'id', 'input', 'isinstance', 'issubclass', 'iter', 'len', 'locals', 'max', 'min', 'next', 'oct', 'ord', 'pow', 'print', 'repr', 'round', 'setattr', 'sorted', 'sum', 'vars', 'open']

[static relic]

!e print([x for x,y in import('builtins').dict.items() if type(y) == type(print)])

[fallen slate]

@static relic :white_check_mark: Your eval job has completed with return code 0.

['__build_class__', '__import__', 'abs', 'all', 'any', 'ascii', 'bin', 'breakpoint', 'callable', 'chr', 'compile', 'delattr', 'dir', 'divmod', 'eval', 'exec', 'format', 'getattr', 'globals', 'hasattr', 'hash', 'hex', 'id', 'input', 'isinstance', 'issubclass', 'iter', 'len', 'locals', 'max', 'min', 'next', 'oct', 'ord', 'pow', 'print', 'repr', 'round', 'setattr', 'sorted', 'sum', 'vars', 'open']

Hey @static relic!

It looks like you tried to attach a Python file - please use a code-pasting service such as https://paste.pythondiscord.com

[grave jolt]

@static relic We don't allow advertisements on this server.

[static relic]

Ok sorry @grave jolt

https://paste.pythondiscord.com/elizixosow.py

[grave jolt]

@static relic If you want to showcase your project, you can upload it on https://paste.pydis.com or on GitHub and share a link in #python-discussion, off-topic channels or in an appropriate topic channel (e.g. #user-interfaces). This channel is for meta-discussions about the Python language itself.
If you have a question, you can check out #❓｜how-to-get-help and claim a help channel.

[static bluff]

Does anyone know of an implementation of PLY/SLY which successfully lexes/parses all of normal python?

Or short of that, an almanac of everything I'd have to cover to do it myself? There's just so much to Python that I'm certain I'd miss a few things if I tried myself

[halcyon trail]

did you look on the python grammar page of the docs

[paper socket]

Has anyone here worked on apache airflow?

Wanted to know if there is a way to pass data between different DAGs? Like using Xcom or some other built in functionality

[red solar]

i still can't overload is, right? (just asking in case new stuff has appeared, cuz i just saw descriptors and they look close but not quite)

[pliant tusk]

You can't overload is

[static bluff]

Yes, but a grammar is a language unto itself. i was hoping for something in Python just because its easier to read. Plus, my understanding is that a grammar is used to build the AST-parser, not the lexer

I've decided to go off PyPy's lexer for now, it's a simplified version of what's found in the built in tokenizer.py module, but it's still not prettiest code

[halcyon trail]

well, it fits the description of an "almanac" that's the part of your question I was answering

[quaint zephyr]

hello which youtuber should i watch to get started with python

[static bluff]

you are technically correct. The best kind of correct!

[halcyon trail]

going to guess this is the wrong channel for that question

since this is for advanced discussion of python...

[mystic cargo]

Hello @hollow hill , your question is off-topic for this channel. You can ask your question in one of the off-topic channels.

[spark magnet]

Sounds like you have an interesting project. What are you building?

[static bluff]

In a nutshell, a python based web browser

It uses wxPython to interface with the OS on an application level and also as the windowing system, and then embeds CEFPython in sed windows to run an html page and and a js runtime

Upon launching a page, BS4 is used to parse the html and create a virtual DOM in Python, and a series property like objects automatically send signals through CEFPython (a shared runtime between Python and Javascript) whenever an element's attribute is changed

Javascript then changes the virtual element's real Javascript counterpart

So what I'm trying to do is build a hybrid language of Javascript and Python that combines Python's superior pretty-much-everything with a couple of Javascript's better components, like arrow notated anonymous functions

That way the language is a bit more palatable to people coming in from a webdev background, and html/css/javascript tutorials, of which there are many, can be translated to the new engine on a one-to-one basis

Plus, for a number of reasons, I want to hide the internals of the engine as much as possible. Essentially hiding what's under the hood so that there is less room for one's own project to interfere with the gears of the engine

The engine itself is actually rather simple. I could have it up and running in a matter of weeks if I wanted to skip the bells and whistles. But term ended and what can I say, I like the challenge. I'd really like to give this custom implementation at least one solid try before i give it up as a bad job

[spark magnet]

wow, this sounds like a big project

[static bluff]

Well like I said, the wx/cef/virtual-dom is already done

Now I just need to be able to tokenize the source code of the new implementation, parse it to an ast, and convert that ast to one that python understands

I even found a fully functional compiler written in pure python that I could use to run the ast without even having to convert it to vanilla python. But yeah, those are three big and complicated tasks

[spark magnet]

it also sounds fun 🙂

[static bluff]

😄 😄 😄

[static bluff]

I'll reiterate though, if anyone knows any examples of a clear, readable, easy to understand tokenizer that can fully tokenize valid python source code- I'd really appreciate to see it

[runic musk]

@static bluff have you looked at https://github.com/dabeaz/sly ?

[static bluff]

Sly is absolutely a fantastic tool for this

But actually writing all the rules within sly is still a job to itself. A working example of python's rules written within sly would save on a lot work.

[runic musk]

That's true. Setting it up does look like it would take some time however, it looks like it would be incredibly maintainable once the initial setup is complete. Barring any drastic changes to Python in future versions, you probably wouldn't need to spend much time modifying it when Python does introduce or slightly change the syntax of some feature. 🙂

[static bluff]

Not to mention, if I actually could write all of python's rule in sly, that would save the next person a lot of time

[halcyon trail]

i was about to ask if it was related to yacc

and I see it is, quite closely 🙂

[static bluff]

So I've finally sat down and started in on coding this. The very first thing in the lexer I'm refactoring an encoding check

Does normal python support differing encodings? And for my purposes, should I care?

[sharp widget]

This sounds pretty funky

[static bluff]

I gotta say, this process is both terrifying and exhilerating

The way its shaping up is to be a lexer, parser, compiler, and interpreter for running python- from within python

And if I can pull this off, it would expose all the parts needed for anyone to add new syntax and whatnot without having to rewrite any C code, and in a way that plays nicely with normal python

That said, I have absolutely no idea what I'm doing 0.0

[deft pagoda]

should note that sly is LL(1) but python's parser isn't

though i guess it doesn't matter if you only need tokenizer

but this exists: https://docs.python.org/3/library/tokenize.html

[static bluff]

I'm using PyPy's tokenizer for now because its ever so slightly more readable than cPython's built in tokenize module- they're nearly identical though

[grave jolt]

@static bluff Have you tried lark for parsing?

It has a python grammar in its examples, but AFAIK it's not totally up to date

[static bluff]

Ooooooooooo

Holy hell, this is one advanced tool

Lark, that is

[swift imp]

So at work we submit jobs in batches to a computing cluster. We would like to profile the speed and memory while on the cluster. Does anyone know how we would go about doing that?

[pseudo cradle]

Is this DataBricks or some other environment?

Why is a module like PyLab considered bad practice? For those unfamiliar, Matlab is its own programming language that consists of mathematics and plotting tools. PyLab was created to offer someone a Python equivalent. PyLab is a convenience module that contains both mathematics (numpy) and matplotlib.pyplot (plotting). Are all convenience modules like this bad practice?

[swift imp]

No

[pseudo cradle]

Ah. At work we use databricks that has dashboards for that sort of thing.

[deft pagoda]

PyLab is bad practice because it heavily imports into the global namespace. Better to just import numpy and matplotlib separately.

[pseudo cradle]

Why is it better to break up modules than having a singular one? I don't see a functional difference between importing func_a from module1 and func_b from module 2 versus importing func_a and func_b from module 3

[halcyon trail]

if I understand it's not reallya bout whether the module is "broken up" or not

[deft pagoda]

if you import from pylab, do you know which load function you get? from numpy or matplotlib? there's a lot of overlapping names and it's confusing.

[halcyon trail]

the modules are still the same modules

it's just that it brings all the names into the top level

so you can more easily have collisions

it basically does the equivalent of a bunch of from foo import *

doing this in your own python code would be bad practice in general unless you were a library author doing it with your own submodules/packages and knew exactly what was happening

so it's still not great practice. that said it's convenient for interactive work.

[iron perch]

I just realized we store stuff into a class then just get it back again

What does a class do with those given data anyway?

[acoustic crater]

organizes it and allows using methods with it

can also mutate the data when constructed/initialized

[static bluff]

Hey all!

So I'm back with my tokenizer problem. I've scoured the internet for every tokenizer I could find and I'm not really seeing anything that speaks to me. I've looked at

tokenize.py (builtin)
sly/ply
lark
pythonparser

My problem is that none of these are extensible, with the exception sly somewhat. Sly however is giving me metaclass issues that won't mix well with the rest of the project. This project is designed to make it easy to inject new syntax and so I need to be doing this from an object oriented standpoint

And so, to quote Matt Damon- I'm left with only one option. I've got to science the shit out of this

I'm just going to build my own tokenizer from scratch. Now, this shouldn't be too hard, there are only a handful of different tokens and the process is essentially one part the detecting of enclosures (strings, lists, tuples, etc) which may or may not be nested or multiline, and also the recognition of tokens like keywords, identifiers, and operators

Python's own tokenize.py is only a few hundred lines of code.

But if there are any compiler gurus on this board who would be up to discuss the specifics of the process with me, simply for the sake of general knowledge, I could use the back and forth and the insight

Also 😦 Today is my birthday, and my cat ran away

[oblique crystal]

what kind of cluster/jobs?

it it has its own job scheduler they might offer some profiling out of the box

[swift imp]

We use drmaa and grid engine

[oblique crystal]

doesn't it offer sych thing ou t of the box? I didn't use it but the other I used at worst allow to see immediate RAM/CPU usage, and best case would also produce plots over time

[swift imp]

Yeah just don't know how to, and was hoping there was a using a python profiling library

[oblique crystal]

There is also DTrace

I think with this kind of things it's better to use tools that are come with scheduler. I don't you can just go and profile a job that is run by scheduler by some 3rd party python script unless it was created for the specific scheduler to work with or it was integrated into the actual code that is being ran

[odd ether]

@unkempt rock Don't dump copy-pastas in our channels

[static bluff]

Found the cat 😛

Maybe it'll be a good birthday afterall

Sorry for the off topic- just super jazzed

[red solar]

there is no way to get the pointer/address to the start of a contiguous memoryview in python? (without resorting to ctypes et al)

[undone hare]

id(my_memoryview) would work but that's CPython specific

[red solar]

oh yeah - but ok cool, just wanted to make sure i hadn't missed anything