internals-and-peps

[static bluff]

https://paste.pythondiscord.com/gakiqobixa.coffeescript

So, please bear in mind, I'm only just getting my bearings with this sort of programming

The long and the short of it is that its similar to an EBNF type grammar except it is bottoom up and non-recursive, so tokens and patterns can only reference patterns that have already been defined. TOKEN, CAPTURE, NONCAPTURE, and CONTEXT objects all correspond to groups within a regex. All of the groups are named, with the exception of non-capturing groups of course. Every line within a token or other group is an alternative, and so when compiled to actual regex they will be delimited by a pipe.

I'll add in some functionality to support conjunctive groups as opposed to alternative groups at some point

[grave jolt]

@static bluff Why not use an existing library like lark?

You can substitute it with your own later, when you stabilize the syntax

[static bluff]

Beyond that I think its fairly self explanatory. Only other thing worth mentioning, unless you guys are curious about my thoughts on actually implementing it, is that a matched token may execute some sort of callback

[static bluff]

Meh. If I were in school or else working for someone who wanted results fast, I'd go that route

But I'm enjoying swimming about in the theory, for now at least

Plus, lark looks heaaaaaaavy

Side note, assuming I did everything right, once converted from grammar form into its realized lexer form, it should be able to parse all of valid python (save for a few little things still missing)

[blissful comet]

PyErr_SyntaxLocationObject() - is there an equivalent mechanism accessible from python?

Context: I want to raise a TypeError from python code with lineno and location. Couldn't find any good examples.

I hate to write:

t = TypeError("foo", ("bar", 1, 3))
t.lineno = 1
t.col_offset = 3
raise t

[pliant tusk]

You could write a subclass of TypeError

Acc gimme a few I’ll write up a demo @blissful comet

@blissful comet do you want the fancy trace back that SyntaxErrors give? Or just an exception that automatically has those attributes

[spark magnet]

@blissful comet what line number are you trying to indicate?

[raven ridge]

How will it be different from the line number in the traceback attached to the exception?

[neon mesa]

speaking of trackebacks (somewhat), is it possible to make a function detect if a specific function is in the stack trace, and if not then raise an exception? (essentially to make sure it can only be called by specific functions)

https://docs.python.org/3/library/traceback.html ah there's a module for this 🙂 (is that what you were gonna say tritone?)

[unkempt rock]

Okay, I have weird metaprogramming question here involving decorators and objects.

I found a solution to a problem I was encountering with dynamically generating schemas with Marshmallow and SQLAlchemy https://stackoverflow.com/a/42892443

def add_schema(cls):
    class Schema(ma.ModelSchema):
        class Meta:
            model = cls
    cls.Schema = Schema
    return cls

@add_schema
class Entry(db.Model):
    ...

Is it possible to put a decorator on a Mixin, say, SchemaMixin, and then any class that inherits this new SchemaMixin will automatically run this add_schema decorator? Or maybe this is not a solution to use decorators with? Please not that I need to have this class Schema availiable to me at the class level, not merely an instance level.

[blissful comet]

I have a python -> rust transpiler that has type restrictions that normal python doesn't have. Like if you try to assign a u16 to a u8. I generate these type errors and want them to be nicely formatted by tools just like python's built in syntax error.

[blissful comet]

I already have the traceback. Just want the extra attributes. I have visitors like: visit_Functiondef(self, node). The node has all the information where the exception is raised. I want it passed to the TypeError object.

[pliant tusk]

then you can just make an Exception subclass that is passed that extra data

[blissful comet]

That's my plan. Was trying to manually copy stuff from node.lineno to exception.lineno. Guess there is no smarter way.

[raven ridge]

It could certainly be defeated, if you were to do this. For instance, they could just monkeypatch the traceback module to always include your function in the traceback. Or they can construct fake call stacks that contain your function in them.

[neon mesa]

it's more to prevent people from doing it accidentally

[red solar]

safety feature 🙂

[raven ridge]

"we're all consenting adults"

[red solar]

exactly lol

mostly because i was thinking of pickling atomic objects so i can pass it between processes

oh crap...

if processA does mp.Process(target=fn, args=(AtomicUint(width=4),)), can AtomicUint.__del__(self) be called before AtomicUint is created in the new process?

[raven ridge]

yes

[red solar]

goddamnit :/

[raven ridge]

AFAICS, Process doesn't save a reference to the args as an instance variable, and so the only reference to that object will be gone as soon as the expression ends

let me check that, though...

[red solar]

i'm too used to std::thread just starting automatically on construction :/

[raven ridge]

!e ```py
from multiprocessing import Process
class C:
def del(self):
print("destroyed")
p = Process(target=print, args=(C(),))
print("Ready to start process")

[fallen slate]

@raven ridge :white_check_mark: Your eval job has completed with return code 0.

001 | Ready to start process
002 | destroyed

[raven ridge]

hm, looks like it did save a reference there - but - that's not what I saw when I ran it locally...

[red solar]

so i probably shouldn't rely on it?

[raven ridge]

well, I certainly wouldn't. It doesn't appear to be part of the public contract of multiprocessing.Process that it saves a reference to the args anywhere

it certainly doesn't expose them to you

[red solar]

[raven ridge]

https://github.com/python/cpython/blob/3.9/Lib/multiprocessing/process.py#L125 - it looks like it does hold reference to them, but removes it in start

[fallen slate]

Lib/multiprocessing/process.py line 125

del self._target, self._args, self._kwargs```

[raven ridge]

so it's entirely possible that the reference is dropped after the child process has started, but before it has unpickled its arguments and created a new reference to whatever thing this is.

[plush isle]

Hi, sorry for the x-post, but I have a question regarding the use (or lack thereof) of @property decorators in the pathlib module's Path object over in #help-croissant
i'd appreciate it if someone could take a look, thanks!
#help-croissant message

[raven ridge]

https://github.com/python/cpython/blob/4d4be477055d0f9e415407bcd193788fae04dc11/Include/pyport.h#L100-L112

[fallen slate]

Include/pyport.h lines 100 to 112

/* Py_ssize_t is a signed integral type such that sizeof(Py_ssize_t) ==
 * sizeof(size_t).  C99 doesn't define such a thing directly (size_t is an
 * unsigned integral type).  See PEP 353 for details.
 */
#ifdef HAVE_PY_SSIZE_T

#elif HAVE_SSIZE_T
typedef ssize_t         Py_ssize_t;
#elif SIZEOF_VOID_P == SIZEOF_SIZE_T
typedef Py_intptr_t     Py_ssize_t;
#else
#   error "Python needs a typedef for Py_ssize_t in pyport.h."
#endif```

[snow kettle]

I think I'm coming to the same conclusions. I'm reconsidering building out my own functionality on top of dataclass because it's going to require an annoying amount of work to get coercion and types figured out.

I figured out that I can use __post_init__ as a mixin baseclass for my other dataclasses, but this feels like an ugly hack to achieve the same thing django and other libraries do.

I'm not sure if attrs is as cleanly tailored to my specific purposes as I'd like, but I'm also not a decades old project with specific usecases in mind.

[halcyon trail]

You're doing the dataclass into positional argument thing right

[snow kettle]

Well, not exactly

There's an extra layer involved to look up how to translate a keyword for a specific environment

[halcyon trail]

Sorry command line string

Sure

But it doesn't seem like you really need to get fancy with mixins or even post init

All you really need is a function that processes a dataclasses annotations; name, type and necessary metadata

And takes an enum to specify environment

[snow kettle]

post-init is so i can coerce a range of types for things like paths

it's an ugly hack

[halcyon trail]

Not sure what you mean by a range of types

[snow kettle]

Union[str, Path] doesn't work in type annotations

[halcyon trail]

Why not

[snow kettle]

at least not in this context when validating

i get subscripting errors

though I'm also looking at this at the end of the day

[halcyon trail]

Not sure I understand what you mean

[snow kettle]

I'm on 3.7

[halcyon trail]

Nor how would post init help

[snow kettle]

To give some more context, I started trying to build a general type specification and validation system on top of data classes rather than just an argument translator.

[halcyon trail]

Uh

Okay, that seems like you expanded the scope of your work by about 10x

[snow kettle]

def as_command_line(
        params: ParamBank,
        env: Env,
        target_format_version: int = None,
) -> List[str]:

this is why

or stuff like it

[halcyon trail]

What do you mean by "type specification system", dataclasses already have one

[snow kettle]

as far as I know dataclasses don't have type validation built into them

that has to be done by code you add or a static type checker

[halcyon trail]

Okay, that's not specification but I understand now

[snow kettle]

I think it isn't strictly necessary if I actually needed to get this done fast

but this is a toy project for learning so I'm open to taking strange detours for practice

[halcyon trail]

I don't think post init is really the way to go for that unless you are doing frozen=True

Because it doesn't guard against the value being changed after init

[snow kettle]

True

[halcyon trail]

I'd look at validators in attra

Attrs

[snow kettle]

that's why I initially asked about validators as well

[halcyon trail]

And yeah, you mentioned.validatoes.too, attrs already covers this pretty well

I will tell you that trying to ensure that static and dynamic types match can be a bit of a fools errand in python

[snow kettle]

I think now that I try to explain this, validation seems like it might actually belong in the final formatter

[halcyon trail]

The static and dynamic types systems are kind of different

[snow kettle]

yes, that's what I'm thinking in this case

I started learning programming in statically typed environments and that influence is still hard to shake off sometimes

[halcyon trail]

Anddoing the verification is really painful once you start dealing with generics

[snow kettle]

though I think it's also paranoia about users in this case

[snow kettle]

yes, this is what I ran into

[halcyon trail]

Yeah

[snow kettle]

I was trying to add a coerce field to the metadata of fields

[halcyon trail]

So I had to do a lot of this stuff in my dataclass to-from json stuff

Which works recursively and handles List and Dict members

[snow kettle]

that would attempt to pass whatever was set through __init__ it

and then set the value to the result

the approach is ugly yet might even be turned into something useful if a lot more time was put into it, but I don't think I understand the internals of python well enough to make that a reasonable amount of time.

[snow kettle]

did you publish a library for that btw?

i might have use for something like it eventually

not here, in other things

[halcyon trail]

No

Some of the code is kinda hacky but if you ping me during the day I can just send it to you

[worldly star]

fuck

lol

i can span

lol

lol

omg no slowmode

but no gonna spam

[snow kettle]

ah, i'll keep that in mind. ty. Also I think I'm sold on attrs from this alone https://www.attrs.org/en/stable/examples.html#defaults

>>> @attr.s
... class ConnectionPool(object):
...     db_string = attr.ib()
...     pool = attr.ib(default=attr.Factory(collections.deque))

[raven ridge]

Spam or not, please stick to the channel topic. This channel is about the Python language itself, its various implementations, PEPs, and so on.

[snow kettle]

@halcyon trail Thank you again for showing me attrs. It does exactly what I need, and I regret not reading the documentation for it more thoroughly the first time 😅:

>>> @attr.s
... class C(object):
...     x = attr.ib(converter=int)
>>> o = C("1")
>>> o.x
1

This is the exact feature I wanted most

[green plaza]

How you can do this bubbles thing?

[unkempt rock]

HI it is possible to have a program that sends messages on discord all my minutes, because some bot allow to earn money depending on the number of messages you send

[sacred yew]

!rule 5

[fallen slate]

Rules

5. Do not provide or request help on projects that may break laws, breach terms of services, or are malicious or inappropriate.

[sacred yew]

thats a selfbot

[signal tide]

quick question, I've somehow just discovered that the default __eq__ returns self == other, but my understanding of self was that it represented the class instance. How does it differentiate that it's comparing an attribute to a value rather than the instance?

[peak spoke]

the default eq uses identity for the comparison

[signal tide]

can you elaborate?

[peak spoke]

It'll be the same as doing self is other or id(self) == id(other)

[undone hare]

Well, that's a CPython impl detail, isn't it

[peak spoke]

No

!d object.__eq__

[fallen slate]

object.\_\_eq\_\_

object.__eq__(self, other)``````py

object.__ne__(self, other)``````py

object.__gt__(self, other)```
These are the so-called “rich comparison” methods. The correspondence between operator symbols and method names is as follows: `x<y` calls `x.__lt__(y)`, `x<=y` calls `x.__le__(y)`, `x==y` calls `x.__eq__(y)`, `x!=y` calls `x.__ne__(y)`, `x>y` calls `x.__gt__(y)`, and `x>=y` calls `x.__ge__(y)`.

A rich comparison method may return the singleton `NotImplemented` if it does not implement the operation for a given pair of arguments. By convention, `False` and `True` are returned for a successful comparison. However, these methods can return any value, so if the comparison operator is used in a Boolean context (e.g., in the condition of an `if` statement), Python will call [`bool()`](https://docs.python.org/3/library/functions.html#bool "bool") on the value to determine if the result is true or false.

[peak spoke]

The paragraph that got cut off specifies that it uses is and returns NotImplemented if the is comparison was False

[signal tide]

huh I thought for some reason it was used when comparing attributes as well as the object

class MoveStatus(Enum):
    VALID_MOVE = auto()

    INVALID_MOVE = auto()
    PUTS_KING_IN_CHECK = auto()

def move() -> MoveStatus:
    ...

def make_move():
    if move():  # if move is valid
      ...
    else:
      ...  # relay errors```
is there a way to get this sort of behavior? (my first thought was to use `__getattribute__` but I need the enums intact and not just turned into bools)

[edgy parcel]

I'm not clear what you're trying to do next -- are you thinking like
if move() is MoveStatus.VALID_MOVE:

[signal tide]

ya that'd work thanks, I think I need to go to bed im definitely over complicating this

[eager trail]

I've been given carte blanche to write a style guide for my company's lambda functions library and I'm trying to think of more peps to include in here

So far I've got 8 and 257, but if anyone has more suggestions im all ears

[paper echo]

lambda functions library?

[gleaming rover]

like, AWS Lambda?

[edgy parcel]

I'm picturing a .py that's just like a thousand lines of the company's favorite one-line functions

[unkempt rock]

dont companies discourage use of lambda functions..

[halcyon trail]

I don't think it's a "company" thing so much as a "python" thing

[static bluff]

Isn't a 'library' of lambdas kinda redundant?

[grave jolt]

so that you can conveniently copy-paste them

[static bluff]

Ahhh

lambda x: x

XD

[static bluff]

I'd absolutely want it hosted online so that you can just head to the page and click one, and it'll copy the thing into your clipboard

Sorting by category, or name would be nice too 🙂

[undone hare]

Hey, identity functions can be useful

[eager trail]

Not for the size of my org

Ah sorry guys, yeah aws lambda

Not lambda lambda

I forgot about lambda as in anonymous functions

I just need to quote peps to back up my initiatives

[static bluff]

XD Hehe, I know I know

[static bluff]

Bamboozled again!

[edgy parcel]

this reminds me of a place I once heard of called stack overflow dot com

[edgy parcel]

Those are certainly the big two PEPs for style

[static bluff]

Does it have a repository of functions? To be clear, I was thinking of the functionality where you can click on a bit of text and it'll copy to the clipboard. Saved you from having to select it manually

[eager trail]

Blindly copying code without documenting it caused the current state of affairs 🙈🙈🙈

[edgy parcel]

it does not

[halcyon trail]

tbh, I think that in a python place of discussion, "lambda" is more likely to be referring to AWS lambdas then anonymous function expressions 😛

[edgy parcel]

party poopers

[static bluff]

What is AWS?

[rich cradle]

Amazon Web Services, just Amazon providing various services for devs

[static bluff]

Rockin

Anyone working on anything fascinating today?

[eager trail]

I suppose I am, since I'm refactoring about 600+ functions over the next few weeks

[paper echo]

write tests

[eager trail]

Yeah. Wondering if it's worth also implementing gherkins

Probably a bit too much, but yeah I'm building up standard mock tests

Once I'm done with the 600 I should be able to implement some sort of way to auto check code before it gets uploaded on jenkins

[paper echo]

try to avoid mocking when possible tbh

consider https://hypothesis.works

[eager trail]

Oooh

Thanks

[jagged knot]

Any server related to LINUX??

[edgy parcel]

#unix is the channel here related to Linux

[halcyon trail]

I only really mock for things that shouldn't be dependencies of unit testing

like you typically don't want your unit tests to need to connect to sockets, databases, etc

Sometimes I'll mock because the alternative might be dragging in a really huge amount of code to the test, but that's definitely the exception

[uncut sierra]

Matrix multiplication is this PEP in Python https://legacy.python.org/dev/peps/pep-0465/

[paper echo]

@uncut sierra yes https://www.python.org/dev/peps/pep-0465/, the @ operator and its associated methods __matmul__ and __rmatmul__ are in python now

[modern bough]

How are they formatted in a pyc file?

[halcyon trail]

anyone with thoughts on pydantic vs attrs?

[charred wagon]

The main point of pydantic is to validate data. attrs does not do this.

[halcyon trail]

attrs supports validators

but it does not validate certain things by default, which pydantic does (in particular, the basic validation that something is an instance of the type)

[uncut sierra]

It doesnt seem to work, do you have an example ?

[paper echo]

!e ```python
import numpy as np
A = np.array([[1,2],[3,4]])
x = np.array([1, 10])
print( A @ x )

[fallen slate]

@paper echo :white_check_mark: Your eval job has completed with return code 0.

[21 43]

[uncut sierra]

I thought it was usable witout any package ...

[[1,2],[3,4]] @ [1, 10]

!e [[1,2],[3,4]] @ [1, 10]

[fallen slate]

@uncut sierra :x: Your eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 1, in <module>
003 | TypeError: unsupported operand type(s) for @: 'list' and 'list'

[paper echo]

!e ```python
class Thing:
def init(self, value):
self.value = value

def __str__(self):
    return f'Thing: {self.value}'

def __matmul__(self, other):
    return Thing(self.value + other.value)

thing1 = Thing(5)
thing2 = Thing(10)
print( thing1 @ thing2 )

[fallen slate]

@paper echo :white_check_mark: Your eval job has completed with return code 0.

Thing: 15

[paper echo]

the @ operator and __matmul__/__rmatmul__ methods work anywhere, but lists don't implement them

[uncut sierra]

to bad 😦

[paper echo]

i'm OK with it

[uncut sierra]

well for me this make sense by itself [[1,2],[3,4]] @ [1, 10]

maybe is we subclass userlist

[paper echo]

what would [["a", (datetime.now(),)], ["c", None]] @ [requests.get('https://example.net')] return

[uncut sierra]

but thanks anyways 🙂

e! "a" * "c

e! "a" * "c"

[modern bough]

How is python bytecode actually formatted in a pyc file?

[uncut sierra]

it will surrely raise some exception flag 🙂

[modern bough]

I'm trying to parse one w/o using python

[uncut sierra]

maybe .pyc is just all bytecode codes

[modern bough]

There's like 0 documentation around it and ceval.h is really complex

[true ridge]

It is an internal format described in https://docs.python.org/3/library/marshal.html

also we have the serializer/deserializer in Python/marshal.c

just fyi, there is also the header part of pycs. this is for parsing the data part

[modern bough]

Right

Is there anyway to do it without the marshal library?

[uncut sierra]

is the soure of the marshal lib available ^

?

[true ridge]

Python/marshal.c

though it is rather complicated but IIRC there are some python ports of the marshal code

[modern bough]

I thought it would be simpler lol

[uncut sierra]

so there you go "pip" just build a wrapper aroud it

[modern bough]

Isn't bytecode normally just a bunch of bytes, each representing an opcode?

[true ridge]

eh, the co_code contains 2 bytes per instruction which is the simples of all

but the arguments of instructions might refer to actual python objects

such as tuples, strings, integers

which needs to be carefully serialized and then deserialized back to their original object form when the interpreter starts up

[uncut sierra]

all structure need to be maintain also, class definition with class variable as an example ...

[modern bough]

Ok, thanks.

All of that is done by marshal.c?

[prime estuary]

Yes. Note that the marshal format is intentionally undocumented other than in the code, since it's allowed to change in any version.

[acoustic crater]

do Exceptions in all implementations of Python have .args?

and is it always set in __new__?

[peak spoke]

They should, BaseException specifies the attribute. Where it's set shouldn't matter but I don't see where else it would be set except for new/init

[acoustic crater]

kk

[modern bough]

My hacky solution is to serialized dis.Bytecode to json for now.

[static bluff]

So I've got two things on my mind I wanna ask you about

First, is there a word for something that both lexes and parses?

Second- a few weeks ago I mentioned something about wishing I had access to my class within its own definition. A discussion was had about why python doesn't do that, one of the reasons being it could cause references to the class-object to be stored outside the definition, which would later become time-bombs in the event an error occurred and the class failed to be built

What I wanted to ask you guys is (and I'm genuinely curious); what would the repercussions be from having a 'failedClass' object or an 'uninstantiatedClass' object possibly be encountered, if the above scenario occurred?

It would be inert- no functionality except for those of the base object, and a truthiness value of false. Thoughts?

[halcyon trail]

i mean that would typically just be called the parser, as well

when people say, for example, "I want to parse C++", they generally mean in a typical context, I want to build an AST from C++ source code, so they are talking about the overall text -> AST transformation

[acoustic crater]

if you're accessing a class within its own definition then that object is just always going to be False

[halcyon trail]

When people are already discussing a lexer, and they discuss a parser and a lexer in close proximity to one another, then often by parser they just mean the part that takes the tokenized-lexed input stream and transforms it into the AST

[acoustic crater]

you can define a mutable container within your class's definition and put the class in there after definition, but why do you even need to do dynamic stuff in a class definition?

[halcyon trail]

that's generally what I've heard in practice and even the wikipedia article on parsing seems to say this as well

[acoustic crater]

hell I ebt you can implement what you want already, gimme a sec

[halcyon trail]

Re the python, it's probably just extra work to support that, isn't it?

the natural way to code all that is to build up the class, and then when the class definition ends, you bind the string of the class name, to the class itself

if you do it that way, then if there's a failure, the name is unbound, which is what you want

So I think the basic answer is, they're not going to make it work that way, which adds both language complexity, and implementation complexity, unless there is a good reason

And I can't really imagine a good reason

[static bluff]

Me neither really. I'm a few steps in to a long journey understanding languages are an art and a science— and I really think I've found my calling

So I'm asking questions, learning, exploring

As for the class question in particular— I can't remember how I got on to the topic originally but after the discussion had been had, I dunno, it just keeps bubbling up to the top of my brain

So I've been mapping the problem out, not that it really is a problem, just trying to see if from different angles. I was just curious you guys thoughts

[velvet sequoia]

Can I have someone look help-lemon

[acoustic crater]

!e

class UninstantiatedClass:
    def __init__(self, class_name, globals=globals()):
        self.class_name = class_name
        self.globals = globals

    @property
    def get(self):
        try:
            cur_class = self.globals[self.class_name]
            assert isinstance(cur_class, type)
            return cur_class
        except (KeyError, AssertionError):
            return False


class test:
    self = UninstantiatedClass('test')
    print(self.get)

print(test.self.get)

@static bluff
Here, you need to call something though to get different behavior when you access it unless you define the object's behavior in C

[fallen slate]

@acoustic crater :white_check_mark: Your eval job has completed with return code 0.

001 | False
002 | <class '__main__.test'>

[acoustic crater]

but in general it's weird enough to have a getter return falsy cuz something is uninstantiated so making an object that returns differently depending on another object's state without dotted access is pretty gross

[static bluff]

I'm totally saving this pattern for later

[acoustic crater]

lol enjoy

[static bluff]

I love abusing Python- this'll be fun to play with

squirrel, hiding nuts

[acoustic crater]

#esoteric-python is the channel for this sorta stuff really

some of the denizens there might be able to figure out a way to make it work without calling anything but it'll probably be CPython only

[compact thunder]

who understand alot about dis

[paper echo]

@acoustic crater wouldnt this fail anyway if the class isn't defined at the top-level?

maybe you'd have to start inspecting frames...

not sure how all that works in python

[acoustic crater]

that's why globals is a parameter

tho I guess it should be a callable

[paper echo]

i used to know how it all worked in R because library developers did a nontrivial amount of frame inspecting to make nice tidy looking APIs

but the R language supports and condones such evil things

[acoustic crater]

meh it works as a dict

just pass in the dict of whatever namespace you wanna use

[paper echo]

!e ```python
def f():
x = 1
class A:
x = 2
print(locals())
return A
f()

[fallen slate]

@paper echo :white_check_mark: Your eval job has completed with return code 0.

{'__module__': '__main__', '__qualname__': 'f.<locals>.A', 'x': 2}

[acoustic crater]

you'd need to use f.__dict__ as globals

[paper echo]

is there a reliable way to look "one frame up" from the current one?

[acoustic crater]

or maybe __closure__ idk what you're tryign to achieve

[paper echo]

think in R it'd just be parent.frame

[acoustic crater]

yeah inspect.currentframe().f_back

[paper echo]

where are "frame objects" documented? anywhere?

[acoustic crater]

so yeah could do self.globals = inspect.currentframe().f_back.f_back.f_globals probably

https://docs.python.org/3/library/inspect.html

[paper echo]

yeah i see it mentioned in https://docs.python.org/3/library/inspect.html#inspect.currentframe but nothing about the returned objects

[acoustic crater]

!e

import inspect

class UninstantiatedClass:
    def __init__(self):
        self.class_name = inspect.currentframe().f_back.f_code.co_name
        self.frame = inspect.currentframe().f_back.f_back.f_locals

    @property
    def get(self):
        try:
            cur_class = self.frame[self.class_name]
            assert isinstance(cur_class, type)
            return cur_class
        except (KeyError, AssertionError):
            return False


class test:
    class test2:
        self = UninstantiatedClass()
        print(self.get)

print(test.test2.self.get)

edit: need to use locals not globals doh

[fallen slate]

@acoustic crater :white_check_mark: Your eval job has completed with return code 0.

001 | False
002 | <class '__main__.test'>

[paper echo]

https://docs.python.org/3/library/inspect.html#types-and-members ah, in this table

[static bluff]

https://docs.python.org/3/library/inspect.html

The basic frame object is documented here. But there are actually three frame types- the actual base frame type that exists on the stack, found in types I think, the frame type created when inspecting the stack, in inspect, and another frametype created when inspecting stacktraces/traceback objects, found in traceback

Python really shit the bed in terms of stack inspection. If it were me, I'd have a stacktrace class that functioned as a list and contained instances of the one and only frame type- then I'd have the base exception object inherit from stack object

I mean, pretty much the only attribute on an exception is its __traceback__ attribute, which is a Traceback object- itself completely useless unless you feed it to traceback.extract_tb

Why they even bothered making it a dunder I don't know, because the whole point of a traceback object is to be a container for frames. To make matters worse- traceback objects have a reference to the 'tb_next' attribute, which lets you climb down the stack- but you can't climb back up. They also have access to their tb_frame attribute, the actual next frame on the stack. That frame can access it's f_back attribute and climb back up the stack, but it does NOT contain a referrence to its traceback object or let you climb down

Shit, meet show

[tough owl]

Hey people,
This is my first post ever, I was hesitating between this channel and #help-* channels. Finally, I chose to post it here since it's a bit advanced. 😊
Is there someone here who knows a tool capable of analyzing the Python files of a project and gets the file dependencies graph? — This is to better understand the different imports and potential areas of improvement (moving some functions/classes/variables from one file/module to another).
Here's an example of what the tool can generate:

I already found a tool called snakefood. (https://github.com/blais/snakefood) The problem though is that it's not compatible with the recent versions of Python 3: It has some syntax errors and is using a library called compiler which is not supported in Python 3. (https://stackoverflow.com/questions/909092/why-is-the-compiler-package-discontinued-in-python-3)

[static bluff]

Does anyone have a hard and fast rule as to when python expects an indent?

Best I can figure, a newline indented by at least one is expected after a color (specifically, after some sort of def-statement such as a funcDef or classDef)

EDIT: Also following if, for, and while loops as well as with statements

[acoustic crater]

always after a newline after a : which isn't a lambda or type hint or in a slice

u can't nest them but you don't need an indent after any : just when it's following a newline ie def test():return None is valid

[static bluff]

So, parsing wise, any time one of these situations (of which there are multiple, scattered about) occurs, a callback could fire which peeks to see if the next token is a newline (or comment), and if so, then check to ensure the proper amount of indentation follows that

[fallen slate]

Hey @static bluff!

Uh-oh! It looks like your message got zapped by our spam filter. We currently don't allow .txt attachments, so here are some tips to help you travel safely:

• If you attempted to send a message longer than 2000 characters, try shortening your message to fit within the character limit or use a pasting service (see below)

• If you tried to show someone your code, you can use codeblocks
(run !code-blocks in #bot-commands for more information) or use a pasting service like:

https://paste.pythondiscord.com

[static bluff]

https://paste.pythondiscord.com/xikojukube.php

It's SUPPOSED to define itself. Lets see how it parses 😄 😄 😄

[static bluff]

I'd like to extend it with GROUP objects that can be named or noncapturing, such that one could built complex, grouped/hierarchical regular expressions. Right now it's just a flat set of alternatives.

Someone very helpfully pointed out that for the purpose of tokenization with regular expressions, you don't actually need any groups so long as you apply longest first and such other rules properly

But it'd be a nice feature 🙂

[rugged harbor]

hello

anyone used https://schedule.readthedocs.io/ to make a list of jobs?

[static bluff]

!paste

[fallen slate]

Pasting large amounts of code

If your code is too long to fit in a codeblock in discord, you can paste your code here:
https://paste.pydis.com/

After pasting your code, save it by clicking the floppy disk icon in the top right, or by typing ctrl + S. After doing that, the URL should change. Copy the URL and post it here so others can see it.

[static bluff]

https://paste.pythondiscord.com/josunoxoro.py

Yeeeeeeeeeeah baby

(Still lots of work to do, but that'll be for tomorrow)

[static bluff]

Anyone working on anything fun?

[undone hare]

Working on #846514617261621292 haha

[brave badger]

Also working on #846514617261621292, also discovering attrs and writing good tests

[safe bane]

I’m working on a pipeline to aggregate articles from a bunch of RSS feeds and then tag them with the topics they’re about; I think it is correct to describe it as a classification pipeline

Having fun using SQLAlchemy, Dagster, Pandas, SKLearn, and SpaCy 🙂

[static bluff]

Cooool!

[safe bane]

Oh I’ve used this before! Ended up not entirely fitting my use case but it’s a very nice straightforward API. What kind of jobs are you trying to schedule?

[unkempt rock]

true

[rapid tundra]

Hey does anybody know how I can use python in VS code? Because I'm trying a paid course and he told me what plugins to install but it still wouldn't update to the normal python

[raven ridge]

That's off topic in this channel, but would fit in #editors-ides

[paper echo]

how does .read work on a file-like object? does it read chunks at a time until EOF?

i assume there's some platform-dependent buffer size

[raven ridge]

that's up to the individual file-like

but for a regular file, yes

[paper echo]

what about stdin?

i assume it's the same or similar

how is the buffer size determined? something hand-tuned for common architectures?

e.g. i see people recommend things like "when using dd set the block size to 4mb for $reasons"

no idea where those numbers come from

[raven ridge]

eh - you want it to be some multiple of the system's page size, which is normally 4kib or possibly 16kib

other than that, it's just a tradeoff between number of syscalls vs amount of memory required

in the case of open(), you can set the buffer size yourself, though I'm not sure off the top of my head what the default is - 4096 bytes would be my guess... Unless it's attached to a terminal, in which case it's line buffered instead

[paper echo]

i see

that's helpful, thanks

[raven ridge]

https://docs.python.org/3/library/io.html#io.DEFAULT_BUFFER_SIZE

[paper echo]

i actually ask because i'm writing something in a non-python language that doesn't have a "read everything at once" feature

[raven ridge]

!e print(__import__("io").DEFAULT_BUFFER_SIZE)

[fallen slate]

@raven ridge :white_check_mark: Your eval job has completed with return code 0.

8192

[raven ridge]

(exactly) 2x my guess 🙂

[paper echo]

When interactive, the stdout stream is line-buffered. Otherwise, it is block-buffered like regular text files.
https://docs.python.org/3/library/sys.html#sys.stdin

aha

oops thats stdout

[raven ridge]

ah, true - forget I mentioned line buffering

that should only be relevant for output streams, not input

that's about when a flush happens.

[paper echo]

so is there no input buffering? is that up to the operating system, and/or program pushing data into stdin?

[raven ridge]

right, the program pushing the data decides when to flush

[paper echo]

just fgetc() in a tight loop?

[raven ridge]

it may be doing line buffering, in which case it only flushes if the stuff it was asked to write contains a "\n"

sure, you can, because FILE* internally holds a buffer

if we're talking C, there's setbuf that's relevant...

https://man7.org/linux/man-pages/man3/setbuf.3.html

[paper echo]

huh... so fgetc could still be receiving buffered data? or is that only for higher-level functions (eg. for scanf)?

[raven ridge]

and there's also the pipe buffer that's relevant here: if another program's stdout is piped to your stdin, there's only so much that the writing program can write before you start reading. The pipe has a buffer to act as slack between the writer and reader, but that's finite - eventually it fills up, and the writer blocks waiting for more room

[paper echo]

but that's taken care of by the operating system though right?

im interested in "how to read from stdin like an adult"

[raven ridge]

fgetc reads from the FILE* - the FILE* itself contains a buffer, so if something has been read by read and not yet read by fgetc or fgets or fscanf or whatever, it's sitting in a buffer inside the FILE* waiting for something to read it, and fgetc can read from it

or, in other words, fgetc checks if the in-memory buffer is empty - if so, it calls read() to read some data into the in-memory buffer. Then it reads one byte from the in-memory buffer.

[paper echo]

and as per the setbuf man page, by default the buffer is some fixed number of bytes

       Normally all files are block buffered.  If a stream refers to a
       terminal (as stdout normally does), it is line buffered.  The
       standard error stream stderr is always unbuffered by default.

https://www.gnu.org/software/libc/manual/html_node/Controlling-Buffering.html

The value of this macro is an integer constant expression that is good to use for the size argument to setvbuf. This value is guaranteed to be at least 256.

The value of BUFSIZ is chosen on each system so as to make stream I/O efficient. So it is a good idea to use BUFSIZ as the size for the buffer when you call setvbuf.

Actually, you can get an even better value to use for the buffer size by means of the fstat system call: it is found in the st_blksize field of the file attributes. See Attribute Meanings.

Sometimes people also use BUFSIZ as the allocation size of buffers used for related purposes, such as strings used to receive a line of input with fgets (see Character Input). There is no particular reason to use BUFSIZ for this instead of any other integer, except that it might lead to doing I/O in chunks of an efficient size.

and st_blkszie is "The optimal block size for reading or writing this file, in bytes" https://www.gnu.org/software/libc/manual/html_node/Attribute-Meanings.html

[raven ridge]

if you care more about speed than memory efficiency, the number of bytes you read with read(2) should probably be a multiple of 4096 bytes - the 8kib Python is apparently using on Linux seems reasonable. If there aren't that many bytes available, it'll return short, anyway.

[raven ridge]

aha, and I'm sure that's where the 8kib from Python came from.

[paper echo]

#include <stdio.h>
int main(void) {
    printf("%d", BUFSIZ);
}

this gave 1024 on my system

and io.DEFAULT_BUFFER_SIZE is 8192

so yeah some multiple of the libc default buffer size

i guess thats just the balance of memory and syscall-intensity that made sense on my particular platform

[raven ridge]

reads return short if there's not enough data available, so really what's most important is making it a multiple of the OS's page size

[paper echo]

wikipedia says that a page is a fixed length block of contiguous memory

that means that the buffer is guaranteed to be contiguous in memory if it's the size of a page?

[raven ridge]

yeah - 4kib on Linux, at least by default.

[paper echo]

#include <stdio.h>
#include <unistd.h>
int main(void) {
    printf("BUFSIZ = %d\n", BUFSIZ);
    printf("SC_PAGESIZE = %ld\n", sysconf(_SC_PAGESIZE));
}

BUFSIZ = 1024
SC_PAGESIZE = 4096

so python chose 2x the page size

[raven ridge]

a page is the minimal amount of memory that is mapped into the process. Imagine that you call malloc and ask for 1 byte of memory, but there is no available space already mapped into the process. malloc will ask the OS for more memory, and the amount of extra memory it'll get will be some multiple of 4096 bytes.

that doesn't necessarily imply anything about contiguity in and of itself, beyond that the OS can work with pages more efficiently than parts of pages

also - there's useful syscalls here... if you're concerned about efficiency, learn about the splice and sendfile syscalls

and possibly mmap, if it's applicable to your use case

[paper echo]

aha

splice and sendfile being potentially useful if you're e.g. writing your own shell?

[raven ridge]

more if you're writing your own HTTP server, I think

splice lets you move data out of a socket into a program without needing to copy it twice

normally it'd get copied off the wire into a kernel buffer, and then out of the kernel buffer into a userspace buffer

splice can avoid that copy

[paper echo]

ah

thats very good to know

not what i need at the moment but definitely could come in handy

i appreciate all this info

[steel crystal]

cant seem to find any help on understand how to use this

been studying and researching all day =/

[safe bane]

What are you trying to do/what API documentation is this? Sharing a link to it could be helpful once you share what you’re trying to accomplish 🙂

[steel crystal]

https://discord.com/developers/docs/resources/webhook#modify-webhook

Im trying to modify the webhooks channel

Everything ive tried today, with alot of success in other areas, but not the endgoal xD

Considering just dropping the embeds idea, and learning how to use Pillow to dynamically create images

[grave jolt]

@steel crystal This channel is for discussions about the Python language itself. If you have a general question, see #❓｜how-to-get-help and claim a help channel.

[steel crystal]

i get ignored in help channel

[grave jolt]

@steel crystal It doesn't mean that you should ask your question where it is off topic.

You can also try #discord-bots if that's what your question is about.

[steel crystal]

im going to bed, so its all good, ill find another solution

Its just sad they would spend all that time programming it and then give zero indepth explanations on how to actually use the api

[static bluff]

What are you guys' thoughts on a reimplementation of how decorators work? Personally, I think they are a bit slapped together and I think they can be hard for novice programmers to wrap their minds around (get it? Wrap? Haaaaaaaa)

shows self out

Anyway, I'd personally build it such that decorators inherit from a decorator class, an methods are decorated using the same syntax except a fully instantiated decorator object follows the @, not a method

class MyDecorator(Decorator):
  
  def __init__(self, *arguments, **keywords):
    self.arguments, self.keywords = arguments, keywords

  def __beforecall__(self, method, *arguments, **keywords):
    #fired before the method itself

  def __aftercall__(self, method, *returnvalues):
    #fired after the method itself

@MyDecorator(1, 2, 3, a='a', b='b', c='c')
def myMethod(a, b, c):
  ...

Something like that

With this system, instead of replacing the callable around which it is wrapped, it is stored within a 'decorators' container on the callable. Whenever that callable is called it executes each of its decorators beforecall methods with a reference to the method itself as well as all of the arguments provided to the call, and after its executed it fires aftercall, with a reference to the method and also any values returned from the method itself

So theres no fighting with signatures, and you dont need to worry about transformation from function to method or other such cases

[paper echo]

Interesting idea

I see the value in it

[raven ridge]

It's far less flexible than a normal decorator. You can't reasonably wrap the called function in a context manager, or catch exceptions from it (like @retrying.retry), or decide not to call it at all (like @functools.cache) or to call it multiple times (like @retrying.retry)

[undone hare]

That should be a fairly simple utility to build

Doesn't this already exist in functools actually?

[static bluff]

I think I see what you're getting at, mostly, but I don't think those would be impossible to implement

[paper echo]

The signature issue is annoying though. Would be nice if you could have a function "inherit" another function's annotations for writing decorators

[raven ridge]

Well, I don't see how the decorator can control how many times the decorated function is called with your model. But even if it's possible, it's awkward.

[brave badger]

Like ParamSpec

[paper echo]

But you can make a decorator that accepts a list of pre-call and post-call hooks

[undone hare]

Would you mean the scope of __beforecall__ leak into the decorated function's scope?

[static bluff]

More awkward than decorators are now? You, as someone who can rightly stand tall as an advanced programmer, don't really get to comment on the difficulty people have with decorators when they're first starting out (no disrespect meant)

[paper echo]

Eh? This exists?

[raven ridge]

Yes, I think what you're proposing is both more complicated than how decorators exist, and less flexible.

[static bluff]

Writing a functions that defines a function which itself calls the function you're wrapping is a mind bender for a lot of people, and it layers one deeper if you try to call the decorator with arguments

[paper echo]

https://www.python.org/dev/peps/pep-0612/

[undone hare]

With the right explanations decorators are very easy to understand if you ask me

[paper echo]

Alternatively, they are mind bending to learn about because higher order functions themselves can be surprising and exotic to some people

I don't remember ever having trouble with it but I know that some people do

[static bluff]

Higher order meaning functions that deal with functions

?

[undone hare]

yep

[static bluff]

Fair, I won't argue that

[raven ridge]

I don't think passing a callable to a method will necessarily be less complex for people to understand. The confusing part of decorators is learning that functions can be passed to functions, and that functions can be replaced by other functions. The closure part of it seems to come much more easily to people

[static bluff]

But at the same time, a decorator system designed to make higher order functionality easier, not just possible, would be preferable to me

[raven ridge]

And of course, you doing need to use closures. You can just make all of your decorators classes, taking the original callable in __init__ and doing the magic in __call__

[static bluff]

Alright then, how about meeting half way

One sec...

[undone hare]

I guess if you show someone who has never seen decorators before

def set_a(_value_of_a):
    def decorator(func):
        a = _value_of_a
        def wrapper(*args, **kwags):
            func(*args, **kwargs)
        return wrapper
    return decorator

@set_a(42)
def print_a():
    print(a)

print_a()
```Yes, they will be confused. Although if you break it down with them, it really isn't complicated.

[raven ridge]

The @ syntactic sugar for decorators makes them easier to read, but harder to explain...

[undone hare]

If you expand

@set_a(42)
def print_a():
    print(a)

to

decorator = set_a(42)
def print_a():
    print(a)
print_a = decorator(print_a)
```is it really? Surely you wouldn't start with a nested call like that first, but still

[static bluff]

class MyDecorator(Decorator):
   def __init__(self, *arguments, *keywords):
     #positional and keyword arguments taken in here and stored
     #for future use

     self.arguments, self.keywords = arguments, keywords

   def __decorate__(self, method):
     #activated by the '@' operator, such that logic is not required
     #within __call__ to either set the method onto the decorator if
     #not already done or do its logic otherwise
     self.method = method

   def __call__(self, *methodargs, **methodkwargs):
      #precall logic here  
      returnvalues = self.method(*methodargs, **methodkwargs)
      #postcall logic here
      return returnvalues

Totally, completely flat, no lost flexibility

[undone hare]

Eeehh

That's really breaking

and it doesn't allow more than one use per instance

[raven ridge]

That's the biggest issue I see

[static bluff]

I'm not 100% sure what 'one use per instance' means

[raven ridge]

You can't decorate two callables with one instance

And: if you like this style, you can just do this in Python. This doesn't need a language change, you can make a library that does this.

[static bluff]

Like what a property does, it can decorate getters, setters, and deleters all in one instance

[raven ridge]

You store the decorated object as an instance attribute on your instance, so one instance can only decorate one object

[static bluff]

Logically I don't see that as a problem, so you'd be concerned about memory I'm guessing

[raven ridge]

Nope, concerned about lost flexibility. You're losing something that can be done today

[static bluff]

Can I see an example?

And, though I'm listening intently to everything you say, because its good learning, I might disagree a little bit

Sure, some flexibility is lost, but readability is gained. Its not exactly write off, its a trade

[raven ridge]

flask.App.route for example.

Every function it decorates is registered into the app, and it can be used multiple times to do that

[undone hare]

Sure, we used to have one for #846514617261621292. We wanted to have a rate limit bucket per user that could be applied to many routes. For that, you could do something like

user_bucket = ratelimit.UserGlobal()

@user_bucket
@fastapi.route(...)
def route_1: ...

@user_bucket
@fastapi.route(...)
def route_2: ...```

I think this is a better example as you really need state sharing here

[raven ridge]

That's a better example than mine, since it needs

Yeah

[undone hare]

hah

[static bluff]

So, in some cases, a single decorator might store state information (or some other mechanics I'm sure) throught its lifespan, applying it on everything it decorates

[undone hare]

yep

[static bluff]

ponders

[undone hare]

I guess you can overcome this in your model by encapsulating each decorator in a larger class, although that feels a bit bloaty

Hey @unborn burrow, please see #❓｜how-to-get-help for Python help!

[static bluff]

Well, I won't back down in my opinion that the decorator system is user unfriendly and that an objective, dunder based language like python is more than equipped to meet the challenge

[raven ridge]

But anyway: decorators today are sufficiently generic that you can implement what you propose on top of them. You can build a decorator factory library function that creates decorators that function the way you propose.

[static bluff]

But I'll concede that the problem is much more complicated than I had originally considered

[undone hare]

I do think your API has its merit. Being able to write pre-call and post-call decorators more easily sounds useful, that could fit inside a module such as functools

[raven ridge]

Yep. I don't think it's a good replacement for decorators as they exist today, but it could make simple cases simpler.

You could make a decorator that makes decorators 😄

[static bluff]

Well thanks y'all! I feel like I learned something

[raven ridge]

This probably already exists as a library on PyPI, I'd bet

[static bluff]

For the record, if I ever seem like I'm not listening or don't appreciate you guys' position on things- its actually the opposite. Every word that comes out of you guys' mouth is invaluable learning for me. Its just that asking the questions or advocating for ideas is how the discourse happens

So, always appreciated 😄

[sand python]

seems like it, yes https://pypi.org/project/Decorum/

[raven ridge]

https://github.com/micheles/decorator/blob/master/docs/documentation.md also

Including the decorator creating decorator I was imagining

https://github.com/micheles/decorator/blob/master/docs/documentation.md#decorator-factories and a decorator for creating decorator factories

[static bluff]

Its just occurred to me, as a side note, that state information could be stored on the class itself. Just a thought

[raven ridge]

Whoa - apparently contextlib.context_manager instances can be used as decorators!

[static bluff]

Not wanting to start up another big debate, just thought it was worth mentioning

[raven ridge]

!e ```py
from contextlib import contextmanager

@contextmanager
def my_decorator():
print("before")
yield
print("after")

@my_decorator()
def some_func():
print("hello")

some_func()

[fallen slate]

@raven ridge :white_check_mark: Your eval job has completed with return code 0.

001 | before
002 | hello
003 | after

[raven ridge]

I did not know you could do this. That's handy.

[static bluff]

COOOOOOOOOL

[raven ridge]

Other than not preserving the signature, that's pretty much perfect

And the decorator lib I linked provides a decorator.contextmanager that works the same but does preserve the signature

[undone hare]

Yeaaahhh, I knew there was something like that but I totally forgot about the existence of contextlib

[brave badger]

It's quite nice for managing contexts based on certain parameters without resorting to __call__

[humble idol]

I always found it annoying that when writing python classes, I need to write self in essentially every line, and often several times. It's self.this self.that self.other all over the code. Especially when using an API like PyQt, which is based on subclassing the classes they provide. But don't have any good idea for avoiding it.

In javascipt I often start a method by sort-of declaring which of the object's properties will be needed, as const { propA, propB, propC } = this, and if the method introduces new properties, then at the end: Object.assign(this, { propD, propE }). Makes things pretty explicit, and avoids having the this word sprinkled all over the place. . But I don't see a natural equivalent in python.

[spice pecan]

You could use attrgetter and unpack the result, I guess

[peak spoke]

I really don't see how assigning them at the top would make things easier. Just makes it more confusing which name lives in the object and what's local to only that method. Majority of the time is not spent on writing code so it's not like you're wasting s considerable amount of time there

[hearty kernel]

you could just use a local variable say label until the end of the method then add self.label = label, when its created, in another method label = self.label

[spice pecan]

!e ```py
from operator import attrgetter
from typing import NamedTuple

NamedTuple isn't the best example as it's iterable on its own, but we'll roll with it

class Example(NamedTuple):
a: int
b: int
c: int

obj = Example(1, 2, 3)
a, c = attrgetter('a', 'c')(obj)
print(a, c)

[fallen slate]

@spice pecan :white_check_mark: Your eval job has completed with return code 0.

1 3

[spice pecan]

But also yeah, what Numerlor said

Using self is perfectly fine in most cases

[humble idol]

I'd claim the exact opposite. I only need to look at the top of a method to see what properties it "imports", and at the bottom to see what it "exports". And I avoid writing this a zillion times in the method body.

[peak spoke]

The methods shouldn't be so large in scope that finding what it modified on the instance is a problem

[humble idol]

Yeah that sort of answers it, except for the inconvenience of still having to type self at least as many times as there are properties affected by that method.

[humble idol]

Thanks for pointing this out - I kinda forgot about attrgetter, yes it seems useful.

[spice pecan]

You still have to duplicate the names, but at least self only comes up once

[hearty kernel]

the methods might need breaking down into smaller methods if there are that many properties effected.

[humble idol]

I never said that using self is in any way wrong, just that it's tedious. I just looked at a method with about 20 lines of code, and about 50 instances of the self word.

[flat gazelle]

I have seen people alias properties to locals at the top of a function before, especially if they are used a lot

[humble idol]

Well python need a lot more time than javascript for property lookups, so it's bad to have a property lookup in a tight loop.

[flat gazelle]

if you have slotted classes, it isnt as bad, but still slower than JS

[humble idol]

Just about everything is slower LOL. That's one reason for not breaking up code into too small methods - function call overhead.

[flat gazelle]

I disagree

[humble idol]

That, in addition to the visual ugliness, is a reason I would usually want to have local references to self properties in a method.

[flat gazelle]

if you have to worry about performance that much, python is not the right tool for the job. Writing bad python code to save nanoseconds is never wise.

[humble idol]

Sure, if your function is called once, or a hundred times per program run, it doesn't matter at all. But if it's going to be called a hundred time per second .. ?

[flat gazelle]

I have yet to see function call overhead be a bottleneck in a real program. Python is not great at CPU bound tasks regardless of whether you make everything a local or not

use numpy, numba etc. or just julia, rust, C, JS, Java. There are cases where python just doesnt work out well, and heavily CPU bound tasks is one of those

[humble idol]

Sure, the CPU heavy work is usually delegated to NumPy or something. But sometimes python is the only, or only convenient option giving you access to the libs you want to use.

[peak spoke]

If anything, having it properly divided into methods makes it easier to convert the bottleneck into an extension module if one does come up

[humble idol]

On an aside, I find Julia code plain ugly - of course not a good reason to write it off.

[peak spoke]

And instead of python micro optimizations that will only take down a few ms per execution at best, it will probably bring a huge speed boost

[humble idol]

Properly divided doesn't always mean methods should be small.

Something which is logically a single step of your task might just require a sizable chunk of code.

[peak spoke]

Sure, there are exceptions but you rarely want a method to be longer than ~10 logical lines

[flat gazelle]

I do agree that there are tasks where the most sensible thing is to just put them in a giant function, give it a docstring and a link to the source of the algo or whatever. Those are few and far between however, and the reason to do so is not to avoid function call overhead.

[humble idol]

Well the __init__() is often seriously bloated precisely by all the boilerplate like self.a = a etc.

[paper echo]

https://attrs.org 🙃

[paper echo]

does this expression produce a generator or an async generator?

(val async for val in agen)

[undone hare]

that's..

I would say the former

[paper echo]

it's not stated anywhere in pep 530

!e ```python
async def g(): yield

async def f():
print(type(x async for x in g()))
print((x async for x in g()).anext)

import asyncio; asyncio.run(f())

[fallen slate]

@paper echo :white_check_mark: Your eval job has completed with return code 0.

001 | <class 'async_generator'>
002 | <method-wrapper '__anext__' of async_generator object at 0x7f05512c9c10>

[paper echo]

@peak spoke showed this ☝️ so i guess it's the latter

[undone hare]

interesting

[spice pecan]

How long has this been a feature?

[paper echo]

https://www.python.org/dev/peps/pep-0530/ since 3.6 apparently

i didn't know about it until recently

[spice pecan]

Me too, I'm surprised considering async lambda is not a thing

[boreal umbra]

with open(...) as i, open(..., 'w') as o:
    o.write(
        re.sub(
             r'...',
             lambda m: ...,
             i.read()
        )
    )

Suppose the content of i is many gigabytes. Is there a way to control how much text is being read in and how much is getting flushed at a time?

[true ridge]

well stuff like shutil.copyfileobj will read N bytes at a time and write N bytes and repeat this until the input file is done.

[velvet sequoia]

Hi does anyone here know OCR?

[true ridge]

instead of storing everything, you could just have a loop and do partial reads

[velvet sequoia]

Or pytesseract

[true ridge]

wrong channel @velvet sequoia

[boreal umbra]

@true ridge thanks! I was wondering if there was an approach that was more abstract than that. Guess not

[true ridge]

depending on the use case you might end up with some extra code to handle atomicity (e.g writing to a temp file and then renaming it) for handling various other forms of interruptions

[paper echo]

this is related to what i was asking about yesterday with respect to buffering

(in this channel, w/ godlygeek's help)

whether it makes sense to re.sub across the entire file, or linewise, depends on the file and the pattern you're replacing

if the pattern is not a multi-line pattern, you can stream lines from the input file to the output file, and cpython internally will do some sensible buffering for you

import re
from typing import Iterable

def sub_lines(lines: Iterable[str]) -> Iterable[str]:
    pattern = re.compile(r'...')

    def _do_sub(m: re.Match) -> str:
        ...

    for line in lines:
        yield pattern.sub(do_sub, line)

if __name__ == '__main__':
    with open(...) as input_fh, open(..., 'w') as output_fh:
        output_fh.writelines(sub_lines(input_fh))

although im not sure if writelines tries to consume an iterable into a list
https://docs.python.org/3/library/io.html#io.IOBase.writelines

Write a list of lines to the stream.
sloppy docs, or part of the spec?

import re
from typing import Iterable

def sub_lines(lines: Iterable[str]) -> Iterable[str]:
    pattern = re.compile(r'...')

    def _do_sub(m: re.Match) -> str:
        ...

    for line in lines:
        yield pattern.sub(do_sub, line)

if __name__ == '__main__':
    with open(...) as input_fh, open(..., 'w') as output_fh:
        for line in sub_lines(input_fh)):
            output_fh.write(line)

could do that to be safe. run it with pypy for extra zoom

(it's probably faster if you don't write it as a function and just "inline" the logic)

import re

pattern = re.compile(r'...')

if __name__ == '__main__':
    with open(...) as input_fh, open(..., 'w') as output_fh:
        for line in input_fh:
            output_fh.write(pattern.sub(do_sub, line))

[raven ridge]

@boreal umbra As long as your pattern doesn't span multiple lines, this seems like the best approach. It will perform pretty well - it reads bytes from the underlying file through the OS in big chunks (probably 8kib at a time, from what we saw yesterday), then copies from that buffer into a string up until it hits a newline.

if the file is multiple gigabytes, you almost certainly don't want to do input_fh.read() without a size, because that will read the entire file into memory, and require multiple gigs of memory (or swap)

reading the file in big chunks on your own is possible, but the trick then is that you'd need to make sure that your pattern doesn't match across a chunk boundary - which would probably be pretty expensive to implement, and require adding your own buffering, in a way that won't be substantially faster over using the file as an iterator.

[boreal umbra]

So the follow up: how much slower are we getting by doing what I assume are thousands of disk reads?

And writes? Can we specify "do 1000 lines at a time"?

[raven ridge]

SSD, or spinning disk?

[boreal umbra]

This isn't a real problem that I'm having so idk

[raven ridge]

realistically, the disk reads themselves are probably not the bottleneck, the OS syscalls are.

and those are relatively efficient, especially in so far as they're multiples of the OS page size

the default buffer size that Python is using seems likely to be 8kib on a Linux machine - when you open the file, you can specify a different buffer, and you may get better performance if you do some multiple of that size, like 64 * 1024 bytes, or even 1024 * 1024 bytes.

you can try providing buffering=64*1024 as an argument to open, for instance, and it might make it a bit faster. You'd have to profile to know how much.

it'd be more likely to be helpful on spinning disks than on SSDs, because spinning disks are worse at random access and perform much better for sequential reads

[static bluff]

So I've been thinking about runtime type enforcement

For my own language that is, and I'm thinking the easiest approach would be to simply have every class define an __enforce__ method which takes in the object being checked and applies some logic to it, returning either True or False

[visual shadow]

is it statically typed or dynamically typed

if latter, there's a cost to that check if your code does it everywhere, that would be the only concern i suppose. i otherwise like the idea.

if former, you wont need the enforce dunder, you could let the compiler do everything necessary related to types

[static bluff]

It would indeed be dynamically typed. I like ducktyping but I also like enforcement, optionally

[flat gazelle]

how would you check the return type of a function?

[undone hare]

It gets very costly very fast with nested types

[spice pecan]

You could assume the annotation is true and then throw a runtime error if it happens to return a different type, i guess

[flat gazelle]

there is also the whole you can't typecheck a mutable list at runtime issue

[undone hare]

my_list = [42] * 999 + ["a"]

def my_function(my_list: list[int]) -> None:
     print(my_list)

my_function(my_list)

Now try to typecheck that at runtime

[flat gazelle]

that isn't even all of the issue

l = []
a: list[bool] = l #fine, the list is empty
b: list[str] = l #fine, the list is empty
a.append(True)
print(b) #oh no
```Typechecking at runtime is full of issues

[undone hare]

You also have to account for edge cases like list referencing themselves

[flat gazelle]

even matlab does it by keeping a tag with the type of the contents

[undone hare]

I don't think it is very viable either

If you really want to do typechecking, it should be at compile time

[flat gazelle]

at the same time, automatic type errors for the simple cases would be useful

prevents every public function from having 6 lines of error reporting

but well, patma handles this ok enough I think.

[grave jolt]

JVM also has an issue with this, apparently

because it has type erasure in generics

so it has some complicated variance mechanics I won't even attempt to understand

[flat gazelle]

Variance is a thing in every language that has genetics. Even in haskell. It's just a lot more prominent with mutable containers. The difference is that java genetics for List<> are sound, whereas checking at runtime isn't.

[grave jolt]

yeah

[grave jolt]

Why do you want that?

[cloud crypt]

from typing import Any, Type, TypeVar, cast

S = TypeVar("S", bound="Singleton")


class Singleton:
    INSTANCE = None  # or whatever name

    def __new__(cls: Type[S], *args: Any, **kwargs: Any) -> S:
        if cls.INSTANCE is None:
            cls.INSTANCE = cast(S, super().__new__(cls))

        return cls.INSTANCE

    def __repr__(self) -> str:  # optional, but why not?
        return f"<{self.__class__.__name__}>"``` I think this is a pretty nice base for a singleton, if that is the singleton you are looking for

by the way, I recently learned that something like ```python
from typing import Generic, TypeVar

T = TypeVar("T")

class Wrap(Generic[T]):
WT = TypeVar("WT", bound="Wrap[T]") # <- this actually does type-check nicely

def __init__(self, value: T) -> None:
    self._value = value

@property
def value(self) -> T:
    return self._value

def copy(self: WT) -> WT:  # <- we can use it like this
    return self.__class__(self.value)```

and then it actually type-checks correctly:

class Derived(Wrap[T]):
    pass

wrap = Derived(42)

reveal_type(wrap)  # Derived[int], pretty logical

reveal_type(wrap.copy())  # Derived[int], too!```

being generic over both T and type

[grave jolt]

🧵

❌

[cloud crypt]

haha

fair

thread-safety is what I forgot

how about a Lock in there?

[cloud crypt]

I think you can even ```python
class Replace(Wrap[T]):
RT = TypeVar("RT", bound="Replace[T]")
RU = TypeVar("RU", bound="Replace[U]")

def replace(self: RT, value: U) -> RU:
    return self.__class__(self.value)```

not in the exact form, but something similar, perhaps

[spark magnet]

I hate singletons like this.

[cloud crypt]

I think we can often sentinel = object(), immutable and useful for things like non-None defaults?

[cloud crypt]

from threading import Lock

lock = Lock()

# in __new__
with lock:
    ...  # call super().__new__``` I presume something like this?

[grave jolt]

yeah

although why would you need a singleton like that?

[spark magnet]

no one needs singleton classes that lie

[grave jolt]

wdym lie?

[spark magnet]

# Make a new thing!
s = Singleton() # oops, not a new thing.

[cloud crypt]

yeah, that

well unless you need an anti-pattern of attaching things to a singleton

[spark magnet]

it's just a way to hide a global and make things hard to test

[cloud crypt]

which is well, an anti-pattern

[cloud crypt]

getting back to this, I feel like it's a fine work-around before we get (if we do) higher-kinded types of some kind, like parametrized TypeVars

[deft pagoda]

but the type is singleton, that's pretty truthful

though asyncio just uses get_event_loop or whatever instead of singletons

[flat gazelle]

event loops aren't singletons though, there is just one global instance, but you can make new ones and use them just fine, e.g. for new threads or even in the same thread.

[deft pagoda]

yeah, but i they're mostly just singletons

like loggers

what cruel person would create two event loops

that necessitates a 3rd loop so you can await each of the original two loops in the 3rd

that's the only sane way to do it

[flat gazelle]

I have made ad hoc event loops when making sync wrappers for async libraries

[raven ridge]

It's a common way to use asyncio style libraries from threaded frameworks.

[static bluff]

class Singleton(type):
  def __call__(prototype:'Singleton', *positional:Any, **encyclopedic:Any):
    
    # I've always found underscores a bit ugly, so I use a colon
    if not instance := getattr(prototype, ':instance', None):
      instance = super().__call__(*positional, **encyclopedic)
      setattr(prototype, ':instance', instance)
    
    return instance

    def __repr__(self) -> str:  # optional, but why not?
        return f"Class({self.__name__})"

I know I'm a bit late to the party, just signed on

But I thought I'd share my own implementation of a singleton

[spark magnet]

@static bluff what do you use it for?

[static bluff]

Well there are times when you want to use a singleton. I have a project that tries to emulate javascript as much as possible for the purpose of DOM control over a webpage. So I have an 'undefined' object that gets used a lot and its important that there be only one

[spark magnet]

@static bluff why not just make only one?

[raven ridge]

The singleton pattern is just a way to create global variables in languages that don't allow global variables. Python already has singleton modules, and 99% of the time the best solution is to create a module, and use global functions and variables in that module to model your singleton

[spark magnet]

@static bluff ```python
undefined = object() # done

[static bluff]

Its just a philosophical choice really. Most Python programmers assume the philosophy 'we're all adults here' and they specify how to use something in the docs. If someone choose to break the rules thats on them. And thats fair. I've always been of the school of thought, though, thats its better to remove the temptation to screw around with the internals as much as possible. thats why I'm building a language that supports privacy and type enforcement

Its really just a matter of taste though

As for your example Nedbat, it works but you can't control the behaviour of the object

[spark magnet]

Then create a class with the behavior you want, and make one of them.

[brave badger]

I quite like:

class _GlobalConfig:
    ...

GlobalConfig = _GlobalConfig(...)

[static bluff]

from typing import Any;

ERROR = "Cannot {} attribute '{}' of undefined object";

class Undefined(object):

    def __new__(prototype, *positional, **encyclopedic):

        if instance := getattr(Undefined, ':instance', None) is None:

            instance = super().__new__(prototype, *positional, **encyclopedic);
            setattr(Undefined, ':instance', instance)

        return instance;

    def __init_subclass__(prototype, **encyclopedic):
        raise TypeError("Class 'Undefined' cannot be subclassed");

    def __getattr__(self, subject:str, *positional:Any, operation:str='get'):
        raise AttributeError(ERROR.format(operation, subject));

    def __setattr__(self, subject:str, *positional:Any, operation:str='set'):
        raise AttributeError(ERROR.format(operation, subject));

    def __delattr__(self, subject:str, *positional:Any, operation:str='delete'):
        raise AttributeError(ERROR.format(operation, subject));

    def __bool__(self):
        return False;

The other thing is that metaclasses are fun

[deft pagoda]

i created a function recently that makes generic sentinels for me with the attributes/ methods i need

[spark magnet]

This is just over-complicated, but if you enjoy over-complication, go for it.

[static bluff]

I do, I really really do

So, and I know I could look this up but it'd take twice as long to get half the answer, what is a generic?

[spark magnet]

@static bluff I'm still having a hard time with your unusual naming conventions, but we've talked about it before, so I guess it's just my problem 🙂

[static bluff]

Also, it's not really over complicated. Four of six of the methods in the undefined class specify the behaviour of the object and would be there no matter what. The init subclass method is a no brainer so no issues there. And I think just about any programmer with more than a few months under their belt can recognize a 'if not exists make one, then return' pattern when they see one

[spark magnet]

But you don't need that pattern, or this defensiveness

(and BTW: semicolons?)

[static bluff]

🙂 So sorry if I seem defensive- thats just the text format killing my tone. In reality I'm sipping a coffee having a great time discussing the pros and cons

Just arguing for the sake of argument I guess

[spark magnet]

I didn't mean you were defensive, i meant your code is defensive

[static bluff]

Ohhhh

Yes, well

[deft pagoda]

can't access attribute with . 😦

[static bluff]

You're talking about the ':instance' variable?

My philosophy of defensiveness comes from having spent about two years building (trying to) build a graphics library for Python by bridging it with a runtime in a modified web browser

I did everything I could to try to replicate the javascript syntax and make it feel natural. I wanted it to be beginner friendly and also friendly for established programmers from a web development background

So, for example, an element object would of course have a style object attached to it. And there was no way to make that style attribute read only. You could use the underscore convention of course but that was only a half measure

If I were building a tool for experienced programmers to use that wouldn't matter. But I was trying to build not a tool, but a whole dialect of Python. As such, greater safety measures had to be put in place. It was a library with huge scope and tonnes of moving parts that programmers of all backgrounds would (in theory) be using for a wide range of tasks- not a specialized toolkit the experienced

So it was important to me to remove the temptation to screw around and to build the tool to be as robust as possible

[paper echo]

Do beginner programmers randomly overwrite attributes? That doesn't seem to be an issue in python

[static bluff]

Not beginners so much as programmers moving from beginner to intermediate- and not randomly of course

Its not just about explicit errors when changing explicit attributes though

For a system with so many moving parts, a change in one place could cause unforeseen problems further down the line

Now for the record, since growing a bunch as a programmer, I've realized I could streamline my original model and implement it in Cython, improving speed and achieving the privacy I want, which would mean only having to type check publicly facing methods

[deft pagoda]

you can just change __setattr__ or like adopt the strategy of namedtuple -- just put attributes in a tuple and use itemgetter property

[static bluff]

True, but a half measure in my mind

Robust is robust, shortcuts are shortcuts

(respectfully)

Just running out for a smoke

[static bluff]

Holy good moly, typing hinting is quite the rabbit hole

[fallen wadi]

Can anyone help me with this

[brave badger]

@fallen wadi Not exactly the appropriate channel for that, we also don't directly provide help with graded coursework but if you feel like you're stuck on something feel free to ask in a help channel #❓｜how-to-get-help

[brave badger]

How's it going so far?

[static bluff]

while True:

  try:
    pounds = int(input("Enter the number of pounds (-1 to quit): "))

    if pounds == -1:
      print('Quitting'); break

  except TypeError:
    print('invalid input, please try again'); continue

  try:
    ounces = float(input("Enter the number of ounces as a real number: "))

  except TypeError:
    print('invalid input, please try again'); continue

  ouncesKilograms = ounces * 0.02834952
  poundsKilograms = pounds * 0.45359237

  print(f"The number of kilograms for {pounds} pounds {ounces} ounces "
        f"is {ouncesKilograms + poundsKilograms}"
  )
  
  break

That aughta do it, but you might want to get someone else to go over styling with you, mine is a bit unique

And yeah, not really the right channel for questions like this, but we're always scanning the 'help-whatever' channels

[static bluff]

I mean, I'm only just collating information right now

[fallen wadi]

Sorry

I didn’t know

[fallen wadi]

Thanks btw

[raven ridge]

My answer was meant to be philosophical, too: the purpose of the singleton pattern is to allow creating a global variable (single instance, accessible from anywhere in the program) in a language that is designed to not have global variables. It came from Java, which doesn't have globals, but does have static variables, so you can create a global by wrapping a function that's available everywhere (public class method) around a lazily initialized variable that there's only one of (static variable).

Philosophically, in a language that has global variables, there's no need for singletons. They're a convoluted way to store global state compared to just using global variables.

[dawn sundial]

is there a mailing list for up and coming python project who need help?

[dawn sundial]

Singletons let you organize the global variables within different namespaces tho.

[spark magnet]

That's what modules are for.

[spark magnet]

Seriously: java-style singletons are unneeded in Python. There is always a better way.

[halcyon trail]

Are Java style singletons even needed in Java

They sure became popular in C++, and in C++ they are also unneeded

I don't think it's ever a good idea to restrict a class to a single instance. If you need a global, you need a global, but it seems presumptuous to assume the class won't be needed as a non global, not to mention making testing harder

[flat gazelle]

java needs singletons in some cases due to static initializer runtime being unpredictable

[halcyon trail]

To control ordering of initialization?

I guess Java lacks C++s static locals which give you lazy initialization of a global for free

[subtle ibex]

Just started working on a codebase that uses async, multiprocessing, and threading all at the same time (to run a simple application)

I don't know much about any of these, but from my vague understanding that seems like a bad idea.

Is this a red flag? Should I work on removing these when I refactor?

I would need to do a ton of research to refactor it out so I am just wondering if it's worth looking into

[flat gazelle]

there are reasons to use all 3 in the same app, but it would be nice if you could simplify it

3 different kinds of synchronization primitives get messy fast

[subtle ibex]

How can I figure out if my code benefits from all 3?

[flat gazelle]

you will have to understand how they are used and how the various pieces interact

e.g. you need threading if you want to get a multiprocessing queue into an async queue, so that is one case where all 3 have their place

[subtle ibex]

gotcha

thanks for the help :)

guess I have some research to do

[flat gazelle]

concurrent programming is one of the more difficult parts of programming, especially in complex cases

[visual shadow]

for a pragmatic approach, perhaps you need to ask, what kind of speed is acceptable for this code base, and what's too slow

then, i'd refactor out 1 of the 3 things at least, and see if im still within acceptable speeds

if yes, i'd keep going and refactor out 1 more.

to decide what to refactor, perhaps consider the specifics. is this io bound workload or cpu bound.

[halcyon trail]

Was going to say, in python specifically the only one of the three that offers parallelism is multiprocessing

But if you don't need parallelism, multiprocessing is otherwise the hardest to use. So should be easy to identify whether or not you need multiprocessing separate of the rest

[grave jolt]

well, some things can run in parallel with threads

for example, I/O

some libraries (like PIL and numpy) also lift the GIL for computations that don't touch Python objects.

[halcyon trail]

Yes, I meant, it will not give you parallelism internally in python

But that's a good point

I've had to use a mix of multi threading and multiprocessing before and didn't enjoy it

I had to write a thread safe muli processing executor trivilly by combining the existing one with a lock

Its definitely a headache, in other languages it's relatively rare you'd need to use multiple processes so it's much simpler

[flat gazelle]

I once had to get results from a subprocess generating some CPU bound results into an async queue for other things, which you can only can do with a thread in order to not block the async loop, so I had all 3 used at once

fortunately, it was pretty simple

[halcyon trail]

Yeah if you can get rid of the multiprocessing layer quickly then it's ok. I n my example I was kind of shocked how adding a bit more complexity to my code, made everything a huge headache

[grave jolt]

I guess you can talk to local HTTP endpoints if you don't want to deal with multiprocessing 😄

[halcyon trail]

Hah

[flat gazelle]

sockets for IPC are pretty common

[halcyon trail]

Multiprocessing is very nice for what it is

I mean, relative to what I guess

Sure I've encountered IPC before

[flat gazelle]

yeah, it is a great way to get around the GIL, still not as simple as powershell's jobs, but those have their issues

[halcyon trail]

But wanting parallelism is much more common ime

And in other languages you can just use thread pools for everything, and life is much simpler up till the one time you want IPC

I had a lot of headaches recently because I wanted to use multiprocessing but also do logging

This whole problem would mostly be a non issue in many other languages, thread pool + thread safe logger, you're done

[flat gazelle]

loguru claims to work with multiprocessing, but I never tested

[halcyon trail]

It's very doable so if it claims to I'm sure it does

There's a recipe for it in the official documentation

It just feels kind of ridiculous running a logging server for this, which is basically what happens

[grave jolt]

You could have a channel/queue/whatever it's called for receiving log messages

[halcyon trail]

I eventually learned enough about multiprocessing to just give each pool worker it's own integer and had them each log to a different file

That's basically how it works I think

[grave jolt]

oh, with a process pool

[halcyon trail]

Yeah

A little more challenging. Not by much once you know how, but still

I want to post the recipe somewhere, simple as the final product is I still think it's useful for people

I posted what I came up with here a few days ago

[surreal sun]

How do you look at python from a less abstracted sense

Like if you want to look at python code with less abstraction and see more of what's going on

[boreal umbra]

what are you finding too abstract?

[surreal sun]

I sorta just want to see how python works behind the scenes in a sense - not like totally behind the scenes just understand more of how python works

[boreal umbra]

do you know about dunder methods?

[surreal sun]

The basics of them, yes
Like ```
init - when initializing a class
call - for when a class is called
add - when adding two instances of a class what will it return
repr - returns a readable string, something for those looking at the code to see - like how you would initialize the class and what are the parameters of said class
str - returns a string for those not looking at your code and more of what the class is

and probably a few more that i mised

missed*

[boreal umbra]

there's tons more. but it sounds like you understand how they work in general

[surreal sun]

not a few more, those are just what i can remember off the top of my head

there are a lot of dunder methods yea

would the dis module help to see bytecode within python?

[boreal umbra]

did you know that instances are quite similar to dictionaries? you can add arbitrary attributes to instances outside of their methods.

[boreal umbra]

the bytecode that Python compiles to is an implementation detail

[surreal sun]

That's cool! So like outside of the class you can add other things to an instance of said class?

What would be something like that

[boreal umbra]

I've never actually used it in production, but consider this:

class Person:
    def __init__(self, name, age):
       self.name, self.age = name, age

bob = Person('bob', 20)
jane = Person('jane', 24)

bob.location = 'new york'

now bob is the only instance of Person that has a location.

[surreal sun]

ohh, that's cool, i didn't know that

is it because it creates a local 'class' variable specific to that instance only?

[boreal umbra]

no, because instances keep all their attributes in a dictionary, and it's read/writeable

[surreal sun]

ohhh

[boreal umbra]

it's the same thing you do in the __init__ method, really.

[surreal sun]

so like you could acesss
bob['location']?

[boreal umbra]

bob.__dict__['location'], yes

[surreal sun]

ah, ok, that's cool!

[boreal umbra]

but look at how init methods work. you're just setting attributes on the instance. nothing magical is going on there.

[limpid forum]

dunder dict is used unless you implement dunder slots
builtin stuff mainly uses slots:

>>> my_list = []
>>> my_list.x = "test"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'list' object has no attribute 'x'

docs on slots: https://docs.python.org/3/reference/datamodel.html#slots

[boreal umbra]

it's the same thing you're doing in bob.location = 'new york'

[surreal sun]

ohh, that makes sense

also why do the signatures of an object start with '0x'?

[boreal umbra]

you mean the ID? that's giving you a memory address. it's a hardware thing, I guess

[surreal sun]

ah ok

and then what follows is what is converted to binary

with the last character being a hexadecimal digit at least for arrays

[boreal umbra]

are you referring to the default __repr__?

[surreal sun]

yep

[boreal umbra]

!e

class A: pass
print(A())

hmm

[fallen slate]

@boreal umbra :white_check_mark: Your eval job has completed with return code 0.

<__main__.A object at 0x7f9932639c10>

[surreal sun]

like

[1,2,3,4,5]

if you went into each object it would be

0x00500
0x00504
0x00508
0x0050C
0x00500
```?

[boreal umbra]

hex is just another way of representing numbers that's base 16 instead of 10

[surreal sun]

ah ok

and then within each "signature" per se there are 4 bits in an integer and each bit is in binary?

[boreal umbra]

do you have experience with C?

[surreal sun]

nope, python is my only language so far

i started python 3 ish months ago

[boreal umbra]

do you plan to pursue computer science academically? you don't have to, but you'll learn more about how computers store memory and such

it's not super important for writing Python

[surreal sun]

well i'm 13 in 8th grade right now so that's a long ways away but yes i do plan to pursue cs in the future

i think what i've learned so far in terms of 'how computers work' is binary, octal, and hexadecimal systems, infix, prefix, and postfix, boolean algebra (Basics of boolean algebra), and a couple more concepts

[boreal umbra]

so you've learned bitwise OR and AND?

[surreal sun]

Yep

[boreal umbra]

oh you said boolean

yeah, boolean algebra will serve you well

[surreal sun]

like

1 OR 0 = 1
1 AND 0 = 0

[boreal umbra]

bitwise operations are slightly different. I've... never used them

[surreal sun]

ah ok

[boreal umbra]

(except for my C class)

[surreal sun]

oh wait so

for bitwise or

[boreal umbra]

!e

print(100 & 33)

[fallen slate]

@boreal umbra :white_check_mark: Your eval job has completed with return code 0.

32

[surreal sun]

if you had

1010101
OR
1100001

it would be
1110101

[boreal umbra]

yep

[surreal sun]

there's also NOT, XOR, EQUIV though, and NAND and NOR

but I don't know much about EQUIV

[boreal umbra]

I assume that's just equality?

[surreal sun]

i think there are more opreators then just those

oh wait probably yea

[boreal umbra]

here's NORway

[surreal sun]

Lmao

that's the NOR gate right?

[boreal umbra]

yes

[surreal sun]

i'm not that great at gates and what not

ah alright

[dawn sundial]

lmaooo

[boreal umbra]

I'm pretty sure you will never have to care about logic gates unless you decide to be an electrical engineer.

[undone hare]

Lmao

[boreal umbra]

!warn 253696366952316929 shitposting in advanced-discussion and making everyone go off-topic

[fallen slate]

:incoming_envelope: :ok_hand: applied warning to @boreal umbra.

[boreal umbra]

@surreal sun if you understand dunder methods, learning the accessor protocol would be a good move. they're key to how non-dunder methods work.

[surreal sun]

ah alright

will do

what's a good tutorial for that?

[raven ridge]

The default repr includes the address of the PyObject structure representing that object in the interpreter's address space. Which is of course a CPython implementation detail.

not really a hardware thing, but - Python generally doesn't directly expose its address space to you, so it's not something you'd ordinarily see

[deft pagoda]

curio has a universal queue somewhere

https://github.com/dabeaz/curio/blob/master/curio/queue.py#L81 here it is, i dunno if this is a nice solution or not

[paper echo]

Maybe anyio has a version of that

[swift imp]

Not, or (disjunction), and (conjunction), xor (symmetric difference aka not equal), nor (joint denial), nand (alternative denial), xnor (equivalence)

[swift imp]

Only time I've used them on integers is for C programming and manipulating the registers on a microcontroller

[raven ridge]

Yeah, they're must commonly used when treating a single integer as a collection of fixed width fields

[terse orchid]

What might be the best practice to log a set of processes in python? I mean if I have an api, and I want to log every step of a single request being processed in different modules but keep the logs for that request together, is there any approach or tool there? Or maybe a conversation of a chat bot I have

[ebon cloak]

hiii

i need help in django plzz

class EmployeeSalary(models.Model):
title = models.CharField(max_length=100)

instead of "title" what other fields can i write ?

[undone hare]

Hello @ebon cloak, please see #❓｜how-to-get-help for Python help

[paper echo]

Are you using the logging module?

[eager trail]

I love the logging module now

[terse orchid]

Hi, sorry for the late reply. I can use any module if it fits the problem i described

[terse orchid]

E.g. log4mongo. But I believe it's not currently maintained. Good idea/project though

[halcyon trail]

what's wrong with logging?

[sacred tinsel]

not snake case 😔

[cerulean cloak]

Hello, is there person who has code of recognize Face in Python?

[edgy parcel]

assert == guido

[iron perch]

def cakes(recipe, available):
    return min(available.get(k, 0)/recipe[k] for k in recipe)``` I personally think this has a bad runtime

What about this?
```py
def cakes(recipe, available):
    total = 6969
    for name, amount in recipe.items():
        owo = available.get(name, 0) // amount
        if owo < total:
            total = owo
    return total```

Because min makes it iterate twice

So I'm thinking that
The first one is O(n²) and buttom is O(n)

[cloud compass]

min() only executes once, so the first one is still O(n)

(a better channel for this would be #algos-and-data-structs )

[iron perch]

Next time in the future I will

[iron perch]

But the list comprehension tho ?

def cakes(recipe, available):
return min(available.get(k, 0)/recipe[k] for k in recipe)

[cloud compass]

is there anything within the list comp that takes O(n) time?

[iron perch]

The whole thing I think?

https://cdn.discordapp.com/emojis/788882232441307154.gif?size=64

[cloud compass]

the list comp itself is O(n)

but O(n^2) would be a nested loop or something similar. here, you have the generator expression (which is O(n)) and then you find the min of that (also O(n)) so it’s O(2n), which is just O(n)

[halcyon trail]

technicality, it's a generator comprehension

[grave jolt]

technically it's called a generator expression 🙂

[iron perch]

https://cdn.discordapp.com/emojis/586031051068014612.gif?size=64

Mine was faster

[grave jolt]

Does TypeVar bother anyone else, or am I just weird? It feels really weird that I have to declare an argument (a type argument) before using it. Imagine doing this:

Self = Argument("Self", is_self=True)
X = Argument("X", annotation=int)
Y = Argument("Y", annotation=int)
Cell = ReturnType("Cell", annotation=str)

class Board:
    def get_cell_at(Self, X, Y) -> Cell:
        ...

[west lily]

Not sure, but why in gods name do you have self capitalized

[halcyon trail]

yeah, it was incredibly weird to me as well

you only need to do this for generics though, and it's because python didn't want to add syntax for generics

so overall it probably makes sense

it would be weird to have special syntax for declaring a generic function when, well, it doesn't really do anything different in terms of program execution. It would basically be extending the type annotation syntax, but most likely needing to allow it to escape just the annotation, which seems probably further in than they wanted

like

def [T] foo(x: T) -> T:
   return x

pretty awful

[grave jolt]

def foo(x: 't') -> 't':
   return x
``` ![lemon_pleased](https://cdn.discordapp.com/emojis/754441880263196742.webp?size=128 "lemon_pleased")

or

from typing import var

def foo(x: var.T) -> var.T:
   return x

well, it's still not as good as ```hs
foo :: a -> a
foo x = x

[gleaming rover]

this makes my head hurt

tbh

also isn't that already used for literals

[grave jolt]

nope, it'd be Literal['t']

[gleaming rover]

isn't Literal needed when you have multiple choices

[grave jolt]

nope, it's always needed

stringly annotations are used for forward references

so right now 't' means the same thing as t

(but can be used if t is not yet defined)

[gleaming rover]

oh right

I forgot about that

[grave jolt]

I always end up with a trash can like ```py
A = TypeVar("A", bound=Hashable)
B = TypeVar("B", bound=Hashable)
K = TypeVar("K", bound=Hashable)
T = TypeVar("T")
C = TypeVar("C", bound=Hashable, contravariant=True)

[west lily]

huh, what's the point of typehinting an object?

[grave jolt]

wdym?

[west lily]

like doing x: 't'

[grave jolt]

not sure what you're asking

[west lily]

well usually I would use it like x: str to let me know that a string is expected, if you're hinting to an object then it's like saying you're expecting a given object. if you know what object you're expecting, why take a parameter at all?

or is there another purpose to that?

[grave jolt]

oh, we were discussing what syntax could be used for type variables

i.e. instead of ```py
T = TypeVar("T")

def f(x: T) -> T:
return x

[native flame]

you often typehint as strings to denote its a type which hasn't been defined yet, I think

[west lily]

ah, that would make sense

[native flame]

x: A

class A:
  pass

wouldn't work

[west lily]

mhm

well thanks, the more ya know

[grave jolt]

well, yes, that's what x: 't' means right now, but that's not what I meant

[west lily]

yeah, I was just asking the practical application of it

[signal tide]

Isn't that what from __future__ import annotations is for?

can be used for*

[grave jolt]

yeah, but it breaks some libraries that inspect annotations

which is also why it becoming default was postponed to 3.11

[gleaming rover]

it could be a literal type

[west lily]

yeah, I guess

[raven ridge]

to paraphrase all of that discussion, x: str can also be spelled x: 'str'

[terse orchid]

I want to log the messages that related to a request or a conversation together. Like in an object or array, etc.

[verbal escarp]

how do i integrate/merge an absolute path with a relative path with pathlib?

say, i have Path("/home/anna/foo.py") and i have a Path("./project/bar.py")

i would like to merge both to get a Path("/home/anna/project/bar.py")

well, in this case i guess i could just / append them

but what if the relative path would be something like Path("../bar.by")?

[stray apex]

hello I am here because I hate python someone can make it love?

[grave jolt]

@stray apex Why do you hate Python?

[stray apex]

the syntax is a bit ugly

with if true:
--print("test")

I prefer

if(true){
printf("%s", "I love c++ and stdio.h")
}

[grave jolt]

Well, nobody can convince you to like indentation, that's personal preference.

But it's honestly a minor thing.

If that's what you hate most about Python, you probably don't know much about it 🙂

[stray apex]

I also hate the way to install libraries

with pip install ...

[grave jolt]

What would you propose as an alternative?

Also... how do you install libraries with C++?

[stray apex]

An beautiful ide with gui

[stray apex]

I can include it directly with the file path

[grave jolt]

That's not a feature of the language. Python also has IDEs with fancy GUIs. It's just that installing libraries is simple enough that you don't need a complicated dialogue box.

If you want to literally include the source of a library into your project, you can do that in Python as well, nobody is stopping you. But:

• how do you put this into source control? you can't just include the source of another library into git.
• if another developer wants to build your project, how do they install the dependencies?
• how do you automatically deploy a project on a server? on 10 servers?
  That's why C++ has third-party dependency management tools and package managers. Python already comes with pip

There are also tools for Python like poetry that make life significantly easier.

[stray apex]

Which syscall use python?

[grave jolt]

?..

[stray apex]

https://en.wikipedia.org/wiki/System_call

[grave jolt]

I know what a syscall is, I don't understand what your question is.

What syscalls in Linux are handled with Python? None, because Python isn't a systems programming language.

Linux is written mostly in C.

[stray apex]

it use C library for syscall?

[grave jolt]

???

[stray apex]

by example when I use stdio.h for python to print something

[grave jolt]

I don't understand your question.

[stray apex]

stdio.h use syscall?

[grave jolt]

Yes, if you want to interact with input/output, you'll eventually use system calls somewhere.

[stray apex]

and python use stdio.h?

[grave jolt]

Yes, Python definitely uses stdio.h for something https://github.com/python/cpython/blob/bb3e0c240bc60fe08d332ff5955d54197f79751c/Include/Python.h#L25 Don't know what.

[fallen slate]

Include/Python.h line 25

#include <stdio.h>```

[verbal escarp]

@stray apex well, i also hate the way classic imports work, which is why i'm building a thing that lets you use stuff from anywhere ( https://pypi.org/project/justuse/ ) 🙂

[stray apex]

ok thanks

[verbal escarp]

auto-installing of packages isn't in yet, but a planned feature ^^

it's pretty much in flux though

[grave jolt]

I'm not really fond of how imports work in Python tbh

I like how PureScript and Haskell do it -- there are only absolute imports.

In PureScript, you can put a file anywhere in your project, and anyone can import it by its name (you declare the name at the start of the file).

[verbal escarp]

i think one big problem with the classic import statement is that it isn't explicit enough and implicitely tries to guess what the user means even though there might be ambiguities

[grave jolt]

I also like how imports work in Node. You specify either an 'absolute' import (from installed packages), or a relative import (providing the path relative to the file, which removes some confusion)

[verbal escarp]

my approach with use() is to distinguish between pure names (for packages), URLs for web-resources and pathlib.Paths to specify a local module

and there are different features available for each case

[grave jolt]

I guess the biggest pushback you're going to get is that you won't get autocompletion/type checking with existing tools 🙂

[flat gazelle]

yeah, the import system of python is one of the weakest points of the language

[verbal escarp]

@grave jolt well, i'm working a lot in jupyter, and it's quite good with autocompleting paths, so at least that works 🙂

use() isn't complete by far, still lots of things to do and fix, but i think it really is getting somewhere

every day i'm stumbling over new ideas where it might be useful.. just the other day i was thinking that one could auto-run a test-module on importing something

if you now have a test.py in a different folder, you can't easily access that without sys.path manipulation or running an external test runner etc.

but now you could use(path_to_test, initial_globals=globals()) at the end of your module to run the tests automatically on import, if you want

pinning versions of packages inline where you import it is also very useful

especially in jupyter notebooks or other flimsy code you're not really documenting externally

[summer parrot]

I was playing around with sorting algorithms and I wanted to time them to see how long they take to sort out an algorithm. So I implemented a Stooge sort algorithm (on of the slowest sorts) and made a graph of it. The Graph shows the time taken to sort the array with size N (I used the same array but sliced arr[:N]). One phenomenon was that time taken to sort the array was increasing in steps instead of it being a curve. I tried repeating the process with a different set of data (I used random.randint) and I got the same results. Any ideas on why this is happening? Shouldn't it be a curve?

[raven ridge]

Your question is a bit hard to make sense of.
If you're asking whether there are syscalls whose implementation is written in Python, no. The implementation of a syscall always lives inside the kernel, so a syscall is always implemented in whatever language the kernel is implemented in.
If you're asking what syscalls Python exposes, it's all of the common ones, and some of the esoteric ones. Take a look at what's in the os module, many of the functions there are thin wrappers around syscalls.

[raven ridge]

Even in C++, you're just using a C library that wraps the syscalls, rather than making syscalls directly. If you're using stdio.h, you're making calls into your C runtime, which lives in a library like the GNU libc implementation, glibc. That C library is the thing that actually makes the syscall to the kernel, usually. Though you could invoke it directly by writing some inline assembly, no one actually does that in practice.

[raven ridge]

To put that a little bit differently, making a syscall directly is extremely rare, regardless of language. You always ask your language's standard library to make syscalls on your behalf - using something like stdio.h from the C standard library, or iostream from the C++ standard library, or io from the Python standard library.

[verbal escarp]

@summer parrot whenever i see timings like that i'm thinking caching

or drop-in replacements

optimizations based on some condition etc