#internals-and-peps

1 messages · Page 121 of 1

halcyon trail
#

Rust seems interesting, will have to see how it goes, obviously in the real world, jobs are still 100:1 for C++

paper echo
#

whereas close to 100% of the code i write i never care about GC at all

halcyon trail
#

maybe when I get tired of finance, it would be fun to go to a FAANG and say I'm interested in doing either C++ or Rust

verbal escarp
#

hehe

halcyon trail
#

Yeah, I mean, that's close to 100% of people

paper echo
#

so i have vanishing interest in learning rust, although knowing how it works has been useful since it has interesting language features (affine types, traits, etc)

flat gazelle
#

Raku can IIRC also enforce function purity

halcyon trail
#

vanishing?

paper echo
#

like, approaching zero

halcyon trail
#

ah, ok

paper echo
#

abusing math jargon

halcyon trail
#

you made it sound like it was decreasing

#

so I was curious why

paper echo
#

like how people say "modulo" colloquially

halcyon trail
#

gotcha

paper echo
#

this const faq is hurting my head

halcyon trail
#

it's pretty cool. My dream general purpose, GC language at this point, would probably be some mix of Kotlin and Rust

#

which const faq?

paper echo
#

the D one i linked above

halcyon trail
#

oh the D one

paper echo
#

have you tried nim? it seems pretty nice, and it compiles and interoperates with both C and C++ so you aren't starting from scratch w/ tooling and libraries

halcyon trail
#

I wouldn't worry about most of that faq

verbal escarp
paper echo
#

idk why i picked up crystal instead of nim, i guess i just wanted to try ruby syntax + i wanted go-like concurrency without go

halcyon trail
#

I have not. Realistically, you're still starting from scratch in many respects

paper echo
#

that is true

paper echo
halcyon trail
#

The thingis that we have to write actual C++ for performance (I work in high frequency trading), and we need to use python for the quant stack

#

So both C++ and pythons are simply givens

paper echo
#

yeah makes sense

halcyon trail
#

The question is, with those fixed, does it make sense to introduce any more languages

#

and the answer, for me, has been "no"

paper echo
#

i think the early adopters will always be hobbyists and small boutique firms, writing web servers

halcyon trail
#

If python type annotations + mypy hadn't existed, then I think I would have started using Kotlin for all the unsexy code

paper echo
#

and CLI tool developers

verbal escarp
halcyon trail
#

Yeah. I mean the benefits of static typing IMHO are pretty big.

#

Especially when a lot of our unsexy code is written very quickly, it does a lot of system operations so it's a bit of a pain to test properly, so we often don't

#

So it's fairly awful to crank out some script quickly to do some boring but important operation, like say backing up production logs across all your servers, and then get a failure deep in the code because of some trivial type issue

verbal escarp
#

i knew typing would be useful, but i haven't met anyone so far for whom it was the main selling point

halcyon trail
#

Main selling point of what?

verbal escarp
#

python

halcyon trail
#

It wasn't

#

we already had python in our stack because of the quantitative stuff

verbal escarp
#

you said "If python type annotations + mypy hadn't existed"

#

to me that sounds like a main selling point 😄

halcyon trail
#

I think you misread 🙂

#

The main selling point was the suitability for quant research

verbal escarp
#

okay

halcyon trail
#

It's such a big selling point that using python, was a given

verbal escarp
#

got it

halcyon trail
#

the question was just "do we also write some of this non-quant, non-perf-critical code in python, or use a third language"

#

types hugely decreased the value add of using a third language like Kotlin

#

but obviously the costs remain the same. People have a third language ot learn, third environment, harder to share code, etc.

flat gazelle
#

Justifying a new language is always hard. Rust brings some actually useful things you can't get elsewhere (safe parallelism, except pony, but you are really working from scratch there), which is why it is growing in popularity

paper echo
#

pony seems even less popular than crystal...

halcyon trail
#

We had a ton of python 2 code, and I made the call to just write new code in python 3 with type annotations, gradually rewrite, etc, instead of writing new code in Kotlin or something similar

paper echo
#

pony has some kind of capability model right?

halcyon trail
#

Pony is the 0 / 0 = 0 language 🙂

#

probably the only thing I will ever know about Pony

paper echo
#

the what

verbal escarp
#

lmao

halcyon trail
#

you read it

verbal escarp
flat gazelle
#

Pony is very very tiny, but it has a company behind it which is afaik yet to be bankrupt.
Yup, reference capabilities. Honestly kinda useful for just thinking about data races

halcyon trail
flat gazelle
#

Pony is an excellent actor runtime on a mildly strange language

#

I generally end up using the unsafe or erroring division

verbal escarp
#

uhhh.. not sure i'm convinced :p

halcyon trail
#

it's pretty weird

flat gazelle
#

I mean, unless you need a fast actor runtime and are willing to write things from scratch for it and will never encounter memory pressure, pony is not the language for you

halcyon trail
#

IME almost every one of these niche languages has something that is, I'm sorry, totally insane

#

and only people grinding axes online will try to defend

#

for Nim, the language has some kind of bizarre snake case/camel case insensitivity

#

when I saw that, I was like... yeah....

flat gazelle
#

Honestly, I like that about nim

paper echo
#

oh yeah that was why i stopped messing with nim

#

lol oof

halcyon trail
#

I'm sorry but its nuts

paper echo
#

i really like UFCS

#

but not that

flat gazelle
#

Makes it really easy to interop with different libraries that use different conventions

halcyon trail
#

It's the kind of feature that everyone who's worked on a large codebase with a large team stops and says "oof"

#

well, almost everyone

paper echo
flat gazelle
#

There aren't large ním codebases. It's a great glue language and "I wish I could write C without C" language

halcyon trail
#

Right

#

But there's also no real major selling point for it

#

that I can see, at least

flat gazelle
#

Yeah, I wouldn't use ním for much anything

halcyon trail
#

the only thing I will say is that there is a big of a vacuum, IMHO, for a reasonably powerful, GC, native, cross platform language

#

it's strange, but... there's really no language there

paper echo
#

is nim not that? since it compiles to C/C++

flat gazelle
#

I mean... Go

halcyon trail
#

when Go gets generics 🙂

paper echo
#

isn't it getting generics soon?

halcyon trail
#

Nim is that but Nim is a nonentity

#

the same as Kotlin native

paper echo
#

also nim seems good for writing CLI tools and stuff

halcyon trail
#

Kotlin native looks cool and I'm rooting for it but realistically I wouldn't touch it right now for anything serious

flat gazelle
#

And I mean, julia will be a strong contender once aot is done

halcyon trail
#

but anyhow in relative terms I mean, there's still, relatively speaking, a vacuum there.

#

I doubt that Julia will ever become even vaguely mainstream outside quant work. Julia is also dynamically typed.

flat gazelle
#

But yes, Right now you have a large number of minor languages

paper echo
#

julia is already a contender for data science

#

julia for general purpose computing is like a side benefit

flat gazelle
#

Hell, Lisp falls there

halcyon trail
#

I basically just want something like a Java/C#/Kotlin/Swift/Rust, language-wise

#

nothing tooo too insane, but reasonable type system (generics). But cross platform, native, GC.

#

generics will help Go, the last thing I read about it though was what's happening with modules and that made me facepalm pretty hard, at least if the people telling me about it were accurate

flat gazelle
#

Yeah, go has some issues, especially with the cross platform part

verbal escarp
halcyon trail
#

it looks like Go's module system is planning to handle different majors of the same package as though they were entirely different packages

#

this has, uh, a number of ramifications, none of which seem great to me

verbal escarp
#

o.O what's the reasoning behind that?

halcyon trail
#

something about sharing the burden of upgrading between users of the library, and the library maintainers, I think?

#

but basically, imagine you have two dependencies, that in turn have a common dependency, which defines some type, and you are getting an instance of that type from one dependency, and passing it to the other

#

when one of your dependencies upgrades its major on the transitive dependency, and the other doesn't, teh module system itself will not complain. It will just say "ok, you now depend on Foov2 and Foov3"

#

Fetch both

#

and then, your code will of course not compile, because you are trying to pass Foov2.Bar where Foov3.Bar is expected

#

it seems pretty insane to me, that this error is happening at compile time and not at package/dependency resolution time

verbal escarp
#

hm.. as i mentioned, i'm working on https://github.com/amogorkon/justuse, which also allows inline auto-installation of packages, version/hash-pinned, so that scenario actually could happen there as well

#

but with classical pip/import, you wouldn't even have the chance to make things work with different versions of the same package without danger of conflicts

halcyon trail
#

yeah, pip is pretty awful for thsi stuff, though, or at least it was

#

you need something that properly solves the dependency satisfaction problem

#

there is no other way

#

In an ideal world, packages should separate private and public dependencies

flat gazelle
#

Yeah, python more or less doesn't support 2 versions of one thing in the same codebase

halcyon trail
#

And versions of private dependencies wouldn't need to agree

verbal escarp
halcyon trail
#

I don't know any language that makes this practical though

verbal escarp
#

i love that phrase, i'm going to steal it 🙂

halcyon trail
#

It's not really possible in python, I don' tthink

#

heh

verbal escarp
#

actually, with justuse that will be possible

#

you could have public dependencies maintained with pip/import

flat gazelle
#

node.js can have different versions of the same dep, but due to sys.modules, it is near impossible in Python

verbal escarp
#

and private dependencies managed via justuse

#

i like the notion of public/private dependencies, that's nice

halcyon trail
#

I'm curious to see how justuse will allow one package to use numpy 1.19 while another is using numpy 1.18

verbal escarp
#

numpy is a bit of a sore point, i'm afraid.. although, we're getting there, slowly

halcyon trail
#

Well, forget numpy then

#

any other package

flat gazelle
#

If justuse completely reifies the import system, it is possible-ish, unless the module wants to grab itself from sys.modules

verbal escarp
halcyon trail
#

what exactly does "reifies" mean in this context

#

I know at least 2 usages of that term and I don't see how either applies here haha

flat gazelle
#

Reify is essentially reimplement in this case

verbal escarp
#

we're managing distributions in a completely seperate way from pip

#

so we don't run any danger of conflicts

#

unless, yes, modules insert themselves into sys.modules and other shenanigans

halcyon trail
#

Well, the very large practical danger here is that there is zero enforcement of private vs public dependencies

flat gazelle
#

It is not impossible to build 2 module objects of the same module with different versions, it is impossible to have that work correctly with the current python import system

verbal escarp
flat gazelle
#

Yeah, wonder if it will be stable enough to see adoption

halcyon trail
#

if package A asks specifically for foo =1.1, and B for foo=1.2, there's nothing that actually enforces that that foo.bar doesn't show up anywhere in the public API of A, or B

verbal escarp
#

me too ^^

halcyon trail
#

and if foo.bar is in their public API, then it's really bad

#

maybe if you packaged a tool that helped enforce that, it would reduce my misgivings.

#

Out of curiosity what was the motivation for this

verbal escarp
#

my main motivation was the lack of module/version annotations

halcyon trail
#

We use conda where I work, which is explicit about only having one version of every package, and also explicit about completely solving the dependency graph correctly (or error'ing out if it's not possible)

#

And it's been a good experience

#

what do you mean by that?

verbal escarp
#

i ran code in jupyter never knowing which version is actually running, got me into trouble a couple of times and i realized how limited the import statement was, which could only be improved on by syntax changes

halcyon trail
#

well, your environment manager would tell you that right

verbal escarp
#

not if you try to run notebooks that are over a year old across different machines

#

so i started with a functional import, building on importlib, which worked surprisingly well and simple

halcyon trail
#

I don't follow, at the time you're running it though, you run it in a specific environment, right

#

Here's a question, suppose I do stuff like this:

import pandas as pd  # this just gets the default pandas, which is such and such version, requires numpy 1.20
np = your_import("numpy", "something something = 1.18")
#

So now I'm using pandas and numpy that are incompatible with each other, no?

verbal escarp
#

damn good question

#

haven't tried it

halcyon trail
#

yeah....

verbal escarp
#

but theoretically, yes

halcyon trail
#

I just don't think tbh inside the code is to solve these kinds of issues, it's an entire world of issues

#

*is the place

verbal escarp
#

well, not like that

halcyon trail
#

you can't really get away from different imports interacting with each other, so I still think the only thing that makes sense in the context of python is to "solve" the dependency world, and know you have a solution that works for everyone

verbal escarp
#

let me get a bit into detail

halcyon trail
#

tbh your reasons for writing this weren't what I expected, I think the issues you described can be handled well enough with virtual environments. I expected you were going to say that you weren't able to resolve dependency graphs.

verbal escarp
#

because use("numpy", version="1.18") would just try to pull the classical install and discover that it's running the wrong version. if you use("numpy", version="1.18", auto_install=True) it would install the package in a seperate folder with no overlap with pandas

#

yes, you might get in trouble down the road if you mix versions, but for those cases, we've got an internal dependency graph planned to track down issues

halcyon trail
#

okay, the auto_install=True is what I mean

#

I just think it's risky to allow code to essentially do ad-hoc, local dependency management, instead of a proper dependency/environment manager

#

And I'm not really sure what the major benefit is

#

If you want to run year-old notebooks on different machines, you create an environment that you think will run it properly on one machine, test it, and if it works, just keep using that environment on all the machines

verbal escarp
#

well, one big main benefit is that you have your versions pinned inside your code, which is especially important for scientific and other "free floating" code that isn't managed in an environment

#

like publications

#

you could post code in a blog and have something that is definitely running anywhere

halcyon trail
#

You could just post the lockfile with the source code

verbal escarp
#

i think the risks are manageable, especially with a proper internal dependency graph that can alert the user of potential problems

#

if done right, it could even help with potential issues when upgrading versions

halcyon trail
#

The thing is that 100% of scripts that use this aren't describing their dependencies, they're pinning them, which is not great for downstream users. mature packages like numpy, etc have, in the grand scheme of things, very very few backwards incompatible changes, you have to be using code that totally ignored deprecation warnings for much longer than a year to have problems. So at the very least, even if you ran you rscript with numpy 1.18, you don't need exactly numpy 1.18, but at least 1.18

#

and even the "at least" part is usually very conservative

verbal escarp
#

i'm thinking of alerting the user of incompatible call signatures etc

halcyon trail
#

If I download two scripts, and one says numpy 1.18, and the other says 1.19. this whole approach creates many more problems than it solves

#

If the first script had a dependencies file that said >= 1.18, and the second. >= 1.19, then a dependency manager would just resolve it to 1.19, which is going to be a better outcome, 99.9999999% of the time

verbal escarp
halcyon trail
#

for testing, deploying, your own code, sure

#

for writing code that's going to be used by someone, in conjunction with other code, no

#

that's why for libraries, you ideally want to have both a dependencies file, and lockfile(s)

verbal escarp
#

i've had so many problems caused by pip upgrading some library because one thing required "at least" version X and breaking other stuff in the process

halcyon trail
#

well, pip doesn't properly solve environments

#

if you already have package X that depends on Y, and now you want to install Z which also depends on Y, you need to ensure that Z and X have compatible dependencies on Y

#

you need to find some version of Z, that has a dependency on Y compatible with your current version of X

#

either that, or, you can ask it to try to install any version of X with any version of Z, from scratch, and then it will try to find the latest X and Z that can depend on the same Y

#

Conda and I'm sure many others, do all this properly

verbal escarp
#

why not just have things their own "private" dependencies and raise the issue of requiring "public" dependencies if a diamond comes up

halcyon trail
#

Once the environment is resolved, your stuff only breaks if library authors have bugs (like incorrectly describing their dependencies)

#

what do you mean by a diamond coming up?

#

In this example, there's already a diamond

#

"my environment" depends on X and Z, X and Z depend on Y

#

diamond

verbal escarp
#

well, it's not exactly a diamond but more a rhombus

halcyon trail
#

"raising the issue when a diamond comes up" (or, modulo shapes that are equivalent to diamonds for our purposes) and simply having public dependencies, is exactly the same thing

verbal escarp
#

not really

halcyon trail
#

can you give an example of how it's different?

verbal escarp
#

well, let's say you have a program (maybe a subprocess) that requires version X of a package while a different program, in the same environment, requires version Y

halcyon trail
#

are you saying that there could be a "diamond" present in the environment, but not in any specific piece of software that runs in the environment?

#

ok, yes, that is what you are saying

verbal escarp
#

pip wouldn't allow that, although there is no direct conflict of those dependencies

#

those two programs have no overlap or exchange, although they share a common "space"

halcyon trail
#

Sure, so in principle, that does give you more flexibility, at the cost of adding a lot more danger

#

the real question is, what's your use case for this flexibility

verbal escarp
#

that's where i'd say they can have "private" dependencies, no problem

halcyon trail
#

Well, it's not really "no problem now", because now there are landmines in your environment

#

You're basically saying "well, if you never run X and Z in the same process, then you're fine"

#

But one day, someone takes th existing environment, and they write a package P, P depends on both X and Z, and now suddenly there's an issue

#

So, you could in principle have a tool that errors, at import time

verbal escarp
#

and then your code should alert you of that potential problem

halcyon trail
#

as soon as both X and Z get imported, boom

verbal escarp
#

not before

halcyon trail
#

Okay. So, like I said, you get more flexibility, but you also have landmines buried in your environment.

#

I shouldn't use that word, that's too charged.

#

You have more flexibility, but you have have a potential issue that you've left latent instead of dealing with immediately.

#

This can be a valid engineering trade-off, don't get me wrong. I just don't really see the need for that flexibility, both in my work, and also in the use cases you've described so far.

#

Practically speaking, 99% of the time, the dependencies just get resolved, everything just works, and it's all great.

verbal escarp
#

true. on the other hand, you couldn't even run those two programs in the same environment, at all

halcyon trail
#

Well, that's the thing, there's a certain assumption behind that statement that isn't really true

#

If those two programs are over-specifying their dependencies, then yes, it will be true

#

but, again, in 99.99999% of cases, X and Z should be fine to specify something like numpy >= 1.18

#

and then you will get a valid dependency up front

#

for both X and Z

verbal escarp
#

i don't think so. just think of python itself

halcyon trail
#

what about python itself?

verbal escarp
#

python versions

halcyon trail
#

I don't understand

#

lets use python itself as an example

#

and numpy

#

numpy's dependencies on pip say that it can be used with multiple versions of python

#

it doesn't say "numpy 1.20 depends on python 3.7"

verbal escarp
#

how much has been added and then deprecated over the years, you couldn't just say "python >=3.1 should work"

halcyon trail
#

that would be incredibly bad

#

that's literally exactly what numpy says 🙂

#

just not 3.1

verbal escarp
#

numpy is a bit of a special case, i wouldn't like to get into detail there

halcyon trail
#

looks like 3.7, 3.8, 3.9

#

well, pick any python package

#

almost none of them will be locked to a specific python version

verbal escarp
#

numpy is compiled to certain platforms and i had to downgrade to make things work at some point

halcyon trail
#

yeah, so with conda, the C packages are also part of the environment

#

so that's all handled correctly and 100% reproducibly

#

sorry, 99.9% percent reproducibly, there's like, one package that conda doesn't handle, which is libc itself. But not normally an issue.

#

More like 99.9999% reproducible.

#

But ok, let's leave that example alone.

#

arrow afaik is a pure pytho package. it looks like it supports python 3.6-3.9

verbal escarp
#

while i am a fan of conda for its reliability, it suffers a different problem

#

it's basically impossible to upgrade things

halcyon trail
#

I haven't had any issue upgrading things

#

not really sure what you mean

verbal escarp
#

well, sometimes it would take hours to try to come up with feasible dependency resolution and then i'd just cancel the operation

deft pagoda
#

anaconda broke my installation once, and i never used it again

#

i will concede that it's as likely that it was user error, but i don't like things that let me mess them up

halcyon trail
#

it's taken me as long as minutes, never hours

#

I've never used anaconda, just conda, so I'm not sure what else goes into it

#

I have 280 dependencies in my lockfile, for perspective

#

so, idk

#

I guess it's possible you had far more and that was part of it

verbal escarp
#

no idea what the problem is in my case, but in my experience, it's easier to just set up a new VM with a completely new conda installation

halcyon trail
#

at any rate though, taking a step back: it is 100% the standard for libraries, and generally any code that is planning to be reused, to explicitly state that they depend on "ranges" of other libraries

verbal escarp
#

rather than to try to upgrade an existing installation

halcyon trail
#

and say what those ranges are

#

If you just say "this code was run and tested with these exact dependencies" then you're basically shifting an insane burden downstream

#

because obviously, as soon as you have say, 10 dependencies, arrow says that it only supports python 3.7, pydantic says it only supports python 3.8, etc

#

that's just madness

verbal escarp
#

hm.. the system requirements are checked for each indidvidual package installation

#

i don't think that's a big issue, tbh

#

but yeah, there's one issue that has been nagging me

#

i'm thinking of security patches

halcyon trail
#

well, it applies for any kind of dependency, even if it was not a system package.

verbal escarp
#

i just got an idea.. let's be crazy for a moment

#

what if there's a mechanism for patch releases that can automatically try and install a new package release (of course with cryptographic signatures etc) but checks if all callable signatures match the current version before actually making it available to the user

#

and raise a warning if signatures don't match to give a heads up

#

then the user could be sure to get all patches, as long as the contracts are fullfilled

#

it would force people to put a little more thought into their APIs, but i think python has everything by now that could make this approach doable

paper echo
#

conda is particularly fucked up for some reason

#

i dont know how apt, zypper, dnm, etc all manage to be not as fucked up as conda

halcyon trail
#

I honestly don't know what you think is fucked up about conda

#

@verbal escarp i just don't really think that's much of a value add; the same type signature doesn't guarantee the same behavior, usually with small fixes the types are all identical anyway, and if the types were not identical then your static type checker would anyway catch this issue. After I update dependencies, I run my full CI, which includes obviously static type checking, if it doesn't work, I roll back. This doesn't happen very often.

#

so the value of doing this aprticular thing earlier is pretty low

raven ridge
verbal escarp
#

that usually isn't an option 🙂

raven ridge
#

why not?

verbal escarp
#

because dependencies aren't called "dependencies" without a reason

raven ridge
#

that's a circular argument.

#

I must always depend on this because I currently depend on this.

verbal escarp
#

if you rely on a certain library and they break something after you build on them for years, it's not an option to just change it

raven ridge
#

sure it is - most libraries are open source, you can fork it, or you can look for an alternative and start using that instead

verbal escarp
#

and there's no guarantee that there is an alternative that doesn't break things just the same

raven ridge
#

maintaining backwards compatibility, or at least warning users loudly and documenting clearly how to make code forwards-compatible when you break it, is the bare minimum that you should be asking of the maintainers of your dependencies. If you can't depend on them to do that, you probably shouldn't be relying on them at all.

verbal escarp
#

not sure i get what the whole point of the argument is, tbh

paper echo
#

its incredibly slow and apparently kind of flaky

#

i love that it exists, i used it heavily when i was doing data science work

#

but its just a weird tool with a lot of weirdness about it

#

its gotten a lot better, was weirder 4 years ago when i first started using it

raven ridge
halcyon trail
#

@paper echo I mean dependency resolution is an NP hard (or complete? I always forget) problem

#

if your dependency solver is working really quickly and smoothly, be very suspicious

paper echo
#

yeah but why is it so much harder in conda than say zypper?

halcyon trail
#

I'm pretty sure that conda internally uses some industrial strength solver

paper echo
#

is conda dependency resolution harder for some reason? more kinds of dependencies?

halcyon trail
#

I couldn't speak to that, we'd need to do a careful apples to apple comparison, with graphs of comparable size and density

paper echo
#

resolving stuff like gcc versions and mkl vs openblas etc?

#

my impression is that its mainly slow because its written in pure python

#

in my quick testing, mamba is significantly faster and gives the same results

halcyon trail
#

according to this, conda uses pycosat

#

which is a python wrapper around picosat

#

which is, an industrial strength SAT solver

#

I don't know if it's the absolute state of the art but this isn't some naive pure python code here

unkempt rock
#

Hey guys quick question can you set/cap/ control cpu clock speeds in python ? For windows I'm trying to make it kinda like the throttlestop app but in python. I have done research but it all just keeps saying only for linux

halcyon trail
#

@raven ridge agree, and similarly I think library authors need to make a tleast somt reasonable effort to say that their package supports some range of dependencies

#

You can't just say you depend on foo=1.21, bar=3.5.7, it's not scalable

#

it depends on foo >=1.1, bar >=3.5

verbal escarp
halcyon trail
#

realistically these things aren't always perfect but, again, you need to make some kind of reasonable effort

#

I dont' see which of your ideas solves this

#

This situation is fundamentally unsolvable

paper echo
#

that makes sense and thanks for the link @halcyon trail

halcyon trail
#

np

paper echo
#

i never heard of metachannel before

halcyon trail
#

Yeah, I don't know why it's slower. I admit I haven't done much real comparison. Obviously, a few minutes for a few hundred dependencies doesn't feel super fast, but it is a hard problem

#

but like all benchmarking, you have to be incredibly anal

#

there could be some "small" detail, something that one takes into account and the other doesn't, that makes the whole computation 10x more expensive

paper echo
#

it's not just that it's slow, it's that installing packages after the env is created is extremely flaky and liable to break or give inconsistent results

#

i understand, dependency resolvers always are heuristic and have to decide where to stop

halcyon trail
#

I really appreciate conda because it does a lot. It doesn't just do python, it does the entire C stack, which means total reproducibility for the entire numpy stack

#

I dealt with those headaches before

#

and they are awful

paper echo
#

yeah its great, i used it to package fasttext long before facebook had updated binaries

#

i love the idea and its quite a technical achievement

#

also there was some weirdness with the CLI, ill-named things, unable to search for the latest version of a package, conda env create -f vs conda create -f using different file formats, etc

#

new releases being very slow to propagate to conda forge and default channels

#

not that hard to host your own private channel though, which was reallly nice

halcyon trail
#

Yeah at my work we use private channels for everything

#

Also installing packages has always been super fast for me

paper echo
#

but i agree, being able to distribute an entire environment with a conda env yaml file is a huge plus, without having to rely on docker which has a lot of its own issues

halcyon trail
#

I agree that some functionality is or was weirdly missing

#

I have code that creates lock files given a dependencies file

#

I thought that workflow was industry standard more or less

#

But whatever, problems solvable with 100 line python scripts aren't problems :-)

paper echo
#

like the conda env list file?

#

i remember i wrote a script to clean up the conda env yaml files so they could be distributed to other people

halcyon trail
#

Yeah so conda doesn't make this distinction at all so I have to myself

#

I have one file that is a conda env list file with loosely specified dependencies

#

Only direct dependencies too

#

Then I have multiple, fully specified, including transitive, files

#

I.e. the whole environment

#

You need to have both dependencies file and a lock file

#

For multiple reasons

paper echo
#

yes i agree strongly

halcyon trail
#

And the former should allow programmatic generation of the latter

#

But I couldn't find a way to do it

#

So I just created a dry run dummy conda environment with the dependencies file + variant packages lol

#

And dump everything from it

paper echo
#

thats more clever than what i did

#

you parsed the dry run output?

#

or you actually created a temporary conda env, got its deps, then deleted it?

halcyon trail
#

The former

#

Except you don't need to really parse anything

#

Theres a --json option

#

Gives you everything in a structured json

paper echo
#

ooh

halcyon trail
#

I can share the code with you if you wan

paper echo
#

yeah that'd be super useful actually

#

i've been wanting to do data things again

halcyon trail
#

Sure np, not at desk but I will tag you when I have it

paper echo
#

been too busy messing w/ idris lol

#

sure thank you!

halcyon trail
#

@paper echo when I removed all the company specific stuff there wasn't much left lol 🙂

# 
import json
from pathlib import Path
import subprocess
from typing import List

SOURCE_CONDA = "source /opt/devtools/conda/etc/profile.d/conda.sh"

TEST_ENV_NAME = "foobarbazqux"


def make_lock_files(deps_files: List[Path], extra_packages: List[str]) -> List[str]:

    conda_command = f"{SOURCE_CONDA} && conda create -n {TEST_ENV_NAME} --dry-run --json"
    files_string = " ".join(f"--file {f}" for f in deps_files)
    packages_string = " ".join(extra_packages)

    conda_output = subprocess.run(f"{conda_command} {files_string} {packages_string}",
                                  shell=True, check=True, capture_output=True, text=True).stdout.split("\n")[2:]

    conda_json = json.loads("\n".join(conda_output))

    return sorted(f"{d['name']}={d['version']}={d['build_string']}\n" for d in conda_json["actions"]["LINK"])
#

obviously you'll want to change SOURCE_CONDA to wherever your conda.sh file lives. Note that while shell=True is not ideal, it's the only "proper" way to call conda from python, afaik, because conda itself is actually a shell function

#

so this shell=True and source'ing dance is the way to go

#

the extra_packages argument is very useful for variants

#

not sure if there is a more formal name for that

#

In my actual code, I have a constant that looks like this currently:

VARIANTS = {"gcc8cxx17": ["gcc8cxx17"], "clang8cxx17": ["clang80cxx17"]}

And when I run this code, I actually iterate over the variants, and produce one lock file for each variant. the key is the variant name, the value is a list of extra packages to put in that variant.

#

in my code, as well, I don't actually take deps_files, I just take the overall project root, and hardcode two relative paths from teh project root that I need dependencies from. One of them is my own team's dependencies, and the other are the dependencies of the software of the rest of the company, which we build/use from source (so this is more relevant to the C++ side). I also have a couple more constants that go through the company-wide dependencies and make certain modifications, i.e. excluding certain dependencies, basically minor hacks that I've needed in order to actually get the dependencies to resolve.

paper echo
#

i think you only need the conda shell function if you're using conda activate fwiw

#

otherwise it wouldn't be able to inject any env vars into the current shell session

#

e.g. my script just uses whatever conda is on your path

#

i think there is a name for those variants but i cant remember what it is right now

#

thats a great idea to make different lockfiles for different toolchains

halcyon trail
#

yeah, different tool chains or even different versions for internal testing

#

I mean say a python package claims to support python 3.6-3.9

#

you probably want to do at least some testing with different python versions

#

so you have a few different lock files

#

obviously as you add ranges of dependencies and have more dependencies it explodes exponentially, so you can't cover everything, but you can have however many you think are appropriate to try to give some kind of coverage

#

your build matrix, basically

#

it's not just conda activate, at least, that's not my understanding, which comes from the guys at my company that specialize in this, one guy in particular who knows conda really well

#

it could have changed recently, but basically conda-as-shell-function is/was the only officially supported way to call any of the conda commands

#

at some point it was a binary and you could call it directly and it mostly worked, but support for that was deprecated

upper spoke
#

so I have an insanely bizarre thing I'm looking into....

I am working on what I hope to be a next-gen MUD server (strange as that may sound.) one of the things that I want this thing to have is a powerful integrated scripting capability, preferably with the ability to run more-or-less untrusted code from users in order to bring life to game objects and power quests and so on. Something kinda like Bethesda's Papyrus as an example!

The problem is that I'd really, really like to write this project in Asyncio because the multi-tasking/concurrent features of asyncio are amazing and fit in well with the idea of making sure the whole simulated game world keeps ticking along while letting different sub-systems access web resources. That means using something like Lua for the scripting isn't very practical - I can sandbox it, but I don't have a great way to make it yield/await from a Python sense and then resume the Lua execution.

However, I -have- had some good luck porting MUSHcode - an awful-looking lisp-like mishmash of bad ideas that works surprisingly well - creating my own Python-based interpreter (the original MUSHcode is interpreted by C, so...). I'd -really- prefer not using MUSHcode though... I'd rather have an expressive, sane-looking language akin to Lua, Javascript, AngelScript, SOMETHING.... but I don't see any way to utilize those in a way that will interact nicely with asyncio.

This leaves me with the strange idea of creating a whole new domain specific language that is designed from the get-go to execute atop of asyncio and, thus, can be paused, canceled, etc, while running inside of an asyncio Task.

But where the -heck- would I even -start- ?

grave jolt
upper spoke
#

hmm

#

yeah the problem with run_in_executor is that while it can net you an awaitable Future object it doesn't actually.. await said future.. there is no way to await something once you are out of async def territory

#

so hmmm

grave jolt
upper spoke
#

when I was using Scoder's Lupa to try and interrupt Lua, I had only one way of doing it - lua's debug.sethook to force a python routine to run and have it 'await' something but the way it did that left me very unsure of just what's happening because the Lua runtime was still executing synchronously for 100 instructions and then pausing at an arbitrary state of executing bytecode

grave jolt
static bluff
#

I wanna apologize to you all for a bit of drama I was at the centre of today

#

I was accused to asking for help and then arguing against the advice provided. As much as I think the person who said this was being unreasonable, there is probably truth to this

#

So I just wanted to say that I've heard this criticism, and though I'm trying to be a gracious student of all you already, I'll try even harder in the future

#

I think this server is the best learning tool I'll ever find 🙂 Hoping to learn a lot from you all

lilac storm
#

can someone guide me on how to get started with building web applications in python..? the only modules I know to use are tkinter and pyautogui as of now

rough leaf
#

Anyone know about the collatz conjecture (in math)? I really want to try to apply it to python and see what happens.

gleaming rover
verbal escarp
#

@upper spoke you have to consider that a scripting language like python is actually what you'd want for scripting a game, so I understand the urge to write a more limited DSL, but most often it is only a matter of time until that people need that DSL to be turing-complete, so you end up with a language that theoretically has the power of python but only a fraction of its features

#

there might be a few ways to go about though. for instance, you could look into the interpreter thing discord runs to run those python snippets in chat

#

or you limit users to define certain artifacts like functions with defined names and then scan the AST before importing them

#

for anything weird

#

or you could try to sandbox those modules as subprocesses, regard them as isolated blackboxes and only care about their input/output

#

not sure which is the safest or most convenient for you, though

halcyon trail
#

The problem is that python tends to be really poor for embedding for various practical reasons

#

So people don't generally do it

#

Most games do either use lua or a custom language for internal scripting

verbal escarp
#

python is used in a number of AAA titles though

halcyon trail
#

Really, which ones

verbal escarp
halcyon trail
#

Well, ok :-)

verbal escarp
#

it's not all doom and gloom ^^

halcyon trail
#

Some these games are written in python completely which is different

#

And many are not AAA or are very old. And it's a fairly short list.

verbal escarp
#

actually, i didn't know sims 4 is using python, that's unexpected and pretty cool

halcyon trail
#

Yeah, sims 4 is really the most decent example

#

But still, it shows that python is a rare choice for this

verbal escarp
#

actually, there are some missing

#

iirc one of the modern civ was also using

#

not sure which one

#

civ 4

halcyon trail
#

I'm sure it's not complete but there are dozens of top tier AAA games per year, and this list has zero from the last 7 years

verbal escarp
#

true, that's sad

halcyon trail
#

The first title I picked at random, horizon zero dawn uses lua. Skyrim and Witcher both use custom languages etc.

#

Eh,don't think it's sad, tradeoffs and all that

verbal escarp
#

it's sad because in other industries python is used primarily for scripting, only gaming didn't pick it up

halcyon trail
#

In what other industries is python embedded usually?

verbal escarp
#

video/audio production pipelines are mostly in python

halcyon trail
#

Is it embedded though

verbal escarp
#

sure

flat gazelle
#

the issue with embedding python is that the interpreter is full of globals

halcyon trail
#

That doesn't sound convincing lol

flat gazelle
#

whereas with lua, you can just run it in 4 threads at once without the things breaking

verbal escarp
#

@halcyon trail no, i'm just too lazy to write a list 😄 just look at blender, CAD, inkscape..

halcyon trail
#

"mostly in python" sounds like it's not embedded but just in python

flat gazelle
#

afaik, PUBG uses python for at least parts of it's codebase

halcyon trail
#

Maybe with C parts inside

flat gazelle
#

yup, blender, CAD, inkscape do use python for plugins

#

but well, it is hard to justify python when lua is just overall easier to work with, and you don't really need the python stdlib for scripting in a game engine

halcyon trail
#

Games also tend to be a lot more resource constrained

#

They have to run very fast on consoles

verbal escarp
#

oh, you mean consoles

halcyon trail
#

Well almost all AAA games are one codebase

#

Used for PC + console

verbal escarp
#

i'd say computer games are the least resource constrained of any product

#

(consumer)

flat gazelle
#

that certainly doesn't hold for AAA games at least

halcyon trail
#

Don't quite follow that

finite sparrow
halcyon trail
#

I feel like you're trying hard to say that game devs don't use python for scripting just cause

verbal escarp
#

no, i think game devs don't pick up python because it's harder to write a DSL with it than with lua

halcyon trail
#

Whereas all the game devs I know think that python just doesn't make the best tradeoffs for being used in that way for a game

#

That's not the reason

#

Not the main reason, at any rate

finite sparrow
#

python is a incredibly complex language that is hard to embed into another piece of software. i would wager thats one of the main reasons its not being used much

#

compared to lua which is so much more simple

flat gazelle
#

there are cases where python is used for embedding, such as in cases where you need a more expressive language than lua

#

or even want to use the standard library

finite sparrow
#

but it has to be a case where you need what python offers, because its too hard to embed it "just because" (without proper reason)

flat gazelle
#

but for games, where most of the code is just if then else, lua is a fine language without having to think about the GIL, sharing state, kind of lacking performance and module caching.

halcyon trail
#

Fwiw I hate lua

flat gazelle
#

same, did not enjoy my time with it at all

halcyon trail
#

And I think writing your own language is nuts

finite sparrow
#

i dont like lua, but there are worse options. like a completely custom langauge.

halcyon trail
#

So I would love for python to be more suitable

finite sparrow
flat gazelle
#

there are some merits to a custom DSL for specialised tasks, I know a company had a custom language for some financial things I am not smart enough to understand

verbal escarp
#

well, i think it would be a good fit exactly because python is used as a main scripting language in content production pipelines, so artists who learned python to tinker with their tools could easily get started with in-game scripting

flat gazelle
#

if you know python, you can learn Lua in a week

#

it is a trivial language

finite sparrow
#

maybe its a good fit if you dont care about the effort it is to actuall do that

#

but when you actually have to do it, you will care a lot about how much effort it is

verbal escarp
finite sparrow
#

...

halcyon trail
#

You're saying it would be a good fit because the language would be familiar to some people who aren't the main people writing the game

verbal escarp
#

yeah

#

would make it simpler to communicate

halcyon trail
#

That's not as important as making it easier to hit the performance goals of the game for the people writing it

#

And reducing the technical complexity

verbal escarp
#

you don't render your content with python, it shouldn't be performance-critical

finite sparrow
#

anything that needs to run once per frame, physics or render, is performance critical

#

unless you want to say that physics scripting should be impossible when using embedded python?

halcyon trail
#

The scripting language sucking up a ton of memory still affects your ability to hit performance goals

#

And while those parts aren't as critical they still need to be reasonable, and luajit is way faster then python

finite sparrow
#

oh i have an idea, why dont they embed cython? that would be fast enough (/s)

halcyon trail
#

Basically... Game devs have very good reasons this. Seems like unless you really understand the situation extremely well it makes sense to give experts the benefit of the doubt no?

verbal escarp
#

i've been involved in game development

finite sparrow
#

have you embedded any scripting language in a game?

verbal escarp
#

no

#

we used a custom scripting language based on XML back then 😐

finite sparrow
#

and what was the reasoning?

halcyon trail
#

There's a lot of game studios making independent decisions about what scripting language to use

verbal escarp
#

i only used the language, i wasn't on the coding side

halcyon trail
#

If almost zero AAA games have chosen python in the last 7 years there's probably reasons

verbal escarp
#

so i can't really tell you what the reason was they cooked up that monster

halcyon trail
#

Ah, in the content production pipeline?

verbal escarp
#

yeah

finite sparrow
#

wait are we arguing about scripting for game users or for the developers? i thought we were arguing scripting for endusers.

swift imp
verbal escarp
halcyon trail
#

Scripting internally for developers

#

Game developers

#

Game devs usually embed a higher level language for controlling high level game state/events

verbal escarp
#

like AI

#

quests, dialogues

halcyon trail
#

Things like "when the character reaches this town, if this didnt happen, trigger this cutscenes"

verbal escarp
#

the content devs need to be able to script those events without worrying about low-level stuff

finite sparrow
#

i thought we were arguing about scripting for endusers because there were comparisons made to blender scripting etc.

verbal escarp
#

ohh

#

well, there is a bit of overlap because model, texture artists use those tools to create the game content

#

so i was saying that those people who create the content and the settings/quest/ai devs would have an easier time if they all used the same scripting language

finite sparrow
#

.xkcd 927

neon troutBOT
#
XKCD comic #927

Fortunately, the charging one has been solved now that we've all standardized on mini-USB. Or is it micro-USB? Shit.

2011/7/20 - #927, 'Standards'
verbal escarp
#

lol :p

#

yeah, kind of like that - but we all can agree that python should be that 15th language, right 😉

finite sparrow
#

eh

#

debatable

#

python is actually used a bunch by game developers, for example in the sims and WoT

verbal escarp
#

yeah, we covered that earlier

#

and made the sad discovery that there's only a handful of lonely exceptions in the past decade that dared to use python for internal scripting

finite sparrow
#

or as i meantioned earlier, things like VRchats UDON which are specifically designed for anyone to be able to create content with them

paper echo
#

im going to jump into this conversation without context and declare that python is bad for embedded scripting specifically because its syntax prohibits succinct one-liners, and lua is unequivocally better for the same purpose, especially because a good embedded scripting environment will provide the "standard library" that lua is missing, and because you can already compile moonscript or teal or fennel to lua if you want cute syntax or types or s-expressions

#

however hy wouldn't be bad for scripting, and i suppose theoretically you can use coconut or hy to write embedded scripts, but then the embedded command line itself or whatever will still be a deficient experience

#

also pharo 9 just came out so i am going to jump ship on functional programming and upload my brain to the object oriented meta-matrix, goodbye world

quasi hound
#

because its syntax prohibits succinct one-liners
really? i use these all the time

#

the one thing i hate is no multiline lambdas

paper echo
#

yeah that's a big part of it, also statements like if being not-really-inlinable

#

lisp/scheme is also bad in general i think because if you're typing a script directly into a little gui box or a command line you typically won't have structured editing or parentheses balancing support

halcyon trail
#

Why does succinct one liners matter so dramatically for embedding?

paper echo
#

it depends on what you're embedding in

#

e.g. i think python is fine in freecad, but horrible in sublime

halcyon trail
#

Okay I think you are talking about end-user scripting

paper echo
#

yeah, if your "scripting" purpose is for dev-side scripting then do whatever you want

halcyon trail
#

Which wasn't the kind of scripting we were talking about

paper echo
#

ok back to my salt cave

halcyon trail
#

Also, lisp bad for scripting, hello, emacs

paper echo
#

yes and elisp is cursed

halcyon trail
#

Why would you type your code into a little gui box

#

Yes but it's divine compared to vim script

#

Even compared to JS maybe ;-)

paper echo
#

heh vimscript is silly but it has some interesting ideas

#

namespaced variables, if/endif syntax

halcyon trail
#

elisp has dynamic scope which is certainly "interesting"

#

And obviously it has all the cool lisp stuff

paper echo
#

apparently that was an old lisp thing fwiw so i don't blame it for being really old and kind of a trailblazer

#

ive been casually keeping an eye on the emacs-in-common-lisp project

halcyon trail
#

there's so many things emacs badly needs, and being written in CL isn't one of them...

paper echo
#

heh

verbal escarp
#

"True is instance of the class named True that has methods for control structures (like ifTrue:) that work with closures" okay, pharo lost me

#

next up: zero code

grave jolt
#

oh, Pharo the language

#

i c

#

yeah, smalltalk and friends are mind-blowing

visual shadow
#

programming languages are basically "i hate the code i wrote yesterday" played out on a much larger scale.

woven robin
#

What problems could arise if we could import from folders that didn't have __init__.py in them? Not sure if I have the situation exactly right but you should know what I'm talking about.

unkempt rock
#

They're called "namespace packages"

paper echo
#

in general i recommend not using namespace packages, unless you have a specific reason to use them

#

i kind of wish implicit namespace packages werent entirely implicit, although i know why they did it that way

static bluff
halcyon trail
#

@paper echo i was just thinking that, it seems a bit odd that they get picked up, completely implicitly

verbal escarp
#

okay, so i'm trying to figure out which artifact is suitable for the current platform/version/interpreter to install from pypi

#

this is the versioning issue from yesterday ** 3

#

numpy is compiled, so in the json pypi provides, there's a python_version given, but also a requires_python and tags to be extracted within the filename

#

and none is compatible with each other :p

halcyon trail
#

not totally sure I understand

verbal escarp
#

cp36-cp36m-macosx_10_9_x86_64 are the tags provided in the filename

halcyon trail
#

you're installing numpy or installing something that depends on numpy

verbal escarp
#

i'm working on justuse, which is supposed to help the user giving the proper hash for the file that should work on their system

halcyon trail
#

I mean you can't really solve this problem at the justuse level

#

you need to consider what C shared libraries are already loaded

#

crazy stuff

verbal escarp
#

the installation is solved, relinking works inline

paper echo
halcyon trail
#

dark past meaning like python 2.1?

#

an empty init.py file has been enough since I've been programming python

#

so, prob since 2.6 or so

verbal escarp
#

given the proper hash to the artifact for the specific platform

#

but finding that artifact.. sigh

halcyon trail
#

you can and it will probably work most of the time but the odds that you're accounting for all the things that you technicall should are low

verbal escarp
#

why, it's solved :p

paper echo
#

it goes through all the historical solutions

halcyon trail
#

ah gotcha

#

i was confused

#

err how do you know it's solved, did you test it on every single combination of compilers, system package, setups, etc? 😛

verbal escarp
halcyon trail
#

as Asimov would say

#

So the universe is not quite as you thought it was. You'd better rearrange your beliefs, then. Because you certainly can't rearrange the universe.”

verbal escarp
#

i'll do that when the universe gives me reason to 😉

halcyon trail
#

Seriously though this stuff is awful. If you thought python dependency management was bad, when you start involving the C/C++ ecosystem it's much worse.

verbal escarp
#

my main issue however is the ad-hoc nature of versioning in python exponentiated with platform and distribution specifiers, it's a bloody mess

halcyon trail
#

not quite sure how python versioning is ad hoc

#

but yeah, the problem increases exponentially, as you say

verbal escarp
#

look at the list of PEPs, if that isn't ad-hoc, i don't know..

#

pip "solved" the issue for them by a gigantic list of "compatibility tags"

#

(which we boldly stole)

halcyon trail
#

even if you know that something is compatible with a platform

#

how od you know it's compatible with the specific native packages you have installed

#

not everything is ABI compatible forever

#

well, only libc is really

verbal escarp
#

don't make me cry

halcyon trail
#

Well, that's why if you want to solve this properly you need something that actually fully solves the native side of it

#

i.e., conda or docker

verbal escarp
#

well, can't have docker on my phone

#

and it's damn nice to be able to install a package inline on my phone!

#

still, need to be able to find the right distribution :<

digital laurel
#

Guys

#

I have just stumbled upon this

#

keep womens nuss

#

What does "nuss" mean here?

verbal escarp
#

i wonder what those shoes have to do with python.. they're not even leather

grave jolt
#

@digital laurel please keep off-topic discussions to the off-topic channels

#

!ot

fallen slateBOT
digital laurel
#

Oh, crap

#

@grave jolt I'm sorry, I thought I was at the English server

#

@verbal escarp

#

It was supposed to be an English discussion

grave jolt
#

okay, no worries

silent leaf
#

hi guys

#

i have urgent task and i need help . i have to find a an api or tool or code can let me : take the html of website but without header , menu , advertising … just the main content in python or php someone can help ?

rough leaf
#

Anyone know about the collatz conjecture (in math)? I really want to try to apply it to python and see what happens.

#

Plz respond ASAP

paper echo
#

@silent leaf @rough leaf neither of these questions are on topic for this channel. also i don't think either of them are posed in a way that is answerable. you should read the information in #❓｜how-to-get-help

rough leaf
#

ok

#

thanks I got the channel!!

#

By the way, what is THIS specific channel for?

paper echo
#

@rough leaf if you click the text next to the channel name at the top, you can see the topic. it says:

Discussion on the use cases, implementation and future of the Python programming language including PEPs, advanced language concepts, new releases, the standard library, and the overall design of the language.

rough leaf
#

Ok thanks

dapper grail
#

So, property decorators are a decorator factory for the property that they decorate.(right?) Allowing the dispensed decorator have the name of the attribute. Am I correct in assuming there is nothing stopping me from using a class decorator in a similar fashion but on all the attributes of that class? it looks like I would just have to define call on the iteration of the classes attributes, right? I know there isn't a good code example and I'm sorry.

#

been trying to get an answer for a thing for a while, and it feels like I'm doing this wrong because I cant find examples of it anywhere and I don't know why, maybe not everyone has as much ability to manipulate classes because of environment restrictions?

#

either that or I stumbled into doing something so stupid and dangerous that everybody else knew not to, lol

halcyon trail
#

you can do that, somebody posted about doing that here recently

#

well, maybe I don't understand exactly what it is you want to do

#

what exactly are the "attributes of the class"

#

like if you annotate the class

class Foo:
   x: int
   y: float
dapper grail
#

I am overriding the getters of all of my classes to return insantiated gui objects if the signature of the application making the request contains a renderer property

#

so that I can get auto controls without having to do the layout every single time, I just have to define the format on the container making the call

#

and if I check for not rendered first, the actual data object should only be minimally affectied in both size and processing speed

#

I don't know if it is a good thing or a terrible thing, but I'm making a thing that is really big so I can get some real elbow grease practice in

#

and this is just kinda the way I wanna pursue it

# 
@rendered
class Person(object):
  name : str

@name.Text([control attributes put here]) 
def name(self):
    return self.name
halcyon trail
#

if the signature of the application making the request
How exactly do you envision that part getting passed in?

#

You could override __getattr__ for your class

#

but that method is only getting passed self and the name of the attribute

dapper grail
#

it wont be passed in, the two layers will just be coordinating off the same object, an api that pushes data in and a gui that displays modifies and replies, it knows that it is not an api and the api knows it isnt a window

halcyon trail
#

I think maybe it would help to show the syntax you want, and explain what you want to happen

dapper grail
#

like I said, I dont know explicitly why, it just looks so clean when I do it that wayt and now it is a compulsion

#

I'm just finishing up the initial entry window for the application currently, but the project is intended to be open source so everyone will get it eventually

halcyon trail
#

err I didn't ask "why", just asking what syntax you are trying to achieve, and what you want it to do

dapper grail
#

I already showed you the syntax I was trying to achieve just up there on the person object

#

the name.Text property is a wrapper that checks the signature of where it is currently running, and if it exists in an application that makes gui requests it returns controls with the class attributes instead of just the attributes themselves

#

atm I am using PyuSimpleGui

#

PySimpleGui**

halcyon trail
#

What do you mean by "class attributes instead of the attributes themselves"

dapper grail
#

so, in a console window, if you have a person object, and you call person.name, you get the value of that attribute

halcyon trail
#

what do you mean by "in a console window..." lol sorry but I'm not really following a lot of your terminology

dapper grail
#

but the decotrator would instread return like, [sg.Text(self.name)]

halcyon trail
#

you mean like "in python"

#

"in python, if you have a person object, person.name gives you back a string, which is the person's name" ?

dapper grail
#

well, yes in python, but if my api is written in python and my frontend is written in python why wouldnt I want to use the same classes between them, and why would I want to burden my API with strippiung values out of controls or even thinking about them?

halcyon trail
#

I'm sorry I have no idea what the whole part has to do with what I asked

#

you're qualifying things with "in a console window", when I don't understand how that's related

#

python works a certain way, whether it's running in a console window, in a gui, in a server, etc

dapper grail
#

do you not know what console, command prompt, a running environment is?

halcyon trail
#

I do...

dapper grail
#

yes it does

#

and if you have a python program that has a gui, it needs more than just its attributes to render them onto a window, it needs to know how to display them, if they are editable controls, where on the window to put them

#

I'm just defining containers that tell objects how they want to be displayed and object that know what kind of controls they can adapt into

#

so I don't understand where your disconnect with understanding is

halcyon trail
#

I mean you're very focused on explaining your problem entirely in terms of ideas that are very specific to GUIs, even though what you're asking about is at its core more of a fundamental question about python reflection, etc

#

I don't have much experience with GUIs, so it makes it hard to follow what you want

dapper grail
#

because you literally asked why I was doing it

#

so I told you

halcyon trail
#

I literally didn't though...

dapper grail
#

I misunderstood when you said here, sorry my bad

well, maybe I don't understand exactly what it is you want to do

halcyon trail
#

Yeah. I mean, I'm trying to understand what you want to do. I think generally you'll have a better chance to get help if you can boil your problem down so that it's free of GUI specific terminology, because then it's just a python problem, and everyone here knows python but not everyone knows GUIs

dapper grail
#

legitimately though, I'm just kinda scrambled atm, apologies

halcyon trail
#

but, then again, maybe somebody will come along with more GUI experience and understand what you want immediately, I'm not sure

# 
class Person(object):
  name : str

class name:
    def Text(f):
        def f2(self):
            return [Text(f(self))]
        return f2

@name.Text
def name(self):
    return self.name
#

like I'm quite sure this is not what you want, but this just something I'm throwing out to show what I've understood so far

#

in this case, name.Text is a decorator that just calls the passed function, and passes the result to Text (which I guess is a class somewhere as well, so that's a bit ambiguous), and wraps in a list of size 1 (for some reason), and then returns it 🤷‍♂️

dapper grail
#

nah, I'm just trying to clean up my code by abusing the thing the property decorator does, let me try somethin real quick

# 
class render_property_displays:
    def __init__(self, func):
        self.func = func

    def Text(self, *args, **kwargs) -> Text:
        

class render_class(object):
    def __init__(self, cls):
        if not cls.rendered:
            cls()
        else:
            for name in cls.__dict__:  
                setattr(cls, name, render_property_displays(getattr(cls, name))

I get about this far and then cant think of how to do the dunder call for the functions to make the decorators return with the name of the function passed in

#

and the Text def there is just going to return the cotrols, that bit of code is irrelevant to the scenario

#

I think I am just a little wrapped up in the idea though and made a mistake somewhere

halcyon trail
#

is render_class supposed to be a metaclass? Or what exactly

dapper grail
#

it is a decorator factory

#

or its supposed to be, I'm still new to that bit

#

I think I mixed up init and call and made some poor assumptions on the code example

#

the simplest way I can describe it is I want a class decorator that builds property-like decorators for the attributes of a class which have wrappers on the get to return a contextualized display rather than a passing a flat value

#

and the contextualized display is irrelevant as previously discovered

#

I know it is bad form generally to just change the data out from under the variable like that because you can unexpectedly return incorrect data types, but I control both ends and all sides of the environment and I'll likely just make everything return as list anyways

#

so if I have

@decorator_factory
class Person:
  name : str=''

I want to be able to do this:

@name.wrapper(kwargs)
def name(self):
  return self.name

and then I can just define transforms to the get of the attribute based on the kwargs of the wrapper

#

or have various wrappers defined

#

I know is all boils down to callable so it should be possible, its just an iterated property decorator that I can self define. I can accomplish basically what I want by overloading property, but I dont really want to do that if I plan on releasing the end result as open source

halcyon trail
#

I understand the idea of a class decorator that applies another decorator to all its properties

#

I don't understand what's happening in the second bit of code

#

are you just saying that decorator gets applied to the method name ?

#

*a decorator

dapper grail
#

The property decorator itself has methods on it like .Text or .Input which will let it conditionally change its return value

#

I dont expect to have multiple per property

#

It doesnt have to have the name of the property realistically, I just think it is cool and am trying to step up my game

halcyon trail
#

sorry still really unclear what you want

dapper grail
#

Fair enough, thanks for listening

lean scaffold
#

E

dusty verge
#

umm

#

Thank you, whoever mod just deleted those

terse hill
#

Not sure if advanced or not. But if python lists are containers of references stored in an array, do the lists still have spatial locality? If you're retrieving a slice of numbers that fits into a cache line, does python internally try to line up the data in the list so that there is SOME form of spatial locality when stored in memory?

spark magnet
terse hill
#

So getting 4 numbers in a list of numbers could easily be 4 compulsory cache misses

spark magnet
#

You might want to use numpy arrays instead

terse hill
#

Because all 4 are not potentially stored sequentially. Got it

#

Im guessing if im interviewing in python, an interviewer would not care about this level of detail, but good to know

#

Thanks @spark magnet

spark magnet
severe lichen
#

Actually, there MAY be cache locality.

#

Since first 256 ints are pre-loaded in memory, they probably SHOULD be relatively close to each other.
I do not know if additional allocations follow some pattern, or CPython tries to fit them in blocks.

spark magnet
#

Yes, but my point was that lists themselves do nothing to ensure locality

halcyon trail
#

I'm not sure how hard cpython actually works to help with locality tbh

#

the JVM is kind of famous for this kind of stuff

#

I actually just saw some pretty crazy benchmarks in Kotlin; kotlin has "sequences" which are like python iterators, i.e. lazy, and it also has the notion of "iterables" which are eager

#

so if you have a sequence s and an iterable like say a List, i, if you do s.map { ... }. filter { ... } it does the maping and filtering point wise

#

with i.map { ... }. filter { ... } it creates entire intermediate collections

#

the crazy thing is that the performance of these is about the same, even for very large collections

raven ridge
# severe lichen Since first 256 ints are pre-loaded in memory, they probably SHOULD be relativel...

Since first 256 ints are pre-loaded in memory, they probably SHOULD be relatively close to each other.
There is no guarantee of that whatsoever. They're each independently allocated. https://github.com/python/cpython/blob/59242431991794064824cf2ab70886367613f29e/Objects/longobject.c#L5753-L5761 Nothing guarantees that the allocator returns objects that have any spatial locality when you call it repeatedly.

fallen slateBOT
#

Objects/longobject.c lines 5753 to 5761

PyLongObject *v = _PyLong_New(1);
if (!v) {
    return -1;
}

Py_SET_SIZE(v, size);
v->ob_digit[0] = (digit)abs(ival);

interp->small_ints[i] = v;```
unkempt rock
#

need opinions, anything that i should change?

image0.jpg
severe lichen
#

TIL.

#

I wonder how close True and False are in memory.

#

Probably the same independent allocation.

raven ridge
fallen slateBOT
#

Objects/boolobject.c lines 200 to 210

/* The objects representing bool values False and True */

struct _longobject _Py_FalseStruct = {
    PyVarObject_HEAD_INIT(&PyBool_Type, 0)
    { 0 }
};

struct _longobject _Py_TrueStruct = {
    PyVarObject_HEAD_INIT(&PyBool_Type, 1)
    { 1 }
};```
severe lichen
#

Interesting

raven ridge
#

though I suppose there's no guarantee that the linker puts those adjacent in memory.

severe lichen
#

I wonder how much experience with C do you need to meaningfully understand CPython code.

raven ridge
#

I'd say the Python code is generally written to be easily read, compared to a lot of projects of similar size.

halcyon trail
#

Two globals declared one after another I'm almost positive are going to end up adjacent in memory

#

The linker doesn't get to control that, it's not loading individual variables

orchid portal
#

struct_longobject _Py_Falst

raven ridge
raven ridge
#

wait, not BSS - they'll be in the .data segment - but still, serialized in some order chosen by the linker.

raven ridge
#

in any event, regardless of whether it's the compiler or linker's responsibility, some phase of building the executable decides what order to place the globals in, and while it will probably choose to keep two globals of the same size and alignment adjacent to each other in memory, it's under no obligation to.

halcyon trail
#

are you talking about the static linker or the shared linker, now?

#

At any rate, we definitely agree that it's not under any obligation to, but I do suspect that it will be

# 
#include <iostream>

int x = 0;
int y = 0;

int main() {
    std::cerr << &x << "\n" << &y;
}

just tried this on godbolt, it printed out

0x601178
0x60117c

so, exactly 4 bytes apart as you'd expect

raven ridge
halcyon trail
#

if you aren't linking to static libraries, the static linker is never used 🙂

#

but yeah, it should not matter

iron perch
#

How would I know if a variable is a reference of another variable or just a copy

raven ridge
#

a .so isn't just a collection of individual object files like a .a is, it's partially statically linked. References that can be resolved statically are resolved, and the remaining undefined references are left for the dynamic loader to handle.

raven ridge
raven ridge
iron perch
raven ridge
#

are you asking me if it's possible to check if two names are references to the same object without checking if two names are references to the same object?

iron perch
#

I mean is there a rule or law about what happens

raven ridge
#

if you do a = b, then a and b are references to the same object

iron perch
#

But what about
iwi = self.smth
iwi.append(69)

raven ridge
#

same thing.

#

= always makes the left hand side a reference to the right hand side.

iron perch
#

But why

        white_turn = self.is_white_turn
        if white_turn:
            white_turn = False
            return
        white_turn = True
#

It didn't changed self.is_white_turn

raven ridge
#

right, because you used = again

iron perch
raven ridge
#

after py white_turn = self.is_white_turn white_turn is a reference to the same object as self.is_white_turn

After py white_turn = False white_turn is a reference to the same object as False

#

the only thing that = does is make the left hand side a reference to the right hand side. After the first =, it's a reference to self.is_white_turn, whatever that might be set to. After the second =, it's a reference to either True or False - one of hundreds of references to those objects, most likely.

#

!e ```py
x = True
import sys
print(sys.getrefcount(x))

fallen slateBOT
#

@raven ridge :white_check_mark: Your eval job has completed with return code 0.

56
raven ridge
#

^ there's 56 references to True in this freshly started interpreter.

iron perch
#

Dies of confusion

raven ridge
#

you're expecting inconsistent behavior from =. You're expecting that the first time you use = in that function, it makes the left hand side refer to the same object as the right hand side, and the second time you use = in that function, instead of making the left hand side refer to a different object, it modifies the object that the left hand side is referring to and gives it a new value.

#

it doesn't work that way, = always does exactly the same thing

#

Seriously, watch Ned's talk on this, or read the slides. It will help you understand it much better.

native flame
#

when you do white_turn = False it forgets all about self.is_white_turn, it only knows about False now

#

if you had the two labels self.is_white_turn and white_turn on the same box, white_turn = False moves one of those labels to another box, leaving the previous label unaffected

iron perch
#

Oh that makes sense

raven ridge
#

Let's imagine that there are no references to either True or False anywhere in the program. Assuming that in your __init__ you're doing self.white_turn = True, now self.white_turn is a reference to True, and there is 1 reference to True and 0 references to False.
Then in your function to change whose turn it is, you do: py white_turn = self.is_white_turn and now there are two references to True (white_turn and self.white_turn) and 0 references to False.
Then later in your function to change whose turn it is, you do: py if white_turn: white_turn = False and now there is one reference to True (self.white_turn) and 1 reference to False (white_turn)
Then you do py return and now there is one reference to True (self.white_turn) and 0 references to False (because white_turn is a local variable that disappears after the function ends)

iron perch
#

I get it now py_guido

raven ridge
#

nice. But seriously, watch Ned's talk, or read the slides. You'll get it better, and it will cement the knowledge. 🙂

verbal escarp
#

@halcyon trail i thought about what you mentioned the other day, the notion of public and private dependencies, and i must say i really like that idea

#

i don't think pip/conda are in direct competition with justuse, it's more complementary

#

i can well imagine that people start a project using pip and classical installs, with an environment and at some point realize they might want private dependencies to be more self-contained for deployment on different platforms without resorting to docker

#

or that a user runs into a situation where pip/conda would just give up because dependencies couldn't be resolved without conflicts

#

i don't think justuse can or should try to replace environments and other traditional deployment methods, but rather complement them

#

the two solve different problems and i think that's good

stuck valley
#

Hi. I'm a programmer interested in getting deep into the horrible internals of Python.

#

Does python 3.9 have an equivalent of python 2.x's __metaclass__ global variable?

verbal escarp
#

why do you need to get into python 2?

stuck valley
#

I'm not. The guide to metaclasses I'm using is a few years old.

verbal escarp
#

ugh..

#

use a more up-to-date guide?

stuck valley
#

I can't find one.

#

I'm a skilled programmer (7 years), but I've only been using python for three.

verbal escarp
#

there are worlds between 3.9 and 2 when it comes to internals

stuck valley
#

I'm aware of that. That's why I asked if 3.9 had an equivalent. Has it been removed entirely?

verbal escarp
#

i haven't worked with metaclasses much so far, but given that the class system was revamped, that's hardly surprising they also changed those implementation details

stuck valley
#

Is there a workaround?

#

I've already read that question and done some research.

verbal escarp
#

well, what exactly are you trying to do?

native flame
#

what did the global __metaclass__ do in python 2?

#

if you just want the metaclass of some class A, type(A) should give you that

stuck valley
native flame
#

oh

spark magnet
stuck valley
#

I hope there's a workaround.

spark magnet
stuck valley
verbal escarp
#

read the source 😄

spark magnet
#

OK, you don't need a workaround then, because you don't have a problem. That's an old-style class feature, and old-style classes are gone.

stuck valley
#

Specifically, my latest project is converting this line in my .pythonrc to 3.9.

__metaclass__=lambda*_:type('',[],{})
#

I'm well-aware of how bytecode works.

spark magnet
stuck valley
#

That line breaks most classes.

verbal escarp
#

???

spark magnet
#

why do you want that?

stuck valley
#

It's for fun.

spark magnet
#

OK, then the workaround is to do a different fun thing 🙂

stuck valley
#

If you can't answer that, I do have another question about metaclasses. It is also a fun thing.

verbal escarp
#

then sorry for python3 to spoil your fun?

stuck valley
spark magnet
#

there are plenty of reasons to not use python2!

verbal escarp
stuck valley
# 
class math(metaclass=lambda*q:__import__(q[0])):0

With this python 3.9 program, how can I convert it into a single call to type(), in the form

math=type(...)

?

#

I apologise for bad practice. This is just a learning experience.

verbal escarp
#

i haven't seen so appalling code in a long time

#

that's just horrible

stuck valley
#

Would you like some more examples of bad code? I'm happy to oblige.
By the way, I normally write clean code.

peak spoke
#

You'd use your lambda thing in place of type

stuck valley
#

Could you elaborate?

verbal escarp
stuck valley
# 
import sys
import marshal
import types

file=sys.argv[1]

magic=b'\x61\x0d\x0d\x0a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

f=open(file)
t=f.read()
f.close()

code=compile(t,'','exec')

new=types.CodeType(code.co_argcount,code.co_posonlyargcount,code.co_kwonlyargcount,code.co_nlocals,code.co_stacksize,code.co_flags,code.co_code,code.co_consts,code.co_names,code.co_varnames,'','',0,b'')

f=open('.'.join(file.split('.')[:-1])+'.cpython-'+str(sys.version_info.major)+str(sys.version_info.minor)+'.pyc','wb+')
f.write(magic+marshal.dumps(new))
f.close()

python compiler that produces 16% smaller files than the official compiler

# 
#The virgin:
print('foobar')

#The chad
import sys
sys.stdout.write('foobar')

#The megachad
__import__('sys').stdout.write('foobar')

#The gigachad
__builtins__.__import__('sys').stdout.write('foobar')

#The ultrachad
getattr(__builtins__,'__import__')('sys').stdout.write('foobar')

#The überchad
getattr(globals().get([*globals()][-1]),'__import__')('sys').stdout.write('foobar')

#The exachad
getattr(globals().get([*globals()][-1]),''.join(map(lambda _:chr(0x2fdfe9cb7f0dba6fdf>>7*_&127),range(10))))('sys').stdout.write('foobar')
# 
text=lambda depth,data,start='',end='\n\n',seed=0:(k:=__import__,sys:=k('sys'),random:=k('random'),(lambda r=random.seed(seed or None),head=[data[i-depth:i]for i in range(len(data))],output=list(start or data[:depth]): (lambda a=any(((output.append((arr:=[data[j]for j in[i for i in range(len(head))if head[i]==''.join(output)[-depth:]]],random.choice((head,arr)[len(
arr)>0]))[1]),1)[end in output]for _ in range(sys.maxsize))):''.join(output))())())[3]
native flame
verbal escarp
#

lmao

stuck valley
#

I'm new here. I didn't realise that channel existed.

verbal escarp
#

although, most of those aren't even esoteric or horrible

spark magnet
#

all of those are esoteric and horrible 🙂

#

ok, not all. the first is fine 🙂

verbal escarp
#

nah, just bad

stuck valley
#

The first one is weak. I'm improving it right now.

peak spoke
stuck valley
#

I'm going to work on a ton of optimisations to nested codetypes.

stuck valley
verbal escarp
# 
sys.stdout.write('foobar')``` that conjures up bad java memories
peak spoke
stuck valley
#

I want the class to take the name of the module to import from the name of the variable. Is this possible?

verbal escarp
#

you can simply replace the module by the class

#

or vice versa

#

in sys.modules

stuck valley
#

?

stuck valley
#

I don't understand. Code example?

dusty verge
#

@stuck valley

#

Idk if this is what you want to do

#

Btw this is (probably) very bad practice

prime estuary
#

It's not that bad, many packages will do this, especially before __getattr__() was supported as a module function. You could do it to make the module callable to avoid from blah import Blah repetitiveness, or lazily import things only when actually used.

swift imp
#

What is this module __getattr__() stuff? Never heard of it.

valid rose
#

__getattr__ is called when you try to access an attribute that doesn't exist, atleast for classes

swift imp
#

That I understand

#

For modules it seems to be some weird mechanism for creating warnings and raising exceptions

halcyon trail
#

i think getattr is called for any attribute

#

not just those that exist

swift imp
#

no

halcyon trail
#

sorry, not just those that don't exist

swift imp
#

__getattribute__ is called no matter what

valid rose
#

that would be __getattribute__

swift imp
#

reading the PEP, it seemed like previously, some really smart people were subclassing typing.ModuleType and setting __class__ of a module to that custom subclass to handle this stuff.

halcyon trail
#

ah I'm thinking of get attribute

#

my bad

valid rose
#

__getattr__ is the guy who is called when the attribute doesn't exist, and __getattribute__ is called for any attribute

swift imp
#

__getattribute__ is one I've never played with, probably should so I can learn how to handle the descriptor protocol correctly.

paper echo
swift imp
#

Subclassing types.ModuleType looks like fun

#

Wonder what kind of shenanigans could be had with that

verbal escarp
#

i'm also researching a way to realize module-level variable guards aka module properties, where that might be of use

#

just to mention two "shenanigans" 🙂

swift imp
slow creek
#

I need help

grave jolt
#

also, tell what exactly you need help with

#

(btw, today is 2021, not 2020)

quasi hound
#

Do you know what a screenshot is

white nexus
grave jolt
#

since 01.01.2021 🙂

verbal escarp
# 
import datetime
def seconds_since_beginning_of_year(dt: datetime) -> int:
        return (dt - datetime.datetime(dt.year, 1, 1)).total_seconds()

print(seconds_since_beginning_of_year(datetime.datetime.now()))
#

all hail github copilot 🙂

charred pilot
#

that was generated by it?

verbal escarp
#

yeah, but i had to sift through a few nonsensical suggestions first ^^

#

can't just blindly accept suggestions

gusty marsh
#

Hmm, it uses github code right? does it abide the licenses pithink

verbal escarp
#

there is nothing to abide by, it generates code and doesn't (at least shouldn't) blindly copy code

gusty marsh
#

hmm,i mean wouldn't you need to license it even if you copy and edit the code, anyway getting offtopic

grave jolt
#

I think we just need to wait for it to be settled in court

charred pilot
#

that's probably going to be a shitshow

verbal escarp
#

there is pretty interesting reading material on that, although we'll probably get a lot more once it hits the main stage

verbal escarp
#

maybe there's a need in the future to have a license validator plugin of sorts to check before releasing stuff

#

maybe not

static bluff
#

Question about Cython

#

I know that Cython is only faster than Python in the ways that it's faster. One of the key ways to leverage the speed of Cython is to use only Cythonic types, to avoid overhead on conversion from the pythonic equivalents when possible

#

I wanted to know if defining a class in Cython with Python syntax is considered 'a cythonic type'

winter ridge
#

any gentleman can help me out ?

static bluff
#

I just can't see to get my head wrapped around this performance thing

static bluff
#

XD

winter ridge
#

😛

verbal escarp
#

@static bluff just test it and look at the numbers if you're really interested

winter ridge
#

i have a small script that i havent done it myself i bought it tho 🙂 it has a small glitch and the developer not fixing it , i was hoping if some1 can find a solution for it

verbal escarp
#

we can speculate (i would say it's cythonic) but other than refering to docs, there is only looking at the code itself and doing benchmarks

verbal escarp
#

why would you "buy" a script?

#

that sounds fishy

grave jolt
winter ridge
#

dont wanna spread the code , the dev will be made 😛

#

mad

grave jolt
#

I guess you can hire someone else (or the previous developer), but this is certainly not the place for that

#

also, wrong channel

winter ridge
#

ok do u know any place i can look at?

verbal escarp
#

wrong platform even

#

gitcoin would come to mind, but not sure

grave jolt
winter ridge
#

ok , thanks

paper echo
static bluff
#

Rolls off the tongue though 😛

halcyon trail
#

someone who knows cython really well: cythonista

slow acorn
#

how do you know you're a Python expert?

unkempt rock
slow acorn
#

waits for an excellent answer

unkempt rock
slow acorn
#

what if you need help but you're too proud to ask and eventually figure it out anyways?

#

that's my problem sometimes...

deft pagoda
#

probably not exactly on topic for this channel

slow acorn
#

ok, how about memory management in CPython, then?

#

Which C file is this handled in?

slow acorn
#

So how do the arenas work?

#

anyone ever have a highly successful usage of metaclasses? what made it successful? would you use the solution again?

raven ridge
coarse urchin
#

Alright I hope it's the right place. I don't know python well enough, but I wanted to ask some implementation details. How do you store variables? Like, for instance, an int, double, and what about reference types?

I assumed there might be some type union mechanism which would store all tiny data structures as value types (like int, etc.) and others as reference types unconditionally allocating them on heap. But in reality, I don't know. Is there a short answer to this?

visual shadow
#

python abstracts memory management away from you. so, you have zero control over this. but short answer? everything is an object in python, everything goes on the heap more or less

coarse urchin
#

Right, but does it mean that every variable I work with is essentially a reference type?

flat gazelle
#

yes

visual shadow
#

yes

coarse urchin
#

So, a list of ints is in fact a list of references to random places all over your memory?