#off-topic

1 messages · Page 3 of 1

junior narwhal
#

long story short I messed up this CLI and also Hatch by originally doing a DMG installer because there is no way to add things to PATH

#

(technically Hatch installers haven't been released since I was waiting on the next minor)

#

perhaps this won't work if you're not in a GUI, lmk

silk jungle
# 
gcc104:~ ichard26$ installer -pkg ddev-5.0.1.56.pkg -target CurrentUserHomeDirectory
installer: Package name is Datadog Agent integration developer tool
installer: Installing at base path /Users/ichard26
installer: The install failed..

literally first time using a .pkg file on macOS

#

if this needs root access then I can't help

junior narwhal
#

can you try without the last flag?

#

I think it does, I chose to copy the way Go does their installer by dumping everything in a single global path and then shipping a file under /etc/paths.d that points to the path

silk jungle
#

Ah, yeah I won't be able to help then. Sorry!

junior narwhal
#

literally first time using a .pkg file on macOS
interesting, are they not as common as I thought for CLI tools? as I mentioned Go has one and also it's recommended for the AWS CLI

silk jungle
#

nah, it's just that I don't have a macOS machine myself. I just happen to have access to one

#

I'm a happy linux user :)

junior narwhal
#

oh okay, what distribution do you use?

silk jungle
#

Haven't switched from Ubuntu. There may be something better but I've already reached my "if it ain't broken, don't fix it" mentality.

boreal bramble
#

I'm with you on going with the boring choice of Ubuntu :)

#

every once in a while I want to try openSUSE Tumbleweed but switching would be a significant effort

junior narwhal
#

I'm still looking for someone on macOS with root access that can try that installer I mentioned above!

boreal bramble
junior narwhal
boreal bramble
#

it's installed but with permissions that don't allow regular users to execute it

# 
sudo ls -al /usr/local
total 0
drwxr-xr-x   3 root  wheel   96 Sep 17 22:18 .
drwxr-xr-x@ 11 root  wheel  352 Aug  2 09:44 ..
drwxr-xr-x   3 root  wheel   96 Sep 17 21:19 ddev

sudo ls -al /usr/local/ddev
total 0
drwxr-xr-x  3 root  wheel  96 Sep 17 21:19 .
drwxr-xr-x  3 root  wheel  96 Sep 17 22:18 ..
drwxr-xr-x  3 root  wheel  96 Sep 17 21:19 bin

sudo ls -al /usr/local/ddev/bin
total 16040
drwxr-xr-x  3 root  wheel       96 Sep 17 21:19 .
drwxr-xr-x  3 root  wheel       96 Sep 17 21:19 ..
-rwxrwx---  1 root  wheel  8211888 Sep 17 21:19 ddev

sudo ls -al /usr/local/ddev/bin/ddev
-rwxrwx---  1 root  wheel  8211888 Sep 17 21:19 /usr/local/ddev/bin/ddev
junior narwhal
#

thanks! I'll have to investigate how to fix that...

junior narwhal
boreal bramble
#

running /usr/local/ddev/bin/ddev --version as root does work in case you wondered

#

no clue if it's properly added to PATH, not sure if there's a good way to check this in a CI environment

junior narwhal
junior narwhal
boreal bramble
#

oh no, I wasn't answering your question about Go

#

welp, I'm not having much luck installing go at all...

# 
curl https://go.dev/dl/go1.21.1.darwin-arm64.pkg -o go1.21.1.darwin-arm64.pkg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    73  100    73    0     0    364      0 --:--:-- --:--:-- --:--:--   368
sudo installer -pkg go1.21.1.darwin-arm64.pkg -target /
installer: Error - the package path specified was invalid: 'go1.21.1.darwin-arm64.pkg'.
junior narwhal
#

curl -L ...

boreal bramble
#

ah, didn't know it's a redirect

#

should I expect the binary to be at /usr/local/go/bin/go

junior narwhal
#

Yes

boreal bramble
#

I can run Go as a regular user then

junior narwhal
#

interesting

#

unfortunately I cannot find their release pipeline scripts, they might not be public

#

as such, I cannot find out what they put inside their distribution.xml file

#

oh wait I think I found it! I did actually put this:

<pkg-ref id="com.datadoghq.ddev.pkg" auth="Root" packageIdentifier="com.datadoghq.ddev">

#

I'll remove root and try again

boreal bramble
# junior narwhal as such, I cannot find out what they put inside their `distribution.xml` file 
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<!--
 Copyright 2023 The Go Authors. All rights reserved.
 Use of this source code is governed by a BSD-style
 license that can be found in the LICENSE file.
-->
<installer-gui-script minSpecVersion="1">
    <title>Go</title>
    <background mime-type="image/png" file="bg-light.png" alignment="left"/>
    <background-darkAqua mime-type="image/png" file="bg-dark.png" alignment="left"/>
    <options hostArchitectures="arm64" customize="never" allow-external-scripts="no"/>
    <domains enable_localSystem="true"/>
    <installation-check script="installCheck();"/>
    <script>
    function installCheck() {
      if (!(system.compareVersions(system.version.ProductVersion, '11') &gt;= 0)) {
        my.result.title = 'Unable to install';
        my.result.message = 'Go requires macOS 11 or later.';
        my.result.type = 'Fatal';
        return false;
      }
      if (system.files.fileExistsAtPath('/usr/local/go/bin/go')) {
        my.result.title = 'Previous Installation Detected';
        my.result.message = 'A previous installation of Go exists at /usr/local/go. This installer will remove the previous installation prior to installing. Please back up any data before proceeding.';
        my.result.type = 'Warning';
        return false;
      }
      return true;
    }
  </script>
    <choices-outline>
        <line choice="org.golang.go.choice"/>
    </choices-outline>
    <choice id="org.golang.go.choice" title="Go">
        <pkg-ref id="org.golang.go.pkg"/>
    </choice>
    <pkg-ref id="org.golang.go.pkg" auth="Root" packageIdentifier="org.golang.go" version="go1.21.1" installKBytes="212211">#org.golang.go.pkg</pkg-ref>
    <pkg-ref id="org.golang.go.pkg">
        <bundle-version/>
    </pkg-ref>
</installer-gui-script>
#

it's part of the pkg file so you can just extract it

#

i.e. pkgutil --expand go1.21.1.darwin-arm64.pkg out

#

was easier to find once I knew what I can code search for :)

junior narwhal
boreal bramble
#

nope, didn't help

#

it's really just the last part where the perms are wrong

#

ddev binary has no read and execute perms

#

but all of the parent directories do

#

meanwhile with go, go binary has read and execute perms

#

everything else seems the same

junior narwhal
#

I'm wondering if perhaps the manifest is okay but rather the executable has some sort of process that go puts it through where the permissions are correct like may be as simple chmod

#

do you think chmod +x <EXE> before I package it would fix this issue?

boreal bramble
#

worth a try

#

though I would probably go with +rx

#

they seem to do something like that in postinstall

#

so you'll probably just want that

junior narwhal
#

it's running, about 8 minutes to go. that makes a lot of sense because I do set permissions for standalone executables earlier in the process but I don't after I merge the Apple ones into a universal binary

junior narwhal
boreal bramble
#

works 👍

junior narwhal
boreal bramble
#

np

junior narwhal
#

now I have to talk to Ee and apologize for not being thorough in my research, and ask for 20 more minutes of his free time to make certificates for this type of installer 😅

boreal bramble
#

Cirrus CI allows you to play with shell directly as well but they now limit resources available to you for free so it's more efficient to run a short build rather than SSH into a machine :P

#

unless you need to test ARM64, then Cirrus is really your only option at current time

junior narwhal
#

oh that's cool

#

yeah for ARM I've just been cross compiling for macOS and Linux

boreal bramble
#

if you go with the ngrok thing, you'll probably want to not use special characters in SSH_PASS as I recall I had some issues with that when I first tried it

dreamy hatch
west basin
#

this looks like something Poetry should warn against at least

#

wonder how many packages are affected

onyx spindle
vast wren
onyx spindle
junior narwhal
robust sandal
junior narwhal
#

the next release is going to be so awesome. a smooth installation experience for everyone and the ability to easily install various Python versions

ionic tulip
#

@sand rapids does scriv have any way to get the next version from something like setuptools_scm ?

sand rapids
#

scriv collect --version=$(setuptools_scm something?)

ionic tulip
#

that works out - also wondering if perhaps a cli command could be used as default version source like (python -m setuptools_scm --strip-dev)

sand rapids
#

you mean, we could add a setting to scriv that is the command to run to get the version? That could work.

#

I like the nature of that: nothing hard-coded, but easy to configure the connections you want. The docs could include example commands for other tools.

ionic tulip
#

or a entrypoint to call

sand rapids
#

yes

#

We already have a setting: version = literal: pyproject.toml: project.version It could be extended:

version = command: setuptools_scm something

or

version = call: setuptools_scm.main(strip_dev=True)
#

(maybe full python there is weird)

ionic tulip
#

lets not go there, - if itsa command, its a string and fair game, if its a "entrypint" it should be pre-configred for scriv

sand rapids
#

what's an example of an entrypoint? A named entrypoint, or module.callable ?

ionic tulip
#

aka there should be a entrypoint like "scriv.get_version" and importlib metadata shouldb e used to fetch a name of that group, no parameters shouldbe passed

#

lets not go down the path of random pythonpath modification and module lookup, its a can of killer worms

sand rapids
ionic tulip
#

@sand rapids i think you misunderstood me - i would add a scriv.get_version entrypoint to setuptools_scm like setuptools_scm = setuptools_scm._integration.scriv:return_version_for_scriv - and then people could configure that in their scriv config

sand rapids
#

ok, so it would be an entrypoint from setuptools_scm, that's fine if it doesn't exist yet.

#

i think setuptools_scm is well ahead of the curve in terms of exporting many entrypoints, so I'm not sure I would limit the callability to only entrypoints.

ionic tulip
#

@sand rapids i an understand the sentiment, unfortunately that opens up the failure vector of having to manage pythonpath ^^ based on year of suffering its not something a tool wants tobe in charge of

sand rapids
ionic tulip
#

@sand rapids also - aware of any premade github action/workflow to render scriv into a release proposal mr?

sand rapids
ionic tulip
#

@vernal hornet is there any ruff online playground to test a example - i want to report a bug with PT022

vernal hornet
ionic tulip
vernal hornet
# ionic tulip seems like i cant make it run the autofixer there

We might've broken autofix in there, we mostly use this to inspect the syntax tree. You're welcome to just send me a file and the command you ran! You can run with --isolated to avoid picking up any configuration (e.g., ruff check file.py --select PT --isolated --fix).

ionic tulip
vernal hornet
vernal hornet
normal ibex
shell oracle
# normal ibex the pypi blog post on malware had me curious. do folks know of cases in the past...

I haven't seen anything similar since we've started up (March 2023).
The bulk majority of packages indicated in that blog post would've been initial uploads of malicious code.
As a result, the pinned hash likely would've been irrelevant-- if something utilized those 'newly uploaded' malicious packages, they'd be pinned to the malicious hash no matter what.

However, that isn't to say it can't happen. In the past, https://pypi.org/project/pystyle/ this project contained a malicious payload authored by the package maintainers themselves.
Pinning to a previous version/hash would've prevented the new version from being installed, and subsequently, dependent libraries likely would've subverted this attack entirely if developers were performing due dilligence in reviewing changes to their dependancy tree. (Which can be a bit tedious so... I'm aware of the implications there.)

#

We monitor packages indiscriminately of the 'reputation' of the package though, and I'm fairly confident that's unilateral across all orgs. So I'm hopeful that, with the current security net we have in place in the trusted reporter/malware detection sphere, a combination of hash pinning and early detection can help avert things like this in the future.

stoic mural
junior narwhal
#

does anyone here have a contact at GitHub/Microsoft that is able to prioritize issues?

ionic tulip
boreal bramble
junior narwhal
boreal bramble
#

but less likely that people here run into that considering this is a Python server

junior narwhal
#

I've always wanted to try C#, to my naïve outsider eyes it appears to be a better Java

ionic tulip
junior narwhal
#

meh, I wouldn't consider the situation icky. Microsoft has went through a bunch of ship of Theseus events and so what they are now does not at all reflect the past and I don't judge them as such

#

perhaps my perception is off but I consider them to be very developer friendly now, much more than Google or Amazon

arctic copper
#

My perception about windows, from a developeping point of view, made a 180 turn for the better.

Thinking about C# being microsoft actually doesn‘t cause a bad reaction, compared to the case of it being from Google/facebook/… (for multiple reasons, not just „is big bad wolf corporation“).

They still have my antipathy for their UX/UI stuff though 😉

vast wren
#

WSL2 might be a better dev environment than macOS tbh

onyx spindle
#

personally, I am so used to linux by now, I can't even think about going back

arctic copper
#

With some stuff to be honest, windows can be actually less problematic than linux. (But this is hard to generalise with a lot of distros that vary a lot in that regard)

Emphasis on some. Generally what probabyl happend is switch to linux, from windows, became less important, but most likely the need to switch to windows, from linux, is unchanged.

boreal bramble
#

I really dislike Java, I don't like the philosophy used for its design

#

a lot of it is, however, fixed by Kotlin

#

things like the lack of operator overloading or syntax sugar for properties (getter/setter)

#

C# has all those niceties

#

there are some weirdnesses in the stdlib and there's a bunch of, perhaps icky, concepts that come from how it tries to be versatile with syntax features but well

#

one little thing about its syntax that I find superior in comparison to how Python does it (not that I can really draw a fair comparison when Python had these for way longer) is multi-line string syntax:

unknown.png
#

the little thing of "Any whitespace to the left of the closing double-quotes will be removed from the string literal"

timber sphinx
#

Oh, that's nice in not having to wrap it in textwrap.dedent()

junior narwhal
junior narwhal
#

@frank shore Coming from the comment in #hatch, I didn't realize you work at Snowflake now! As I assume the most senior and experienced Python expert there can I please put in a small request? 🙂

#

I'm honestly not exaggerating, this happens so often it's a meme on my team

#

If you could please (if you agree) kindly explain that libraries should not have such tight bounds. It makes sense perhaps for other languages, but it does not work for Python. Maybe when it was created it was by folks who were coming from other ecosystems and they set the policy

normal ibex
#

this^ has literally gotten several projects at openai to drop snowflake

frank shore
#

Yeah, I can't publicly promise anything, but I totally appreciate the feedback and will share it internally. Off the record I am sympathetic to the ask 😄

normal ibex
#

btw, if you do feel it necessary to continue to manage the dependencies of your users, maybe one compromise solution is to add a "paternalistic-dependency-management" extra to snowflake-connector-python et al that has all the same deps but with the pinned specifiers. then you could recommend in your docs that users default to running pip install snowflake-connector-python[paternalistic-dependency-management]

(extra name to be workshopped, of course 😉 )

frank shore
#

Hmm, interesting idea. I think I get what you're suggesting, but wondering if you have any real world examples to point to?

hexed briar
#

Airflow uses a constraints file for achieving something similar. I know I've suggested this (=extra with stricter constraints) on pip's issue tracker but don't have a link handy for it either. 😅

frank shore
#

No worries, and thanks for the Airflow pointer.

normal ibex
lusty folio
#

😐 I keep libraries relatively clamped even in library code unless the library in question has a strict policy on breaking changes that allows me to clamp into the future. If my code breaks because a dependency upgrade was incompatible, that's my fault for allowing the version prior to testing it. I feel like the python ecosystem and people expecting a version upgrade prior to it being tested is kindof backwards.

frank shore
#

@normal ibex I'll have to play with that, but on first blush it looks like an interesting way to kind of solve both problems of generally wanting to avoid upper pinning, but allow it for, as @lusty folio says, clamping things down to match your testing matrix.

normal ibex
# lusty folio 😐 I keep libraries relatively clamped even in library code unless the library i...

yes, but in snowflake's case they a) refuse to actually pay attention to the dependency in questions's breaking change policy (e.g. cryptography allows ±3 major versions), b) don't actually test and release upgrades in a timely manner.

if you're on top of your releases and dependency upgrades (and ideally don't have many dependencies), paternalism is much more workable*. if you're not, then you end up preventing folks from picking up security upgrades and causing paying customers to stop using you for easily preventable reasons.

i would love if we had https://discuss.python.org/t/proposal-overrides-for-installers/23666 so it's possible for me, a user, to control my environment, but packaging folks don't seem super receptive to that one.

*although i still find myself agreeing with basically everything in https://iscinumpy.dev/post/bound-version-constraints/ (the section mentioning "rapid updates" and rich is relevant here)

lusty folio
#

(I've seen the linked article before, but I don't find it persuasive at all)

#

From the very beginning of the article's TLDR:

Capping dependencies has long term negative effects, especially for libraries, and should never be taken lightly. A library is not installed in isolation; it has to live with other libraries in a shared environment.

Assumes people aren't using venvs (not my problem) or that they would rather run untested code to make things work together than havent been tested to run together (I won't support this)

mild pollen
#

Why does it assume people aren't using venvs?

lusty folio
#

see the or there along with this section of the referenced statement:

it has to live with other libraries in a shared environment.

#

it doesn't have to unless at least one of the two sides of that or is true

normal ibex
lusty folio
#

And if you view your license that contains "provided without waranty" blah blah means it's not your problem that you didn't test it, I don't want to use your code. I as a maintainer am responsible for what I publish, whether legally so or not.

lusty folio
west basin
#

the article discusses soft capping and argues that soft capping a major or even a minor version is just as good as not soft capping at all (the semver "contract" is broken all too often and it complicates testing pre-releases, software matrices, etc.). If a library wants to advertise the versions of a dependency it has been tested against, this is the wrong mechanism to do so

lusty folio
#

the article discusses soft capping and argues that soft capping a major or even a minor version is just as good as not soft capping at all
Agreed, but I disagree with the conclusion strongly. The logical conclusion isn't to softcap assuming some level of compatability will be be retained for some amount of time (unless you're willing to make exceptions for notable ones like cryptoraphy that have promised very specifics) but to restrict to exactly what is tested.

It would be completely unreasonable to not have a minimum version based on what you need, why would you allow a maximum that you also dont know has what you need?

Some of the arguments in the article approach reasonableness, but don't reach it. "unmaintained software that isn't kept up to changes?" That's a risk, fork it and maintain it if you need it.

west basin
#

I think, in the majority of cases, that would result in hard pinning. Libraries will at most test against the earliest version they support, not every version in between. It would also result in an incredible amount of churn, having to constantly bump your dependencies and re-release, and PyPI I suspect is simply not equipped to deal with the increased load. Doing such a thing would also be environmentally morally....suspect. The conclusion is a pragmatic one given the current constraints of the system. I don't know if it's the most logical or the most ideal one (that depends on what you are trying to optimise really), but I think compatibility guarantees are going to have to come through some other means and not version capping

lusty folio
#
  • It's not that hard to have a testing matrix with a few versions.
  • I'm not building libraries that require rapid feature updates from dependencies, nor would I depend on a library that was in rapid development without pinning as that would be irresponsible too.
  • Environmental impact is going to be measurably worse on many other things including those suggested in the article to help you "detect" upcoming incompatibilities that you no longer prevent proactively by just sticking to what you've tested. I have no reason to run CI on a recurring cron job, I've tested every version I support, and this doesn't feel like a reasonable point to bring into this discussion.
#

and to this:

but I think compatibility guarantees are going to have to come through some other means and not version capping

There's no other mechanism than to test, and then support once you can confirm it to be compatible. The article, as well as things people posted here and in other references all acknowledge that versions can't accurately convey what will and won't break a consumer of a library, but people are reluctant to face that head on.

west basin
#

What I mean is that we need to break the bump version and republish loop which would be extremely costly, much more so for binary packages. Currently all of a package’s metadata is derived from the METADATA file in the wheel. Uploaded wheels are immutable. I don’t disagree with the core premise, I’m just not sure how the information ought to be distributed.

lusty folio
#

Funny enough, I somewhat disagree with the premise of wheels themselves, for each of security, performance, and enviromental reasons.

#

I think the largest, frequently depended on either directly or transitively, "build time is costly" common libraries all actually provide strong deprecation policies that can be relied upon though, so it would be more reasonable to use those to inform the window under what I said with

unless you're willing to make exceptions for notable ones like cryptoraphy that have promised very specifics)

also just noticing the typo there, but oh well

#

I also think that some of this is people coping with solvers having historically been worse, and the lack of things other dependency management tools have, and deciding the risk is worth less annoyance, because that's the dichotomy they are presented currently.

#

https://www.anaconda.com/blog/patching-source-code-to-conda-build-recipes conda has a way for people to add patches to libraries, and also has editable metadata.

Now, editable metadata definitely has some security considerations, but if people were pinning (and especially if they were also generating hashes while doing that), those considerations don't come into play. The defensive posture of check the dependency update, then publish prevents those issues too.

lusty folio
# west basin What I mean is that we need to break the bump version and republish loop which w...

oh, and to add to this, a wheel is just a zip file. If you're only editing metadata, you can grab your old artifact, unpack, modify metadata, repack, upload new version. Some people won't like this due to liking to be able to verify it matches a build artifact, but that can probably be resolved in some way with an extra metadata key that says it's only a metadata change and what version it was changed from to compare the innards.

junior narwhal
vast wren
#

restrictive version pins are almost always the wrong thing to do for software that is meant to be distributed to others

frank shore
# vast wren restrictive version pins are almost always the wrong thing to do for software th...

Personally I agree. It's a hard lift to convince sometimes though especially for corporate software because of the testing and compatibility uncertainties. Whereas in OSS, I can just let my users tell me if one of my dependencies gets updated in an incompatible way, laissez-faire doesn't go over as well when considering customer experience. I'm sympathetic to both points of view.

vast wren
#

Eh

#

People who want to release like fully tested combinations should just not publish to PyPI or similiar things

#

They should ship an artifact with minimal dependencies on the system that contains all their tested deps

#

Python doesn’t make this super easy of course, and sometimes the product also makes this hard. Like from a past job I believe the aws cli is now shipped via a self updating bundle through the aws website. But boto3 is shipped thorough PyPI because of the different use cases

#

At the end of the day the only way to ensure you’re shipping tested combinations is to use exact pins, and if you’re doing that you should just ship a bundle. If you’re not doing exact pins then you’re not shipping tested combinations anyways and you should just not use restrictive pins

mild pollen
#

I don't think the AWS CLI is self-updating (I wish!).

lusty folio
# junior narwhal security teams wouldn't allow this FYI, the hash would always be based on the ar...

What I'm saying here, this would be a new release still (new version, new hash, etc) but with such a key there providing that additional info, someone could unpack both wheels and compare everything except the metadata (perhaps even automated and enforced on the backend) as identical.

Someone would still need to update the versions they accept, but doing so wouldn't require people building binaries for wheels to rebuild the binaries, and that extra bit of data gives people an answer to "I didn't see the build system emit this, where'd it come from? (the answer being a prior build, and which to confirm it against).

#

there's no negative security impact to this, it's purely something that can already be done, but having an additional metadata key that lets people know what you did and give them a means to confirm it

frank shore
#

Maybe there's a way to think of version compatibility more expansively, like "here's what's tested" vs. an expansive uncapped maximum. Let users decide explicitly what they want. I.e. either a well tested and guaranteed compatibility or they can take on the extra responsibility of letting versions float up. That gets to @normal ibex 's paternalistic suggestion (although I'd like to find a better word for that).

normal ibex
frank shore
lusty folio
# frank shore Maybe there's a way to think of version compatibility more expansively, like "he...

The problem here is that this only really works for one level in the chain. You can't really use extras conditionally passed down. I don't think hard capping is as problematic as people have said it is, but if this was a desired thing to express separately, I think it needs standardizing so that a user just tells pip(or other) "give me dependencies based on maintainer confidence" or "give me whatever isn't forbidden"

vast wren
#

If everyone hardcapped as you suggest you'd run into a lot more version conflicts

frank shore
#

It's dependencies all the way down

lusty folio
#

If people hardcapped, but bumped metadata on a reasonable time frame, it changes from:

We don't know if the next version is going to break us, and therefore our users, users can complain if it breaks them.

to

We don't know if the next version is going to break us, depending on how heavily we lean on this dependency and trust in the stable maintainership of it, we can do anything from read the changelog for anything breaking, to fully test our usage, then bump versions. Users can help inform us they need this and contribute to it.

The latter slows down propagation of updates, but reduces the risk of a problematic update. In terms of conflicts, this really shouldn't be problematic if people don't overly restrict both high and low simultaneously and update the upper bound with some level of regularity.

vast wren
#

except if multiple things all depend on X, and all have different ideas of what version of X is needed

lusty folio
#

And? If they have incompatible needs of X then they shouldn't be used together.

#

python doesn't have a mechanism for this outside of vendoring and replacing the entire namespace yourself

vast wren
#

you have no idea if they're incompatible though, you're just defensively assuming it might not be

lusty folio
#

if people don't overly restrict both high and low simultaneously and update the upper bound with some level of regularity.

#

I'm trusting maintainers to be responsible rather than assume they won't be, and that if they aren't, people shouldn't rely on them anyhow.

vast wren
#

I'm expecting humans to be humans, and have wildly different cadences for updates

#

Also tbh, it's kind of crummy to just take up more resources on PyPI just to bump versions like that

#

and I don't think you're actually acheving anything anyways. Unless you'er using == pins then you don't know if any specific set of versions are going to break you or not, you're just randomly guessing

lusty folio
#

If each version in a range is known to be good for the behavior you directly rely on, I think that's the "reasonable level" and perfectly expressible with ranges (and exclusions as needed)

vast wren
#

version ranges aren't frozen in time though, new versions can be released within those ranges

lusty folio
#

You're right, I'm assuming a versioning system with monotonically increasing versions here when thinking about it that I shouldn't be, more to handle later.

#

As for the pypi resources issue, you're right, and this is largely only an issue because of wheels and immutable metadata.

I think there's potentially a way to address all of this (without the security concerns some people might be having with me bringing immutable metadata up again), but I'll need some time to formulate my thoughts on it a bit more. I don't like the current lack of a clear better way to convey what versions are known to be okay vs suspected to be.

#

advisory metadata which exists in addition to the actual sdist/wheels could help if it was editable and was limited in what it could do such that it being editable was not an issue, for instance if there was an editable field for "known good versions" (the key name would need to be better). This would only be usable to further restrict to those versions, but it's then at an ecosystem level for solvers to optionally use (preferably by user indicating to), and doesn't widen versions or add dependencies in something editable.

#

(in a sort of opposite function to yank being a maintainer pulling their own version, this would be indicating a version of a dependency with a stronger confidence from the person depending on it)

#

I'll come back to this with more thought on it, I think there's room for improvement, but it's clear there's a lot that needs considering still.

hexed briar
hexed briar
lusty folio
# hexed briar You seem to be thinking of something akin to Conda's https://docs.conda.io/proje...

Not really. I was thinking less of editing any existing metadata here which would exist in the wheel/sdist and instead a new bit that would be in the simple index API (much like yanked is) that can further limit versions from requirements, but some of the thought about conda not really having this issue due to that, as well as what I linked aboive about being able to add source patches to build recipes did lead to thinking about this as a possible solution.

Because of existing promises that uploaded versions are immutable, any metadata which could be "Tacked on" needs to neccessarily never increase the things it could cause to be installed, or it changes the existing security promises of pypi.

Having it be an optionally used field that's recognized by tools allows maintainers to go "hey, we know/think this version is good", users that want to stick to what maintainers have confidence in, while maintainers allow more then has an ecosystem specifiable way to be done that doesn't introduce a way to add things that could be installed by editing an old version.

I think there's also a way to do better than wheels in terms of resources used (disk, bandwidth) by package indexes, but that's really not directly related, and just something I've been tinkering with due to my own dissatisfaction with wheels for a few reasons.

junior narwhal
dreamy hatch
#

great news! although paid features:

With today’s launch, our macOS larger runners will be priced at $0.16/minute for XL and $0.12/minute for large.

lusty folio
#

more reasons for me to get my ass in gear on finishing up the thing I'm working on to hopefully serve as a long term replacement to wheel in spare time. Trying to make it easier to (correctly, and from a security focused approach) have binary resources be reusable. only so many hours in the day though.

#

this is a large enough thing that I want to have a working implementation though first, everything from building it, to showing how it could be securely validated, to how it can be efficiently served up by an index

junior narwhal
robust sandal
#

The only downside is the missing locking. Though admittedly, that's not a minus vs. nox. I'll have to compare. The cookiecutter does test the Hatch CLI.

#

Does Hatch support other backends (like scikit-build-core, meson-python, etc) yet?

junior narwhal
robust sandal
#

(I keep mentally comparing it to PDM/Poetry, rather than tox/nox)

#

I know at least one scikit-hep dev has switched to Hatch

#

Okay, I'll look into it soonish 🙂

ionic tulip
#

i'd like to gather some opinions - i'd lik to make a github workflow avaliable that proposes releases on release branches and applies tools like towncrier/scriv/regendoc + provides a way to have maintainers review/trigger the deploy (similar to what we currently have in pytest)

im curious what people consider as must haves to make em switch

junior narwhal
#

that sounds interesting but I would have to hear more details

ionic tulip
#

data gathering for a release pipeline proposes

mild pollen
#

I just saw a car with a PEP 621 license plate. 😄

sand rapids
real dove
#

I wonder - which part of the code is responsible for generating all these exes in the Scripts folder? Is there any easy way to force a regeneration on that ?

onyx sphinx
real dove
#

Thank you 🤗

somber badge
#

byebye

junior narwhal
#

lol https://discuss.python.org/t/37502/44

Discussions on Python.org
User experience with porting off setup.py

This. Please, for whatever is precious to you call these features by their names not PEP numbers. It is like discussion of two Reformed Theologians “Because of Mt 22:23 and of course consider 1Co 4:2, and you cannot miss Rev 8:6, it is obvious that you are an idiot.” I hate it when they do it (and yes, I am a practising Christian) and we do...

fierce horizon
dreamy hatch
#

the point is to not use jargon (PEP numbers) but use titles when referring to them

boreal bramble
#

But you can always lookup the PEP if you need to know the details, the PEP itself doesn't indicate what the feature is if you don't have them all memorized.

#

"project.metadata table in pyproject.toml" is self-explanatory if you're vaguely familiar with the concept, "PEP 621" is not

#

There's also another point that may not necessarily be as relevant to proposal discussions specifically but makes sense overall - the PEPs aren't supposed to be a (user-facing) documentation of a feature, that's what Python docs or Python packaging docs are for.

dreamy hatch
#

yep, and generally PEPs are the historical proposals, specs are meant to be the living docs

frank shore
#

FWIW, I find this Chrome-ish shortcut to be super helpful: https://peps.python.org/pep-%s/ which I bind to pep. So search bar pep 404 leads me to the right place pretty quickly. Huge thank you to whoever implemented the redirect from "short numbers" to the actual URLs.

Also, TBH, I never anticipated so many PEPs back when I created PEP 1, or how difficult it really is to keep a mapping of PEP numbers to topics in your head. I was just trying to implement the lightest weight derivative of Internet RFCs that I could think of to help Guido make decisions without having to read millithreads of mailing list emails!

dreamy hatch
normal ibex
vapid kite
#

@hexed briar (if the ping isn't too annoying) did I hallucinate that somewhere there's a furo setting that makes it use the full browser width for content rather than heavily padding the sidebars? I can't seem to find it, so I'm doubting myself -- obviously if not I can fiddle with CSS

#

I'm trying to make this page feel less cramped, especially now that I added that .. sidebar

Screenshot_2023-11-02_at_4.33.30_PM.png
#

(Also hi of course!)

hexed briar
#

Definitely hallucination. Having a fixed content width that respects the UX guidelines written by anyone who cares about accessibility (which all say that fixed width is what works best on the web) was an intentional design choice. Not having an easy opt-out is also intentional.

#

That's... the longest .. sidebar I have ever seen.

vapid kite
#

Interesting, TIL.

#

I assume similarly there's no direct setting making that fixed width simply be larger?

hexed briar
#

Yup.

vapid kite
#

Got it, ok thanks for confirming!

normal ibex
junior narwhal
#

I usually elide the parentheses but I understand the rationale for the default of true for consistency's sake and I think I should go with that default

ionic tulip
kind moon
junior narwhal
mild pollen
#

I think it generally makes sense to have a blank line between any two compound statements.

dreamy hatch
#

my colleague favours adding a newline between consecutive ifs and I can see the benefit. we looked for a lint rule but couldn't find one 🙂

junior narwhal
#

nice, I'm not the only one that was looking! if it isn't too much trouble would you mind commenting on that issue? (or thumbs up)

dreamy hatch
#

✅ 👍

junior narwhal
#

almost finished now, I spent all weekend upgrading Ruff on Hatch to the latest version and enabling most things

#

literally all weekend

frank shore
normal ibex
frank shore
ionic tulip
junior narwhal
junior narwhal
junior narwhal
#

at work I have to build a new endpoint and have the liberty of doing whatever I want so I assessed the landscape and chose Starlette. it is such an awesome framework and well designed, I think I like it just as much as Flask and with time it might surpass that for me

#

I looked into FastAPI first of course but the docs were incomprehensible to me

#

another one stood out and actually I wrote my PoC with Litestar but two people I trust very much advised me against that and linked me some stuff showing that the original maintainer is not... great. that person did step down but for now I'm avoiding it even though it appears to be (and technically is from testing) excellent

fierce horizon
#

My partner is also not too pleased with FastAPI. It has a nice golden path if you use it with manually written Pydantic models, but it’s not super interoperable if you e.g. use SQLAlchemy models

arctic copper
dreamy hatch
#

works for me

arctic copper
#

Weird, going through discord works, but using safari on my phone gives me
Request canceled from preloader

Need to check from firefox.

arctic copper
#

Seems to be a weird iOS safari thing, just needed to clear the cache. Seems to be a know problem with certain sites sometimes.

dreamy hatch
junior narwhal
lusty folio
#

definitely been liking ruff's formatting, and this is yet another thing I'm happy to see

dreamy hatch
#

Looks a bit like JavaScript

junior narwhal
#

saves a bunch of space

frank shore
#

A great tool just keeps getting better

junior narwhal
#

soon it will be available directly with hatch fmt! I just have to do two more things before I can start the release process

#

today and tomorrow there is a hackathon at work and I am exclusively working on Hatch so it will happen soon

frank shore
#

Sounds like a dream job @junior narwhal !

junior narwhal
#

for these two days, yes haha

hexed briar
#

To be fair, that's Black's preview style that Ruff has implemented -- Black has a backwards compatibility promise that means it'll only change in the release in the first release next year.

normal ibex
frank shore
#

Maybe it's time to think about democratizing default style choices rather than just leaving it to Black to drive?

onyx spindle
#

I think Python is too far gone at this point. people have opinions, stdlib has opinions (it being PEP 8), black has opinions... the only solution I would see is for Python to have a "built in" formatter (like cargo fmt for rust). black dominated the market because it reduced bikeshedding to line length setting...

shadow zealot
#

FWIW a large portion of stdlib never really follows PEP 8

onyx spindle
#

that too

#

I mean, tbh, most people wouldn't mind if stdlib just got nuked with formatting in one commit

frank shore
#

The stdlib waaaay predates PEP 8 or any autoformatting. The hitch has always been that the diff/PR will be huge, of limited benefit, and makes backporting fixes much more difficult.

onyx spindle
#

well, stdlib would have to be formatted with "the formatting tool", to avoid "I won't use the formatting tool, because even stdlib doesn't do it"

vast wren
livid plover
#

let me know what you guys think! criticism appreciated!

#

i want to devote a lot of time to that project and start regularly refactoring a number of recognized python projects, including pipx that i recently forked (outdated type hinting + some code can be rewritten)

#

hit me up if you want to help me, i am just a jobless poor student and eager to learn the hard way 🚀

#

the product vision is generally stated in the linked repo's readme.

#

i will compile a list of projects that autorefine will be meant to run on so that collecting custom rules that need LibCST codemod implementation is easy. plenty of large projects outside the Python's stdlib need a sophisticated and easy-to-review, easy-to-justify, easy-to-deliver and easy-to-incorporate grand refactoring. the project aims to semi-automate this and open the room for experienced developers to deliver the best practices with ease and keep the projects they maintain up-to-date with the newest Python trends with an optimal effort by centralizing the linting and typechecking rules from living quality assurance projects such as Ruff. not only will it bring a business optimization, but i also think it will be helpful in propagating education of the Python's ecosystem.

livid plover
#

as i am thinking about it now, i will reach out to some software development authorities to ask for feedback and help formulate business requirement for the project - what else could make them use the tool in their projects. feel invited!

ionic tulip
dreamy hatch
frank shore
vast wren
#

No project should have their own style

#

there's so much more useful things to spend effort on

arctic copper
#

Counter point: But there are so many people that struggle to read code if it is in a certain style.

I have real problems understanding bigger lists or dicts if they are formatted with black for example.

And something that I do to help my attention is adding colored regions via comments (and an vscode extension). The comments alone would make other people mad for example. So other peoples style is bad for me and my style is bad for other people

onyx spindle
#

is requires.txt file still used or is that some old standard not used anymore? from what I have found it's only for old egg format, but I just wanted to make sure

junior narwhal
onyx spindle
#

great

hexed briar
normal ibex
#

it has different connotations! i didn't want to go as far as presume controversiality 😄

jaunty marlin
#

"Maybe controversial" lol

#

The double negative is def confusing

west basin
#

I like it

#

bit of confusion never hurt anyone

hexed kindle
#

It's always confusing that the release page for minor CPython releases never contains a changelog of any sorts.

livid plover
kind moon
#

Wait

livid plover
#

without the import __modern_types__ line, you would get an error, either that types aren't subscriptable (Python 3.8) or that types.GenericAlias doesn't implement the | operator (Python 3.9).
for Python 3.10+, the library is useless.

#

also, typing_extensions doesn't often have to be imported at runtime, many times it serves for typing purposes only (but not always, since it lets you do variadic generics etc.).

#

the replacement is needed for pydantic models to work with PEP 585 and PEP 604, as stated in the readme.

lusty folio
#

that sounds like an issue with pydantic either not specifying an appropriate minimum version of python or implementing what's needed themselves if they are intended to work with that.

livid plover
#

but what I am trying to do is to make the developer able to drop the deprecated types from their code already in 3.8 so that they don't have to migrate to it later (by later I mean about 2 years from now)

#

which is just a convenient time saver and a fighter against legacy-keeping

#

best practices are important :)

lusty folio
#

monkeypatching the standard library and all of the issues that can entail to avoid just running pyupgrade in 2 years doesn't seem worth it to me, but if it's worth it to you then go for it I guess.

#

removing the deprecated types is an already automatable process for people when they are ready to drop 3.8 and 3.9

livid plover
lusty folio
livid plover
#

I'm wondering if the least surprise principle is the reason I'm still seeing class Foo(object): and setup.pys without pyproject.toml nowadays.

lusty folio
#

?

livid plover
#

in projects that support Python 3.8+

livid plover
lusty folio
#

that's a far bit different from thinking patching the standard library rather than using the officially supported method is uneccessarily hackish and could lead to surprises

#

I have some code that supports 3.8-3.12, and there's a ._compat file that conditionally defines Dict (upper case D) as either dict in 3.10+ or imports as appropriate.

#

you can get consistency across your code base without having to have surprising outcomes

livid plover
#

the monkeypatching doesn't seem to be that deep, it's just minimal to achieve the goal (wrapping typing._eval_type in-place, marking typing.Dict and analogous as instantiable, dynamically adding a typing._GenericAlias base and overwriting the typing._GenericAlias.__subclasscheck__() implementation). the rest is just some frame stack-related magic.

livid plover
#

as I am thinking about it, I've decided to even more reduce the monkeypatching, since I found a way to do so.

#

thank you @lusty folio! you've convinced me to limit the monkeypatching much more.

#

the library will now only overwrite typing.ForwardRef._evaluate and base replacements on ast.

#

this could be even a feature for typing_extensions.

#

this will be a much, much safer approach. even a trustable one.

lusty folio
#

Hmm, I don't think that should need replacing, but it's been a bit since I looked at deciding how to handle 3.8-3.11 spanning support for typing (naturally included 3.12 without issue)

#

If you get it to just needing that, and the purpose is for pydantic, it may be possible to get pydantic to change how it evaluates forward refs to support this without modifying typing.ForwardRef

livid plover
#

i will make a PR

#

(and an issue beforehand, describing the entire thing :P)

#

the AST-based solution looks very simple. i'm testing it now.

lusty folio
#

Not quite what I meant. ForwardRef can come from standard lib, you'd want a backportable function to evaluate a forward ref that didn't rely on ._evaluate. You can see an example of this in discord.py's source, handling edge cases in forward refs with a 3.8 min python

#

(at least if the goal is entirely no modifying stdlib for it)

livid plover
#

in the AST-based solution, there is only one modification to the stdlib: wrapping ForwardRef._evaluate with a function that reads through the expression and finds names to overwrite in the global and local namespace with subscriptable/or-able generic aliases.

#

everything based on a simple registry {class_without_pep_585_or_604: generic_alias_instance}

#

a very safe and unhackish solution.

#

it starts from

image.png
#

and you can extend it

#

yay! it works!

livid plover
silk jungle
#

Discord is dying dumpsterfire

onyx spindle
#

yup

junior narwhal
#

what happened?

livid plover
#

you can think of it as a netsplit but in discord haha

junior narwhal
#

oh okay thanks! the one time I'm not actually online there is an outage, perfect lol

kind moon
#

For me it took them out of my list entirely and gave me a single icon at the bottom that said

IMG_2007.png
junior narwhal
#

I got my first hate email lol someone sent to my open source email (I only list that on GitHub) in Russian saying to off myself. they must have encountered a really bad bug in something I wrote 🤣

onyx spindle
kind moon
#

I haven’t gotten anything yet

#

I worry about the day my projects actually get used

arctic copper
dreamy hatch
kind moon
#

We - the staff at https://discord.gg/python - Are putting together some links to good resources for creating/maintaining/publishing Python packages.
We've got the official tutorial front and center, and we've added links to the docs for the four backends listed in it.
If anyone has some other useful resources, we'd happily accept suggestions.

dreamy hatch
junior narwhal
#

Happy Holidays everyone!

ef05bd1f-afe3-4bb2-aeab-f246d071d2df.jpg
silk jungle
#

switched to a new laptop. building Python 3.11.7 from source fails to use the system OpenSSL on a nearly fresh install of Ubuntu 22.04.03... dumpsterfire

#

and yes, I do have libssl-dev installed

#

does Python 3.11 support OpenSSL 3, I thought it did...?

dreamy hatch
#

yes. I'm on macOS and have 3.0.11:

❯ python3.11
Python 3.11.7 (v3.11.7:fa7a6f2303, Dec  4 2023, 15:22:56) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
re, sys, time)
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 3.0.11 19 Sep 2023'
>>>
silk jungle
#

yes

#

well, passing ac_cv_working_openssl_hashlib=yes ac_cv_working_openssl_ssl=yes to bypass the failing API check seems to work ¯_(ツ)_/¯

#

looks like the configure cache was stale, oops :P

silk jungle
#

I've finally done it, I'm reading the arch wiki to debug a system issue on Ubuntu.

#

My wireless adapter is erroring out randomly.

junior narwhal
#

Happy New Year 🎆

junior narwhal
mild pollen
#

I'd expect IDEs to be smart enough to recognize __all__ elements as identifiers.

undone sinew
#

Haven't seen that pattern before, but it makes sense. Any time you use a variable rather than a constant string you benefit from automatic checking of the value.

lusty folio
#

I've only done a pattern similar to that in extreme cases where the exported symbols may vary significantly by system, and tend to prefer a fixed set of re-exported names. That's a recipe for re-exporting a name unintentionally, and (at work) we consider anything included in __all__ as a supported interface subject to breaking changes. exported names should be intentional. Also VSC and most vim configurations I've seen are each smart enough to recognize strings as identifiers there, and let you know if something isn't actually defined there, so I don't see the benefits of a variable being "so it gets checked", only so that you define it one place and update it in one place.

frank shore
junior narwhal
#

I'm creating a Dockerfile for a Windows image and I must say I dislike PowerShell very much. Its quoting semantics are bananas

mild pollen
#

Really? What's wrong with them?

ionic tulip
silk jungle
#

what—

image.png
#

huh, so if I call /opt/python3.11.7/bin/python3 -m virtualenv venv the sys.path is right

#

how much fun

junior narwhal
#

When you install a C++ workload in the Visual Studio installer, it always installs 32-bit, x86-hosted, native and cross compiler tools to build x86 and x64 code.
64-bit is a flag... so painful

hexed kindle
#

Only if you cross compile, no?

junior narwhal
hexed kindle
#

Ah, for setting up the env. Yeah.

frail oxide
#

Thought someone in here might appreciate this ahah

suss_-_Discord_-_01-16_2.png
robust sandal
junior narwhal
#

yeah there was a thread on the forum about his post from last year

#

I view it as unproductive to discuss because a much better use of time is to simply land features and fix bugs

#

I don't participate in Discourse as much as I used to and it has been paying off, Hatch has been much better maintained

shadow zealot
#

The comment below by N has a point (although also has some of its on issues on takes). And the author calling name on CPython (“deletionist”) doesn’t help. So entitled…

fierce horizon
# junior narwhal I view it as unproductive to discuss because a much better use of time is to sim...

It’s unproductive. I think not because code is so much more important than discussion and identifying pain points, but because the author doesn’t understand why things are how they are:

PEP 723 solves a very niche problem: dependency management for single-file programs. Improving life for one-off things and messy code was apparently more important to the packaging community than any other improvements for big projects.
Like … Chris, you’re writing about packaging. You’re now a part of the packaging community. Why are you criticizing yourself?

He needs to get that there’s no big shadowy committee that decides what to focus on. If he has an idea that he thinks would benefit the majority, he can start discussing it with others, write a good enough PEP that it clearly benefits the majority, and do (or get someone to do) a prototype proving it.

PEP 723 got accepted because that happened, not because “the packaging community” thinks lockfiles are not important.

cosmic dock
# fierce horizon It’s unproductive. I think not because code is so much more important than discu...

i'm pretty sure the point of that was an example, like "why does pep723 exist?" to talk about some sort of a larger issue

Why are you criticizing yourself?
this seems like an illogical extrapolation? just because you criticize a part of a larger community/organization you're in doesnt mean you're criticizing yourself?

He needs to get that there’s no big shadowy committee that decides what to focus on.
i feel like he knows that, i think his main complaint was that there should be some sort of concrete group where the entire community rallies around a "One True Packaging Tool"

(although i do agree he makes some bad points)

cosmic dock
vast wren
#

eh

#

everyone always has opinions on the one simple thing people should do to fix packaging

#

weird how they almost never seem to put in the effort to get that one simple thing to happen

lusty folio
#

I mean, the only thing I can go off of from that that remained cohesive throughout was that they think there should be one tool that the entire ecosystem uses, and that seems both like a non-starter. They didn't even consider conda and the reasons it exists and is used.

vast wren
#

Sadly we had one tool

#

it was called easy_install

ionic tulip
junior narwhal
#

is there a recommended way to get a filesystem path exactly as stored (e.g. with the proper casing) without resolving symbolic links?

#

within a virtual environment the path that sysconfig returns is in all caps for some reason

image.png
junior narwhal
junior narwhal
#

found on X, I disagree with this but funny variant of the midwit meme nonetheless

GEdZvMiasAA4Frb.jpg
#

tool.hatch.build.targets.wheel is verbose... but is better than not being explicit and guessing what ends up in each type of distribution!

onyx spindle
#

If pip was pluggable, i would agree with the meme, but otherwise, I am in the middle and don't see a reason to move from it

onyx sphinx
#

If pip-tools did multiplatform lock files I'd probably move from the middle

junior narwhal
#

I've gotten really unlucky the past few years with software wiping critical parts of my system. First at work a while back there was a bug where upgrading a product wiped system32 on Windows which I hit during QA, then I upgraded software on a server that I use to seed torrents of books and such that removed everything I had there for nearly a decade, then today I go to fix a bug and my Python installation is wiped https://github.com/pypa/virtualenv/pull/2686

I'm gonna start using Docker for everything 😅

dreamy hatch
livid plover
#

uhm... why doesn't pip have a stable programmatic API? droid_sad

#

(i came to whine)

cosmic dock
livid plover
cosmic dock
#

lmao

cosmic dock
#

also calling an anecdote a datapoint is the nerdiest thing ever and i love it

livid plover
junior narwhal
cosmic dock
onyx spindle
cosmic dock
#

its hard™️

#
  • you can alreasdy check what packages you have
livid plover
cosmic dock
#

lmao

hexed briar
onyx spindle
junior narwhal
livid plover
#

🚀 🚀 🚀

#

you have no idea how grateful i am now

#

THANKS!!!!!

hexed briar
hexed briar
livid plover
#

i knew you are an expert

#

i am a fan of your open source contributions!

livid plover
onyx spindle
livid plover
#

can't, too coupled haha

#

i'll check --report and --dry-run in a moment

#

after checking them first time, i must have been thinking they didn't do what i wanted

#

seems like an oversight of me

#

i didn't dig deep enough ig

livid plover
#

@onyx spindle look how easily I could have solved it 😔 ^

#

i wish i asked here before

#

the only thing that concerns me yet is the fact it downloads wheels for caching and --no-cache-dir doesn't disable that.

#

i imagine these are needed on the filesystem to get metadata, no?

#

but i would rather not install 1+ GB of data just to resolve torch with no intention to install it

#

hm.

onyx spindle
#

torch is bad in terms of metadata

hexed briar
livid plover
#

yaaaaaay 🔥 🚀 🚀 🚀

#

can't wait!

dreamy hatch
#

that's a first, got a request to connect on LinkedIn asking me to check a (week-old) PEP 541 request... I'm not even a PyPA member let alone mod/admin. I wonder if they're spamming other people with PSF things on their profile?

Hey Hugo, I filed a request on PyPi support and its pretty important to me and my team, can you please have a look? https://github.com/pypi/support/issues/<snip> I'm reaching out because I don't see these request being actively handled, sorry for the bother and thanks!

hexed briar
junior narwhal
#

do speakers at PyCon have to purchase a ticket? I thought not but the email says that I must register which entails purchasing a ticket

#

I heard from two previous speakers that indeed they always had to pay, so I'll do that now

#

that is a bit of an odd policy I think, from what I heard about other conferences speakers always get a ticket by default (except the rare academic-type conference)

dreamy hatch
junior narwhal
#

I believe work will cover the cost and if not I can but I just assumed speakers would be guaranteed a ticket since attendance is mandatory

marsh kite
#

It depends conference to conference. I believe Pycon asks speakers to pay for a ticket since it funds financial aid for others to attend, and frequently those speakers are sponsored by companies

junior narwhal
onyx spindle
shadow zealot
#

This exists for a super long time actually, not sure why they chose this specific time to announce it

clear wigeon
pseudo bison
#

Hii .. Have you resolved this issue .. or how is it going on with devpi generally .. ?

junior narwhal
robust sandal
#

Haha, I got a little scared by that one too. But yes, underscores are safe. They aren’t hidden by the file system.

junior narwhal
#

PATH has always worked for me on Windows in every environment that I've ever worked in, I never modify the registry

#

installation is often just an unpacking step and PATH update

boreal bramble
#

you don't need Python in PATH for py launcher to detect it which can be useful if you have multiple python versions since the executable on Windows is not versioned. Of course, proper ordering of PATH entries also solves this but 🤷‍♂️

#

(unless you use Chocolatey packages which have python3.x shims)

shadow zealot
#

I think the registry is automatically modified when you install Python. I never click the add Python to PATH checkbox.

#

Don’t recall if there’s a choice to not modify registry off the top of my head

kind moon
#

<@&815744082823348274> phishing link in every channel

cosmic dock
#

the worst thing about phishing links

#

is that

#

i have to recheck every channel

#

so everything is marked as read

#

so i can actually know when a message is poste

#

d

#

lol

junior narwhal
livid plover
#

something excellent might hatch from that talk!

cosmic dock
#

oh god

timber sphinx
#

but other than that, I'm kinda excited for the talk

junior narwhal
timber sphinx
mild pollen
#

There's also pnpm.

dreamy hatch
#

There's also bun.

timber sphinx
dreamy hatch
#

I don't really know, I saw a project with:

This is a Next.js project bootstrapped with create-next-app.

Getting Started
First, run the development server:

npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev
livid plover
#

🤯

#

so many options

#

one could feel like an airplane pilot

junior narwhal
dreamy hatch
#

and the new user never asked another question...

cosmic dock
clear wigeon
#

did you link to the github in your proposal? or did that all fit in the proposal window in the submision?

junior narwhal
clear wigeon
junior narwhal
mild pollen
#

I wish Python standardized on python3 on all systems.

lusty folio
cosmic dock
mild pollen
junior narwhal
#

there are many such posts and I find it greatly concerning that all engineers don't have an internal gauge of how fast network requests can be...

quartz yew
#

ah yeah

#

specially with METADATA not being available without downloading the wheel

#

which thankfully is now being sorted out

#

I have tried to say this on twitter many times, but this isn't a python vs rust issue, it's a lack of funding for Python packaging tooling issue

onyx spindle
junior narwhal
onyx spindle
#

Let them install pytorch packages, let's see those miliseconds in action 😄

cosmic dock
#

xd

cosmic dock
quartz yew
#

there are optimizations, but not to the same degree

#

uv for example caches resolver outputs, etc

#

pip maintainers are more focused on just keeping things working, than to actively trying to make it super fast, as opposed to uv

#

and uv is also not bound by backwards compatibility, unlike pip

#

all this allows them to make a super fast package manager

#

but, and you can clearly see this, it is definitely not a pip replacement and won't be for a long time

#

same as setuptools still being around after we standardized build backends, it's likely not going anywhere

#

but I think this kind of inovation is very good for the ecosystem IMO

cosmic dock
#

incentivizes everyone else to speed up their packaging

west basin
#

Isn't using __name__ for loggers within a package a bit of an anti-pattern? The name's supposed to be used for dot-namespacing so that you are able to capture logs coming from a specific package - if you have a module in your package "foo" called "bar" and its logger's name is just "bar", you kinda foo'ed the bar

lusty folio
west basin
#

ah, of course. I wonder why I thought it wasn't qualified. Thanks!

dreamy hatch
#

naming things: pipenv has a --support flag to print out some handy info to help with bug reports. we're going to add something similar to Pillow. I'm sure I've seen other projects with similar ones, but can't remember. can you think of any?

shadow zealot
#

Homebrew has brew doctor

west basin
#

Scoop has scoop checkup

junior narwhal
dreamy hatch
#

very nice, should have known Hatch would go the extra mile 🙂

timber sphinx
junior narwhal
#

ha yes the flare command is quite nice, I didn't realize but yeah I'm going for the same self-service behavior

#

by first gen you're referring to Agent 5 which was in Python but it was untenable for performance reasons so the Agent was rewritten in Go for versions 6 & 7 (7 is the same as 6 except only Python 3 whereas 6 ships both because customers may be relying on Python 2 syntax to run their custom integrations)

#

my aspiration is to slowly introduce Rust which will start a trend and hopefully most of it gets rewritten within the next 5 or so years but I simply don't have the time I once did

#

I probably could do that mostly by myself if I didn't maintain Hatch, meticulously reduce the amount of sleep I get, take Adderall, and also somewhat neglect personal relationships... but that sounds like a terrible life which I won't to do

junior narwhal
#

on February 6 I sent an email to pycon-program-committee@python.org but they haven't responded, is there a different email I should have used? that was on the talk acceptance email as the one that should be used to ask for accommodations and stuff IIRC

hexed briar
junior narwhal
#

true good idea, I just sent another email

hexed briar
#

My sense is that they have a lot on their plates and that they also don't want to do too many rounds of communication around changes and stuff.

Like, I discussed the packaging summit and a bunch of packaging talks conflicting thing with them (through a couple of other organisers) and even though we agreed on something; it's not gonna get communicated until their next checkpoint to avoid too many rounds of changes.

junior narwhal
#

yeah I'm just worry that my caregiver literally won't have a seat lol so I'm asking if I need to get another ticket

junior narwhal
dreamy hatch
# 
<meta property="og:url" content="https://code.visualstudio.com/updates/v1_87" />
<meta property="og:type" content="article" />
<meta property="og:title" content="Visual Studio Code February 2024" />
<meta property="og:description" content="Learn what is new in the Visual Studio Code February 2024 Release (1.87)" />
<meta property="og:image" content="https://code.visualstudio.com/assets/updates/1_87/release-highlights.png" />
silk jungle
#

it's slightly annoying that testpypi requires 2FA, but hey at least I found an use for bitwarden 2FA integration

junior narwhal
#

it seems like every year the topic goes viral on social media and I find it just as interesting as the last

Screenshot_2024-03-14_161816.png
#

it's so fascinating to me, almost incomprehensible

fierce horizon
#

It's hard for me to put a finger on it. I can get flashes of 6, but can't keep it fixed in my mind’s eye as if it was real.

junior narwhal
#

same for me but I think that just has to do with attention span. I can picture complex scenes but then the focus goes back to what I see after several moments if my eyes are open. if my eyes are closed then I can last longer

marsh kite
#

Wanted to ask admins: Is it ok if I share a job posting for my team specifically for a job working on open source Python packaging projects (e.g. pip, Warehouse, etc)? I don't see anything in the rules, so I figured I should ask before posting. I think it is extremely relevant to the community and figured #general would be a great forum to share in.

junior narwhal
#

fine with me, although I would prefer a link to something rather than a long message if possible

hexed briar
#

Thanks for asking @marsh kite!

marsh kite
#

For sure, want to make sure to be respectful of the server!

devout elbow
timber sphinx
fierce horizon
#

Microsoft Garnet might also be a contender.

boreal bramble
#

@vast wren we're getting hit with spam again that

vast wren
#

I sent them to the shadow realm

hexed briar
#

Thanks @vast wren

boreal bramble
#

thanks!

shadow zealot
#

When is Discord even going to do something about marking a channel as unread even if the message in question no longer exists…

whole jewel
#

that would require competence, so probably never

modern sigil
#

Regarding PEPs, there is a section covering reference implementations. Is creating implementations a required part of championing a PEP?

shadow zealot
#

Not required

junior narwhal
#

I think that actually is required now for packaging PEPs

#

or, at the very least, Paul will not accept such proposals without implementations (and others including myself don't seem to have an issue with that)

shadow zealot
#

OK I should said not technically :p

junior narwhal
#

I think I fundamentally don't understand how the send method of generators work, I've been banging my head up against the wall for like an hour or more

shadow zealot
#

Think of it as calling next() but the generator also receives the value given from the outside

junior narwhal
shadow zealot
#

Because for also calls next on its own. You get the same thing if you swap out gen.send with next(gen)

#

The generator is advanced twice per iteration in the above example, one by send and another by for

junior narwhal
#

if I swap out the for loop for a call to next each time the behavior is the same so I'm thinking what I want actually is impossible and the behavior expects send to come first not yielding the value first

shadow zealot
#

Not sure I understand your description correctly but yes that’s not possible, send triggers a value to be yielded from the generator before the value sent in is received by the generator (edit: corrected terminology)

#

But I think you can hack it by doing an empty iteration first?

junior narwhal
#

what do you mean?

lusty folio
# 
from collections.abc import Generator


def gen(i: int) -> Generator[int, int | None, None]:
    x = yield i
    while x is not None and x > 0:
        i -= x
        x = yield i
        print("gen was sent", x)

if __name__ == "__main__":
    v = 50
    g = gen(1000)
    g.send(None)
    try:
        while v := g.send(v):
             print("Recv:", v)
             v //= 2
    except StopIteration:
        pass

Little toy program to demonstrate. Only place I actually use generators like this is unfortunately in closed-source code

shadow zealot
#

^ The yield before the actual loop in gen

lusty folio
#

sorry, first yield there needed to be different, but it's the g.send(None) that's important here too

#

if you're starting the yield/send from what the generator already has, you can prime it like the above.

shadow zealot
#

It’s usually easier to first try to write a version without loops (by manually expanding the thing you want to iterate on)

#

Like how you first learned to do loops if you still remember that

junior narwhal
#

I'm getting the same output

def generator(values):
    value = yield None
    for value in values:
        modified_value = yield value
        print(modified_value)

gen = generator([1, 2, 3])
gen.send(None)
for value in gen:
    gen.send(value ** 2)
gen.close()
#

and with

def generator(values):
    value = yield None
    for value in values:
        modified_value = yield value
        print(modified_value)

gen = generator([1, 2, 3])
gen.send(None)
while True:
    value = next(gen)
    try:
        gen.send(value ** 2)
    except StopIteration:
        break
gen.close()
lusty folio
#

next in the while loop is replacing the for loop there, try without the line with next and
value = gen.send(value ** 2)

junior narwhal
#

I see

def generator(values):
    value = yield None
    for value in values:
        modified_value = yield value
        print(modified_value)

gen = generator([1, 2, 3])
gen.send(None)
value = 0
while True:
    try:
        print(f'value: {value}')
        value = gen.send(value ** 2)
    except StopIteration:
        break
gen.close()

❯ python gen.py
value: 0
value: 1
1
value: 2
4
value: 3
9
dreamy hatch
shell oracle
#

From a security perspective it's an actual nightmare.
Nobody is installing pycryptosqllite3 (I hope) but if ChatGPT makes something sound compelling enough as a solution for your problem, you may just install libcrypto3.

#

(Both of which are defanged variants of actual malicious packages we've detected.)

junior narwhal
shadow zealot
#

I thought nobody found out who Jia Tan actually is yet

lusty folio
#

afaik, just people memeing. If the whole thing ends up attributed, either to an individual or group, that would be pretty big news.

silk jungle
#

Solar eclipse! 🌙

20240408_151458.jpg
timber sphinx
silk jungle
#

I got out shined, literally :D

cursive gale
#

<@&815744082823348274> ^

sturdy saddle
onyx spindle
#

new kind of spam?

silk jungle
#

This is my alt, we're testing out moderation filters :P

onyx spindle
#

oh, ok

kind moon
onyx spindle
hexed briar
lusty folio
hexed briar
kind moon
silk jungle
hexed briar
west basin
west basin
#

also, I've been catching up on Anduril infiltrating the NixOS and Haskell communities and I feel like I've been living in a bubble. The level of support they enjoy from FOSS folk is astounding and a bit sickening

lusty folio
# west basin Thoughts on this? https://lobste.rs/s/qv8hht/pep_744_jit_compilation#c_uzogqn

Misses the mark on the actual differences between cpython and pypy, that pypy is lagging behind cpython on versions in a significant fashion, and that pypy can't utilize all c extensions, and a bit more besides.

From my own experience, cpython + appropriate selective use of numba/taichi, numpy, polars and a few others vastly outperforms pypy. pypy is only faster in the case where you aren't using complex extension code already.

#

also the idea that cpython is "shackled to" supporting a c api, rather than realizing that the c api is why python has been able to be so widely used seems to misunderstand why dev speed prioritization leads to python use, and how that doesn't need to mean impossibly slow programs.

#

pypy is great, but there's a limited number of experts making slow progress on it, and the speedups one might get from it are rather conditional on all of your actual bottlenecks being that the python interpreter is slow. the initial jit in 3.13 isn't trying to compete with pypy's current progress, it's an initial step in incremental progress.

hexed briar
#

Also, it's well known and understood that the C API isn't flawless. Every core developer sprint / summit etc I've read about/been to has had some discussion about C API's flaws/improvements needed to make it better for CPython and alternative Python implementations as well.

dreamy hatch
#

yep, and see PEP 731 that created a C API Working Group to help deal with its problems, like those listed in PEP 733

junior narwhal
#

also, I've been catching up on Anduril

west basin
#

the other thing I'm curious about is the impact of the JIT on start-up performance - PyPy's significantly worse than CPython's from what I recall and that's a general issue with JITs

junior narwhal
jaunty marlin
silk jungle
timber sphinx
silk jungle
#

pip isn't going to use poetry :P

timber sphinx
onyx spindle
silk jungle
#

I mean, I am not a pip maintainer so you can not legally hold me responsible for that statement, but I think it's a safe bet.

onyx spindle
#

yeah, Poetry isn't that popular here 😄

silk jungle
#

I wonder why :)

dreamy hatch
silk jungle
#

I like these colour highlights in the CPython docs, very nice! :)

image.png
dreamy hatch
#

good to hear! also for "Changed in..." and "Deprecated in..."

318055645-b60456f8-5e69-4494-b474-640fbdb1524f.png
#

(note the above screenshots are edited demos, iirc I added the "New in" just as an example 🙂)

lusty folio
#

oo, I like the attention that draws to different kinds of changes being noted

dreamy hatch
west basin
#

@dreamy hatch Hey, wondering if you'd know if it's an oversight that no-gil isn't mentioned at all in the docs/release notes for 3.13 or if that's intentional?

dreamy hatch
#

Yeah, most likely an oversight! The beta cutoff is on Tuesday, Sam and co have been busy getting stuff in before then, and docs changes are allowed in afterwards. Let me check in with them, we'll definitely need docs, especially during beta.

cosmic dock
#

packagin-

gets banned /j

west basin
#

hm?

junior narwhal
frank shore
silk jungle
junior narwhal
#

have any of you ever had a case where you had to for example submit a crash report? I'm trying to gauge whether or not debug symbols are useful for the average user when running an application like Hatch

#

I'm beginning to ship a Python installation that just contains Hatch and it is quite large because Python is not stripped nor are a few wheels

junior narwhal
#

(I ended up stripping)

hexed briar
junior narwhal
hexed briar
#

No strong opinions on the distribution mechanisms. I was literally asking a question of "is it feasible".

junior narwhal
#

for the Python binary perhaps but also I strip a few wheels that have not been stripped so I'm not sure about those

#

zstandard is one of them

junior narwhal
#

I'll be leaving for Pittsburgh soon, I don't know who's going to PyCon but I'll see you tomorrow! I downloaded Discord so I'll be able to contact a little when I get there, not really sure where I go at the convention center but I'll find my way 🙂

dreamy hatch
junior narwhal
hexed briar
#

If anyone is interested, the Maintainers Summit has a bunch of space available.

#

Room 402

fierce horizon
hexed briar
#

Yo, I'm familiar with that. My expectation was that having it be kept separately would allow someone else to upload it to a debuginfod server. :)

mild pollen
#

I wish PyPI would host debugging symbols for wheels.

junior narwhal
#

that's pretty cool, I've never heard of that spec

junior narwhal
#

has anyone experienced OpenSSL performance degradation for 3.x? upgrading from Python 3.8 to 3.9 with OpenSSL going from 1.1.1u to 3.0.9 is thrashing every CPU we can test on when running many HTTPS requests (using requests) concurrently

#

profiling indicates the function do_handshake in the standard library's ssl.py is the culprit and is where most time is spent

lusty folio
junior narwhal
#

wow such a bummer, thanks for the link

#

I wish there was a Python binding to Rust's ring

lusty folio
#

Are you able to use boringssl via python-cryptography for whatever specific functions are in need?

junior narwhal
#

I don't think so, I'm just using standard requests

lusty folio
#

(I dont remember everything that happens off hand in standard library's ssl)

junior narwhal
junior narwhal
#

I wish there was a Python binding to Rust's ring
what an absolutely bonkers thing to say lol my brain was forgetting cryptography existed

silk jungle
#

:D

limpid iron
junior narwhal
junior narwhal
#

oh my goodness I just went through a roller coaster of discovery

#

for those who are unaware, Windows runners on GitHub Actions have incredibly terrible disk performance (there are many open issues on various repos and the conclusion is always hardware related)

#

when I say terrible I really mean that, orders of magnitude difference

#

all of these issues are pretty much unanswered by Microsoft and I've been wondering why for years

#

come to find out the reason for the lack of communication is not some mistake or insufficient staffing but rather they slowly are rolling out a solution to improve the situation which can't really be done any faster than how they are already handling it

#

(to be clear I haven't spoken to anybody, piecing everything together through GitHub issues, blog posts, and company announcements)

junior narwhal
shadow zealot
#

Oh you’re saying… if I’m interpreting the pieces reasonable… the GHA team may have known there’s a solution for a while, but they can’t say anything about it because it’s a part of Windows 11, nor invest time for an alternative solution because there’s already one in the pipeline.

junior narwhal
#

yes exactly

boreal bramble
#

I saw mentions of dev drive but I thought that was just some unaffliated person creating an action to use dev drive to speed things up

junior narwhal
#

the underlying file system ReFS has existed for a while but only recently are they evangelizing it and adding many more features while increasing performance

boreal bramble
#

did anyone in any of the disk performance issues/discussions from GH/MS actually mention that this is the direction they're trying to work towards for GH Actions?

junior narwhal
#

no

boreal bramble
#

I wouldn't be too surprised if I missed it

junior narwhal
#

except one singular comment I found on the pip issue tracker, let me find it

boreal bramble
#

interesting, thanks

junior narwhal
#

I can't find it though

#

maybe he was thinking of Azure Pipelines and there is no equivalent yet for GitHub Actions

junior narwhal
hexed briar
#

^ @dreamy hatch

#

Opengraph descriptions work

junior narwhal
#

oh man, the pure joy one feels when getting a Discourse like by Brett Cannon meaning no more paternity leave and lock files proposal will be forthcoming 😌

coral marsh
junior narwhal
coral marsh
jaunty marlin
#

I hope it doesn't feel like a race! We're interested in engaging with the PEP and I think exploring what a lock file that meets our requirements would look like is really valuable.

#

Welcome back 🙂

normal ibex
coral marsh
coral marsh
#

Not really. I just hoped Andrew knew about the PEP ahead of time so they don't feel like they wasted their time when there's other work also going on

hexed briar
vernal hornet
silk jungle
# 
#!/usr/bin/env python
#
# Hi There!
#
# You may be wondering what this giant blob of binary data here is, you might
# even be worried that we're up to something nefarious (good for you for being
# paranoid!). This is a base85 encoding of a zip file, this zip file contains
# an entire copy of pip (version 24.1.1).

Yes, I was wondering why this file was 2+ MB big :)

onyx spindle
#

🤣

#

is that get-pip.py or something like that?

#

I remember seeing it somewhere

silk jungle
dreamy hatch
onyx spindle
#

discord stupid

dreamy hatch
#

😄

onyx spindle
#

wants to download file

silk jungle
#

It's based on the file extension. The link doesn't even download.

#

I feel like Discord's developer mode should disable this warning.

#

Although I guess you could argue that developers are even more at risk of running arbitrary scripts.

hexed briar
#

Given that Discord is the platform, of choice for various kinds of scams... and the typical age in the past... yea, I like that warning.

silk jungle
#

of course 🙃

Screenshot_20240701-103856_Gmail.jpg
junior narwhal
#

this is where AI responses would be useful, something polite and direct in response to such questions

dreamy hatch
#

the AI responses are right there: "What?"

onyx spindle
# silk jungle of course 🙃

pip maintainers: lock packages behind private modules and explicitly state to not use pip as a library
people: use private modules
pip maintainers: remove private code
people: 😮

silk jungle
#

It also appears to come from a work email address too :/

onyx spindle
#

what was that Einstein quote about the infinity of the universe?

mild pollen
#

I'm picturing this person desperately switching between vendored copies of six as more and more projects remove theirs.

marsh kite
#

The thing that really baffles me is that it isn't even internal pip code, which I could somewhat see reasonably doing, but six, which is available seperately

onyx spindle
#

well, people do weird stuff

marsh kite
#

Yeah!

hexed briar
# silk jungle of course 🙃

In case anyone ever asks me why I have "Don't email or tweet me for tech support." on my GitHub bio... this is why.

marsh kite
#

Yeahh. I've gotten an email threatening to sue me before because "broken software = malware" according to the user

strong blade
#

Just refund them the $0 they paid for the software, easy /j

frank shore
onyx spindle
frank shore
#

I don't remember, but probably! Likely most users are 1) unaware of installer; 2) see that they get pip automatically with their Python so that's the thing they want to use.

silk jungle
#

we should totally reintroduce the concept of mucking around the internal application state to modern OSes, it will improve flexibility a ton

onyx spindle
#

I mean, I guess it's nice to have a pip interface for package installation

frank shore
#

I also kind of wish python -m install worked out of the box (note: install not installer). I'm not underestimating the user experience, education, migration effort, etc. involved in that throwaway comment 😄

onyx spindle
#

well, I wish Python had packaging stuff built-in, but we can't have everything, right?

#

I mean, either make pip all-in-one installer and project manager or make another project official

frank shore
#

I'm not sure how we get there or what the shape of it should look like, but I think that package installation should be a built-in experience.

junior narwhal
#

hard disagree, if you mean that the experience would be shipped with a Python distribution

#

the best way would be a tool, not Python distribution, that users install

marsh kite
#

whether its a tool or Python distribution, I think what I want is a default packaging tool included in a default Python installation method

junior narwhal
marsh kite
#

I think it would be good to include an integrated tool that does more than pip, including environment management, command line program addition (a la pipx), environment locking, etc

junior narwhal
#

but why must it be tied to a Python distribution specifically?

marsh kite
#

I think it would be good for usability and compatibility purposes (i.e. you can say ah user has Python 3.x, they must have packaging tool at least foo.bar), and having an "official" tool would help with a lot of common confusion on which tool to use for what and what workflows should look like

junior narwhal
#

I'm not disputing the "official" tool because I think that is the best path and I'm trying to make Hatch suitable for all purposes indeed. I'm questioning why you cannot just download a "tool" the same way you do any other package manager and then it would do everything for you, including manage Python installations as Hatch does

#

the "everything tool" doesn't make sense to be bundled with every distribution because it in fact would manage distributions themselves

#

more or less I'm asking why this is so different than every other language

marsh kite
#

Yes, to restate my point a different way, I think a tool like hatch that becomes the default should be distributed as the official way to install Python

#

Much like rustup is the default

junior narwhal
#

in those words, yes I agree with you

#

my ideal future (which I'm pretty sure will actually happen) is that python.org will provide installers for "the tool" and then it would also ship relocatable distributions that "the tool" will use for Python installations

#

i.e. installation would simply be unpacking an archive

boreal bramble
#

maybe the corepack way?

frank shore
#

Much the same way pip is a tool but "comes with" Python (via ensurepip and python -m venv creation).

silk jungle
#

Pip is already a pretty low-level tool fwiw.

frank shore
#

I'm perfectly happy if package building doesn't come with Python out of the box. That's a more specialized experience than package installation and it's okay with me if we let package maintainers install their tool of choice. I just want venv+install to be out-of-the-box experiences

frank shore
silk jungle
#

<insert rant on expectations>

junior narwhal
#

I just want venv+install to be out-of-the-box experiences
it already is 🙂

boreal bramble
lusty folio
onyx spindle
junior narwhal
#

the development experience on Linux is subpar all around so I'm okay with stating that Windows and macOS is what we should prioritize (again, for development experience specifically)

frank shore
#

I think many of the pieces are already there, but what we don't have is a vision for the experience that brings all these parts and pieces together (including in DPO threads, wheels by default, venvs by default, etc.)

lusty folio
# junior narwhal the development experience on Linux is subpar all around so I'm okay with statin...

I'm the other way on this. python dev experience can be fixed on linux. Simply unpacking the correct archieve from https://github.com/indygreg/python-build-standalone/releases/ into /opt/ and adding it to path is, for the vast majority of users, going to be a better experience than getting python through their distribution (exceptions for RHEL, which most users won't have, since RHEL isolated system python, then made installing python do what users expect, not what the system needs from python)

junior narwhal
#

wheels by default, venvs by default
the former doesn't need some massive effort, any installation tool can do that by default if they want to, it's just that pip would have to go through a transitory period

the latter is not a good idea in my opinion and UV rejected that recently for good reason

frank shore
#

And PEP 582 was rejected

junior narwhal
#

it's almost but not quite the same downside as that project-local directory installation method that got rejected, I forget the PEP number

#

yes that 🙂

lusty folio
#

I'm in favor of wheels by default as well.

venvs by default is a more complicated tradeoff, I think it would be a good thing, but I think there are too many people that just use python for 1-off scripts and depend on what they need in system packages, and that the script block is too new to depend on people having started using

junior narwhal
#

the biggest hurdle for Linux is the fact that there is no standard installer mechanism

frank shore
#

What I feel like we're missing is a comprehensive vision for where we want to be, even if it's 10 years off. That would be the roadmap toward which we strive to transition the tool landscape to. But I think the Cargo/Rust experience is making Python's inherited wild-west approach pretty user unfriendly.

lusty folio
#

which admitedly, is archaic feeling

junior narwhal
#

I don't think so, what I mean is something akin to MSI for Windows and PKG for macOS

#

there are a few attempts but honestly I don't know if any of them are "winning"

lusty folio
#

Not really, no.

junior narwhal
#

flatpak for example looks nice on paper but I found out a few weeks ago it really wouldn't work for CLI apps, it's tailor-made for GUIs

lusty folio
#

flatpak isn't suitable for dev tools at all

#

snaps similarly

boreal bramble
#

snap's fine for dev tools but only in --classic mode

#

and then there's the whole proprietary store thing

lusty folio
#

I don't think the snap format is good to begin with ^

junior narwhal
#

this is what I mean when I said Linux is in a bad spot, unfortunately the best way to install Hatch on Linux is to get pipx from your distribution and then pipx install hatch

lusty folio
#

I should make an actually portable ELF binary later that just grabs the correct prebuilt archive and unpacks it to the right place as a PoC

junior narwhal
lusty folio
#

uhh

#

not what I meant by that

boreal bramble
#

I like snaps but the store situation is awful and keeps you at Canonical's whim at all times (because you can't use a repository that isn't Canonical's Snap Store and Snap Store has a lot of requirements that you need to fit into to be able to publish a package, especially in classic snap variant)

lusty folio
#

I mean an actual portable python installer that installs python to the correct place, not making a python script a portable binary

boreal bramble
#

and I'm sure it has some issues on non-Ubuntu systems, even if it's technically meant to work everywhere (*on all major distros)

junior narwhal
lusty folio
#

hatch bootstraps using python

#

I'm talking about a binary that has all of it's dependencies statically linked, and runs on multiple archiectures (using something like https://justine.lol/cosmo3/) grabs a reproducible build, and puts it on the right place on the system

#

ie, a standalone installer

junior narwhal
#

rather than a tool just for that purpose

#

ah I see, I tried that in the past

lusty folio
#

partially because I don't believe in the 1 tool for all things direction. I don't think it will ever "just work" to solve the problems of distro python problems on linux, and want a solution that depends on nothing being distro provided to begin with.

#

(I dont see debian changing their policies any time soon to better align with user expectations of python)

fierce horizon
# junior narwhal the biggest hurdle for Linux is the fact that there is no standard installer mec...

I think the Rust experience is perfect on Arch:

  • either you want to develop on your system, then you install+use rustup for Rust toolchain management
  • or you want to build in a chrooted/containerized env, so you just install cargo/rustc which are the latest versions

I much much much prefer having a package manager over juggling one separate self-update process for each single application plus some way to manage a dev environment.

fierce horizon
junior narwhal
#

also fun fact, Greg the person behind those Python standalone distributions, was one of the main people behind ^

frail aurora
#

The "problem" is that there hasn't been a ton of movement on the non-git backends which makes it a hard sell for hosts (github, gitlab, etc) to contribute/add in adoption

#

I doubt facebook will ever really open source major parts of their backend, but JJ seems interesting

#

The dream is something like a pluggable backend (maybe in the way Libgit2 provides backends) that could abstract over cloud storage. Think packfiles in s3 and ref data in dynamo

#

If performance could be made reasonable, it might mean that git hosts aren't really relevant beyond what they provide as a PR/workflow tool

silk jungle
#

In more off-topic discussion, does anyone have a TL;DR article or whatever explaining the situation on from __future__ import annotations and from __future__ import co_annotations?

#

I've been vaguely following the various discussions on DPO, but it has gotten unwieldy to keep track

junior narwhal
#

I've never heard of the latter

silk jungle
#

This is why I need a summary :)

ionic tulip
lusty folio
lusty folio
#

So, the non-editorialized short form:

Original implementation ran over the version to include (3.13)'s feature freeze. I don't think there's public info from Larry (original implementor, PEP author (649)) about handing it over, but Jelle has picked up a continuation of it.

The original 649 proposed immediately just turning from __future__ import annotations to the new behavior, since then, there have been several discovered issues with that approach, including, but not limited to needing to import inspect to interact with annotations reasonably.

Jelle opened PEP 749 for SC's consideration, amending the scope and plan for 649, replacing it. There are a few open questions still on how to proceed from here.

This is probably the document which best shows off the known deficiencies with the original plan right now.

Personal opinion: It might be better to drop changing the stored annotation form entirely, and instead have pep 749 only be annotationslib which would interact with the current state of the annotations future, but I'm waiting to see more of how things shake out, because if some of the warts with the current questions have good answers available, there were and still are reasons people don't want strings there.

dreamy hatch
ionic tulip
frail aurora
#

Is there anywhere besides srcht for the remaining hg users? is it all self-hosted now?

ionic tulip
silk jungle
marsh kite
#

I don't think its nit-picky, responsiveness is important

#

would it be possible to do an adaptive download chunk size based on the time it took to download the last chunk?

#

So you start small and increase the download chunk size until the last chunk was "slow" by some metric?

lusty folio
#

This feels like something that could be improved more with downloading multiple dependencies at once, either threading or async based, and have the updates to tty on a separate thread

#

so that the granularity of updates isn't tied to the chunk size

clear wigeon
#

doesn't pip use rich vendored in?

#

afaik rich has something built in precisely for things like this

lusty folio
#

as for too nitpicky or not, personally, I think if I'm installing in an interactive environment, I'd prefer the responsiveness to more speed, but I'm rarely on a connection that slow anymore...

marsh kite
#

so it can't figure out the dependency graph up front then download everything right away

lusty folio
#

there's some exceptions to that with wheel only downloads and being able to use metadata to append to the download queue

#

but yeah, it's non trivial to do that

silk jungle
#

*well in this situation for the progress bar. On a second read, you probably mean all of the TTY code which no is not pushed to another thread.

lusty folio
#

Is it? at least when I played the video It looked like updates to the progress bar where dependent on the chunks in some way and that the estimate didnt update independently of the chunks fetched

silk jungle
#

The progress updates are pushed to rich every chunk yes, but it isn't rendered to the TTY on every chunk.

#

Currently the progress bar renders (up to a maximum?) to 30 times per second which I definitely agree is too high.

#

If you're suggesting that another thread "sniffs" the download progress asynchronously, then I don't even know how you'd go about that.

lusty folio
#

No, downloads in background threads pushing progress back to main thread, main thread updates tty on a combination of refresh rate/number of updates, whichever happens first.

#

which is more complicated than the gains to responsiveness is probably worth absent parallel downloads

silk jungle
#

Yeah.

lusty folio
#

I think you're on the right track with the feedback there though, if it's in a tty, the responsiveness is going to matter for people on slow connections to not give up on an install

silk jungle
#

I am not entirely sold on my suggestion either though. The thing is that the progress updates cannot obviously occur any faster than the chunks are downloaded, so the higher the chunk size, the less responsive the feedback.

lusty folio
#

well, if the updates happened interleaved faster than the chunks, it could still update the eta printed, which is a sign it hasnt hung

silk jungle
#

I suppose, yeah. I'd have to look at the rich API again.

lusty folio
#

(I dont know if rich actually supports that out of the box, I dont use rich that way)

silk jungle
#

It's set to auto refresh (render) 30 times per second, but evidently it's not recalcuating the ETA between chunks.

silk jungle
#

I already spent too long on this one review. I'll take another look later.

marsh kite
lusty folio
#

quick searching, havent found any out of the box stuff in python for this, doesn't look like requests or aiohttp have anything to behave this way automatically built in

silk jungle
lusty folio
#

yeah, doesn't look like rich is designed for this

lusty folio
# lusty folio <https://www.researchgate.net/publication/333859237_Bandwidth_prediction_in_low-...

I might take some time to tinker with an Idea I have seeing this later, would end up as a useful snippet to live somewhere for people to use/copy if I can get the idea working, but I'm not gonna commit to working on this in depth right now, not sure how much an improvement it would actually be/value of it. Needs exploration to see if it feels any better before seeing if it's worth other things using such code

silk jungle
#

We don't need to get pip using the bleeding edge art for download streaming responsiveness, don't worry about working on this in-depth. The general discussion has already been helpful!

lusty folio
#

I have a "todo list" with interesting problems on it that when I get free time, sometimes I pick one and work on it. then, sometimes that results in publishing stuff for people to use (including myself in other projects), so if I end up digging in deep enough, it wouldn't just be for pip

#

I think for now though, the answer to your question on responsiveness (with existing code in mind) is your thing about a lower chunk size seems super reasonable to get better responsiveness

clear wigeon
silk jungle
clear wigeon
#

https://rich.readthedocs.io/en/stable/progress.html

from time import sleep
from urllib.request import urlopen

from rich.progress import wrap_file

response = urlopen("https://www.textualize.io")
size = int(response.headers["Content-Length"])

with wrap_file(response, size) as file:
    for line in file:
        print(line.decode("utf-8"), end="")
        sleep(0.1)
silk jungle
#

That's still fundamentally linking the progress bar updates to the individual read calls (or chunks), no?

#

What I was asking about was a way for rich.progress to update its ETA between chunks so pip doesn't appear to hang.

clear wigeon
#

ah, tqdm has that

ionic tulip
#

@silk jungle i beleive there is a tool to run the spinner updates in a thread that updates regular, but that needs a slightly different controll flow

silk jungle
#

Rich progress bars use a live instance internally. I'm not interested in a spinner update, I wanted the progress bar to update its ETA regularly even if there is no progress advancement.

clear wigeon
#

Will is active there

silk jungle
#

Later perhaps.

junior narwhal
#

always a trade-off, this one is better maintained but the import time is 75% more...

#

it's unfortunate nobody really cares about import time

onyx sphinx
junior narwhal
#

that's true, and there has been a dedicated effort the past year for various modules!

#

I guess I meant library authors

lusty folio
#

if the standard library made it easier to reduce import time, I'd work on it more.

#

most of the code I write nowadays in python though arent 1-off scripts, it's long running applications or code meant to be used in long running applications, so there's practical limits on how much effort I'll put into that

dreamy hatch
lusty folio
#

deferred imports, mostly. low import time competes with doing the "right thing" with type hints.

#

if TYPE_CHECKING blocks neuters the ability to do runtime introspection, but is something some people use paired with from __future__ import annotations, but this just results in annotations containing things that can't be resolved at runtime

#

I prefer not to make introspection worse for import time

junior narwhal
#

that's not really a feature request for standard library maintenance but rather the language/interpreter itself, no?

strong blade
lusty folio
#

the ideal would be at the language/interpreter level, but standard library tools existing for it would still become things I'd consider using

junior narwhal
lusty folio
#

makes sense for cli tools that are used that way

#

responsiveness is good

onyx spindle
junior narwhal
#

I agree but it will never happen I think because of dynamism

#

then again the GIL is going away so who knows lol maybe one day

icy jetty
#

hey all

#

curious if anyone knows --- is the new support spec. going to be involved on the repo dealing with 541s and such, or focusing on other issues somewhere else?

frank shore
#

I can't speak officially, but I think it's things like 541s, org accounts, and limit bumps.

icy jetty
#

👀

#

let's hope. seeing a year and a half backlog of 541 reqs after opening one was not the most encouraging

icy jetty
silk jungle
#

It's unfortunate how poorly the org accounts roll-out went.

#

I understand the reasons why, but it reflects poorly on the packaging ecosystem when a major feature roll-out is botched.

dreamy hatch
# icy jetty curious if anyone knows --- is the new support spec. going to be involved on the...

Yes. From the job ad:

This role will take responsibility for all user-facing support of PyPI including account recoveries, project name requests (PEP 541), Organization request review, Organizations feature support, PyPI resource limit requests, abuse reports, and end-user support. This will include the existing backlog as well as triaging and responding to tickets in a timely manner once the backlog has been processed. Additionally, the PyPI Support Specialist will provide feedback and guidance to the PyPI Administrators and contributors regarding improvements to our systems and tools to better support their role. Improvement, consolidation, and maintenance of PyPI end-user documentation will also be a priority of this role.

Responsibilities
The PyPI Support Specialist will:

Process support requests in the following areas:
Account Recovery
Project Name Requests
Organization Requests
Project Limit Requests
Malware/Spam/Abuse Reports
End-User support
Provide feedback and guidance to PyPI Administrators on process and tooling features that will improve the speed and ease of processing support requests
Improve, consolidate, and maintain PyPI end-user documentation
Produce periodic reports of the current status of PyPI support response time for the community
Willingness to travel to annual PyCon US conference

https://web.archive.org/web/20240320225217/https://pythonsoftwarefoundation.applytojob.com/apply/nyYHuOha9h/PyPI-Support-Specialist

PyPI Support Specialist - Python Software Foundation - Career Page

Apply to PyPI Support Specialist at Python Software Foundation in Remote.

icy jetty
#

might have to wait it out and see

onyx spindle
junior narwhal
#

I'm so sad, Pydantic v2 models are actually incredibly slow to import/load and therefore I can't use it in Hatch as I had planned on cryingcowboy

lusty folio
junior narwhal
lusty folio
# junior narwhal it's 80% faster, what in the world...

it has a few less features than pydantic does (personally, I've never needed the other things pydantic does, but I don't transform data before validating it), but it's extremely fast, only requires a c compiler (rather than a rust one) (and only if wheels aren't available, which they are for the same platforms as pydantic) and the objects it creates are both smaller than the corresponding ones you'd write in pure python, and faster than the equivalent classes in python.

junior narwhal
visual furnace
hollow wind
#

@visual furnace yeah, that's an interesting case. As a general principle, I don't think packaging works very well in combination with installing new packages into a running interpreter. It's too easy to fall into assuming that nothing changes "under your feet" - and a lot of tools (like pip, for example) do that. Frankly, doing anything else would be a nightmare, particularly as the import machinery is owned and managed by the core devs and the packaging tools are external...

silk jungle
vast wren
#

appears so

#

it happens every now and again

#

often times they email us

#

and ask us to fill out some vendor form for them

#

and we tell them no

icy jetty
#

wonder how long the support queue will take to clear when the new rep active

#

few months is my guess

hexed briar
#

And, also +1 -- I also want that.

marsh kite
frank shore
hexed briar
junior narwhal
#

the former should be the first step because the latter would depend on those for Python management

hexed briar
#

Yea many of the latter just use binaries by downloading them from https://nodejs.org/dist or build from source (similar to pyenv).

silk jungle
dreamy hatch
#

This release is a milestone: it fixes Black’s first CVE security vulnerability. If you run Black on untrusted input, or if you habitually put thousands of leading tab characters in your docstrings, you are strongly encouraged to upgrade immediately to fix CVE-2024-21503.
https://black.readthedocs.io/en/stable/change_log.html#id7

marsh kite
#

It is odd to me that as a project owner I cannot click a button to disqualify a GHSA against code in my repo

silk jungle
#

Security was a mistake.

#

We should just accept all software as broken and insecure.

lusty folio
#

pip still has one that docker scout reports that's even been disputed regarding --extra-index-url

#

I don't think it's reasonable to assign black a cve for this, language servers dont all have cves for executing untrusted code, and black as a tool isnt intended for use on arbitrary untrusted files

#

cve process is insanely broken, and I'm on the side of report more things

silk jungle
#

I don't maintain black anymore so I don't care that much, but god this makes me want to engage less with security.

lusty folio
#

but the reports being bogus harms signal:noise

clear wigeon
lusty folio
#

we really need a better way of doing cves that preconditions the cve

if [this] then [vuln]
then when [this] is never a reasonable concern for an aplication, the cve is dismissed

silk jungle
lusty folio
#

it's like inheriting a vuln for something statically linked or for library code when you can provably show you never trigger the vuln conditions, and that tree shaking even removed the vulnerable logical branch, but still being pestered to do a release

#

we need smarter analysis of cves so that security isn't working against developers but with

dreamy hatch