#general

1 messages · Page 4 of 1

distant briar
#

I'm talking about the things you mentioned above

#

it's a whole package deal

blazing lantern
#

PEP 735 should address the dev dependencies case

inland prawn
#

oh, yes, please

echo ruin
#

Does PyPA (or the PSF since I suspect someone may know the answer for that as well) publish its list of banned github users publicly perchance?

queen hornet
#

I don't think so. at least for https://github.com/python repos, I normally report directly to GitHub to begin with, they will often suspend the account directly

#

and that helps other orgs too, the spammers often target multiple orgs

echo ruin
#

I tend to do that as well, but that works only for spam obviously -- the reason for asking is a certain notorious user who files ostensibly spam bug reports who I banned, and who then created what I suspected was a ban evasion account but never bothered to research -- and then now a few minutes ago I happened across a comment on a repository that indicated y'all agreed and had banned the other account.

#

(And obviously I'd love to not have to waste any brainpower whatsoever on these sorts of things)

lusty quarry
lusty quarry
lusty quarry
#

I have a Dr. appointment tomorrow so I'll be offline but Wednesday I'll announce #pyapp finally

blazing lantern
lusty quarry
blazing lantern
elder tinsel
lusty quarry
#

yeah I was just wondering, figured if it came from Red Hat the benchmark must be legitimate for them to change packaging

elder tinsel
#

oh, I did change that to use base

#

oh, but not because of python, it's even in the comment there about pip installed things might not behave properly if they assume something present

lusty quarry
lusty quarry
#

@high stone wow I just read your email about us being upgraded to enterprise courtesy of GitHub! thank you for that and to William for starting that discussion with them

lusty quarry
vital pier
honest geyser
#

👋 I'm looking at build front/backends for building trivial (like.. one source file) projects for testing package resolution. I've been using hatch/hatchling and am in search of faster solution as building the sdists/wheel is the slowest part of my pipeline. I tried switching to pypa/build/hatchling but that's ~5x slower. Does anyone have any quick recommendations or ideas about which build tools are the fastest?

#

I just tried hatch/setuptools and ended up with >10x slower 🙂

uneven karma
#

Have you tried setuptools without hatch?
Maybe flit?

lusty quarry
honest geyser
# lusty quarry can you please explain what your setup is like? which part is slow exactly? `bui...

I'm generating "projects" from a definition in a JSON file e.g.

❯ tree build/example-4a6eaaab
build/example-4a6eaaab
├── example-4a6eaaab-0.0.0
│   ├── pyproject.toml
│   └── src
│       └── example_4a6eaaab
│           └── __init__.py
├── example-4a6eaaab-a-1.0.0
│   ├── pyproject.toml
│   └── src
│       └── example_4a6eaaab_a
│           └── __init__.py
└── example-4a6eaaab-b-1.0.0
    ├── pyproject.toml
    └── src
        └── example_4a6eaaab_b
            └── __init__.py
#

Then, I'm iterating over them and calling hatch build

#

It's not slow, but it's orders of magnitude slower than anything else I'm doing

lusty quarry
#

is the second time the fastest out of all?

#

try to run it a second time

uneven karma
honest geyser
# lusty quarry is the second time the fastest out of all? 
INFO:root:Building 'example-4a6eaaab' in directory 'build/example-4a6eaaab'
INFO:packse.build:Generated project for 'example-4a6eaaab-a-1.0.0' in 1.08ms
INFO:packse.build:Built package 'example-4a6eaaab-a-1.0.0' in 0.38s
INFO:packse.build:Generated project for 'example-4a6eaaab-b-1.0.0' in 0.81ms
INFO:packse.build:Built package 'example-4a6eaaab-b-1.0.0' in 0.39s
INFO:packse.build:Generated project for 'example-4a6eaaab-0.0.0' in 0.88ms
INFO:packse.build:Built package 'example-4a6eaaab-0.0.0' in 0.30s
INFO:packse.build:Built scenario 'example-4a6eaaab' in 1.08s
example-4a6eaaab```
Seems roughly constant
lusty quarry
#

can you show the configuration?

indigo token
honest geyser
lusty quarry
blazing lantern
#

I don't know if their message means they are missing the key or there's nothing at the path location

lusty quarry
#

interesting, thank you!

inland prawn
#

did anyone ever seen or heard about any effort/works to standardize a use of dependencies from external sources (other than PyPI) in a "better" way than --index-url/--extra-index-url that pip offers?

uneven karma
#

(whats wrong with that way?)

inland prawn
#

it's mostly that with many package managers available, each has it's own way of dealing with packages from different sources. this is especially an issue with pytorch libraries since they come from outside of PyPI. would be nice to have it standardized (like PEP 621 standardized project metadata)

lusty quarry
#

nothing has changed, and I think a lock file is a prerequisite for that

elder tinsel
#

I can't think of any differing behavior here which would benefit from additional language. There's conda, but conda isn't directly fetching requirements in a compatible manner to pip to begin with, the goals are different there

#

Do you have an example of something that differs you don't think should?

inland prawn
#

I meant more in the ways of better UX. shifting responsibility of providing the extra source from user to packagers/authors that require that source

elder tinsel
#

They already can be, packages that do that just can't be uploaded to pypi. That's intentional, and there are related security concerns

#

any pep 508 dependency specification (Which can include a source to get it from) should be handled fine by tools, whether indexes allow them is seperate

lusty quarry
#

which I haven't put too much thought into however I plan to tackle that next year as part of lock file stuff

elder tinsel
#

(git links require git+repo_url but work too)

lusty quarry
#

no I think you are missing the point. the incoming spec is numpy==x.y.z and during resolution you want to override that explicitly

elder tinsel
#

I'm not, pep508 specifications explicitly allow this.

lusty quarry
#

yes I totally agree with you but the issue is the UX of how to override that

#

pip has no solution for this

elder tinsel
#

I can use that in a requirements file or pyproject.toml today with pip

lusty quarry
#

I think PDM has an option

elder tinsel
#

I cant upload a package to pypi that does that.

#

it's not the package management preventing that

lusty quarry
#

sorry we are still talking past each other 😅 let me give a more concrete example

you have a giant list of requirements called requirements.in and you install it like this pip install -r requirements.in. on Windows specifically there is a transitive dependency that simply does not work as defined but you have a fix prepared in an internal repository. you want to somehow override that definition on Windows before your lock file is produced

#

@inland prawn also feel free to chime in if I am misinterpreting what you're asking for

elder tinsel
#

I thought that works currently with pip and specifying that? 🤔 if it doesn't, then I think at work I need to review Why something is working as expected tommorow when I'm on the clock.

lusty quarry
#

it works if you manually change the input file to point to your patch or any other overrides, sure

#

but that is not really an override, that is simply changing your inputs

elder tinsel
#

I don't think specifying you depend on a specific patched version is the wrong way to do this 😐

inland prawn
elder tinsel
#

so maybe im missing a piece of the puzzle still

#

I also dont think publicly distributed packages should rely on patched versions that could conflict with the same namespace, and should contribute upstream neccessary fixes

lusty quarry
#

at work we definitely have to. we support customers who run on CentOS 6 and we do not want literally everyone to use old versions of stuff like cryptography so we have patches to do that just on old platforms

elder tinsel
#

right, but there you're the app, not a library, so patching is in your purview, you know the full environment and that doing so isn't an issue.

mortal shore
#

pip does not let you override dependency specifications, it requires them all to match, if the numpy spec in the dependencies does not overlap with the numpy spec in your requirements file or passed on the CLI or what not it will detect it is not resolvable and error out

#

you can fake it by making a lock file and using --no-deps

inland prawn
mortal shore
#

do you mean from the package author or from the package consumer

lusty quarry
#

we intentionally have no packaging channel for some reason so I will post here: is there any way that we can reduce the import cost?

#

it doesn't look like much is going on there

mortal shore
#

It's probably from compiling the regex I'd imagine

lusty quarry
#

I thought the same but the impact is negligible

#

I can reduce the imports by 4 ms if I set those custom type definitions on top to None

#

oh typing takes 3.2 ms

queen hornet
inland prawn
lusty quarry
#

I created one a while back but it got deleted promptly because someone felt like it wasn't useful or something, I can't remember

mortal shore
#

I think tags precomputes a big list of tags

lusty quarry
lusty quarry
lusty quarry
agile burrow
#

what's the sleep 2 in there for?

lusty quarry
agile burrow
#

curious

ivory phoenix
#

(the header of the table)

image.png
ivory phoenix
lusty quarry
#

yeah I use it frequently but for standard library imports I like to just run the import multiple times to get an average

bright parcel
lusty quarry
#

yes that tool is how I track responsiveness in CI for Hatch but for a one-liner invocation of Docker it's easier to do a for loop in Bash

blazing lantern
lusty quarry
blazing lantern
inland prawn
lyric hedge
#

did we get raided by the equivalent of an angry farmer with a pitchfork? :)

indigo token
#

With that description, I’m sad that I missed it. Like, an anti-packaging luddite?

hollow trout
#

It was just a spammer.

lyric hedge
#

we don't keep logs so I have no idea

#

I just saw an admin ping and a whole lot of unread channels :P

mortal shore
#

Just a spammer

lusty quarry
#

how did it go away? I saw the ping but I don't know how that translates to all of those messages going away at the same time

lyric hedge
#

IIRC discord has a purge ban feature

mortal shore
#

I did a ban and told it to delete the last 7d of messages from that user

topaz pine
#

Good to know I wasn't the only one confused by a bunch of empty "unread channels" 😅

obsidian lily
#

hi

toxic cypress
#

There's a Python package I'd like to develop. This package will need the following steps at build time:

  • Clone a Git repo that holds native code I'll need
  • Compile this native code for the big three desktop platforms (Windows, macOS, Linux), though with room to expand if necessary
  • Generate ctypes bindings from this repo's headers

I know the language well but haven't written a polished Python package before. Any recommendations for build tools I should use? Or maybe it would be more helpful if I described my project in detail?

inland prawn
#

Setuptools sound like a good choice. Kinda default when it comes to compiling and distributing native code based packages

toxic cypress
#

Okay. How does setuptools do that, broadly speaking? Am I expected to call os.execute a lot? Or are there Pythonesqe wrappers for common C/C++ build systems? The library I want to build is written in C, and I intend to write a CMakeLists.txt for it. I don't want to write any extra C code if I can avoid it

#

Also, fetching the library. To keep the build process simple, I'd like to fetch the library itself (and any dependencies) via git as part of the build process.

#

Should I do that in Python, or can I use CMake's FetchContent module for that?

#

...eh, I'll use FetchContent. Probably easier, now that I think about it. One less thing for Python to manage.

#

That leaves generating the ctypes bindings, then. Got any suggestions? I know of ctypesgen, CFFI, and SWIG. But I can't decide between them.

woven plover
#

what library do you want to make a binding for to begin with?

toxic cypress
# woven plover what library do you want to make a binding for to begin with?

libretro-common, which is used by libretro cores (emulators) and frontends (things that play the emulators). https://github.com/libretro/libretro-common

GitHub
GitHub - libretro/libretro-common: Reusable coding blocks useful fo...

Reusable coding blocks useful for libretro core and frontend development, written primarily in C. Permissively licensed. - GitHub - libretro/libretro-common: Reusable coding blocks useful for libre...

#

Frontends are expected to expose their implementations of certain APIs (like filesystem access, performance counters, etc.) through function pointers that cores can ask for at runtime. However, libretro-common includes reasonable defaults.

#

I want to write a libretro frontend in Python for the use case of running tests on cores.

#

It's one thing to parse the headers to get ctypes definitions for the structs and other types I need. But if I can hand off these functions (where applicable) to the core from a compiled library, mimicking RetroArch's behavior will be much simpler.

hidden flame
#

Hey all, I'm looking for people who might be interested organizing the packaging summit at PyCon US 2024. Is anyone already on it?

lyric hedge
toxic cypress
lusty quarry
#

I would echo the previous recommendation, please definitely try scikit-build-core first

toxic cypress
#

You got it.

inland prawn
#

@toxic cypress as to fetching the upstream native code of libretro, I think a better idea would be to submodule it into your own project, rather than fetching it on build time

toxic cypress
woven plover
toxic cypress
inland prawn
#

I'd say that build should be always possible offline when you have an sdist of the project and build dependencies. That's what I have always seen in any major packaging system. it also makes your build fault-tolerant in case upstream commits/repos are removed

toxic cypress
lusty quarry
#

is anyone here today proficient in compiling on Windows? I'm blocked currently on installing pymqi on Python 2 (don't ask lol)

  cl : Command line warning D9002 : ignoring unknown option '/std:c11'
  pymqe.c
  code/pymqi/pymqe.c(1053) : error C2275: 'Py_ssize_t' : illegal use of this type as an expression
          c:\py2\include\pyport.h(183) : see declaration of 'Py_ssize_t'
  code/pymqi/pymqe.c(1053) : error C2146: syntax error : missing ';' before identifier 'property_value_free'
  code/pymqi/pymqe.c(1053) : error C2065: 'property_value_free' : undeclared identifier
  code/pymqi/pymqe.c(1059) : error C2065: 'property_value_free' : undeclared identifier
ivory phoenix
rapid glacier
lusty quarry
#

so weird https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11_0_1-release-notes#Kernel

New in macOS Big Sur 11.0.1, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286)

inland prawn
#

is there any list/matrix of tags needed to cover all possible platforms and architectures out there?

elder tinsel
# inland prawn is there any list/matrix of tags needed to cover all possible platforms and arch...

probably not, there's a (minor) problem with assuming that can be exhaustive as platform is tied to {sysconfig, distutils} get_platform(), which is allowed to vary by interpreter. If you only care about Cpython, I think that reduces it to something that could be exhaustive on platform, you'd then need to find an exhaustive supported architecture list, and that seems daunting since in theory, all you need is a c compiler and a bunch of deps that also only need a c compiler and some c libraries for cpython to exist on the architecture.

You could probably get a pretty pragmatic list by getting every option possible for platform on cpython, and every architecture for which there is a wheel already published on a well known index though.

west drift
#

You can look at the list of tags supported by pypi, which is the closest to "supported platform" there is, otherwise the list afaik unbounded, people are compiling cpython for all sorts of platforms

lusty quarry
elder tinsel
inland prawn
elder tinsel
#

Stuff like this comment in thread:

If you do your own build for other platforms, you could get a different tag. This includes the two (current) ARM variants of Windows, as well as occasional (unsupported) variants targeting UWP or WinRT.

seems to kill any idea that it's going to be exhaustive even on platform though unless you limit platform to things listed in the pep linked above.

west drift
lusty quarry
#

actually, why do you want a list? if you build for Windows and macOS and use the manylinux spec, wouldn't that cover literally everything?

#

(the manylinux spec in combination with the list of supported architectures I linked)

inland prawn
#

I plan on writing a package joining Rust and Python and wanted to avoid "you didn't cover my platform" issues by providing all possible wheels

elder tinsel
#

🤔 you should be able to just look at the platform+architecture support the rust compiler has and use that, that will be more limited than the possible places you could build cpython

west drift
inland prawn
mortal shore
#

someone is going to cry that you don't support NonStop or something

blazing lantern
blazing lantern
blazing lantern
lusty quarry
inland prawn
#

Distro-specific ones, like rpm, apt etc (and probably needs a special case for Nix, since it's kinda like homebrew, not bound to a distro)

lusty quarry
twin dagger
#

Hello - I have a question about a package name release request for an invalid package (no code or uploads, 8 years without anything added). Should I ask here, or in the pypi channel? I've made a ticket, but judging by the other tickets it's been a long time since any were actioned

blazing lantern
latent elbow
#

Hey does anyone know where I can download pypa3?

latent elbow
lusty quarry
#

help with what?

latent elbow
#

Do you know where I can download pypa3?

lusty quarry
#

what is that?

latent elbow
#

Idk I was just told to download it to start on this project

#

This is pypa isn't it?

lusty quarry
#

perhaps you have the wrong server, this is for packaging discussions about a programming language

latent elbow
#

This is the pypa discord is it not

twin dagger
# blazing lantern You can ask in the <#885686683507494913> channel, but if it's about ETAs or how ...

Yeah I figured as much, I think we are going to have to go for a different name. We can't wait 6+ months for the name to be released.
The project name we want has never had any code or installable package in nearly a decade, so it's just name-squatting. TBH I'm surprised PyPi allows this
Maybe in the future projects should have to have a minimum amount of code. Especially as it seems the only actioned tickets are those resolved by users...

#

@mortal shore ^

mortal shore
#

Eh, well we don't really allow it

#

but there's only ~4 people with the power to remove a package

#

and we're all pretty busy

#

and processing the name requests is a lot of time and effort

#

and it's hard to add new people, because granting someone the power to reassign names like that requires a very high level amount of trust

#

and most of the people who the 4 of us know well enough to trust with that power already, are also pretty busy themselves

twin dagger
#

Fair enough - I do think maybe there should be an automatic flag for projects uploaded with no code or package though

#

Especially if nothing is added after 8+ years

mortal shore
#

It's currently not possible to claim a name without uploading something, if there's a name on PyPI without any uploaded files it either means:

  • They uploaded a file and then deleted them
  • They did it many years ago before it was required to upload a package.

We could automatically release names with no files associated with them, but I suspect once the initial "burst" got released, very few extra ones would because people claiming names would just upload a dummy file instead.

There area also some squats that we generally consider valid-- for instance Microsoft could decide to squat the name "microsoft" and that would be fine.

merry rune
hollow trout
twin dagger
cunning lagoon
#

Hi, I'm not sure where this belongs best, so I'm writing here in general:
Since I read this blogpost I try to implement best practices regarding dependency management. My goals are:

  • Use only pyproject.toml as single ground truth defining the dependencies.
  • Support multiple python versions and in the future also multiple platforms.
  • Generate pinned requirements files or Pipfile.lock for CI testing (with multiple python versions) and development.
  • If possible use hashes.
    I'm currently using multiple pip-compile commands, to generate requirements/dev.txt and requirements/test.txt but they only generate requirements for the current virtualenv. If I understand it correctly, going further with pip-tools I would have to make an automation, which enables various environments and runs pip-compile for each and saves them to separate requirement files, e.g. requirements/py38/dev.txt. In the future, I would probably also add another layer and create requirements/linux_x64/py38/dev.txt. I have never seen this approach and it looks very verbose. But it is the suggested way of pip-tools as per their README.md: https://github.com/jazzband/pip-tools?tab=readme-ov-file#cross-environment-usage-of-requirementsinrequirementstxt-and-pip-compile)

I found multiple open issues:

So, if I understand correctly, this is an open problem? But I guess there are solutions. I checked some popular large packages, but did not find one, which really checks all marks.

What are your approaches? One compromise could be to only support one development environment (e.g. linux, x64, python3.10) and let the CI tests install via pip install ., i.e. from pyproject.toml. But I'm not 100% ready to accept that. What if a Mac user, or god forbid a Windows user joins my team?

blazing lantern
lusty quarry
#

I agree with Brett, and I'm sorry about the current situation

gritty roost
inland prawn
#

Poetry goes with assumption, that all wheels use the same set of dependencies based on markers

gritty roost
#

It seems like a good assumption. It makes the problem tractable

#

And I guess as Poetry and PDM continue to gain popularity, there is increasing pressure on package authors to make their packages work with those tools.

ivory phoenix
lusty quarry
#

I amended my comment lol

ivory phoenix
#

i agree that the reference implementation from the PEP is easier to translate into other languages

#

filter+lambda from Python can intuitively be written in JS/TS/Dart

#

but when it comes to Python-only, i would argue that it's better to use generator expression instead of filter+lambda most often

#

primarily due to readability and "Pythonicity"

#

filter wins when there are C functions, so filter(str.islower, s) > (ch for ch in s if ch.islower())

cunning lagoon
#

Thanks Brett, ofek, jvolkman and Secrus for the input! I'll take a look into Poetry or PDM. I refrained until now, since I want to keep dependencies at a minimum and work with standard tools.

lusty quarry
cunning lagoon
# lusty quarry it's not something that we directly support but you can use Conda

Yeah, no thanks. In my 6+ years of programming Python, I never had a good experience with it. I always ran into unsolvable environments. Waiting hours for that slow program to solve them. Sometimes it solved them after more than two hours. Really started understanding the halting problem, by watching that spinner animation.

lusty quarry
#

I don't use it anymore but I can confirm that they have fixed the slow resolution. Over the past 1-1.5 years they rewrote the resolver in Rust using state-of-the-art algorithms.

cunning lagoon
obsidian lily
#

PEP 723 seems uhh

interesting to say the least

lusty quarry
obsidian lily
#

imo, i dont really understand the justification of not using a requirements.txt format, as the format would use something similar for the dependencies section

queen hornet
#

you can define everything in a single file. this lets you have many single-file scripts in the same directory (like a bin or something)

obsidian lily
#

why not like

# /// script
# my_package==3.14
# ///```
cunning lagoon
#

I'm no expert on this. But the PEP says: "And furthermore, the requirements file format is pip-specific, so tools relying on it are depending on a pip implementation detail."

obsidian lily
queen hornet
#

ah sorry, you asked about the format. there was a lot of discussion about the format, with concerns pyproject.toml/TOML is hard to learn. but Microsoft did some user testing and the result was that it wasn't such a big deal after all

so I think the idea is that if a single script is later promoted into a project, you already some TOML format to create the pyproject.toml

obsidian lily
#

requirements.txt and pyproject.toml arent mutually exclusive

#

pyproject.toml's way of defining requirements are just to slap em in a list with essentially the same format

obsidian lily
queen hornet
#

sure, but that was a concern

obsidian lily
#

what build tool does everyone use for normal python projects (setuptools, pdm, hatch, poetry, ...)?

queen hornet
#

some usage graphs here ^

obsidian lily
queen hornet
#

I usually use Hatchling for pure Python projects, or setuptools for projects with C extensions

obsidian lily
#

i personally use setuptools

#

for like everything :P

lusty quarry
#

@obsidian lily I think you might have missed the part in the spec where you can define the Python version requirement which would preclude your idea of a straight requirements.txt dump

lusty quarry
mortal shore
#

I mostly use setuptools, but that's more or less because I've always used setuptools and nothing it's done has made me care enough to figure out another tool

uneven karma
#

Me too
I also use setuptools' support for requirements.txt files to help Snyk understand my dependencies

inland prawn
#

First packaging tool I came across was Poetry and I loved the all-in-one feeling, so I just went with it all the way to maintaining it lol

indigo token
#

Hatchling. It feels like it has no ceiling. It allows to do as many things as you could imagine, and if you can imagine more, it allows to easily extend it with ad-hoc and shareable plugins.

And Maturin, because Rust is a pleasure to write.

lusty quarry
gritty roost
obsidian lily
west drift
obsidian lily
#

wow a lot of people are joining

blazing lantern
blazing lantern
queen hornet
obsidian lily
#

nooo setuptools

lusty quarry
gritty roost
#

Are you cross-compiling wheels with zig?

lusty quarry
#

yes because the beta for GitHub ARM runners is not yet ready (and apparently what we'll be getting is a blank image with nothing preinstalled)

gritty roost
#

I've been using zig for a couple of years now with Bazel and it makes cross compiling (relatively) easy and fun. Even allows me to build macos wheels on a linux build host.

inland prawn
#

@lusty quarry care to share your cross-compilation setup?

lusty quarry
#

it's not yet complete but I will link it here when it is, it's a public repo

lusty quarry
blazing lantern
lusty quarry
#

macOS did not Linux

fading veldt
#

Are there any plans for a packaging summit at pycon this year?

queen hornet
fading veldt
#

Great! I'm very interested in joining, I have two or three PEPs I hope to work on

lusty quarry
#

did the accepted PyCon talks get announced yet?

queen hornet
#

not yet, I heard in the core.py podcast they're pretty much decided on, but not yet officially

gritty roost
fading veldt
#

Is there a list of known projects implementing a given packaging standard? E.g. Pip and poetry for installers, setuptools, hatchling, flit, etc for build backend...

It would be useful to know what exists out there

#

Especially for new PEPs so authors can make sure they get implemented in the right ecosystem e.g. I want to propose a change to metadata, but what projects are important to that? I know a few (backends, installers, etc), but not all of them.

noble matrix
# queen hornet Yes, <@333125359811952640> is helping arrange it

More specifically @high stone , @tidal kiln, @hidden flame and myself 🙂 Now that we got the date coordinated with the other summits, Saturday afternoon of the second main conference day we're getting the various bits ready and plan to announce in the next few days

fading veldt
#

Thank you all for organizing 🙂

high stone
inland prawn
#

I hope one day I can attend

honest geyser
inland prawn
queen hornet
elder kettle
#

It's possible I'll finally attend this year 🙂

queen hornet
inland prawn
#

well, I guess until we get a good, stable standard for extension modules packaging, setup.py will still be there

hollow trout
#

It's weird to have this framed in terms of setup.py vs pyproject.toml, when it's perfectly valid to have both.

inland prawn
#

well, I guess it's "the old way" vs "the new way"

#

some projects also keep setup.py for GH discoverability (which now works with pyproject.toml too)

spiral urchin
#

Yeah, doing vs. doesn’t work also since many projects use pyproject.toml only for non-packaging configs (remember when Black was the only reason people use the file?)

#

The argument on extension modules is also somewhat awkward considering literally everything in setup.py is non-standard…

inland prawn
#

yeah, but it is a kind of standard for extension modules. might be non-standard, but there is nothing as popular right now (outside of maturin maybe)

restive venture
#

hi! quick question: is it guaranteed that setup.py will always be executed with the cwd of its containing directory?

#

that is, if I'm writing code to single-source package version, can I just use relative paths or do I need to manually do a Path(__file__).parent?

#

(this is a pre-existing project with a setup.py, so I can't use [tool.setuptools.dynamic] or setuptools_scm or anything else)

hollow trout
#

I don't think this can really be guaranteed, since someone can always run it from whatever directory they want. That said, I think setuptools generally assumes that the current directory is the project root.

#

Also, PEP 517 specifies that build hooks are invoked with the working directory set to the project root, so assuming someone uses standard mechanisms to install your project, that should always be the case.

restive venture
#

thanks!

lusty quarry
#

whelp, looks like I'll finally have enough motivation to work out the logistics of attending PyCon 😄

image.png
blazing lantern
lusty quarry
inland prawn
#

(I am clueless about previous lockfile attempt, I only know it didn't support sdists, was based of files not releases and that it failed) what's wrong with Poetry's resolver? What issues doesn't it address, compared to what you want to achieve with a "standard lockfile"?

lusty quarry
#

what do you mean by being based on files rather than releases?

#

in theory we could standardize on one of the existing file formats

inland prawn
#

but the question stands

#

(not that I am trying to force something, I am genuinely interested if there is something wrong with Poetry's lockfile)

lusty quarry
#

I think poetry (if I understand correctly) dumps basically everything in the file. this in my opinion is wasteful and causes extra complexity during installation. my view is that as much work as possible needs to be done during resolution to keep installation logic to the absolute minimum

inland prawn
# lusty quarry I think poetry (if I understand correctly) dumps basically everything in the fil... 
[[package]]
name = "build"
version = "1.0.3"
description = "A simple, correct Python build frontend"
optional = false
python-versions = ">= 3.7"
files = [
    {file = "build-1.0.3-py3-none-any.whl", hash = "sha256:589bf99a67df7c9cf07ec0ac0e5e2ea5d4b37ac63301c4986d1acb126aa83f8f"},
    {file = "build-1.0.3.tar.gz", hash = "sha256:538aab1b64f9828977f84bc63ae570b060a8ed1be419e7870b8b4fc5e6ea553b"},
]

[package.dependencies]
colorama = {version = "*", markers = "os_name == \"nt\""}
importlib-metadata = {version = ">=4.6", markers = "python_version < \"3.10\""}
packaging = ">=19.0"
pyproject_hooks = "*"
tomli = {version = ">=1.1.0", markers = "python_version < \"3.11\""}

[package.extras]
docs = ["furo (>=2023.08.17)", "sphinx (>=7.0,<8.0)", "sphinx-argparse-cli (>=1.5)", "sphinx-autodoc-typehints (>=1.10)", "sphinx-issues (>=3.0.0)"]
test = ["filelock (>=3)", "pytest (>=6.2.4)", "pytest-cov (>=2.12)", "pytest-mock (>=2)", "pytest-rerunfailures (>=9.1)", "pytest-xdist (>=1.34)", "setuptools (>=42.0.0)", "setuptools (>=56.0.0)", "setuptools (>=56.0.0)", "setuptools (>=67.8.0)", "wheel (>=0.36.0)"]
typing = ["importlib-metadata (>=5.1)", "mypy (>=1.5.0,<1.6.0)", "tomli", "typing-extensions (>=3.7.4.3)"]
virtualenv = ["virtualenv (>=20.0.35)"]

is that wasteful?

#

what would you take away?

lusty quarry
#

I view (which I think? Brett shares as do the folks at prefix.dev) that the future lock file will actually have targets and basically have a resolution result per target

inland prawn
#

the only thing I see is description field

#

so, multiple lockfiles?

lusty quarry
#

the folks implementing rip are doing it with top level targets which is the conclusion that I came to last year (which like I said if I remember correctly Brett also concluded)

#

basically yeah you could say that

elder tinsel
#

It becomes more noticably a problem when you see a set of hashes where you actually support a version range and multiple platforms with native dependencies

lusty quarry
inland prawn
lusty quarry
inland prawn
elder tinsel
#

no?

lusty quarry
#

yeah I don't think it would cause that much extra computation during resolution

elder tinsel
#

and standardizing this makes resolving with locks sourced by multiple dependencies to be simpler, as you should just be able to merge the constraints

lusty quarry
#

and if we hit pathological cases then I'll simply write the logic in Rust

inland prawn
blazing lantern
lusty quarry
#

the other thing that no one talks about that I will definitely have to put in the PEP is the concept of patches

#

so specific dependencies in the resolution can be modified to point to a fork for example

elder tinsel
#

I feel like you're retreading the ground that conda has already solved, may want to look there.

#

(Not saying to just use conda, but look there for what problems they ran into to skip past needing to relive those issues)

blazing lantern
lusty quarry
blazing lantern
elder tinsel
#

Oh, I actually meant on handling patches/forks

inland prawn
# lusty quarry yeah I don't think it would cause that much extra computation during resolution

well, it makes lock verification longer. right now with each package listing it's dependencies, you can quickly look up if you have it in lockfile already. with only file pointers, you would have to get the file, get the metadata, resolve dependencies to their files and then you can be sure that it's fine. if wheel metadata had files/hashes of dependencies included, then your solution would be fine

elder tinsel
#

but glad to see that was taken into account there too

lusty quarry
elder tinsel
#

if you're concerned about the cost of file open + read per dependency, I have a feeling you're looking at the wrong cause of slowness

inland prawn
blazing lantern
#

Also note my work is to also fill in API gaps, so even if it doesn't lead to a lock file standard, at least the pieces to build an installer will be there

elder tinsel
#

and there's no reason the resolver can't cache it's knowledge internally in a single db

blazing lantern
inland prawn
blazing lantern
#

And if you want to propose Poetry's format for standardization please do! I want something and I'm not going to be that picky about whose approach ultimately gets chosen

elder tinsel
#

I'd be pretty happy with a single dependency_hashes.toml with sections per platform

inland prawn
#

I am just a bit tired with the ecosystem going rounds and rounds reinventing the wheel, building the same tools again and again and not getting together, using what's already there and potentially improving it

elder tinsel
#

but I'd rather not shove a load of hashes into pyproject.toml

inland prawn
inland prawn
lusty quarry
#

I'd be okay seeing a PEP but since you are so close to Poetry and so experienced, please consider what you've learned and think about if you might come up with a different format from first principles

inland prawn
#

I am not saying Poetry's format is perfect

#

I was just curious on opinions about what is wrong with it

lusty quarry
#

you would probably be the best person to explain that actually 🙂

inland prawn
#

I might be a bit too biased for that 😛

ivory phoenix
#

👀

inland prawn
#

and I am not that experienced yet, still exploring the packaging wildlife and trying to make sense of it 😄

lusty quarry
#

also, now might be a good time to write the PEP because I have no free time and I have a history of getting nerdsniped (my two accepted PEPs have been in response to ones that were open that I was not fond of) :p

#

so no shot I could write a competing one before May

ivory phoenix
inland prawn
#

well, I doubt I will feel competent enough to write a PEP

#

but the idea of it is interesting

ivory phoenix
#

gotta get some poetry folks to help

blazing lantern
blazing lantern
blazing lantern
inland prawn
gritty roost
lusty quarry
#

I don't understand what it's doing

blazing lantern
#

At least as long as the format makes a fast install (e.g. linear reading of packages to install)

inland prawn
#

Don't tempt my ADHD! I already have too many projects started as it is 😛

blazing lantern
ivory phoenix
#

me too 😭

#

i even made time-based milestones for the next 60 years to try to manage them all somehow

image.png
#

hopefully i'll live that long

gritty roost
# lusty quarry I don't understand what it's doing

yeah, there's a lot to unpack if you don't have bazel experience. But basically:

So ipython depends on 7 packages on all of the defined platforms and then one or two additional packages depending on the specific platform.

obsidian lily
west drift
blazing lantern
#

So I wouldn't say "need" as much as say "could have"

honest geyser
obsidian lily
#

wow

#

its been a lot more active lately

blazing lantern
honest geyser
lusty quarry
honest geyser
#

We're very interested in the future of multiplatform lock files

lusty quarry
#

yeah that's the hard part lol

#

@honest geyser how does it know the platform tags to look for without a Python flag?

honest geyser
#

It's re-implemented in Rust

#

Is that what you mean?

#

(I'm not the expert on this part)

lusty quarry
#

sorry I should be more explicit, without knowing the target Python what are the defaults? for example if I'm targeting Python 3.11 it seems there is no way to instruct it as such

honest geyser
#

You can compile with a --python-version, otherwise we will infer it from your current / installed Python

lusty quarry
#

You can compile with a `--python-

merry rune
#

Would UV be able to make venvs and replace pipx and build too? That would be amazing. 😉

lusty quarry
#

don't know if you heard but pipx feature is coming to Hatch as a top-level command group but with a different name

blazing lantern
inland prawn
#

they plan on making uv pipx too

blazing lantern
west drift
merry rune
#

Ah, great. Then all the parts are there to provide a build replacement too. All you need are an installer and a venv creation tool. I might even be able to help with that.

merry rune
#

it lacks support for some of its [pip] legacy features, like .egg distributions.
I thought pip dropped support for those a while back?

manic belfry
merry rune
#

Actually, maybe the other direction would be better. If uv had a Python API (only in the pip package version, of course), then build[uv] could be added which would optionally allow you to get the speed boost of using it over venv/virtualenv. (We already support the same optional feature with virtualenv vs. venv).

#

And I assume most other tools like nox, etc. could optionally use it as well if installed.

obsidian lily
#

uv represents an intermediary goal in our pursuit of a "Cargo for Python": a comprehensive project and package manager that is extremely fast, reliable, and easy to use.

i feel like this already exists with pdm/poetry

#

they already have a lot of good dependency/project management utils

torpid veldt
#

I'm having some trouble with pip-audit showing dash-core-components and dash-html-components as vulnerable - is there a pip-audit channel?

#

the vuln id is GHSA-547x-748v-vp6p

spiral urchin
#

I was about to say that’s where pip-audit gets the vulnarabilities

torpid veldt
#

but it doesn't say dash-html-components is vulnerable

spiral urchin
torpid veldt
#

but it's not vulnerable because it doesn't have any code in

#

it's just an import for the main dash project

spiral urchin
#

Need to tell the database maintainer (whoever that is) to fix that I guess. @strange knot should know more about what to do (might need to reach him elsewhere; I don’t think he’s particularly active here)

torpid veldt
#

I was looking to PR the advisory-database but it doesn't seem to have a purl for dash-html-components eg purl: pkg:pypi/dash

queen hornet
torpid veldt
#

but it defaults to pypi so it's not that

queen hornet
#

so is the problem that:

  • a vulnerability was fixed in the dash package on PyPI
  • dash contains a sub-package called dash-html-components
  • dash-html-components used to be a separate package on PyPI, but is now just an "empty" wrapper that issues a deprecation warning and imports the main package
  • dash-html-components on PyPI shouldn't therefore be marked as vulnerable?
torpid veldt
#

yes

#

interestingly the same applies to dash-core-components but it's not marked as vulnerable

#

dash-html-components and dash-core-components v1.x is vulnerable but 2.x is not

queen hornet
#

@mild dew can probably help advise, he's also a pip-audit maintainer

torpid veldt
spiral urchin
#

We got a spammer here

zealous swan
torpid veldt
#

There's a false negative and false positive

torpid veldt
zealous swan
torpid veldt
blazing lantern
#

This is mostly for @inland prawn , but what usernames should I make sure get cc'ed at https://discuss.python.org/ for my next lock file discussion? (I already have Ofek for Hatch, Frost for PDM, konstin and Charlie for uv/Rye, and Paul somewhat implicitly for pip.)

inland prawn
queen hornet
inland prawn
blazing lantern
lusty quarry
#

nvm, I understand, there are multiple unnamed lock files and one would iterate and check compatibility with the child tags array. brilliant

honest geyser
zealous swan
#

So had a thought regarding reproducibility of source dist -> wheel while reviewing the lock proposal. I've heard many folks thinking about sdists as a tarball of VCS. Many reproducible builds guides note that you should use SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) for git-based builds, but that information is lost when creating a source distribution. Should there be a capture of this value in the sdist (metadata field?) that gets automatically pulled from VCS if available? Then build backends can start utilizing this value since it'll be available in more builds (rather than relying on users to set this value themselves).

elder tinsel
#

shouldn't be neccessary. The original timestamps for file modification are preserved in zip (And therefore also in wheel)

zealous swan
lusty quarry
#

that would be useful but just FYI that's not the main issue/trouble with source distributions. the hard part is entirely about the dynamism of metadata like dependencies

elder tinsel
#

I don't see that as the job of the source distribution to handle, that's the job of redistributors to make the correct choices in using the source distribution

#

It's entirely reproducible using the creation date exposed by pypi (And other indexes) or from git if they pull from there, the redistributors just need to pick whatever date is appropriate

zealous swan
#

@lusty quarry I agree that there are more than one issue with sdists 😛

zealous swan
elder tinsel
#

same difference. Whoever is pulling the sdist can use the date of the upload for a consistent date

zealous swan
#

To be clear this is a super-low priority issue, it's just something I thought of while reviewing since reproducibility was mentioned.

#

I think we can both agree that things that don't happen automatically are usually as good as not happening? Most folks aren't setting SOURCE_DATE_EPOCH per call to pip install.

elder tinsel
#

most people don't benefit from doing so. if your install has meaningfully different behavior on a different date, it's probably broken outside of the case of people that care about bit for bit reproducible builds.

inland prawn
elder tinsel
# zealous swan I think we can both agree that things that don't happen automatically are usuall...

if it became both feasible and a goal of pip to ensure reproducible builds* from sdists, it might be worth making it happen automatically. I don't think this is a generally reasonable goal for the primary audience of pip or even most security conscious developers. A chain of trust from trusted source code and a trusted build system is extremely sufficient. In the cases where it isn't, I think people still over-value reproducible builds* and we should be more focused on finding better ways to ensure code equivalence so that those interested in the security aspects of reproducible builds don't end up tying hardware to the reproducibility.

*bit for bit, not just identical code and mathematically equivalent machine code

lusty quarry
#

speaking of this, I literally can't get stuff to compile reproducibility on CentOS 6

#

I'm setting the environment variable, etc. and for some reason it just likes to produce new stuff every time

#

which sucks because it's forcing us to increment the wheel build numbers every time we have to upload

lusty quarry
#

does anyone have an example of a large project using the build number part of the wheel spec?

queen hornet
fading veldt
#

Very interesting! I'm also quite interested in this actually.

blazing lantern
lusty quarry
#

nice, thank you!

frigid jewel
#

is there a way to query how many libraries create a "bindings" (site-packages/bindings/) package or similar?

quasi spire
quasi spire
frigid jewel
#

i don't think it's convention or anything however

quasi spire
#

does it install bindings as a namespace package?

#

if not, then it's doing the wrong thing

#

what library is this?

frigid jewel
#

zaber-motion

frigid jewel
quasi spire
#

I don't see anything referring to "bindings" there though?

frigid jewel
#

hmm i might be wrong as to how they create that package then

quasi spire
#

hm, I looked at the wheel and indeed it did come with a bindings/zaber-motion-lib-linux-amd64.so

#

but the bright side is that it will show up as a namespace package indeed

#

so it's less bad than I feared

#

but no, I don't think there's any way to scan for similar packages very easily

frigid jewel
#

gotcha

#

there's another library which drops it's DLL in Lib/

#

i feel there should be a rule against that

quasi spire
#

which library, and how does it do that? or do you mean site-packages/Lib?

frigid jewel
#

nope

#

one second.

#

it lives in venv/Lib/

quasi spire
#

which library?

frigid jewel
#

it's gmsh

#

was looking for it my bad

#

Directory of C:\Users\Divine\Desktop\OSS contributions\nuitka-2649-gmsh\venv\Lib

01/15/2024 02:37 PM <DIR> .
01/15/2024 02:37 PM <DIR> ..
01/15/2024 02:37 PM 86,433,792 gmsh-4.12.dll
01/15/2024 02:37 PM 356,182 gmsh.jl
01/15/2024 02:37 PM 41,059,108 gmsh.lib
01/31/2024 02:26 AM <DIR> site-packages

quasi spire
#

oh, but gmsh actually distributes that as package data

#

too bad they don't distribute a source package

frigid jewel
#

so this is okay in PyPa/pypi eyes?

quasi spire
#

this is the proper way to distribute such things

frigid jewel
#

that doesn't feel right, feels like it should at the very least be under site-packages/gmsh/

#

also, this is the first library i've seen that moves something higher than site packages.

toxic cypress
#

Can anyone suggest a Python module for cloning git repos? I'd like to clone a repo (that doesn't represent a Python module) within setup.py and use it to generate ctypes definitions

queen hornet
mortal shore
#

talking to the network from within a setup.py is a bit of an anti pattern fwiw, particularly if you're sharing this with others

toxic cypress
toxic cypress
mortal shore
#

I would just bundle the header files or whatever it is you're downloading, rather than downloading them

toxic cypress
#

Why?

mortal shore
#

there's no guarentee that you're going to have access to whatever the URL is that you'r etrying to fetch from on your user's machines

#

e.g. many places run internal mirrors of PyPI and otherwise block internet access in CI

toxic cypress
#

I don't expect that will be the case with this project, though

woven plover
#

Why not just use a git submodule?

gritty roost
#

If you're set on network access in setup.py, maybe consider including a mechanism that allows the user to bypass it provide the necessary repo files themselves.

oak quiver
#

iirc pypi will look for references to making network requests in setup.py to prevent malware? Something to consider if you want to publish the package.

toxic cypress
#

I've decided to just use a git submodule for now, as I was overthinking it. I'm working on another project unrelated to Python that fetches its dependencies over Git, but in a way that's much more flexible (very easy to pick a specific fork/branch/commit/local installation)

#

So that's what got me thinking about doing so here

placid wyvern
#

Warehouse used to have some built in YARA rules and whatnot, but they were torn out a few months ago IIRC.

#

I do, however, get notified every time someone uploads what we consider to be atypical setup.py structures. It's quite noisy. Think of my sanity before you import networking packages into your setup.py's. 😆

oak quiver
#

That’s actually hilarious lol. How often are you getting pings about that?

placid wyvern
elder tinsel
#

huh... maybe I should wait before I make the next post I was going to make about the current state of supply chains, I didn't realize those yara rules werent in effect and I had my own thoughts on them

placid wyvern
#

setup.py is a very common vector to attempt to deliver some sort of malicious behavior.
So the extra focus on that particular venue is well placed.

#

To be clear we do not interface with PyPI in any automated way, so... your packages are safe if you're making network requests. I just get to read them. 🙂

fading veldt
#

I am working on a build backend that runs in WASI because arbitrary code execution is annoyingly useful but being able to set capabilities about what an installer script can do would be nice

placid wyvern
#

That sounds neat! Yeah-- it's a really complicated subject; if we start locking down the behaviors of build systems, they become less useful, but more secure. It's a give and take. I don't have a very strong intuition for how we got here (or even, the build systems that exist out there) except that... I've seen just about everything under the sun at some point, and there's quite a bit of 'unique' out there on PyPI.

#

My current endeavors are largely focused on detecting APT activity, which is something that A.) Happens reasonably frequently and B.) Is... understandably quite difficult to detect.

fading veldt
#

The one thing I am having difficulty figuring out if it is realistically doable is building C/C++ code

obsidian lily
storm canyon
#

<@&815744082823348274>, spam ^

inland prawn
#

<@&815744082823348274> spam

high stone
#

FWIW, the spam was dealt with in ~1 minute. 😬

obsidian lily
#

admin efficiency 📈

lyric hedge
#

next time we should use incident_investigating when actioning a ping to avoid duplicate admin work :P

#

i'm mostly joking but pradyunsg was 🏎️

obsidian lily
lyric hedge
#

i've been an admin since a while

obsidian lily
#

hu h

lusty quarry
#

okay, I think I'm overthinking and thus being stupid. is there a way using packaging to extract the extra from the string foo; extra == 'bar'? I can't find a way

#

I can use internal stuff I suppose but it looks so hacky

image.png
indigo token
lusty quarry
civic compass
#

how do I flag a package as out of date?

torpid veldt
#

And the old ones get a little red banner

quasi spire
#

are there any pypa projects using codecov? it looks like the new codecov github action needs to be explicitly passed a token

queen hornet
quasi spire
#

hm, bandersnatch is also not providing the token

#

doesn't look like any of the pypa projects using codecov has a working solution in place

#

alright, I just got the token from codecov.io and started using that, problem solved.

civic compass
torpid veldt
civic compass
torpid veldt
#

Released: Oct 22, 2015

#

That's pretty outdated

torpid veldt
civic compass
#

how would I make a fork?

torpid veldt
#

You'd have to have on already and get it fairly popular

#

What's the outcome you're after? You can't take down other people's packages for being outdated

civic compass
#

get it updated to use wget commands

torpid veldt
#

Looks like the source code got deleted off bitbucket

queen hornet
fallen shuttle
#

alright, I just got the token from

toxic cypress
#

You know how PyTest expands assert statements if they fail? I really want this, but my testing needs are spoken for. Can anyone suggest another library that just explains failing asserts?

Screenshot_2024-03-12_184332.png
#

...ah, you know what, PyTest may actually be okay for my needs after all

spiral urchin
#

They do this by overwriting the pyc code generated by your Python code. Take a look in _pytest/assertion/.

toxic cypress
#

Neat.

#

Does anyone else do that?

inland prawn
#

You mean for tests or overall?

toxic cypress
# inland prawn You mean for tests or overall?

For tests. This is for a test suite, but the thing being tested isn't actually written in Python. Only the test scripts are in Python; the thing being tested is a compiled binary with a particular C interface

#

So...I guess since the tests are being orchestrated and run elsewhere, I suppose I'm looking for assert expansion outside of a test framework

spiral urchin
#

I think you need some sort of framework to make this possible. Pytest does into this by hooking into module loading at import time, to replicate it you’d also need a way to manage modules besides plain imports. (Or maybe it’d be possible with some importlib magic? Not sure.)

toxic cypress
#

Hm...that would add unnecessary complexity for me right now

#

Thanks for the tip, though

toxic cypress
#

Can ctypes be used to define variadic callback functions? I ask because I'd like to implement the following C interface in Python;

enum retro_log_level
{
   RETRO_LOG_DEBUG = 0,
   RETRO_LOG_INFO,
   RETRO_LOG_WARN,
   RETRO_LOG_ERROR,

   /** Defined to ensure that sizeof(enum retro_log_level) == sizeof(int). Do not use. */
   RETRO_LOG_DUMMY = INT_MAX
};

/**
 * Logs a message to the frontend.
 *
 * @param level The log level of the message.
 * @param fmt The format string to log.
 * Same format as \c printf.
 * Behavior is undefined if this is \c NULL.
 * @param ... Zero or more arguments used by the format string.
 * Behavior is undefined if these don't match the ones expected by \c fmt.
 * @see retro_log_level
 * @see retro_log_callback
 */
typedef void (RETRO_CALLCONV *retro_log_printf_t)(enum retro_log_level level, const char *fmt, ...);

struct retro_log_callback
{
   /**
    * Called when logging a message.
    *
    * @note Set by the frontend.
    */
   retro_log_printf_t log;
};

The Python function I want to write needs to be compatible with the definition of retro_log_printf_t.

lusty quarry
#

you'd have better luck in the Python server

toxic cypress
#

Not so far.

inland prawn
#

Well, definitely more than on packaging server

toxic cypress
#

I know, I know...I figured it was worth a shot anyway, since this server is full of Python users

lusty quarry
#

try asking one of the top AI chat bots, my favorite currently is Claude 3 Opus, anecdotally seems slightly better than the best ChatGPT that is available

fading veldt
torpid veldt
torpid veldt
lusty quarry
inland prawn
#

<@&815744082823348274> Please ban

high stone
#

Done, and also oof.

topaz pine
#

Just saw all the channels with the "unread" mark and thought "oof, another spammer here" :/

#

thanks for removing it! 🙂

frigid jewel
jade zephyr
#

Yep, just ran into that in the middle of deploying a new release.

jade zephyr
#

I can't imagine it's intentional, the message is likely correct just misconfigured restriction.

jovial holly
#

sorry for that! it was late and i just needed the inbound to stop :/

jade zephyr
#

We understand, thanks for your work 🙂

vital pier
#

@queen hornet I just noticed some of the css changes on the pydocs theme - just wanted to say it's really cool to see these changes!!

queen hornet
#

thanks! the font change makes a big difference 🙂

spiral urchin
#

<@&815744082823348274> here we go again…

high stone
#

Dealt with that. Also bumped up the moderation settings on the server.

queen hornet
#

what setting is it now?

lyric hedge
high stone
#
  • verified email + more than 5 minute old account for joining the server (which were required earlier as well).
elder tinsel
#

docker scout started showing this old CVE (despite not showing it before recently) for pip that's arguably a non-issue and scored inappropriately to people https://scout.docker.com/vulnerabilities/id/CVE-2018-20225?s=gitlab&n=pip&t=pypi&vr=>%3D0

Internally at work, we were able to quickly deal with supressing the noise it caused, determining that its a non-issue, but it might be something someone from pypa should reach out to someone from docker about.

uneven karma
elder tinsel
high stone
#

I'm wondering if anyone knows of what other servers have done for dealing with the spam bots.

We're at all but the highest level of verification (high requires a validated mobile number, which even I don't have on Discord).

elder tinsel
#

It's just something mods have to deal with on public servers. There are "low bar" methods that stop some of the fully automated scrape public invites + spam bots like having a verification channel that users have to post a specific message to gain access based on reading something and following directions + a discord bot to assign a role

uneven karma
#

I just set my server to require a phone number from the start
Haven't had a problem with spammers yet
Having a role exempts you from the phone number requirement, so if someone doesn't have a phone I just give them a role and all is well

fading veldt
#

I help moderate a public server that constantly gets spammed, what we do is use a bot to help filter messages and moderate. Basically it will block sending a message if it is clearly spam (based on message content and keywords) it works pretty well.

#

Happy to help with setting that up. The bot we use is called Vortex, but there are many out there

queen hornet
#

with some servers you're stuck in the welcome room until a bot or human validates you

fading veldt
#

I feel like human verification might be extreme, but adding a "react to the packaging_platypus icon to confirm you're a human" might work well, I've seen that used in other servers

inland prawn
#

yeah, but that's not a big issue to script over it

elder tinsel
#

it stops a lot of the really low effort spam that just joins servers it scrapes links for, then sends messages to every channel it can access that way still

fading veldt
#

Which is a lot of the bots

west drift
#

Hey i'm currently doing research for a new non-pip-copy interface for uv, with focus on the interaction with PEP 621

#

My question is: Are you writing a tool that reads pyproject.toml's project.dependencies? Are there any non-resolver users, and if so, what information do you need?

#

I'm trying to figure out what information we need to provide so that the table is useful to others

ionic beacon
#

@low drift <@&815744082823348274> the survey in the pinned message 404s?

inland prawn
#

it's over 1.5 years old, I guess the survey ended

high stone
#

There's no longer a pinned message. :)

uneven karma
high stone
#

Isn't a 100% solution, but also, I expect stuff like this shouldn't really affect anything around packaging workflows.

west drift
high stone
#

Yea, basically post-facto analysis of the pyproject.toml files.

#

Not a part of the development workflow though; sits outside as a way to look at how people are using a thing (eg: measuring adoption etc) as well as tracking migrations etd.

west drift
#

gotcha, thanks for the info

empty blaze
#

Is there a strong reason to ever use pipx over venv?

inland prawn
#

Those are 2 different tools

empty blaze
golden falcon
#

is there a strong reason to ever use venv over PYTHONPATH? These are all tools which make it easier to accomplish certain tasks. Creating isolated envs in the case of venv and installing/running Python CLIs in the case of pipx

hollow trout
#

To be pedantic, PYTHONPATH will not suppress system-wide site-packages, whereas venv will. 😛

#

pipx, as far as I know, is just a convenience wrapper over venv.

inland prawn
#

yeah, pipx is pip+venv with nice interface

golden falcon
indigo token
#

@empty blaze to answer your question: If you want a tool available from the command line in multiple projects, you should use pipx rather than managing that tool’s venv manually.

empty blaze
high stone
#

OK, I think I managed to configure automod so that it prevents these messages in the future.

high stone
#

And, now this server should be more spam-free. Thanks for bearing with us mods as we've tried to figure out what to do with these stupid spam bots and how to keep them out.

topaz pine
#

Thanks to you, moderators! This is a safe server thanks to y'all 🙂

manic belfry
queen hornet
#

pip install git+https://github.com/pypa/pip.git@refs/pull/12300/merge to install from PR

queen hornet
#

I got this trying to upload a wheel or sdist from Jazzband to PyPI, what am I missing?

Uploading distributions to upload.pypi.org/legacy �[31mERROR �[0m InvalidDistribution: Metadata is missing required fields: Name, Version. Make sure the distribution includes the files where those fields are specified, and is using a supported Metadata-Version: 1.0, 1.1, 1.2, 2.0, 2.1, 2.2.

it used the latest build and pkginfo to make the packages, and latest twine to upload to Jazzband.

but then it failed uploading from Jazzband to PyPI. maybe Jazzband has an old twine or something? I don't know what versions it has

(https://github.com/jazzband/prettytable/issues/298)

#

also using hatchling and hatch-vcs

lusty quarry
#

yes the upload tooling needs to be updated for metadata 2.3

queen hornet
inland prawn
#

jazzband has a separate package index?

queen hornet
#

yes, it's just used as a staging area before they get uploaded to PyPI

uneven karma
#

They have their own special release CI

#

I don’t really understand why

uneven karma
#

Is there anything actually stopping you from just… not using it?

queen hornet
queen hornet
#

as a lead, you usually don't have PyPI access either

#

this was all set up before Trusted Publishing, so I think it could be safely replaced with that and cut out the middle step

uneven karma
#

firHmm
Anyone with write access can create a release, right?
That doesn’t really help there.
But GitHub has allowed you to require approvals for CI jobs for… years. That sounds like a much simpler way to handle this.

#

Would also allow the leads (assuming they have admin on their own repos) to update the rules themselves

#

I don’t know the history here, and I’m just barging in here, but that sounds like a good spot for a feature request, at least to review the viability.

queen hornet
uneven karma
#

I thought it had been around for a while
I'll see if I can find it

queen hornet
uneven karma
#

oh

#

oof

#

That's a... giant bottleneck

queen hornet
#

yup

uneven karma
#

There's many comments on the "need more admins" issue
Did he never pick anyone else?

queen hornet
#

I don't think so

uneven karma
#

Jazzband is probably way older than that

#

Anyways, I do think that would be a cleaner solution, but apologies for barging in and rambling

queen hornet
toxic cypress
#

I'm having problems in generating documentation for a Python package I'm writing. Which channel is on-topic for that?

inland prawn
lyric hedge
#

I was trying to ban them but couldn't find the ban button, I suppose someone else got to it first, haha :)

#

I was like "this is a really inconvenient time for my ban button to disappear... ah they're already gone"

trim haven
#

I think I must've beat ya to it

high stone
#

We've got so many banned accounts in the last few weeks. 🙈

inland prawn
#

is interpreter tag cp310-none-any the same as cp3_10-none-any?

#

(both are produced by packaging.tags on the same machine, just using different functions)

lusty quarry
#

I've never seen the latter

inland prawn
#

I guess confusion appeared from here

The version is py_version_nodot. CPython gets away with no dot, but if one is needed the underscore _ is used instead.
#

anyway, solved my problem 😄

lusty quarry
#

I'd like advice:

I want to have Hatch ship UV in the next release. What I'm doing now is the same thing I do for Ruff which is manage a dedicated environment just for that. What I'm realizing however is that nearly the only reason I'm doing this is because I don't want to have redistributors open issues on the repo for packaging changes/hardships.

Is there another reason to not ship dependencies directly that are only used via certain flags/config options?

#

At a high level it feels very strange that I'm letting project-level decisions be influenced by what is good for redistributors.

elder tinsel
#

I wouldn't bother making it easier for redistributors at the cost of your own design. people interested in hatch are going to install it with or without redistributors doing anything with it, and distros already got the stuff they need to keep system python/pip seperate over the last few years.

lusty quarry
#

thanks that is reasonable and makes life easier for me. also I've been experiencing trepidation over refactoring CLI config parsing with Pydantic because that is Rust as well so indeed might as well bite the bullet and apologize when the time comes for any distribution packaging hurdles

this is great!

west drift
#

Are you putting uv in PATH?

lusty quarry
bright parcel
# lusty quarry I'd like advice: I want to have Hatch ship UV in the next release. What I'm doi...

As a former Debuntu foundations dev (aka redistributor) I would concur that you shouldn't gate your design choices on what downstream Linux distributors might or might not want. That said, I'd recommend that where possible if you can structure your code so that it isn't super difficult for them to unvendor, then they would appreciate making their lives a little easier. Don't sweat it too much, but be open to suggestions in the future as the Linux packagers make any requests or suggestions. Does that thread the needle enough? 😄

lusty quarry
#

@elder tinsel @bright parcel for example, look at this issue that was opened earlier today https://github.com/pypa/hatch/issues/1391

this is just an issue with a test dependency not even the project itself so imagine when I make the change, this is what I worry about

#

I do my best to support distributions but it's really hard to say no for open issues. I even do stuff like have a test marker for tests that require network access because someone asked for that at some point

elder tinsel
# bright parcel As a former Debuntu foundations dev (aka redistributor) I would concur that you ...

for clarity, I also agree with this to an extent. adding emphasis on my prior statement to the words "at the cost of your own design" is where I personally draw that line.

I don't see it as a major issue in the current state of the ecosystem if a redistributor decides that redistributing hatch is more effort than it is worth, as users still have access to hatch via the redistribution efforts for python and pip, and the system dependencies are properly protected from incompatabilities via the mechanisms added to prevent accidental modifications to system python and system python packages.

bright parcel
# elder tinsel for clarity, I also agree with this *to an extent*. adding emphasis on my prior ...

Fully agree that you shouldn't make design choices you disagree with just to satisify redistributors. In the issue that @lusty quarry linked to, and as you say, adding a marker to skip tests for optional missing dependencies makes sense. If you're going to make uv a hard dependency, then it's up to you if you still want that marker or not, but I'd also say that it isn't outside the bounds to leave that as a task for the redistributor.

One thing to keep in mind though is that a lot of these distribution packagers are themselves volunteers. I also agree with @elder tinsel that hatch has plenty of ways to be available on those platforms without having to be in the package index. And it sounds like uv is just a temporary omission from Fedora, so that'll get sorted out eventually too.

queen hornet
placid wyvern
#

Anticipating this becoming an issue for our little ecosystem here shortly, but so far all quiet.

lusty quarry
#

I might add a short bit about build performance to my PyCon talk. At work we are about to switch a monorepo from setuptools to Hatchling and there is a 25% reduction in the time to build the wheel!

gritty roost
lusty quarry
#

also, I'm not sure that would help unless it ignores the build backend that is defined and incrementally builds the archive

bright parcel
#

I'm super interesting in this @lusty quarry . It sounds like your situation closely mirrors mine. I'm still planning our (into) monorepo migration and that's related to the pm about my bug/question/pebkac. I get a fair bit of pressure to bazel, which I'd like to avoid. Build performance w/hatch would be a compelling data point.

lusty quarry
#

yes I always try to avoid that when I can

bright parcel
#

Bazel performance seems to be the biggest stated reason why people want to use it

lusty quarry
#

I'm curious what makes it fast for Python wheel building in particular, is that true or just an assumption people have?

bright parcel
#

I haven't really had time to confirm it, but I believe it's aggressive caching. I have more direct experience with Gradle, which also benefitted from aggressive caching, but didn't really play well with common Python workflows, like being able to pop into a venv and narrow down to a small number of pytests. Gradle (or maybe the way it was used) had lots of problems, like hiding configurations and tool calls inside Groovy or Java files which made it really difficult for a Python dev to discover, modify, etc. From what I've seen of the Bazel configs, it seems... maybe a bit better, but still tends to hide things in places that are unnatural for a Python developer.

lusty quarry
#

what I don't understand is specifically wheel building, how would that be any faster?

bright parcel
#

It's a good question, and I'm not really sure Bazel really is faster for wheel building, unless they're doing some more aggressive caching of dependencies?

topaz pine
#

Not sure if I should ask this here, but I've been working on some translations from PyPA to my native language (spanish), and I have a question about the type of language PyPI and the docs are using. Unlike english, in my language there're two (maybe more if you consider regionalisms) ways to say "you", one is formal ("usted") and the other is more familiar ("tú"). In order to keep consistency on the translation, should I keep the formal way or use a familiar way to say "you/you're"? What's the type of language used by the docs in general?

spiral urchin
#

Maybe use a similar tone to Python’s documentation? We have formal and informal second-person pronouns as well and I believe most documentation uses the informal form, but I assume this varies more from language to language than from a document to another

topaz pine
queen hornet
gritty roost
#

should a wheel ever contain __pycache__? A user submitted a PR to one of my projects to always strip out __pycache__ directories when installing a wheel (because they're running into some problem), and I'm trying to figure out whether this is the correct approach.

indigo token
#

Wheels shouldn’t contain these. Pip (and uv if you ask it to) create them when installing a wheel. Otherwise they’re generated on first execution of each package.

lusty quarry
#

it should never, no

#

Hatchling can't even traverse such directories and there is no way to override that

gritty roost
#

Okay, thanks!

lyric hedge
# 
simplewheel/__init__.py
simplewheel-2.0-1.dist-info/DESCRIPTION.rst
simplewheel-2.0-1.dist-info/metadata.json
simplewheel-2.0-1.dist-info/top_level.txt
simplewheel-2.0-1.dist-info/WHEEL
simplewheel-2.0-1.dist-info/METADATA
simplewheel-2.0-1.dist-info/RECORD

Is this a valid wheel? As far as I understand, wheel build tags should not be included in the metadata directory name...

#

(I'm trying to optimise pip's metadata backend and my patch broke the test suite because of this)

fading veldt
#

PEP 427 says that the format should be {distribution}-{version}.dist-info, so I believe that the build tag should not be in the name of the dist info directory

lyric hedge
#

coolio, that's what I thought

#

hopefully there aren't too many of these technically invalid wheels out in the wild

fading veldt
golden falcon
#

Might be worth reporting on the setuptools tracker

lusty quarry
stone dove
#

can anyone suggest a good pdm based cookiecutter template ?

plucky spindle
tidal kiln
#

if anyone has thoughts, let me know

stone dove
merry rune
lusty quarry
tidal kiln
#

mainly when dealing with quirks of different environments

#

it's useful to quickly look things up, and then also test against specific environments in test suites, etc

#

that was my main purpose for that at least

plucky spindle
#

Feel free to create an issue or discussion for that.

lusty quarry
plucky spindle
stone dove
#

I have used both but for me markdown is much more familiar. I mean mkdocs is much simpler for me 😅

#

Hence I prefer mkdocs

plucky spindle
merry rune
#

We've also only gone with Sphinx + myst in cookie; the myst team is great. Eventually we might provide an option for mkdocs though. I like how fast it is.

plucky spindle
#

Any recommended themes other than material? The material team really did a good job, I even want to pay for the insider features when used for my personal projects. But as a template project, it may not be suitable as the default option for everyone.

stone dove
plucky spindle
stone dove
plucky spindle
#

I should be more precise, again. haha

lusty quarry
#

just FYI when Hatch does a docs command the default will be MkDocs with the material theme

inland prawn
#

meh, sphinx with furo FTW

lusty quarry
#

if you are a new person with no context MkDocs is simply easier than Sphinx

#

and a larger ecosystem of plugins

inland prawn
#

but yaml 🤮

lusty quarry
merry rune
#

FYI, I'd love a "templates" system where there could be two defaults, mkdocs and sphinx. Sphinx + myst + furo is very nice. Does mkdocs support man pages? That's something distro package managers request once they start picking up your package.

lusty quarry
merry rune
#

I think mkdocs makes sense as a default, I just think it also makes sense to have integrated support for sphinx too. 🙂

merry rune
#

What's the status of PEP 639? It seems setuptools and hatchling have added the (supposed to be in) METADATA 2.4 fields and supporting the pyproject.toml syntax, but the PEP is in draft. I don't want to add support to pyproject-metadata until something is standardized. Which means scikit-build-core (and meson-python and pdm-backend) doesn't support the draft yet. But apparently distros (at least Fedora) are asking for these fields. The PEP has been draft for nearly 5 years.

queen hornet
lusty quarry
lyric hedge
#

trail of tears is quite the reference... oh boy

lusty quarry
#

I'll probably eat lunch on my lunch break next time

mortal shore
fading veldt
#

Do you mean any unknown Metadata fields? Or unknown license fields specifically? I'd imagine allowing unknown fields would allow for easier experimentation/demonstrations in metadata changes.

#

(I could definitely see the benefits of banning arbitrary fields on the other hand)

lusty quarry
mortal shore
#

like if you have a tool that doen'st properly escape and you sneak a newline in there, the second line can end up looking like a field with a weird field name

queen hornet
lusty quarry
#

I'm pretty sure it's approximately correct but I wasn't actually around at the time, is this generally correct about what happened with packaging in the early days? I was not there but I've read the story a few times from various folks https://github.com/pex-tool/pex/pull/2408#issuecomment-2105932894

i.e. packaging was not good, there was no desire to improve within CPython, and therefore Anaconda was born. is that right?

bright parcel
#

The topic is convoluted, complicated, shrouded in the mist of history, subject to many different perspectives, and organic growth. I have my own interpretation which is probably too much to go into here (maybe at Pycon!). I was around for most of it.

lusty quarry
#

I would definitely love to hear more about that next week!

inland prawn
#

would be nice to know more about that (especially since some of us won't make it to pycon 😦 )

bright parcel
#

Probably won't have time until I get to Pittsburgh

mortal shore
#

IIRC the Numpy folks kept asking Guido to fix problems they had with packaging stuff, and he didn't care about it and suggested that they go make their own thing

#

and they took that and ran with it

#

when I first got involved in packaging, probably the biggest problem was there was no mechanism to make decisions, this predated the idea of PEP-Delegates (I don't recall if we were the first people to use BDFL-Delegate, or just one of the first), so Guido decided on all PEPs, and generally didn't care about packaging much so there wasn't a drive to use the PEP system.

#

Plus a lot of the early attempts in CPython focused on more or less a complete lift of the ecosystem, kind of like creating a whole new ecosystem of packages without worrying about how to straddle the line between old and new systems

lusty quarry
#

I don't know who has access, but could someone please create a token that would be used on the Hatch repo to publish images for each release? https://hub.docker.com/u/pypa

high stone
lusty quarry
#

okay, I optimized Hatch's installation heavily the past day and I don't think it can get any faster now

#

now it fetches a prebuilt distribution and simply unpacks, super quick

#

you can try with a binary or installer e.g.:

curl -LO https://github.com/pypa/hatch/releases/latest/download/hatch-aarch64-apple-darwin.tar.gz
tar xzf hatch-1.10.0-aarch64-apple-darwin.tar.gz
./hatch --version
lusty quarry
hollow trout
#

I think if you do it similarly to the example cited in the post, then you can no longer run it with python - you have to run it directly.

hollow trout
#

What's the relationship between distlib and packaging? Seems like they have very similar scopes.

inland prawn
#

From what I have seen, distlib is more legacy and has some features that packaging doesn't

maiden hornet
mortal shore
#

Warehouse is probably the closest thing to the right place

uneven karma
#

What's this?
The PSF offers free web hosting?

inland prawn
#

Probably an old relic

mortal shore
#

pythonhosted.org is the domain name for where user supplied content gets stored, like artifacts and such, to keep it from being able to be used to attack the primary domains like pypi.org and python.org (see gifar attacks, etc)

#

at one point in time PyPI let you upload your docs for your projects

#

we stopped allowing that, but didn't delete the old docs incase they were the only copy etc

#

but we do allow projects who had previously uploaded docs, to delete them

uneven karma
inland prawn
mortal shore
#

in general we did a worse job of it than both github pages and RTD

#

and we decided rather than continue to invest in it, it was better to just recommend using something made for it

inland prawn
#

makes sense (although it would be nice to have docs alongside the pypi package, a lot of packages don't provide link to it

mortal shore
#

it also wasn't very widely used, and in a lot of the cases where it was used, it was just old versions of the docs because the project themselves had previously switched to using something else

mild dew
#

there's no packaging-specific channel, so dropping a general question here: does anybody know if there are function(s) to "rehydrate" the results of parse_wheel_filename and parse_sdist_filename?

context: i have a tool that's going to need to take the full filenames of distributions, and needs to fully normalize them according to their respective specs. parse_wheel_filename and parse_sdist_filename return the normalized components of each, but i don't see (maybe missing it) any APIs to turn the returned components back into filenames

(i could do that conversion manually, but it's slightly more annoying than just joining on -. hence the question 🙂)

high stone
quasi spire
#

there was a new packaging release (24.1), but nothing in the changelog. What's with that?

quasi spire
queen hornet
quasi spire
#

perhaps packaging should have a link to the correct changelog on the PyPI?

inland prawn
#

probably just a mistake in release

quasi spire
#

I avoid this in my own projects by autogenerating the GH release from the canonical release notes in the docs

high stone
#

Oh man, looks like I goofed up on the release process. 😅

#

This is what I get for cutting releases while not a 100%.

gritty roost
#

filelock==3.15.0 appears to be wreaking havoc

gritty roost
#

3.15.0 yanked; everything is green again 🫠

blazing lantern
uneven karma
ivory phoenix
ivory phoenix
drowsy moss
ivory phoenix
#

or maybe direct me into somewhere i could learn more?

drowsy moss
# ivory phoenix interesting. would you like to describe some examples?

OS-level software is managed by system package managers. Using an additional/external thing to overwrite those files won't consult the system one and replace parts unconditionally. Other programs that rely on those parts may end up being broken, even said system package manager itself.
You have to know that you'll never ever be relying on whatever's broken (and somehow learn what it broke), or rely on an image that installs additional Python that the system doesn't rely on (or install one yourself, in a separate location). You have to keep this in mind all the time, meaning some extra mental overhead whenever you use the container.

ivory phoenix
drowsy moss
# ivory phoenix thanks for explaining. hats off for spreading awareness <:HatTip:125448551045437...

I forgot to mention that this is why PEP 668 is a thing that pip implements and most distros adopted by now (pip allows overriding it via --break-system-packages) :

Note that --user has similar problems isolation-wise (though, not as disasterous for the system).

indigo token
#

--user mainly just doesn't have a clear use case.

System package manager for system dependencies, and pipx for tools that aren't packaged for for system. Venvs for development. What's left for the user-global env?

hollow trout
#

Personally, I like to use --user for quick experiments/scripts that depend on libraries.

indigo token
#

Packages installed there will put their binaries in ~/.local/bin where they conflict with pipx, so I prefer one-off venvs or the new script dependency spec

hollow trout
#

Yeah, I need to learn to use this script dependency stuff, it would probably serve as a replacement.

woven yarrow
#

I converted coverage.py to use trusted publishing, it worked great (and I wrote a small bug report for urllib3 along the way!) I'm confused about the "provenance" step. I didn't see mention of it in the two guides I was semi-following. Do I need it?

queen hornet
#

it's not needed for trusted publishing

woven yarrow
#

I guessed it wasn't, since I made a release without it!

#

what is it for? When would I need it?

queen hornet
#

I think that's for optional SLA build provenance, a way to prove the artifacts were built by this actual workflow

#

so for example, just before you upload your artifacts to PyPI:

    permissions:
      attestations: write
...

      - name: Attest build provenance
        uses: actions/attest-build-provenance@v1
        with:
          subject-path: "dist/*"
woven yarrow
queen hornet
#

and run something like:

$ gh attestation verify tinytext-3.6.1.dev98-py3-none-any.whl --owner hugovk
Loaded digest sha256:52817601de33f793ca452c9ca85a1bee4ad95601148a7a745f11d9c234e4f647 for file://tinytext-3.6.1.dev98-py3-none-any.whl
Loaded 1 attestation from GitHub API
✓ Verification succeeded!

sha256:52817601de33f793ca452c9ca85a1bee4ad95601148a7a745f11d9c234e4f647 was attested by:
REPO             PREDICATE_TYPE                  WORKFLOW
hugovk/tinytext  https://slsa.dev/provenance/v1  .github/workflows/deploy.yml@refs/heads/main

$ gh attestation verify tinytext-3.6.1.dev98.tar.gz --repo hugovk/tinytext
Loaded digest sha256:a6f78c9b2f3126ea745a82dafb729c1926bfede51df3825e0b6ba46a40006d2a for file://tinytext-3.6.1.dev98.tar.gz
Loaded 1 attestation from GitHub API
✓ Verification succeeded!

sha256:a6f78c9b2f3126ea745a82dafb729c1926bfede51df3825e0b6ba46a40006d2a was attested by:
REPO             PREDICATE_TYPE                  WORKFLOW
hugovk/tinytext  https://slsa.dev/provenance/v1  .github/workflows/deploy.yml@refs/heads/main
queen hornet
#

it's also good to have a toy project no-one cares about to try things out on 🙂

woven yarrow
#

that i completely understand

#

my projects are all over the map in terms of discipline and add-ons like this. adding trusted publishing to coverage.py revealed a github actions job state I'd never seen before (Waiting), so I had to update watchgha. That made me look at the issues there, which made me add a test (it had no tests!). Then I almost added trusted publishing to watchgha, but I'd already done a lot, so it's still uploaded from my machine.

queen hornet
#

they've done a great job with trusted publishing, it's really easy to set up and use. along with other automation, means I can also release from my phone!

woven yarrow
queen hornet
#

I think you probably could do it in one job, but it's better to separate them, can have fewer permissions for each job

#

and also can generate the dists every time, even if not uploading, to test they build properly

woven yarrow
#

in tinytext, you use the GitHub Release publish to push to pypi, is that right?

indigo token
queen hornet
woven yarrow
#

right

queen hornet
#

also upload to TestPyPI for merges to main

woven yarrow
#

right

#

So many possibilities of how to organize the pinball machines

queen hornet
#

yup!

uneven karma
queen hornet
#

one is for TestPyPI, the other is for prod PyPI

uneven karma
#

I don’t understand why can’t the same signature can’t work for both.
The file hashes aren’t changing, right?
And you’re not specifying the destination at attestation time, at least as far as I can see.

queen hornet
uneven karma
#

If the same files are going to different destinations, then why create a different signature for each destination?
In head thats completely backwards from the idea of having a single signature to verify against no matter who you get the file from.
Am I just missing something here?

uneven karma
queen hornet
#

that job runs for every PR, even when we're not uploading anything. my thinking was, I don't want to "litter" the various registries where the attestations are stored, for artifacts which are thrown away

uneven karma
#

Understandable

#

I’ll definitely experiment with it
I’m trying to get better at all this, I’ve always lagged behind on the security side.

jovial holly
#

Headsup! The infrastructure behind bootstrap.pypa.io has been updated.

https://github.com/pypa/bootstrap has the relevant bits but the summary is that instead if it running on a single host in the PSF infra it now runs in our k8s cluster.

The major change for maintainers of stuff published there is that instead of things being completely automated, a Pull Request is opened on the repo to initiate a new deployment. (I'm up to someone helping me solve the problem of how to have a change detected, commit made, and GHA actually run ci.yml)

There shouldn't be anything you need to do as maintainers!

I belive the interested parties (maintianers of existing stuff actively changing things hosted on bootstrap) are:

  • @high stone
  • @mossy pulsar
woven yarrow
#

If i have two separate github actions, one that builds dists and one that publishes them to pypi, should i generate attestations where they are built, or where they are published?

queen hornet
#

but if you attest during build, those artifacts could already be downloaded and the attestations tested locally, if that's important

#

the two repositories where they're stored:

Everything is wrapped up as a Sigstore bundle and persisted to GitHub’s attestation store. In the case of a private repository, that’s the end of the flow. In the case of a public repository, the attestation is additionally written to the Sigstore Public Good Instance and lands on Rekor, the project’s immutable ledger.

https://github.blog/2024-05-02-introducing-artifact-attestations-now-in-public-beta/

woven yarrow
#

I have it now just before the publish step, which seems good.

frigid jewel
#

i figure this might be a good place to ask as any

do you guys know of a tool where i can automatically specify python versions, and create venv per version automatically using a pyproject.toml or something

currently, on windows, i iterate thru all the versions py -0 can find, create a venv per version, run the stuff i need to run and cleanup the venv.

but i'm wondering if there's already a cross platform solution for this already.

lyric hedge
frigid jewel
lusty quarry
frigid jewel
#

similarly automated dev. workflows
fits my use case

lusty quarry
#

you can just as easily create an arbitrary environment with a matrix and run arbitrary commands within each

frigid jewel
#

do you have an example of that?

frigid jewel
lusty quarry
queen hornet
elder tinsel
#

probbaly getting flagged as looking like a disguised zip

spiral urchin
#

looking like a disguised zip
(Not an inaccurate observation)

hollow trout
#

What a masterful disguise.

high stone
queen hornet
#

and you might install more dangerous code with it!

blazing lantern
#

There's no specific channel for that sort of thing here that isn't packaging-specific

queen hornet
obsidian lily
#

literally sums up the entrie packaging ecosystem 😔

Screenshot_2024-06-28_at_08.png
lyric hedge
#

OK, so eggs, distutils, and easyinstall are deprecated/not-something-you-should-rely-on so those don't count at least :P

obsidian lily
#

loll

obsidian lily
high stone
inland prawn
#

But they vendor their own version so you don't have to care about distutils

vague mist
obsidian lily
lyric hedge
#

Sorry about that!

#

I've allowlisted the server and also set the automod punishment to mute for 10 minutes, instead of an hour. I'll also set a custom block message to explain that we don't allow non-allowlisted invites here.

vague mist
#

Thank you so much for the context and for sending out the server invite again, @lyric hedge!

high stone
#

@blazing lantern OK if I add a #packaging in here?

lusty quarry
inland prawn
#

would be useful, since that package is kinda core to most things packaging(sic!)-related

lusty quarry
#

I can't find where you stated your reasoning but it was right around this time #general message

high stone
#

Pretty sure it would've been "I don't want one more place to field questions about it".

#

But, given that we're already fielding questions about it here... having a dedicated channel seems prudent.

blazing lantern
high stone
frigid jewel
#

why isn't it possible to pip install -r pyproject.toml without it installing the package itself?

uneven karma
#

What do you do when the dependencies are dynamically generated by the build system?

indigo token
indigo token
frigid jewel
uneven karma
frigid jewel
uneven karma
frigid jewel
#

if i have a pyproject where i specify the dependendies and want to install them, i dont want to create a separate requirements.txt just to install the dependencies

uneven karma
# uneven karma How is it not relevant?

What if there’s 1000 lines of downloading C libraries above defining the install_requires in a setup.py?
How do you only install those deps without side effects?

frigid jewel
elder tinsel
#

There's an extremely relevent conversation going on on discourse right now

#

that comment explains it isn't possible from pyproject.toml, and relevent specifications

uneven karma
frigid jewel
elder tinsel
#

What's the value in that?

#

you're giving a set of steps here that you want to be possible, but not what the benefit of those steps being possible is

frigid jewel
elder tinsel
#

yes

frigid jewel
#

i only want to install the dependencies, i don't want the package to be installed locally and such

elder tinsel
#

that's still what you want, not why

frigid jewel
#

because i'm not writing a package nor anything of the sort

elder tinsel
#

then pyproject.toml isn't the right tool?

#

if you're not making something meant to be built, don't set things up so that tooling expects to build it, just provide the requirements seperately

frigid jewel
#

what if i still want to throw my script into a repo, while maintaining all the other stuff a pyproject provides? author, license, email etc

#

sure i can do that directly from the repo, but once it's off the repo it can still live on the pyproject.toml

elder tinsel
frigid jewel
#

FWIW, i think that pep is a bad idea.

elder tinsel
#

well, that's what you have within the standards

#

either specify your dependencies separately, treat it like a package, or inline the requirements

frigid jewel
#

well seems like other people want a fourth option like i do 🙂

elder tinsel
#

you're the first person I've seen wanting this who wasn't okay with just providing it in one of the supported ways

#

that's not to say there might not be other people who want that, but I'm not sure where "seems like other people..." is coming from

uneven karma
#

Yeah, I don't have any problems with the specs

frigid jewel
# elder tinsel you're the first person I've seen wanting this who wasn't okay with just providi...

specify your dependencies separately
sure, that's what i'm currently doing, but then i have to essentially keep two copies of the same thing, both in the pyproject.toml, and in the requirements.txt

treat it like a package
sure, that's cool, but this can be a barrier of entry to newbies in the python world

or inline the requirements
personally i'm against this, and i know other people in the security field, like @placid wyvern here, are against this because arbitrary modules could be installed by simply running a python file directly, or what if the requirements get installed to the wrong interpreter (if you have multiple versions in your system), or you accidentally run the script outside of an venv? sure some of these things can happen without it, but having it inlined just makes it more likely imo

elder tinsel
#

you don't need your requirements in pyproject.toml if it isn't a package though

frigid jewel
#

no, but i like the flexiblity it provides of having the author, version and other info within it.

elder tinsel
#

I'm also in the security field, and I don't see inline script requirements as a security issue. running code you don't trust and havent reviewed is dangerous whether it grabs dependencies or not

#

grabbing dependencies should be viewed with that lens

uneven karma
#

Having a consistent well-structured installation method is a barrier to entry?
In my current tasks at my day job, the exact opposite is currently being a barrier to entry to me.
Consistency in knowing what to expect is a godsend for beginners.

frigid jewel
#

or, in some cases, i can put the development and release dependencies within the pyproject rather than maintaining both a normal requirements.txt and a dev-requirements.txt

elder tinsel
#

... you're describing a package not a script

frigid jewel
uneven karma
#

Regardless of which method you pick, my package will function exactly the same way.

frigid jewel
#

there's no one way of doing things, there's multiple of them.

elder tinsel
#

and yet by staying within the standards All of them work

#

users dont actually have to care

frigid jewel
elder tinsel
uneven karma
#

What do you have against PEP 723?

elder tinsel
#

short version, pyproject.toml isn't for specifying dependencies, it's for specifying details about the package and how tools (including build backends and installers) should behave. you can't determine dependencies without a build backend

frigid jewel
uneven karma
elder tinsel
#

^

#

someone could just as easily put a malicious dep in a package than a script

uneven karma
#

Or pipx run my_script or whatever the actual method is

elder tinsel
#

review what you use

uneven karma
#

You're not suddenly absolved of reviewing random code from the internet just because it was packaged according to a different spec

elder tinsel
#

there's also nothing stopping a script from installing stuff prior to import with os.system

#

the specs are for well-behaved things to play nicely with eachother, they can't rule out malice

frigid jewel
#

so many people run python code without reviewing the source and dependencies, we see that a lot over the python discord, there having to be an additional step on running the code i.e installing the dependencies separate, has saved a lot of people hides

elder tinsel
#

that's a user education issue

frigid jewel
#

for your os.system, the script not knowing how to correctly resolve the right interpreter has also saved those very same users, i.e the dependencies get installed to a different interpreter

uneven karma
uneven karma
#

Half of that is in the setup.py, which runs on install anyways

frigid jewel
# uneven karma How?

we ask them what are you installing? then they share the link to the github, or share the list of dependencies and such, and the more experienced helper can spot the shady list of dependencies or the github name and such, and educate them on that

frigid jewel
#

in which case, easier entry point for bad actors

elder tinsel
#

so... your issue here is that the tool does the right thing

uneven karma
#

I'd assume a very high percentage of first-time users only have one interpreter