Need to finish PEP691 support this weekend. That's my goal
#bandersnatch
unique wren
unique wren
Wow - Been a bigger diff than I'd hope - But we have package PEP691 support: https://github.com/pypa/bandersnatch/tree/pep_691_1
Need to fix root simple index.html to generate the two other files too ...
And add some more JSON file generation into existing tests - I expected them to exist but they don't
narrow grailBOT
unique wren
Bah - close - seems I do have a bug with the packages writing out the JSON file ... Will try work that out tomorrow or some stage this week.
narrow grailBOT
unique wren
Busy bots
narrow grailBOT
unique wren
Fuck yeah - PEP691 works + passes CI - https://github.com/pypa/bandersnatch/pull/1154
What a battle
stuck aurora
nice!
I was gonna try to review this week sometime, but you don't have to wait for me
unique wren
It was harder cause I extracted most of the simple API stuff into a new simple.py module
and wanted high testing naturally
Would love a second eye over it, which I don't get often on bandersnatcho
Guess I should add to the docs too ... Will do that tomorrow. Now I go get drunk.
bandersnatch does not exist no HTML tho with this diff ... ๐
So I'll open an issue for that once we merge this to fix
narrow grailBOT
narrow grailBOT
unique wren
So I donโt spam others. @vale swan I cut this release so I can merge tomorrow or Monday my 6.0.0 pre release for bandersnatch. This is such a large change I might do my first ever alpha release
vale swan
Hmm I did want to review it, but I don't think I'll be able to do that in a day ๐
I haven't even kept up the related PEPs so I'd have to catch up on those first and then review your PR
unique wren
Thatโs all cool. Others attached are most in the know but they rarely review
Some co authored the PEP with me
/ wrote most of it
Donald has been rad with it all
I need to do the banderx diff after merging this to test serving the new json files etc.
Then once thatโs done itโs release - then work on getting the xmlrpc calls bandersnatch does replaced - we have ideas for that ๐
I also want to hassle again updating PyPIโs bandersnatch โฆ once this is all done.
teal moss
does bandersnatch (or someone know of a mirror which) support lazy mirroring? When a client asks bandersnatch for a package and it does not have it, bandersnatch will check pypi and download the file before providing it to the client?
unique wren
Devpi
teal moss
ty
unique wren
narrow grailBOT
vale swan
the problem with the coverage bot is that it updates the comment as coverage data from all of the various jobs roll in
the GH email is sent with coverage data from only one or two jobs, additional data is not reflected in the email
and thanks to python version and OS specific code paths, until all jobs have uploaded their coverage data, the comment is effectively useless
huh.
I wonder if this would help
unique wren
and thanks to python version and OS specific code paths, until all jobs have upl...
Yeah, thanks. I understood why. Just donโt want the spam.
Thanks for finding the N builds. Could be a good hack. But I hate how if we change number of builds how we keep that up to date
Another human tweakable to be forgotten about โฆ
vale swan
hmmm, I wonder if ${{ strategy.job-total }} would work
wait no
codecov wants N builds in a file
unique wren
We would have to combine jobs into 1 yaml like you did with black first I feel to get a correct total too
vale swan
oh yeaa I forgot CI is separated too
unique wren
Which my head likes for simplicity โฆ
But I could live with it if it helps this ...
I was pretty proud to get a ~4% coverage increase with my last PR โค๏ธ
Old man can still do some code every now and then
teal moss
codecov wants N builds in a file
it doesn't have to be exact, it just has to be a minimum
the text in the first screenshot still applies to the config applied in the second
unique wren
I get that, but still, if we change CI job counts in 6+ months I'm going to forget this potentially ... just trying to see if there is a way to have some level of adapting here etc. ... e.g. last year we went from 3.6+ to 3.8+ and the amount of CI jobs ran dropped drastically.
teal moss
ah
narrow grailBOT
Bumps docutils from 0.18.1 to 0.19.
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can als...
Bumps keystoneauth1 from 4.5.0 to 5.0.0.
Dependabot will resolve any conflicts with this PR as long as you don't alter...
lone oriole
Alright, I'm going to just guess what I"m doing is a user error but for the life of me I can't seemed to figure out why Bandersnatch is not syncing all packages.
I am trying to create a complete offline pip repo and it seems like it is working but of course, out of thousands of packages that are online, two are not being updated. specically pip, and falcon
I see their names and others in the todo file but it never downloads it
I try bandersnatch sync falcon and nothing
I have ran mirror --force-check and now I am running verify which I can assume will take a really long time
The only plugin I have enabled is `[plugins]
enabled =
size_project_metadata
[size_project_metadata]
max_package_size = 100M
`
everything else is default bandersnatch 5.2.0
what could cause bandersnatch to just ignore a package when you specifically type bandersnatch sync falcon?
--debug doesn't really shed any light afaiks
unique wren
Please open an issue and post your full config and โdebug output from the sync for me to look at and Iโll try repro.
unique wren
๐ Will try look over the weekend
I'm all about working smarter, not harder
I don't even run bandersnatch anywhere anymore ๐ฆ
I've been wanting to update PyPI's instance, but have never been give the go ahead (more so cause I wanted to test instance before doing the prod one)
@vale swan How do I set reminders here?
I've seen you do it
unique wren
not the issue, the automation. ๐
[plugins]
enabled =
size_project_metadata
[size_project_metadata]
max_package_size = 100M
I'm gonna lean towards this plugin maybe having a werid bug - I've never personally used it
vale swan
<@429445826251587616> How do I set reminders here?
That's a feature of the Python bot the organization and community develops
You could just set a reminder in #bot-commands in the PyDis server, it's meant for command spam
lone oriole
```
[plugins]
enabled =
size_project_metadata
[size_project_metadata]
max_packag...
Hmm, I'll try removing it but Im concerned with my repo blowing up in size
Is there a more reliable way you recommend to limit those extremely large pip packages?
So of the cern and nueral net pip packages are several hundred MB each. To compound the problem, some of the worst ones post nightly builds
unique wren
Yeah, I plan to do an allowlist for falon and pip and use everything else from your config changing the base directory obviously
lone oriole
Yeah, kinda figure since you said this weekend.
unique wren
lone oriole
bandersnatch took 1.5 weeks with a gigabit downlink. ๐
unique wren
It's 12.7t
(I added this)
The other option is to generate a list of packages to deny: https://github.com/cooperlees/pypistats
I wrote that to generate said config
I've used that before
pip install git+https://github.com/cooperlees/pypistats.git should work
But as you can see, i haven't touched the code in a long time
tf-nightly 682.6 GB
In the mid 2010's - This was the total size of pypi
tropic fossil
I wonder how many packages could be removed
unique wren
People are against removing anything
But a lot could and no one would care. But that's more on package maintainers to help control
tropic fossil
hoarders
unique wren
Python just does not want left pad like outcomes
tropic fossil
I mean, 13TB is not that much... but still...
unique wren
If you don't know what that is - A javascript library dev deleted his library in rage and caused lots of pain.
tropic fossil
Heard about it
but JS package ecosystem is much more messy than Python (mostly because of almost non-existent standard library)
stuck aurora
fundamentally PyPI can only grow in size unless people remove old releases
but PyPI is useful as an archive
tropic fossil
well, I wonder how many packages for python < 2.7 are still there
unique wren
Lots.
There is a metadata plugin that could filter based on that for bandersnatch
But, relies on accurate metadata that's not enforced / autogenerated from automation
teal moss
That's a feature of the Python bot the organization and community develops
hmm, I'll maybe add reminders to monty in a week or so then
narrow grailBOT
Bumps types-freezegun from 1.1.9 to 1.1.10.
Commits
See full diff in compare view
Dependabot will resolve any conflicts wit...
narrow grailBOT
Bumps python-swiftclient from 3.13.1 to 4.0.1.
Commits
See full diff in compare view
Dependabot will resolve any conflic...
vale swan
dependabot pls no >.>
vale swan
my inbox ๐ญ
exotic tiger
anyone know why bandersnatch might be missing packages which are below the max_package_size i have set? if i add the project to the allowlist and run a sync on it, it will then download. it does not seem to be included in my nightly mirror run
unique wren
anyone know why bandersnatch might be missing packages which are below the max_p...
It does seem that plugin has a bug we need to dig into - I didn't write it so I can't remember it's match / metadata it uses
narrow grailBOT
unique wren
anyone know why bandersnatch might be missing packages which are below the max_p...
https://github.com/pypa/bandersnatch/issues/1169 - This sounds similar ...
exotic tiger
yeah that does seem to be what i'm seeing
unique wren
So I misunderstood it, seems it sums the package size โฆ not the individual release sizes โฆ
exotic tiger
That explains why adding it to the exceptions list worked.
narrow grailBOT
unique wren
Wow - I just got an email about a failure with Azure CI
I haven't cared about bandersnatch CI there for years. I wonder what I've forgot to remove
I added it way back when to get Windows CI when someone wanted support ...
But removed (I thought) when GitHub actions gave us Windows
I can see it here: https://github.com/pypa/bandersnatch/settings/installations
narrow grailBOT
Bumps types-filelock from 3.2.5 to 3.2.7.
Commits
See full diff in compare view
Dependabot will resolve any conflicts with th...
narrow grailBOT
narrow grailBOT
unique wren
6.0.0 go time
narrow grailBOT
narrow grailBOT
Bumps python-swiftclient from 4.0.1 to 4.1.0.
Commits
See full diff in compare view
Dependabot will resolve any conflicts...
narrow grailBOT
narrow grailBOT
narrow grailBOT
chilly stirrup
lol, that package looks great on PyPI
unique wren
Does PEP440 even talk about decimal places?
Surprised Google Cloud's bucket thing accepts names that long
tropic fossil
I was hoping for version = math.pi, but it's just a hardcoded value ๐ฆ
snow sparrow
Version is a string so thereโs no no notion of decimal places (but arguably there should be a character count limit?)
unique wren
Ya, thatโs good start ๐
narrow grailBOT
exotic tiger
hey folks, i'm running a nightly bandersnatch mirror but i'm noticing packages are not being updated. it is writing a diff file which does usually include some updates but obviously there are others that its missing. any idea what could be causing this?
i do have exclude_platform and site_project_metadata plugins enabled but neither should be excluding the most recent package i've noticed is more than a year out of date
unique wren
Yeah not sure off the top of my head. Would have to see a paste of your logs to take an educated guess
Want to paste it?
exotic tiger
Logs were unfortunately not being preserved since Iโm running this from a container. I fixed that and started a mirror force check. The manual mirror runs I did before that would sync up a few packages at a time but it kept missing the one particular one I knew was out of date. Iโll have to check again once this current mirror sync completes.
exotic tiger
well a full mirror definitely picked up the missing package updates. the incremental mirrors every 24 hours seem to pick up a lot more updates too. does the mirror command check for updated serials only for the packages listed in the global index?
round heron
I use bandersnatch with happyness for more than a year. In the last months I tried to contribute by adding some features interesting for my setup, which enables requirements file search from a folder with the glob feature. My setup is docker based, I made my own docker compose file with nginx to serve the packages and a small ftp server to upload new requirements file in the requirements folder. I do this to speed up the automated tests in CI/CD in my company, where the runners get the packages locally instead to retrieve them from Pypi.
I wanted to change the ftp server in a custom tornado server, which only supports a simple endopoint for the upload of requirements files, to avoid to open several ports on the networks for the ftp support. Could it be an interesting feature to add to the repo and share it along with the docker compose file, or it would be better if I implement it in a separate repo?
unique wren
Feels separate. I don't get why a requirements file in your projects repos does not work ...
unique wren
well a full mirror definitely picked up the missing package updates. the increm...
Each run does a call to xmlrpc API to ask for all the packages that have changed since the serial you last synced locally to
round heron
Feels separate. I don't get why a requirements file in your projects repos does ...
of course I have them in their respective repos. I have a pipeline step in these repos which uploads them to bandersnatch. This allows me to be reasonably sure to find them when I need these packages to be installed during automatic testing.
unique wren
So you upload the requirements file and use the bandersnatch requirements file plugin to download all the packages?
round heron
yes, since the whole pypi archive is around 10 TB, and in our company we use a relatively small amount of packages, I thought this would be a nice and scalable approach to sync only the needed packages, but not limit this list to a static one.
This is why I needed the glob in the requirements file list...
round heron
Just an update, yesterday I had a try with the tornado web server, for the moment is a private repo, but the idea is to make it public as soon as it starts working properly.
The server works properly, nginx has a location to redirect the traffic to tornado and everything goes smoothly on this way, but I noticed something strange.
But I noticed an issue on bandersnatch, not sure if it was there before and how to better test it. If I add a new requirements file after the first sync of bandersnatch, it is registered by allowlist_name.py:114, the package list is registered by allowlist_name.py:31, with the new packages, but they are never downloaded.
I will do some more test with a shorted package list and, in case, open an issue which could be easily replicated
narrow grailBOT
narrow grailBOT
unique wren
Thatโs a tough one. Do people think a parallel json metadata file reader and stat each file could be cheaper? I donโt know what โduโ does under the covers โฆ
narrow grailBOT
narrow grailBOT
narrow grailBOT
unique wren
Damn, s3path needs 3.11 support ๐ ... Decent refactor but prob worth it to make tests pass minus s3 ... should probably isolate storage plugins tests more vs. core bandersnatch filesystem support
But that's probably going to be another day.
unique wren
Would have been quicker to do the PR ... haha
So I did it finally
Guess I should do a release once I merge this
vale swan
I'll take a look soon
unique wren
No worries, I'm in no hurry
It's hard to stay motivated in a project you don't use at all and haven't for years
Wish someone would take it over ๐ฆ
vale swan
I'm in the same boat, I don't use bandersnatch either
unique wren
I'd be more motivated if I could update the PyPI install + move it to use the s3 plugin but I've never been able to get a test instance + s3 bucket
vale swan
PyPI is still on v4, right?
unique wren
Something super old
in a big fat POSIX network mount costing a lot of PSF amazon credits ๐ฆ
vale swan
IIRC I remember reading something about PyPI dropping their bandersnatch mirror? I might be totally misremembering though.
unique wren
Ya, started that discussion but couldn't even get a response on if that's the plan or not
I've not had good success with trying to help PyPA / PyPI in my life ๐ฆ
ec2-user@mirror.dub1.pypi.io: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
I don't haz access anymore to see what it's running it seems too
vale swan
We're maintaining legacy software now ๐ฆ
unique wren
A lot of people around the world seem to use it
Especially the 3rd world
With poor internet
vale swan
well mostly you, but I digress
vale swan
A lot of people around the world seem to use it
that's good to know, the repository itself feels very dead
it's not nearly as "cool" or "critically important" as something like black or pip so it gets less attention
not saying that's a bad thing (it's an infra project), but just an observation
vale swan
set up once and forget about it until PyPI changes something or bugs out
narrow grailBOT
vale swan
that stale response issue on warehouse hasn't seen activity in ages
unique wren
Nope. Most things bandersnatch related get that kinda of attention
unique wren
The docker container has over 200k downloads ... way more popular than pypi
raven knoll
Hi, I am researching on the use of _last-serial of the JSON rest api for incremental synchronizations, I see it is used on bandersnatch but I am unable to understand that serial generation logic/scope, did not find any spec mentioning it, there is an index _last-serial and per package _last_serial are they related to a global serial that is incremented at each update ?
unique wren
1337
narrow grailBOT
narrow grailBOT
vale swan
I can take a look sometime tomorrow if that'd be helpful
unique wren
O missed this. I reviewed etc. - cheers tho.
vale swan
looks okay on a desk check
one of the tests seems more verbose than needed but it works
@unique wren Protocols are like ABCs but you don't need to actually inherit from them. As long as the object implements the interface (ie. methods, properties and maybe attributes?) of the Protocol, it's compatible. It's duck typing enforced by static type checkers.
Requiring Python 3.10+ later in the year seems a bit aggressive IMHO, but I don't know any user so take this as an uninformed opinion based on norms than anything quantitative :)
unique wren
Requiring Python 3.10+ later in the year seems a bit aggressive IMHO, but I don'...
Majority docker users which is now 3.11 โฆ
vale swan
Right.
unique wren
Iโm aggressive. If people donโt like it they can run the project ๐
vale swan
I keep forgetting about that
TBH I haven't written any 3.10 code ๐
What's even new in 3.9/10? I don't remember
tropic fossil
pattern matching
tropic fossil
TBH I haven't written any 3.10 code ๐
though life of library maintainer, huh?
vale swan
yup
I think mypyc still supports Python 3.5 (although mypy itself is 3.7+ so not sure how that works, perhaps mypy still supports checking 3.5+? wait it stills supports checking Python 2 code, of course it can check 3.5...)
tropic fossil
I feel you... stuck on 3.7-compatible coding myself
vale swan
All I want is Python 3.8+. I was sad with the amount of times I could've used the wulrus operator yesterday :(
tropic fossil
yeah, but keep up hope, it's only 6 months till 3.7 EOL
vale swan
ah yes, | as Union is a 3.10 feature
unique wren
I also feel if people want to use old python they can also use old versions of libraries etc.
No one is forcing them to upgrade bandersnatch etc.
chilly stirrup
What's even new in 3.9/10? I don't remember
I probably care most about generic collections in 3.9 and type.__or__ in 3.10. As for 3.11, fine-grained error locations (incredibly cool, though same also goes for 3.10's improved errors) and typing.Self
tropic fossil
Does mypy support Self? I remember there was some delay on it
tropic fossil
The PR in mypy was merged day after latest release
snow sparrow
Also toml
torpid pivot
hello! can anyone help me understand the allowlist/blocklist plugins in the bandersnatch config
i want to mirror only a specific set of packages
i thought I could do something like:
[plugins]
enabled = allowlist_project
[allowlist]
packages = package1 package2 package3
but it just goes ahead and downloads everything
the documentation is unclear about how the different plugins interact
unique wren
https://github.com/pypa/bandersnatch/blob/main/src/bandersnatch/tests/ci.conf - CI integration test uses that and allow_release โฆ so itโs our most tested (outside of unittests) plugin โฆ
Maybe new lines are needed? Havenโt checked the code.
torpid pivot
oh, i will try with newlines, thanks for the hint
unique wren
If so, docs PR welcome.
torpid pivot
in that CI test, there is cleanup = true in the mirror configuration, what is this option?
unique wren
I can not recall, knowing me a typo. Iโd have to grep the code to see if it does anything if itโs not documented.
torpid pivot
can't see it in the docs. no matter, i'll have a look later
will try the newlines now
unique wren
I sadly donโt use bandersnatch at all anymore and am stuck with it ๐ฆ
torpid pivot
oh, no
i am trying to use it for a work-related thing, if it ends up in place, maybe we can help with maintenance
(we need a partial mirror of pypi internally because our CI doesn't have internet access)
unique wren
Remember to allowlist deps of deps. pip freeze is your friend.
You using docker container?
torpid pivot
Yep, it was the newlines
my bad, the docs consistently use that style, i just assumed it was whitespace-separated and only needed to list a few packages so put them on one line
unique wren
Welcome to add a validation check of param and error or update docs to help future people if you feel nice โฆ would even accept white space if people want long lines I guess
torpid pivot
i'll make a note to hopefully come back and submit a PR either extra validation or making it more lenient
right now i have to complete the task at hand... getting a python CI job to work with this new mirror
unique wren
Another good selling point to your management - the mirror is DR if PyPI is down and you need to make a release โฆ
Blows my mind how many companies donโt archive their dependencies โฆ
unique wren
(DR == disaster recovery)
torpid pivot
yeah, i'm familiar with DR, although most contexts where PyPI is down would probably not be a disaster
in that context it's not an easy sell to management; we don't write any python ourselves, just have a few dependencies written in python. we already discussed even eliminating those dependencies so that we don't need all this build infrastructure, but for now this is the easier approach
if we can't reach PyPI, we just can't update the version of those dependencies until it comes back up, not the end of the world
any idea what's going on here? i have an allowlist_release set, but it's downloading a random other collection of packages
lol wtf, i have to pay to send messages longer than 2000 characters?
2023-01-19 00:08:20,125 INFO: Selected storage backend: filesystem (configuration.py:131)
2023-01-19 00:08:20,126 INFO: Selected compare method: hash (configuration.py:177)
2023-01-19 00:08:20,310 INFO: Initialized release plugin allowlist_release, filtering [LIST_REDACTED] (allowlist_name.py:177)
2023-01-19 00:08:20,411 INFO: Syncing with https://pypi.org. (mirror.py:57)
2023-01-19 00:08:20,411 INFO: Current mirror serial: 16477978 (mirror.py:278)
2023-01-19 00:08:20,412 INFO: Syncing based on changelog. (mirror.py:301)
2023-01-19 00:08:21,410 INFO: No project filters are enabled. Skipping filtering (mirror.py:99)
2023-01-19 00:08:21,410 INFO: Trying to reach serial: 16478060 (mirror.py:310)
2023-01-19 00:08:21,410 INFO: 24 packages to sync. (mirror.py:312)
2023-01-19 00:08:21,410 INFO: No metadata filters are enabled. Skipping metadata filtering (mirror.py:76)
2023-01-19 00:08:21,410 INFO: Fetching metadata for package: arcanepython (serial 16477988) (package.py:58)
2023-01-19 00:08:21,411 INFO: Fetching metadata for package: arize-phoenix (serial 16478060) (package.py:58)
2023-01-19 00:08:21,412 INFO: Fetching metadata for package: b3fileparser (serial 16478000) (package.py:58)
2023-01-19 00:08:21,504 INFO: Storing index page(s): arcanepython - in /var/cache/pypi/web/simple/arcanepython (mirror.py:698)
2023-01-19 00:08:21,509 INFO: Storing index page(s): b3fileparser - in /var/cache/pypi/web/simple/b3fileparser (mirror.py:698)
[it continues with 21 other packages, none of which are listed in the allowlist]
LIST_REDACTED = <Requirement('aiosignal==1.3.1')>, <Requirement('attrs==22.2.0')>, <Requirement('colorama==0.4.6')>, <Requirement('bandersnatch==6.1.0')>, <Requirement('distro==1.5.0')>, <Requirement('cffi==1.15.1')>, <Requirement('pip==22.3.1')>, <Requirement('yarl==1.8.2')>, <Requirement('ruamel-yaml==0.17.21')>, <Requirement('python-socks==2.1.1')>, <Requirement('humanfriendly==10.0')>, <Requirement('packaging==23.0')>, <Requirement('wcwidth==0.2.6')>, <Requirement('docutils==0.19')>, <Requirement('lxml==4.9.2')>, <Requirement('jmespath==1.0.1')>, <Requirement('ruamel-yaml-clib==0.2.7')>, <Requirement('async-timeout==4.0.2')>, <Requirement('charset-normalizer==2.1.1')>, <Requirement('aiohttp-xmlrpc==1.5.0')>, <Requirement('cryptography==38.0.4')>, <Requirement('multidict==6.0.4')>, <Requirement('filelock==3.9.0')>, <Requirement('prompt-toolkit==3.0.28')>, <Requirement('awscrt==0.16.4')>, <Requirement('pycparser==2.21')>, <Requirement('aiohttp-socks==0.7.1')>, <Requirement('frozenlist==1.3.3')>, <Requirement('idna==3.4')>, <Requirement('six==1.16.0')>, <Requirement('aiohttp==3.8.3')>, <Requirement('urllib3==1.26.14')>, <Requirement('python-dateutil==2.8.2')>
(yeah discord needs to make some money somehow) feel free to use this https://paste.pythondiscord.com
This isn't actually hosted by this server, but it's hosted by the Python Discord server where I'm staff. (There's a fairbit of overlap in membership between us two).
torpid pivot
i'm constantly amazed that open source projects choose discord. they justify the use of a closed platform by talking about the features, and then you have to use an external pastebin anyway
anyway, where does bandersnatch get the idea to download e.g. arcanepython? it's not in the list of allowlist requirements. something going wrong with hashing?
tropic fossil
i'm constantly amazed that open source projects choose discord. they justify the...
it's modern, accessible and offers a lot for free
vale swan
#off-topic please move this discussion to here
torpid pivot
so is .NET, but I'd still rather use Python
yes, sorry
in case anyone is interested, you must enable the allowlist_project plugin as well as allowlist_release, otherwise allowlist_release will filter the versions of those packages, but all other packages will still be allowed to be installed
unique wren
if we can't reach PyPI, we just can't update the version of those dependencies u...
What happens if it never came back up?
unique wren
in case anyone is interested, you must enable the `allowlist_project` plugin as ...
Yeah that makes sense. If itโs not explicit in docs, another thing to add.
torpid pivot
@unique wren then we would replace those dependencies
either there's a successor to PyPI, in which case we switch to that, or there isn't, in which case we can't use those dependencies any more because there is no ecosystem supporting them. the situation of "we have a stale mirror of a package cache that used to exist but no longer does" is no better than not having the mirror
after adding a package to the allowlist, is there something i have to do other than just run bandersnatch mirror?
it completes successfully, and it lists the new requirement in both log entries Initialized project plugin allowlist_project and Initialized release plugin allowlist_release, but it doesn't download the new package
(after it completes, there is no directory for the added package in $mirrordir/web/simple/)
i guess this is https://github.com/pypa/bandersnatch/issues/1297
unique wren
either there's a successor to PyPI, in which case we switch to that, or there is...
If they were important you could get python devs to help maintain if mission critical.
unique wren
after adding a package to the allowlist, is there something i have to do other t...
Adding a new package would need a forced full sync.
Bandersnatch only look for changes from last synced serial. So if you add a new package you have to ask it to do the expensive get every package known to man kind and then resync packages that your plugins allow
There is a sync a particular package sub command, but I donโt know if it has the โforce or โfull cli that โmirrorโ sub command does. If it does not it should โฆ
But like I said, I have little time and donโt use it anymore so have very little incentive to spend huge time on the project
torpid pivot
mm, i just made the job calling bandersnatch keep a hash of the list of packages, and if it changes it adds --force-check
unique wren
Thatโs a winner. Nice.
narrow grailBOT
narrow grailBOT
potent pasture
Does bandersnatch support mirroring local pypi index ?
unique wren
Thatโs what it does โฆ
potent pasture
Does bandersnatch support mirroring
unique wren
Mypy 1.0 found a bug - crazy it's finally 1.0
dense totem
I said, "Oooo, 1.0.0. How exciting." when I saw the pull request to upgrade this morning.
unique wren
Been a long time
narrow grailBOT
narrow grailBOT
unique wren
These old pythons so slow. Need to deprecate!
narrow grailBOT
unique wren
I don't know how you survive in the dark ages
Now we wait and see if anyone complains about 3.11 dockers
@stuck aurora Any interest to get PEP691 to https://mirror.dub1.pypi.io/simple/ ? I feel the official mirror should support it of all mirrors ๐
As a side happy to ansible / docker and even store in s3 ...
Should also get resource wins / speed with py3.11
unique wren
Few docker downloads
vale swan
It's only 0.0025% of the global population smh.
unique wren
More than I expected anyways
Cause I bet most people pin their container and forget about bandersnatch until it breaks
vale swan
That is indeed quite a few. I'm just carrying on with what seemed to be sarcasm :p
vale swan
Cause I bet most people pin their container and forget about bandersnatch until ...
That's like CI in general I feel like
unique wren
I never do that
vale swan
If it weren't for dependabot or other similar utilities, things just wouldn't be upgraded ever until it breaks
unique wren
Dislike it profusely
I use to have dirty scripts that did it
"Lots of small upgrades better than pinning, waiting and big bang upgrades"
Lots of small amounts of pain rather than lots at once
vale swan
Yeah... I'm a little scared for when I'll have to upgrade my Ubuntu 20.04 LTS install to the next one. Smaller (and rolling) updates are generally nicer to deal with.
unique wren
Leant that very early in my career as my first job was a nuclear reactor where some of the computers were 1/2 as old as I was running ancient sendmails and OS's
And I vowed to never get into that hole where ever i could avoid
unique wren
Yeah... I'm a little scared for when I'll have to upgrade my Ubuntu 20.04 LTS in...
Wow. All my VPS's (2 of them), 2 home routes and 3 or so raspberry pi's went immediately to 22.04 weeks after it came out
I'm even tempted to move to Fedora Server so I get upgrades more
vale swan
I don't expect it to be painful (except for maybe my touchpad drivers) but I just can't be bothered reconfiguring everything.
unique wren
I feel ya
My Fedora desktop is all very supported Linux hardwares so I can be more risky and just upgrade all the time
vale swan
My laptop is a tool to get work done. I've mostly stopped playing around with it and tweaking it.
I do occasional clean up and backups but other than that, not much. Standard updates are done via Ubuntu's automated reminders to update.
vale swan
If I can, yeah. When I get a new laptop, I'll 100% dual boot Windows/Linux again.
... or just wipe it, I haven't used Windows in a long while. It doesn't hurt to have it though. Never know when I may need it.
unique wren
Have you seen any good replacement for myst-parser - They been blocking sphinx 6 for a long time ...
https://github.com/executablebooks/MyST-Parser/pull/664 - They are claiming close to a release again ... so I guess wait it out
Has been a long wait
vale swan
you ask for a release, you get oneโจ
unique wren
Amen. Time to merge our docs PR
To be fair I asked months ago to no response
But better late than never
Iโm sure a lot of me (me included) thought black was dead
teal moss
Have you seen any good replacement for `myst-parser` - They been blocking sphinx...
heads up with sphinx 6, there's a few bugs with multiple jobs
if you use multiple jobs to build, it's not going to be the same as if you use one job
most of them are known and patched on master but still waiting for 6.2 or 6.1.4 to include them
unique wren
Cool. Cheers for the heads up. We only have one job and Iโd expect our CI to see them?
quaint schooner
Cool. Cheers for the heads up. We only have one job and Iโd expect our CI to see...
Yea, it should be OK in CI
myst-parser had a hard time with Sphinx 6 FWIW. There was a long laundry list of things that changed and needed fixing. ๐
unique wren
Wonder if we can help them have a sphinx with main sphinx periodic CI run โฆ might open an issue this week after I see what their CI is like โฆ
So they can squash bugs more regularly
unique wren
Anyone else's read the docs start failing this morning like this: https://readthedocs.org/projects/bandersnatch/builds/19855559/
Seems python3.8 is lost in the read the docs environment?
Doc building CI is passing ...
unique wren
Seems to work again
unique wren
Bandersnatch getting some action of late โฆ Iโve had so little time to look ๐ฆ
narrow grailBOT
unique wren
Lame we didn't see the PR here
azure magnet
Could add it?
unique wren
I just need to go look at settings. I thought I had new PRs is all ...
I think we stopped them due to dependbot weekly spam ๐ so I'm torn enabling them again now I think about it
Ahh yeah, it's all PR events basically or nothing. I could live with new created ... but does not seem separated out ...
narrow grailBOT
narrow grailBOT
unique wren
If anyone has some time to spare, would love a review / look over of https://github.com/pypa/bandersnatch/pull/1457/files
vale swan
I can take a look tomorrow.
unique wren
No hurries or worries if busy - Just am a king of typos or silly mistakes ... ๐
Or forgetting things
Most of it was typing code mod by pyupgrade
Patch coverage: 92.03% and project coverage change: +3.70 ๐ โค๏ธ
junior zephyr
@unique wren Hey we are the group of the developers who emailed PyPI for the vulnerability in bandersnatch a few weeks ago. Would you mind if I directly message you in Discord to discuss that?
unique wren
Sure. But lets just submit the PR and merge it.
unique wren
Anyone know where i remove these required 3.8 checks and change it to python3.10? https://github.com/pypa/bandersnatch/pull/1457
unique wren
I don't remember ever doing this - Someone else must have
vale swan
In particular, step 7 "Optionally, enable required status checks. For more information, see "About status checks."
unique wren
Thanks - TIL
unique wren
Read the Docs doesn't have python3.10 for building available. I feel like when we moved to >=3.8 only too we hit the same problem ... 3.10 came out over 1.5 years ago ๐ฎ ... crazy.
unique wren
https://readthedocs.org/projects/bandersnatch/builds/20713525/ - Was our bad - We had outdated stuff i the .readthedocs.yaml
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
unique wren
We even have a PR up and asking s3path if what we did was the right way around their change moving forward โฆ
I try to keep bandersnatch up and hip
narrow grailBOT
unique wren
@stuck aurora - Added in PEP700 support to bandersnatch - Would love an eye over it when you get time to check I can read the PEP ๐ (was bored in an airport lounge and thought I should do some python work for a change): https://github.com/pypa/bandersnatch/pull/1557
@vale swan Got to use your PR number page of awesome ๐ Thanks! Hope all is well.
vale swan
I'm currently head deep writing some JS (will need to rewrite this function later) for a personal project
unique wren
Condolences for having to write javascript
vale swan
It's honestly not that bad. SvelteKit + FastAPI is pretty nice to work with :)
turbid crater
I'm currently head deep writing some JS (will need to rewrite this function late...
Any reason for mixing .then and async functions?
Also you can use throw in an async function instead of Promise.reject
vale swan
Any reason for mixing .then and async functions?
Can you tell I don't write a lot of Javascript? :D
vale swan
Also you can use `throw` in an async function instead of Promise.reject
Ah that's good to know, thanks!
unique wren
bandersnatch, the new Javascript channel! haha
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
royal trellis
Hello, Can someone assist me:
I am only trying to download 1 specific version of python (all files for windows/linux/egg/tgz/ etc...) of Python 3.6.8 -- to host in my offline lab.
Does this config file do what I need?
Please verify that this config file below will only download Python python 3.6.8 (and no other versions)?
# cat /etc/bandersnatch.conf | grep -v '^;' | sed '/^$/d'
[mirror]
directory = /mnt/mylabnas01/repos/pypi
json = false
release-files = true
cleanup = false
master = https://pypi.org
timeout = 10
global-timeout = 1800
workers = 5
hash-index = false
simple-format = ALL
stop-on-error = false
storage-backend = filesystem
verifiers = 3
compare-method = hash
[allowlist]
platforms =
py3.6.8
royal trellis
Does anyone know?
royal trellis
Please help me ๐ฆ
unique wren
@royal trellis I don't think that's going to help you that much as there are a lot of wheels with py3 only and you'll possibly want them.
I'd recommend making a vitualenv via python3 -m venv /tmp/venv and pip install all the possible modules you'd want in there and running pip freeze to see all the packages you need (as pip will download all dependencies) then just add all them to your allow list and maybe block out windows and macos wheel etc.
Filtering PyPI packages is never easy sadly
narrow grailBOT
royal trellis
Hello, Can someone please help me. I posted my question on stackoverflow
I am trying to create a offline repo for pypi for only python 3.6.8 packages with bandersnatch mirror. It downloaded everything but skipped the "requests" package, and I dont know why?
Please, can someone kindly help me. Thank you so much!
unique wren
That's still going to download a lot of packages. As you'll get all py3 wheels ... I'd suggest just getting the last X versions as well ...
I don't think there is going to be a huge amount of bandersnatch gurus on stack overflow ...
rancid rapids
Hello, Can someone please help me. I posted my question on stackoverflow
https:...
requests requires at least python3.7, I'd need to double check a few things, but I'm pretty sure that alone would do it.
you might want to use python versions that are at least still recieving security updates.
royal trellis
Thanks for the feedback guys!
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
woven stirrup
Hello everyone, I've got a quick question and I'm not sure if it ever has been asked as I can't find any post on internet stating so.
Is there a way, with bandersnatch, to partially download the content of Pypi ?
Say because of my internet or because of my storage I want to download packages one letter at a time, is that doable ?
If so, would it work with a file that contains a list of packages I want do download for letter a ? (And then have it on my own frozen mirror ?)
Thank you !
woven stirrup
Thanks a lot, I wasn't even aware I could do that through the allow and block filters, I always assumed they would only be specific to RegEx or similar expressions.
Thanks a lot !
woven stirrup
Well it turns out that the allowlist and Blocklist plugins require specific versions to be taken into consideration, which is not going to make the task easy at all
woven stirrup
Okay I made it half work by enabling the plugin project_requirements. I do have a question though, if anyone has ever used the plugin. Bandersnatch seems to print INFO : package XXX allowlisted but then does not proceed to download and or write anything to disk. I'm scratching my head over this because I do not quite get why it behaves like this.
My immediate concern is this line in the filtering_configuration page of the documentation: Any packages not matching the version specifier for Allowlist packages will not be downloaded and I fear this might be true to requirements too
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
wild lynx
Hi, I'm having trouble mirroring a gitlab pypi repo. The index URL follows the format https://auth-token-name:AUTHTOKEN@gitlab.com/api/v4/groups/0000000/-/packages/pypi/simple, and bandersnatch seems to parse that just fine from the config file.
Unfortunately Gitlab returns a 415 Unsupported Media Type error in the first request from bandersnatch. What am I doing wrong? I'm sure I'm not the first to mirror a gitlab pypi repo.
I've searched google/ddg for about 45 min, and there is barely anything on the subject.
unique wren
Bandersnatch relies on the xmlrpc APIs PyPI itself hosts that pip does not use. Due to that it does not support mirroring of private mirrors โฆ
Would be open to discussing ways around those APIs and abstracting it to use something else โฆ e.g. we could just pull the Simple API and try grab every package or something each run in small private mirrors โฆ
But it would be a big PR and design exercise
wild lynx
Damn, that's unfortunate. Since supporting private repos is non-trivial, I'll have to search elsewhere for a solution.
Thanks for the swift reply! ๐
Do you by any chance know how I'd go about mirroring a private repo?
unique wren
Depends how itโs stored. If on a filesystem โฆ rsync could even work.
Do you have GitLab support? Ask them?
wild lynx
Thank you for your time ๐
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
Hello,
I'm using v6.5.0 and when I attempt to use bandersnatch verify --delete I get a key error.
I read the docs, and from what I understand, storage-backend defaults to filesystem, so I didn't think it would be required to add it into the configuration file.
https://bandersnatch.readthedocs.io/en/latest/mirror_configuration.html#storage-backend
The mirror section looks like this:
...
narrow grailBOT
unique wren
^^ Would love any help here
I am an s3 n00b but would love to be able to offer bandersnatch in >= 3.12
unique wren
This has to be the saddest channel on this discord server ๐ฆ
Most other channels get action, none here โฆ haha
unique wren
s3 will be the death of me: https://github.com/pypa/bandersnatch/pull/1728
(and my lack of ever using it due to being @ Meta/Facebook for 11+ years)
Least the maintainer of s3path has been awesome helping me: https://github.com/liormizr/s3path/issues/172
Would love help from anyone who's s3 savvy with bandersnatch ...
wanton steeple
Popping in to say hello! ๐
Is the S3 work still top of the list? I havenโt used S3 and Bandersnatch together, but Iโm happy to try and chip in where I can.
unique wren
Popping in to say hello! ๐
Is the S3 work still top of the list? I havenโt us...
Hello. S3 still has some code Iโd like to refactor. Weโve made it to 3.12 but Iโd love to fix some s3path private API weโre still hitting. I talk about it here with the s3path devs here:
https://github.com/liormizr/s3path/issues/172#issuecomment-2119279403
So would love help fixing that if you see a way.
Iโd also love help moving to the latest pytest, it seems to hilights some asyncio bugs we have: https://github.com/pypa/bandersnatch/pull/1656
I havenโt spent huge time on it but Iโd love to fix it before I release the next version โฆ (which I really need to do)
wanton steeple
Thanks! I wish my time available to contribute was more regular, but if I have time this weekend/next week I will prioritize these.
unique wren
Understandable, and that would be awesome ๐
unique wren
Feel free to ask any questions here - tag me tho so I get a notification ๐
narrow grailBOT
flint portal
@unique wren I'm in early phases on exploring an idea on making projects uploaded to PyPI temporarily "hidden" for whatever reasons - and was curious to understand how a mirror would behave in that circumstance.
I can imagine a few cases:
- the mirror has already retrieved the package, no new updates listed, no new fetching, no impact
- the mirror is aware of a new package, but has yet to retrieve it, and when it tries to, it's no longer there <== what happens here?
- the mirror saw a package, it went unlisted, and now it's listed again <== rare but possible outcome
Let me know if that makes sense or not - happy to discuss further!
(also I was digging around PEPs for a mirror protocol, didn't find one - is it basically "follow PEP 503"?) found https://peps.python.org/pep-0381/
unique wren
Don't think it's PEP'd, but we're all driven from packages returned via the expensive XMLRPC list_packages_with_serial and changelog_sinze_serial calls:
https://github.com/pypa/bandersnatch/blob/main/src/bandersnatch/master.py#L201
I feel you make those RPC calls return packages how you want, bandersnatch will do the right thing ...
flint portal
Thanks for the pointers, very helpful.
From here it looks like the second scenario I mentioned would not break: https://github.com/pypa/bandersnatch/blob/6eaaf79597e76bad194ee6d65918d69927830491/src/bandersnatch/mirror.py#L129-L135
unique wren
I feel if you just don't share the new package via the RPC calls until you don't want to hide it anymore, wouldn't bandersnatch just stay working as is today.
It just won't sync packages until they are "visible" ...
Am I missing anything here? I feel that would be the simplest approach.
flint portal
Right - that's if I don't ever show it, nobody knowns, nobody mirrors it, until we do, which is fine.
The case I'm thinking of is if we did show it at one point, and now no longer show it, but it's not a deleted operation
so it could resurface
stuck aurora
changing visiblity probably has to put an event into the changelog
otherwise mirrors are going to end up in an inconsistent state
flint portal
otherwise mirrors are going to end up in an inconsistent state
Can you help me understand this more? I'm looking to understand what "promises" PyPI has made that we have to adhere to, specifically vis a vis mirroring, and from looking at the little code I have in bandersnatch, it seems like if there was a package at one point, and now it's no longer found, it's no biggie
stuck aurora
they'll be in an inconssitent state wrt PyPI itself. The mirror will have files exist on them that don't appear to exist on PyPI anymore
or if the file gets restored, PyPI will have files that don't appear on the mirror
the system is designed to be somewhat eventually consistent, mirrors look at /simple/ to determine what files should be made available, but they can't reasonable sync all of PyPI every time, so they use the change log to determine what /simple/$FOO/ pages to sync.
If a file goes missing, without an entry in the change log, mirrors won't fetch /simple/$foo/ for that page, and will continue to serve that file (until they happen to sync that file for some unrelated change).
If a file gets added, without an entry in the change log, mirrors won't fetch /simple/$foo/ for that page, and won't be able to serve that file (until they happen to sync that project for some unrelated change).
flint portal
Following this logic, it's similar to how after PyPI removes a project, it increments the changelog, the mirror sees a new entry in changelog_since_serial() and decides what to do next?
FWIW, it looks like bandersnatch will remove the files on PyPI 404 if running in delete mode, which means that if the file visibility returns, they would re-sync it?
(again, likely a rare case)
stuck aurora
they would resync it next time they syn'c that project
which they wouldn't sync that project until it showed up in the changelog again
basically the rough algorith is:
changes = changelog_since_serial()
projects_to_sync = {change["whatever index project name is"] for change in changes}
for project in projects_to_sync:
make_local_state_match_pypi(project)
if the available files change without a change log entry, mirrors won't "see" that the list of files has changed
flint portal
cool, thanks for clarifying that. I will follow up with you and other pypi admins with a document I'm working on soon
unique wren
bandersnatch also doesnโt delete things very well. Would love some form of event based delete / cleanup too.
If that exists Iโd love help knowing how to tune bandersnatch to do that cause we donโt clean up well today except via โverifyโ which takes a very very long time โฆ (it effectively crawls the data store and compares to latest JSON API package entries โฆ)
No idea if it works at all for non file system mirrors too โฆ
flint portal
bandersnatch also doesnโt delete things very well. Would love some form of event...
We currently emit some strings in the changelog entries, but there's no standard today on their content
Unrelated, I noticed that there's some hard-coded checks in place for https:// URLs in mirrors - would you be open to either allowing a config/cli arg to override that for local mirror testing against a localhost warehouse?
unique wren
No, would totally be open.
Just never supported it cause no one runs the xmlrpc locally โฆ
Iโve wanted to kill the xmlrpc usage for years
flint portal
Us too
unique wren
Will diff bandersnatch the moment itโs live
narrow grailBOT
flint portal
narrow grailBOT
narrow grailBOT
Hi again, I'm trying to reduce the mirror size because now the deletion takes too long.
(approx 1 day with xargs + rsync)
I've found that there are 440k packages that only have 2 versions, which are quite obviously not used by anyone else but the authors themselves, and want to find a way to filter them.
Is there any way we can do this?
narrow grailBOT
narrow grailBOT
unique wren
Anyone seen error with types-pkg_resources and the face it's latest version is yanked with pre-commit?
cooper@cooper-fedora-MJ0J8MTZ:~/repos/bandersnatch$ /tmp/tb/bin/pre-commit run -a
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
...
[INFO] Installing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/home/cooper/.cache/pre-commit/repovugpwdl0/py_env-python3.12/bin/python', '-mpip', 'install', '.', 'types-filelock', 'types-freezegun', 'types-pkg_resources')
return code: 1
stdout:
Processing /home/cooper/.cache/pre-commit/repovugpwdl0
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Collecting types-filelock
Using cached types_filelock-3.2.7-py3-none-any.whl.metadata (1.4 kB)
Collecting types-freezegun
Using cached types_freezegun-1.1.10-py3-none-any.whl.metadata (1.4 kB)
stderr:
ERROR: Ignored the following yanked versions: 0.1.0, 0.1.1, 0.1.2, 0.1.3
ERROR: Could not find a version that satisfies the requirement types-pkg_resources (from versions: none)
ERROR: No matching distribution found for types-pkg_resources
Check the log at /home/cooper/.cache/pre-commit/pre-commit.log
PR that introduces this: https://github.com/pypa/bandersnatch/pull/1788
Let's see if the mypy mirror has an issue
Ahh, repo has no issues :\
weak sage
Let's see if the mypy mirror has an issue
I don't think it's an issue there, as the failing dependency is specified in bandersnatch: https://github.com/pypa/bandersnatch/blob/c274da48cf8ec02d4536fa403ea1635122b2dc3e/.pre-commit-config.yaml#L34
unique wren
Now your wanting of a config param to include non PEP440 compliant version packages, I would accept that. Bonus points to use that linked library that can handle them (that seems warehouse uses?)
I on purpose took the stance to break non compliant versions once upon a time but can accept thatโs too harsh.
Just need to log were including evil non standard packages on the run log I feel and naturally add tests
royal trellis
Hi Guys, Im kind of at a loss, I have tried to use banderswitch several times, but after the complete sync, it never gets the idna package which is a dependency of "pip install requests", so Im not sure why this is happening (after >800GB download of pypi for my off-site dev lab environment)
Here is my bandersnatch, what am I doing wrong?
; Enable all of the plugins, with their default settings
[plugins]
enabled = all
; Allow all of the below platforms
[allowlist]
platforms =
py2.7
py3.6
py3.7
py3.8
py3.9
py3.10
py3.11
py3.12
py3.6.8
py3.8.6
py3.8.7
py3.10.6
; Dont bother downloading python versions we dont even use any more (i.e. <CentOS7.1)
[blocklist]
platforms =
py2.4
py2.5
py2.6
py3.1
py3.2
py3.3
py3.4
py3.5
; Dont bother downloading really big packages (Some are >100GB)
[size_project_metadata]
max_package_size = 2G
[mirror]
release-files = true
[latest_release]
keep = 5
I posted this on stackoverflow a few months ago, but no one has helped ๐ฆ
unique wren
Have you set the config to exit on error and see if it fails on idna?
What errors are in your run output / logs?
I would dare say itโs filter plugin bugs. They are all far from perfect sadly. Especially if you use multiple together โฆ
โโdebugโ output might help us work it out too
royal trellis
Hi cooperless, thanks - let me check
royal trellis
Hello sorry about the delay are you here? I am very sorry
Basically. I need to make a pypi mirror offline but just for my Rocky9 amd64 VMs. Been struggling with this since Nov2023, and definitely need help - was hoping someone on discord could help.
Ideally: Can you provide me with a simple bandersnatch.conf that will will download pypi (and not miss the idna package, which is a requirement for pip install requests).
My final goal is for my Rocky 9 VMs on my offline homelab, so I want toL 1. keep everything python3 any OS, use the latest 5 versions, and no package size limits), can you give me your bandesnatch.conf for this?
unique wren
Sorry, I never got tagged here. Do you have a config you tried? It sounds like doable, but we donโt test every permutation of plugins working together so there might be bugs โฆ
wanton steeple
@unique wren re: pytest 8.x https://github.com/pypa/bandersnatch/pull/1656#issuecomment-2510192557
This error with delete_path is weird. I poked at it some and made some notes, but I'm not sure what a real fix would be like. I may play around with pytest-asyncio's loop scopes, since it seems something to do with the event loop attached to a cached storage plugin being closed.
unique wren
Yeah, there are fixes needed for pytest asyncio update too that might hint at the root cause from memory
Thanks for taking a look. Happy to dig if ya get stuck to see if fresh eyes help.
wanton steeple
@unique wren After taking a break and coming back to it, I think I have a PR for you: https://github.com/pypa/bandersnatch/pull/1829
I updated both pytest and pytest-asyncio at once, since the changes in the two versions seem closely related enough. The change to delete_path looks redundant but is simple and works (on my machine, lulz).
I didn't add a changelog entry, but am happy to do so if needed.
...maybe spoke to soon - looks like its failing CI for macOS and Linux. Time to spin up a VM! ๐ฌ
unique wren
Different event loop for windows thab POSIX โฆ
wanton steeple
Solid point.
The test failure trace for the failed CI jobs look kinda strange to me - is it supposed to be invoking the swift storage backend? I peeked at the logs from a previous successful PR but couldnโt tell.
reproducing the tests in an Ubuntu container is probably a thing for tomorrow-me. Will be nice to have that around for tests.
Shouldโve known better than to jinx myself ๐
unique wren
Iโll see if I can take a peak tonight and if I can finish your PR Iโll merge it โฆ
I havenโt given any love to bandersnatch in a long time
wanton steeple
I got the tests to run in a Python 3.11 container yesterday, but had some weird file system problems - some storage backend tests that wrote to /tmp failed with hash mismatches ๐
So might be back to the full VM idea. Itโs not like it was writing to a bind mount or something. ๐ค
But still - progress! Haha
unique wren
I moved backed to using the storage backend loop ... but I can not workout what is closing the loop
@wanton steeple
This simplification of the test works: https://pastebin.com/5AuLgDF7
Passes, and tests enough for me, but I'd love to workout the loop closing issue as I feel it will bite us somewhere
If we call / await delete_path s econd time I get event loop closed which I don't get ...
I could remove the blob exists check as the delete_file (for filesystem anyways) also does a check ... so we were wasting a stat sys call ...
But I wonder if S3/swift do? So my patch could introduce a bug
But would love any asyncio savvy people to help us try workout how we're causing the event loop to close here in this test
Also, weird it works with 3.12 but not 3.11 on MacOSX
Highly doubt it, but wonder if there is a bug / what difference we are hitting with 3.11 only with event loops ...
I can't get integration tests to fail on my mac with your PR on my mac tho:
2024-12-03 22:00:48,127 DEBUG: Closing Master's aiohttp ClientSession and waiting 0.1 seconds (master.py:102)
Bandersnatch PyPI CI finished successfully!
So yeah, I'm thinking it's all test env bug / feature ...
narrow grailBOT
turbid crater
So yeah, I'm thinking it's all test env bug / feature ...
You could try using the anyio pytest plug-in, it's a bit more robust than pytest-asyncio
It uses asyncio.Runner to manage the loop, so it should never be closed unexpectedly
wanton steeple
At a previous job a few years ago there was a project where we ended up swapping to the anyio pluginโฆ very different project, but it was helpful there, so Iโm open to trying it.
I also meant to fiddle with pytest-asyncioโs event loop fixture scope. For example, if we set it to session scope, it should be reusing a single event loop for all the tests, and if we still encounter a closed loop then it must be something โourโ test code is causing vs the test framework.
Though, Iโm not sure what the downsides are to reusing an event loop across the whole suite like that as a long term thing.
Iโm still meaning to get the tests running in Linux containers on my PC if possible, to avoid having to wait on CI. (And avoid me prematurely announcing that Iโve fixed it ๐)
wanton steeple
Also, weird it works with 3.12 but not 3.11 on MacOSX
At least in CI, the stack trace for the test failure on macOS + 3.11 looks weird to me. It looked like it was showing that the delete_path test ends up using the Swift storage plugin which doesnโt seem right to me. I still think I must be reading it wrong ๐
turbid crater
At a previous job a few years ago there was a project where we ended up swapping...
Generally with re-using the same loop is it's very easy to leak state between tests
unique wren
I'd love to know why 3.12 works and 3.11 does not tho
But I might try move us to this anyio plugin and if we work, move on
I don't think we have a bug per say, and it's test fun ...
I bet we could move to running loop and manage a single loop better tho
I async'd this non arync code base long long ago
wanton steeple
Generally with re-using the same loop is it's very easy to leak state between te...
That makes sense. In general thereโs already a lot of things getting shared across tests, that I would like to refactor to reduce that, so moving in the opposite direction seems like a bad idea ๐ฌ
I agree itโs a test environment thing. When the actual tool runs thereโs only one event loop for the duration of the program (right?) and no way it can be invalidated (I think?) so whatever this case is seems like it should never be possible.
wanton steeple
But I might try move us to this anyio plugin and if we work, move on
I wonโt have time to work on this more until Friday evening at the earliest, but if I get to it before you do Iโll try the same. If it works then I can move on to GH issues and maybe refactoring some of the old unittest-style tests.
wanton steeple
I finally got the tests on main running in a Linux container locally yesterday! I made an Ubuntu image with 3.11 and 3.12 installed and run Tox inside. Thatโs all green, so now Iโve got a baseline.
unique wren
Nice.
But do you mean 3.11 fails and 3.12 passes?
wanton steeple
On main they both pass, I havenโt had time to try it after upgrading pytest etc
unique wren
O right
wanton steeple
I made good progress this afternoon and feel SO CLOSE to fixing all the event loop issues.
I used the โpytest-randomlyโ plugin to shuffle the test execution order and found many cases where the tests were implicitly order dependent because we have many tests that indirectly use an event loop while not being marked as async. Similarly if an earlier test initializes the storage plugins and and another later uses the cached instances, the loop saved on the plugin will be closed since the loop was only scoped to the previous test.
Iโm adding marks to test modules that need an event loop to be available even though the tests arenโt async, and a fixture for resetting the storage plugins between test modules, and that strategy seems to be working well.
turbid crater
I made good progress this afternoon and feel SO CLOSE to fixing all the event lo...
Are these tests using the deprecated get_event_loop() loop?
It's removed in 3.14a2 or 3.14a3
wanton steeple
Thank you for the heads up! For the tests Iโm marking modules or classes with pytest-asyncio so it an appropriately scoped loop exists, but the test bodies themselves donโt usually need it, rather the code under test.
I do think the code under test is using get_event_loop (places like the storage plugin initializer) and with the tests shuffled around one of the ways I was finding things was the warnings. Weโll need to change anywhere get_event_loop was used to get_running_loop, right?
turbid crater
Thank you for the heads up! For the tests Iโm marking modules or classes with py...
Well sort of
Usually you want to defer binding the loop as well as switching from the gel to the grl
Before:
class Foo:
def __init__(self, ...):
self.loop = asyncio.get_event_loop()
After:
class Foo:
def __init__(self, ...):
self._loop = None
@property
def loop(self):
if self._loop is not None:
return self._loop
self._loop = loop = asyncio.get_running_loop()
return loop
(feel free to ping me, I almost missed this message)
turbid crater
Thank you for the heads up! For the tests Iโm marking modules or classes with py...
Do you configure filterwarnings=error in pytest?
It should catch all these issues
unique wren
Yeah we get warnings and I've always tried to cut them down
We've jsut been stuck getting the test suiete to run with latest pytest-asyncio
I think once @wanton steeple get's that working or moves us we can make sure all the warnings are sweet again
I also need to release a 3.12 release and then test on 3.13 ... lol
I just don't personally use bandersnatch myself anymore so it's hard to dedicate time to it
I really appreciate @wanton steeple here โค๏ธ
turbid crater
You can also do a trick with self.dict.setdefault
wanton steeple
Do you configure filterwarnings=error in pytest?
I had not - TIL! Thank you
wanton steeple
Since it ended up being non-trivial, this is the Docker Compose setup I have been using to run the tests against Ubuntu + Python 3.11 & 3.12
https://gist.github.com/flyinghyrax/e3bed496cb049e71a5a5893e697f0b20
unique wren
<@813813914181697566> CI is now passing for [#1829](https://github.com/pypa/band...
O wow; happy holidays, bannersnatch!
unique wren
Back to 2 PRs open now
I'll try fix the type checking tonight / this week
Unless someone beats me
And might see if I can workout how to test that other PR that was contributed ...
wanton steeple
Thanks for tackling all those PRs! I'm happy I was able to get the pytest stuff working without having to rewrite lots of tests. ๐ (So now I can rewrite lots of tests... but one module at a time. ๐ฌ )
Happy preferred winter holiday!
narrow grailBOT
narrow grailBOT
unique wren
Finally the 3.12 release thatโs long overdue.
I think most of our users are docker these days and use latest โฆ
narrow grailBOT
unique wren
Wow. Tests pass with 3.13 โฆ just needed a s3path dep upgraded.
Nice work by then supporting 3.13 nice and early this time
narrow grailBOT
narrow grailBOT
narrow grailBOT
glad fox
hi bandersnatch maintainers! your thoughts/opinions on exposing status markers (e.g. "this project has been archived") via the index APIs would be greatly appreciated on this thread: https://discuss.python.org/t/pre-pep-discussion-project-status-markers-in-the-index-apis/79356
shadow trellis
Hey, all. I'm generating a local mirror with bandersnatch, and my organization's requirements involve either an extremely long allowlist or an extremely long denylist. The way I read it, the only way to implement this is to add 60k lines to the config file under [allowlist].
Does the config file processing allow include parameters for pulling in external files?
Alternately, is there a better way to go about this?
Presume that the lists are entirely arbitrary.
unique wren
hi bandersnatch maintainers! your thoughts/opinions on exposing status markers (...
Seems fine โฆ prob worth opening an issue and link to the pep to see how we can implement it on the bandersnrach side.
Itโs a fairly static setup at the moment so would need to use files/httpd/nginx features that S3 support to serve it โฆ (havenโt read any of the thread yet)
unique wren
Does the config file processing allow include parameters for pulling in external...
Itโs all pythonโs built in configparser. And Iโm not aware of any support for include files there
shadow trellis
Yea, configparser doesn't include that. The best you can do is search through the config app-side and layer described configs.
But it has to be intentional on the app's side.
shadow trellis
Makes sense. Maybe I'll add a filter plugin, then. doesn't look too hard.
Actually, it looks like the requirement_path allowlist will do what I'm looking for.
unique wren
Bug fixes / tweaks welcome
all the plugins are exactly this tho, someone had a niche need and they filled it.
we don't have good ci testing the interoperability, ordering etc. etc.
shadow trellis
I get that. I've forked, and will submit patches.
narrow grailBOT
wanton steeple
This change was me - I was not aware that any projects consumed Bandersnatch as a library. It looks like Pulp is also using the Mirror etc. classes.
unique wren
Yeah โฆ itโs a big user
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
unique wren
/kick @teal python /ban
grizzled matrix
uh... that looks potentially malicious?
yeah, in combination with the same user's comments on https://github.com/pypa/bandersnatch/issues/2052#issuecomment-3523530737 that is definitely sus
https://github.com/Henko551/build/tree/patch-1 seems to be just messing around. No idea what the user is trying to accomplish.
narrow grailBOT
narrow grailBOT
unique wren
Been awhile
unique wren
uh... that looks potentially malicious?
Sure does / did
narrow grailBOT
unique wren
Ok - 7.0.0 time. Going to:
- Remove swift support - https://github.com/pypa/bandersnatch/pull/2081 (no one help maintain it)
- default us to PEP691 JSONv1 output for pacakges to sync (>=6.7.0 allows config enableding this)
- removing the need for xmlrpc - cc: @stuck aurora + @umbral cradle
- So we can prob make a deprecation plan for PyPI ... I'll open an issue when I ship the feture to 7.0.0.dev0
- removing the need for xmlrpc - cc: @stuck aurora + @umbral cradle
- remote py3.11 support
teal moss
@unique wren this is still live but I'm pretty sure you can delete the issue if you have permissions to
unique wren
I don't think I can
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
unique wren
Fun fun webserving side issues here I feel
unique wren
Wow - no upgrade PRs open for the first time in a long time
Thanks to a new docs myst-parsers released finally!
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
unique wren
Sweet.
thick dagger
Lol, I never heard of bandersnatch, but I know all maintainers, except @unique wren ๐คฃ Oh, Theuni created it ๐
unique wren
None of them but me have touch it in years ...
Has Christian returned to any open source work?
He basically handed me the keys and ran
FWIW - I hate the name
thick dagger
Which Christian? There are so many ... Zagy, Theuni, Kauhaus, Schmidt.. ๐ I think you won't get employed at their company unless you are named Christian ๐ But I think you talk of Theuni? Yes, he does a couple of open source projects, basically they now have a hosting company, and everything is open source they do.
unique wren
Yeah, talking about Theune
thick dagger
Most notably may be https://github.com/flyingcircusio/batou and https://github.com/flyingcircusio/backy
unique wren
He would probably hate that bandersnatch is all strictly typed python now. he seemed to hate it when I showed up with typed PRs
Wonder if he has come around
Nope - All of batou is untyped. Yuck.
Backy too
thick dagger
He basically handed me the keys and ran
That's at least better than what happend this week at work. The admin left and kept the passwords. Was all day resetting root passwords and stuff of linux boxes ๐
unique wren
Yeah, sounds it. I know my alcohol would get a beating ๐
vale swan
๐
unique wren
I've been asking for a second active bandersnatch member for Y E A R S
So thankyou ๐
vale swan
I mean, I have a long way before I'll be writing any code for bandersnatch. At least running bandersnatch once would be a good start lol
unique wren
You can just run the CI locally. That's how I test it these days.
I really need to update the PyPI instance version to 5.0.0
I just can't get a test instance from Ee
I have access to the prod one. LoL
vale swan
wow.
unique wren
haha ๐ You gave good reason. All good
unique wren
You're an evil man
But, each to their own.
Bandersnatch is very strictly typed these days. It found subtle bugs.
pallid barn
cool
unique wren
I got it all asyncio, windows suppot and it's >= 3.8 these days.
pallid barn
Nice ๐
unique wren
Do you also hate black formatting?
pallid barn
I just cleaned up one of my early asyncio-based projects a few weeks ago and similar to my initial experience with twisted a long time ago: one has to get it quite wrong initially to understand.
Yes, of course! ๐
unique wren
LoL - It's all black formatted too
unique wren
I contribute to black. It was invented at the desk next to me @ #dayJob
pallid barn
I guess that makes it more palatable.
unique wren
I helped chose most of it's early syntax choices. So it does great for me.
Well glad you're all good and stuff
pallid barn
I'm absolutely for automatic formatting and I'm using yapf both personally and in my team. But we really really did not like some of the style choices.
pallid barn
Yeah.
unique wren
Yapf is slow
pallid barn
I used to come to PyConUS regulary
oh it definitely is slow ๐
but it's faster than me formatting it ๐
unique wren
Agree a formatter is better the no formatter
pallid barn
I had to stop coming as I had to focus more on building my company and got married and had kids, and build a house, and so on.
pallid barn
I'm also happy to use black when a project uses it. It's just really interesting that for some reason a good number of tools / styles / ... over the last years went into exactly the opposite directions from what I grew up with / used to. And many times when I try to force myself to adapting those styles they keep triggering me in such heavy ways that I just can't.
Took a while to switch from HG to Git and I still get tripped up about the horrible usability from time to time.
So, as jugmac poked me about the discord here - you guys don't really need anything from me, right?
unique wren
haha - I gave in at a PyCon and moved bandersnatch to github for the fact contributors just know it
unique wren
So, as jugmac poked me about the discord here - you guys don't really need anyth...
I need nada
nice to hear from you tho
unique wren
I can live with both.
hg with all the Facebook plugins is superior tho.
pallid barn
it's really weird that even after a few years the git "UI" does not want to fit my brain
but i do enjoy some of the features that were not as direct in hg, like rebase+force push still feels dirty but works quite nicely.
we jumped hg as the proper hosting facilities all went belly up
unique wren
Yeah, that all died sadly.
pallid barn
once bitbucket dropped the ball and gitlab never picked it up we basically just bowed to the majority. and that's something i've become a bit disenfranchised with some open source communities.
I personally enjoy open source a lot for the variety.
But maybe 5-6 years ago i started feeling a "winner takes it all" vibe in some communities (black communicating in a similar fashion) and that just does not resonate with me.
unique wren
We never expect black to go the way it did. It was made for FB / IG + Lukasz and I just wanted it for our projects. We just wanted a Python gofmt
*expected
Got sick of people formatting differently everywhere across projects and our company
pallid barn
Yeah, that's where I'm all aboard. And I understand that communicating strongly/boldy is a good marketing move.
It just makes it a complete non-starter for me to even want to get involved.
And I get that you didn't intend that - unexpected consequences and such. ๐
Oh and yeah, if you're around in Germany I'm more than happy to have you over for a beer!
Alright. It's 6:30 over here and I have to get the kids some breakfast.
I'll keep the discord server in my rotation, so I'll be happy to talk.
unique wren
@pallid barn Sweet mate. All the best.
winter mauve
๐
vale swan
@unique wren where should the CLI help output go in the docs? my intuition is somewhere in the installation docs (by extension we'll need to rename it) because there's so little content in there: https://bandersnatch.readthedocs.io/en/latest/installation.html
unique wren
Itโs a simple thing. It gets simple docs.
I feel in the main page / QuickStart section was my thought
vale swan
I don't think the RTD docs have a quickstart
the index / main page would also work well
I'll go with that first
unique wren
Better reactions here than IRC
vale swan
๐
split jungle
I'd propose to not put the help output in the docs, but actually generate the indexed/referenced documentation for it via https://pypi.org/project/sphinx-argparse-cli ๐
unique wren
@vale swan is dynamically generating the help output
split jungle
ah cool ๐ nm then
split jungle
oh yeah that's what I thought
but IMHO sphinx-argparse-cli is better than showing the output of -h ๐
take a look at builds - https://pypa-build.readthedocs.io/en/latest/index.html#python--m-build
to get an impression what you'd get
vale swan
I'll look into, it looks pretty slick (/me wonders if there's anything for click based CLIs)
split jungle
click under the hood just uses argparse ๐ so should work for it
vale swan
But it appears sphinx-argparse-cli needs a function that returns an instance of argparse.ArgumentParser so I'm not sure how I would get click to provide that ...
vale swan
I totally did not forget to add a changelog entry ๐ ... will fix
it's really useful the changelog entry check, just still gotta extract it into its own repo
split jungle
But it appears sphinx-argparse-cli needs a function that returns an instance of ...
would need to look at how click is implemented, but I'd guess you'd have something like click.parser ๐
vale swan
ah ok, so you know less about using click with that than I had expected ๐
I may or may not look into it for black, probably will get delayed into "abandoned" territory though
split jungle
wait, click has its own parser ๐ https://github.com/pallets/click/blob/29df8795dc146ddea328e458068185d3314820e5/src/click/parser.py
never mind me then ๐ I was assuming it's using argparser under the hood ๐