#bandersnatch

1 messages · Page 2 of 1

split jungle
#

I like click less and less by each day, argparser is so much simpler and more powerful 😆

#

guess you're stuck with programoutput then 🙂

vale swan
#

yep, on the upside, that means less work for me 😄

split jungle
#

looks a bit less polished on the user side, and harder to link to it 🙂 but c'est la vie

vale swan
#

that coverage increase on my PR surprised me too @unique wren

#

not sure where those three extra hits came from lol

unique wren
#

Function

#

*From adding a function so 100% of it is covered so more lines == greater %

#

Just shows how useless having 100% coverage can be

#

It's a good benchmark tho

vale swan
#

oh, that makes sense, I should've thought of that

vale swan
#

@unique wren is travis-ci disabled for bandersnatch?

unique wren
#

@quaint schooner broke it

vale swan
#

lol

unique wren
#

Ideally I’d just like to move all our shit to GitHub actions. Only thing we need to move to the coverage secret

vale swan
#

It's just the codecov that matters right? We can move it

unique wren
#

That’s the one

#

@quaint schooner can prob move our secret. I’ll have to check if I have it in my password safe or generate a new secret

vale swan
#

If I'm reading the docs correctly, the codecov-action support tokenless uploads for public gh repos

quaint schooner
#

I see my name against breaking, what did I do?

#

Oh, Travis CI. LOL

vale swan
#

Yeah, it's just the coverage we need to move. No biggies, I expected something to happen with Travis CI one day.

quaint schooner
#

Lemme re-enable that. I do suggest moving away from Travis CI tho, since PSF has to pay $$$ for it now.

vale swan
#

I'll try to get coverage + gh setup soon then (although the last one I tried doing this with coveralls, it was painful, I'm hoping codecov isn't as painful)

quaint schooner
vale swan
#

Yep, I'mma see if I can move codecov in a quick timeframe first tho

#

Thanks for the quick reponse!

quaint schooner
#

:)

unique wren
#

Cheers @quaint schooner - we’ll try deprécate Travis for bander

vale swan
#

how did you make that typo @unique wren 😆

unique wren
#

The accent? No idea why Apple did that to my spellings

vale swan
#

Oh, you're on your phone, less or more typo-prone on it?

unique wren
#

More. I typo everywhere

vale swan
#

oh no

vale swan
#

@unique wren got the migration PR up

#

it was way less painful than last time with coveralls for psf/black haha

unique wren
#

Awesome. Thanks man

#

Lol at your branch name

vale swan
#

Time to shut down travis ci for good now? i.e ping pradyunsg again?

unique wren
#

@quaint schooner low pri - we can kill our Travis now. Can’t we just delete the Travis config file?

vale swan
#

That got rid of the push travis builds but somehow travis PR status checks are still showing up (and they're required too 🙃 )

#

Although travis hasn't built anything these past days so I wonder if just marking the travis check as not required will get rid the "forever yellow" status

quaint schooner
#

Yes. Just delete.

unique wren
unique wren
#

Yeah - i can't find said setting 😦

vale swan
#

😦

#

nothing in branch protection?

unique wren
#

Found it. Damn UI changes. lol

#

Green ticks! ✅

#

now wonders if we can get CI success / fail in here like IRC

#

Seems to be a few

unique wren
#

@vale swan you looked into what we have to do to move bandersnatch to Sphinx to 4.0.2?

vale swan
#

unfortunately myst-parser still doesn't support sphinx 4+

unique wren
#

Grr - "Move to this, but we're painful hard version pinners"

vale swan
#

Yep, I personally don't mind it too much tho.

unique wren
#

I dislike pinning. I'd rather let my CI tell me if it works or not and automate more things to move forward, rather than baking in tech debt / guaranteed work when new versions come out.

vale swan
#

Eh, I view it as making sure a release isn't published which is marked as supporting something when in reality it doesn't and therefore causing never-ending frustration when pip installs it thinking it's fine (although TBF they should be testing the latest versions of their dependencies on a regular basis)

loud knot
unique wren
#

If, we could run it.

#

This is why I offer bandersnatch with the tested versions in requirements.txt and the bare minimum deps in setup.cfg to give people who need it version freedom.

vale swan
#

I guess the "good first issue" label actually did something, huh.

unique wren
#

NFI - I've tried it for years

vale swan
#

MyST-Parser 0.14.0 doesn't unblock sphinx 4 IIRC

unique wren
#

I went and read issue and code - It does not

#

and our CI confirmed - CI never lies, unless it's shit CI

quaint schooner
#

Yea, it's still got "sphinx (<4,>=2.1)"

vale swan
#

@unique wren do you want to try pre-commit.ci on bandersnatch? I guess if it doesn't alleviate your pain with pre-commit we can always remove all things pre-commit 😄

#

also as usual, we can use bandersnatch as a testing ground before applying changes to psf/black :p

unique wren
#

Yes please. It will make me hate it less and I’ll move flake8 back into it.

vale swan
#

@quaint schooner I requested pre-commit.ci to be installed on bandersnatch. I may have also requested that it should be uninstalled on all other repos by accident :/ Might be the same thing with the Travis CI installation request a while back. My apologies in advance! Also thank you!

#

The configuration UI isn't very clear IMO.

#

It wasn't clear whether you have to relist the already installed repos while also adding the repo(s) you want to add access to when updating the config.

quaint schooner
#

Yea, I'll take a look, in case someone else hasn't approved it already.

#

Should be good now.

unique wren
#

Thanks champion.

vale swan
#

So wait, what are you concerned about @unique wren ? About the tox lint env becoming unknowingly broken?

unique wren
#

We no longer test “tox -e lint” so it could break

#

We should remove it and just say run pre-commit for listing

#

*linting <— that one was apple

vale swan
#

Oh haha, I didn't know there's currently no reference to linting in the contributing docs!

#

I assumed that people running tox -e lint locally would be fine, but we don't even mention it once in any sort of docs 😄

split jungle
#

I personally always leave a tox env to set up pre commit

#

See the tox/virtualenv projects

vale swan
#

Eh, I don't really mind either way. I've never been a big fan of tox (although I like its simplicity!), especially for my own needs. Bandersnatch dev still requires a well set up environment for certain commands (primarily the integration test).

#

But I do like the minimal initial dev requirements for virtualenv!

teal moss
#

another note:

#

DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.

vale swan
#

Yeah, I guess we can turn that on for the time being although I expect our rather simple packaging needs to not hit any issues with this sort of change.

vale swan
#

I find having uvloop tests run serially still fast because the environment setup cost of a separate workflow

#

I guess the only main difference would be that uvloop specific issues won't be reported if non-uvloop tests fail, but I'd argue those tests failing are more important :p

#

@unique wren I'd recommend not adding parallelization to bandersnatch since 1) it's only gonna shave a few seconds at most, and 2) there is a bug that leads to the whole session crashing during test collection

#

I'm actually considering getting rid of it for black due to that bug being so frequent, although over there the speed gains are much higher so I'm not sure.

unique wren
#

If you don't think it's a big win I don't care. Tests are decently efficient.

unique wren
#

Feels like layers on layers to me

#

I get the env advantage I guess

#

But that bring me back to my dislike of pre-commit

unique wren
split jungle
#

You can do it manually too, but IMHO makes it easier for casual contributors 😊 but ultimately these decisions are up to the maintainers 😊

#

The layers are there no matter what, it's just about doing it manually or semi automatically at the end of day

unique wren
#

With our parallel testing in different python these days, tox only really gives local dev an advantage. It's overhead for CI these days.

vale swan
#

Yeah, I agree with the sentiment, and I'm considering adding full support for tox for black development (we often end up being the first OSS project people contribute to!), but I'm not sure whether the simplicity is something bandersnatch should work on.

unique wren
#

*different python versions

split jungle
#

Up to you though, I digress 😊

vale swan
#

I'd disagree (only because I know the projects I work on pretty well) but it is way better for newer contributors

stable sleet
#

I'm also a fan of [nt]ox having a lint session, and it just runs pre-commit. I'd use pre-commit directly personally, but it's a nice entrypoint for someone to only have to have one thing installed. Though pipx run pre-commit isn't unreasonable, either 🙂

unique wren
#

But then you need pipx on top of pip 😄

stable sleet
#

Having [nt]ox is great for newer contributors, and some of the exotic jobs locally too. Or just when I work on too many things and can't remember if I have a venv folder setup or not. 🙂

#

pipx run tox, pipx run nox. You always have to have to install something 🙂

vale swan
#

I think we all agree on the point it's better for newer contributors 🙂 It's more I actually don't really care tbh enough to fight for it

unique wren
#

Totally agree for new people. Neither can I be bothered to argue. I just hate layering things on top of things that bring me little benefit. But again, I get it for making things more accessible / automated etc.

vale swan
#

I don't like the layering either and I don't think we get enough (edit: contributor) traffic to warrant the additional complexity.

stable sleet
#

I don't really have a feel for how much it helps new contributors yet, it just seems like it should. Personally, I find tox hides too much of what's really happening, and looks too ugly for my tastes, while nox reads better - it's not hiding what's being run as much. But it's slower. It did help me with a pip contribution, actually. But personally, I almost never use them locally for something I work on day-to-day, and if you aren't having issues with lots of contributors wanting to contribute but they find it hard, then there's no pressing need to use them.

vale swan
#

slightly OT: but yeah I agree I prefer nox over tox because of config readability

unique wren
#

We could almost:

  • pip install -r requirement*
  • lint - pre-commit run -a
  • Unit tests - pytest ...
  • Integration Test run_test.py
unique wren
stable sleet
#

Tools like that are not a substitute for good practices and good documentation for how to do things manually (and I rather don't like using them in the CI)

unique wren
#

tox was super useful when bandersnatch was on bitbucket and I need a py2 and py3 env to port it to Python 3

vale swan
#

Our contributing documentation could certainly do with some work

#

Window contributors are actually treated like they don't exist in the example commands 😬

unique wren
#

FWIW - There was very little when I came to the project. I had to read tox files and work it all out

unique wren
#

(The person who finished my work)

vale swan
#

Yeah I'm not saying it's your fault, it's something totally up my alley to work on 🙂

unique wren
#

O, if you want to sure. I just don't have a use case, and if I did have windows I'd still run bandersnatch in a docker container imo

split jungle
plain imp
unique wren
#

It's still a draft. Does it look like it will become real?

#

@plain imp - Thanks for raising - Will make an issue to say support if this goes past draft.

snow sparrow
#

It likely will. The discussion has died down after my recent edit and I plan to ask for a pronouncement maybe late this month if things keep quiet.

unique wren
#

Cool. It's not a super hard diff for bandersnatch. pull more files + add more into HTML right?

#

(I haven't read the diff in detail)

#

Is this planned for PIP to use?

vale swan
#

@unique wren my guess would be that pre-commit.ci won't catch the mypy upgrade until next monday

#

no clue whether it would update an existing pr or open a new one tho

unique wren
#

I see I have a current open merge - so I'll merge that too in case it blocks if another one is open

unique wren
#

Damn, was hoping the pinning happened in one PR for new typing stub deps ...

vale swan
#

I've found the typeshed separation to be rather annoying, I can look into it later tho

#

although pyup's handling isn't making it much better either 😐

unique wren
#

lol - yeah ... shit happens.

unique wren
#

Ok, cleaned up that mess while I had some down time. Painful.

vale swan
#

🙃

unique wren
#

@quaint schooner Is it supported (is there a public enough API) with the latest pip resolver bandersnatch could use to finally allow people to put on packages and automatically sync all its dependencies?

#

*in

#

In some new pip resolver plug-in

quaint schooner
#

No, there isn't.

unique wren
#

Didn’t think so. But I’ve read 0 lines of code. Cheers.

#

Was that a stretch goal I remember reading?

quaint schooner
#

Yea.

unique wren
#

Ok. I was not dreaming.

vale swan
# 
[2021-08-01T03:15:28.759Z] ['info'] => Found 2 possible coverage files:
[2021-08-01T03:15:28.759Z] ['info'] coverage.xml
htmlcov/coverage_html.js
[2021-08-01T03:15:28.759Z] ['info'] Processing coverage.xml...
[2021-08-01T03:15:28.760Z] ['info'] Processing htmlcov/coverage_html.js...

Hmm I wonder if this is why v2 of the codecov action is acting weird lol

#

I'm impressed that they even attempt at processing the JS file

#

Damn it's been so long since I've checked out this repository I forgot master is still the default branch's name.

vale swan
#

Well OK, setting files to coverage.xml didn't help and same thing with relative_files=True ... ugh I don't understand what's wrong here.

vale swan
#

So relative_files=True does do what as described by its name, it's just that I expected the wrong behaviour ... got it

#

Still nowhere closer to fixing this triple reporting of bandersnatch coverage

unique wren
#

Happy to change that - Just haven't got around to it

unique wren
#

I am stoked we might finally get the S3 plugin merged

unique wren
#

@vale swan Renamed master to main for you - Hope Id on't break any branches you have etc.

#

I'm sure you'll manage 😄

teal moss
unique wren
#

An announcement where?

unique wren
#

Now - Time to teach myself S3 and Amazon Static pages to serve a PyPI mirror to see if we can use it for PyPI itself

#

Is there anyway I can get free S3 to test shit? Could prob ask Ernest to setup some test stuffs

#

*sorry Ee

#

@vale swan ❗ Current head c3bded1 differs from pull request most recent head bfaa941. Consider uploading reports for the commit bfaa941 to get more accurate results

#

Saw that on a PR for bandersnatch - maybe that has something to do with our coverage screw ups

rain basin
unique wren
#

9.1T of generally small files

rain basin
unique wren
#

PyPI, to my knowledge isn’t S3 anymore and even if it still is, don’t think the r/o buckets are exposed cause we want everything pulled vía Fastly CDN

sterile arch
unique wren
#

Ya, thought I remembered hearing that. Bandersnatch still mirrors to Amazon gluster mount (or some form of posix file system) for backup/DR or something. We were talking of moving that back into S3 when bandersnatch supported that …

spice wave
#

@unique wren Hey, I'm trying to sync some packages from pypi with a proxy by setting the env var http_proxy, but the connection doesn't seem to be going through the proxy. Do you have any ideas why?

unique wren
#

Howdy - running 5.0 or main (master)?

spice wave
#

still 4.4 (lol), actually now Pulp is a 3.8 project so we can move up to version 5

unique wren
#

Is pulp async?

spice wave
#

parts of it yes, the rest api is a normal django app, but some of the tasks like syncing are done with async code

unique wren
#

5.0 is fully async (prob still has a few io calls that could be wrapped in threads) if that helps the move too

#

Anyways - sorry - forgot to keep looking - $dayJob ... it should work - You sure you had case right?

spice wave
#

Not sure, I might just have my proxy setup incorrectly. Going to do some more testing and if it doesn't work I'll try upgrading to 5.0 and test some more.

unique wren
#

I doubt going to 5.0 will help

#

It’s all just aiohttp - so check their docs to see if it helps you work it out. Will, as always, accept PRs

#

P.s. damn boto* update a lot. I get a PR ever few days since adding them for S3 storage support

vale swan
#

Yea I've heard apparently it's automated for every API change that they do ...

stuck aurora
#

it is

#

a lot of botocore is autogenerated from the internal service decleration files

#

so anytime any AWS service modifies its external API in any way, it triggers a new release for the day

snow sparrow
#

boto* and azure-* are the weirdest things by far on PyPI in terms of release policy. Boto update counts are off the charts because every single change triggers a new release, and Azure stuff are split into such atomic parts (but still functionally interdependent) a user-side update of one single package cascades into updating them all at once. Not sure which one I dislike less.

unique wren
#

Makes dependbot super annoying

#

And the pinned deps spamming my inbox cause the CI always fails

vale swan
#

yup, and on top of that boto3 is quite restrictive on what botocore it must be installed with

rain basin
#

I had to unsubscribe from many projects because of dependabot. And it also pollutes search result with its changelogs.

unique wren
#

Maybe we (unfortunately) exempt these two from auto update …

image0.png
#

And set reminders to do it manually ever 6 months or something …

vale swan
#

I'm in favor

unique wren
#

Might do the PR in a bit.

hollow dove
#

Isn't there a way to ask dependabot to do monthly updates only?

sterile arch
unique wren
#

O shit - look at this.

#

I'll do this ^^ then look into this schedule ... thanks guys!

#

O, we're pyup too

vale swan
#

that explains my github notification on my phone ^^

unique wren
#

We could move to dependabot for all I guess

#

But lets do the pyup for now to quiet things down

vale swan
#

What is coverage doing?

#

Oh, nvm. I forgot that not all of the jobs had run yet.

unique wren
#

O pyup has it all

#

Gonna edit the config instead

#

Will do every month

sterile arch
vale swan
#

We could change it for sure but it hasn't been that noisy or bothersome IMO. @unique wren thoughts?

unique wren
#

Pre-commit hasn’t been noisy. Need latest black ASAP

#

Most important

#

200% bias

split jungle
turbid crater
#

How does bandersnatch handle upstream packages being deleted?

unique wren
#

bandersnatch sync can, but it has to walk the full mirror and packages and workout what it "can" delete as there is not friendly API to work that out

#

I think it's time I release bandersnatch 5.1.0

unique wren
#

Ok, done.

obtuse sage
#

not found in documents

unique wren
#

verify sorry

#

bandersnatch verify --delete

obtuse sage
#

ok

#

very slooooow speed

unique wren
#

All ideas and fixes welcome. It has to pull the metadata and walk the file system - PyPI has no friendly API to tell us what's been deleted and their path layouts and sharding is hard to calculate

#

So slower your filesystem, the slow this will be unfortunately. And the mirrros sizes are huge these days

obtuse sage
#

Ya, I know it

#

I am holding a mirror site in China

#

Pretty slow sync speed, huge size, and cost days to verify --delete

unique wren
#

@obtuse sage - Ok, I don't know how to respond to that

#

What's your goal with that statement?

obtuse sage
#

nothing

#

just complain

sullen holly
#

Hi friends. When I do a curl for https://pypi.org/pypi/backports-entry-points-selectable/json it converts the - in backports-entry to a . I think this is related to bandersnatch_safe_name, but that function only converts [^A-Za-z0-9.] to - and - is obviously not converted to . here nor is every - converted. So how does bandersnatch and-or pypi convert this name?

snow sparrow
#

You have the logic the other way around; backports.entry-points-selectable is how the developer names the package, so it’s shown in the JSON payload. But the URL follows PEP 503 normalisation logic and converts the dot to a dash.

paper aspen
#

so it doesn't quite follow PEP 503

snow sparrow
#

Eh uh TIL. The JSON API really shouldn't be depended on by any new code

sullen holly
#

so xmlrpc was replaced by the json api but the json api shouldn't be depended on by new code? so i should scrape the html endpoints like pip does?

paper aspen
#

the name is not normalised in the title

#

I don't know where it pulls it from, either sdist or wheel metadata or their filenames

snow sparrow
#

The JSON API never replaced anything, whoever told you that was very misinformed

sullen holly
#

The XML-RPC API will be deprecated in the future. Use of this API is not recommended, and existing consumers of the API should migrate to the RSS and/or JSON APIs instead.

snow sparrow
#

Hmm OK. There’s a messaging problem I guess, the JSON API can be used as an replacement to XML-RPC, but it does not really replace XML-RPC in the sense of being a successor. It’s more like the two are (in the practical sense) equally legacy and one died before the other.

sullen holly
#

This all seems like an XY problem distraction. Fundamentally if I want to make an airplane mode for pypi to cache packages locally and survive blowing away my venv, which API should I be using?

snow sparrow
#

But I'd say reaching for an API is kind of an A/B problem in the first place, there're a ton of tools that do this without you needing to deal with any API

#

e.g. pip wheel if you just want to keep a copy of a few packages, bandersnatch is you want a comprehensive mirror, etc. (which reminds me why are we having this conversation in the bandersnatch channel?)

sullen holly
#

It's here because I got confused and thought pypi was a deployment of bandersnatch when it's really warehouse

unique wren
#

Yeah this is a static artifact mirror only 😄 With some file saving for JSON

lament ravine
#

Hello, I'm having some difficulties getting packages to download.

The first usage of bandersnatch it downloaded the tarballs. I deleted them in trying to understand how things work. Now every time I mirror (even with --force-check) it will populate the simple index with the index.html but the packages subtree is empty. Is there some cache record file I'm not aware of that I need to clean?

I tried deleting the mirrored-files file which didn't help and I try to run verify which gives this error:

$ ./_env_bandersnatch/bin/bandersnatch --config ./bandersnatch.conf verify --dry-run

2021-12-01 10:27:32,373 INFO: Starting verify for /home/salotz/resources/bandersnatch with 3 workers (verify.py:242)
2021-12-01 10:27:32,374 ERROR: No JSON metadata files found. Can not verify (verify.py:249)

Any help is appreciated.

unique wren
#

@lament ravine - Hello - Sorry I missed this - verify is not what you want here: bandersnatch sync pkg1 pkg2 ... for the packages you deleted should help.

#

Are you not saving the JSON files in your config? If so, maybe we need a check for that.

#

To say "verify needs json files to validate and your config had it disabled" - Or we can maybe make it just download them and not save them

#

Can you pastebin your --config ./bandersnatch.conf file please?

unique wren
#

Anyways - released 5.1.1

#

Small bug fix and our CI all passes again

lament ravine
#

@unique wren no worries. I think after evaluating the different tools and such a little more that we don't actually need bandersnatch in favor of a hosted index/mirror. Thanks for helping nonetheless.

unique wren
#

Ok. So devpi or something else fitted your use case more?

lament ravine
#

Something like artifactory. Also devpi for local testing.

vale swan
#

Argh I can't even get mypy to type check bandersnatch cleanly without using pre-commit

#

My best guess is that installing bandernatch's dependencies is triggering some check bandersnatch fails hence why pre-commit's mypy is happy.

paper aspen
#

How do you type check bandersnatch without installing its dependencies?

#

Do they all implicitly have type any?

paper aspen
#

yeah, the pre-commit hook passes --ignore-missing-imports to mypy so they all have type any

unique wren
#

Yup. The typing needs a lot of love again. Any the s3 contribution needs some cleanup. I was happy to get the feature so was lax on the reviewing etc.

#

Also want swift and s3 tests to have dedicated CI and normal filesystem seperate

unique wren
vale swan
#

seems to be working fine on psf/black, I'll take a look

#

hmm, seems to have worked on third attempt haha

unknown.png
#

third time's the charm 😉

unique wren
#

Any idea what the race is or bug?

unique wren
#

No worries. might debug one day, just didn't want to redo what you've done. 🙂 Thanks

mortal night
#

Are there any good writeups on deploying Bandersnatch on a NAS (QNAP or Synology)? Google has not been fruitful.

#

Most NAS wont allow access to CRON, so running bandersnatch mirror on a schedule could be problematic. I think the solution would be to add cron and ntp to the container and just do everything inside.

turbid crater
turbid crater
ember nimbus
#

Hi guys,
I am running bandersnatch 4.4.0 and I would like to upgrade to the latest version. Is there any specific guide to upgrade without ruining my existing configuration?

low hornet
#

Good Evening, i have been trying to find a way to bypass the SSL requirement when running bandersnatch mirror, ihave added the trusted agents to my .pip but am still getting SSL errors. the place i work at uses a proxy that breaks SSL. has anyone else had this issue, or know of a solution. TIA.

vale swan
#

I'm sorry, but why is CI running for all of these commits 👀

#

hmm well it could be helpful in identifying which dependency bump causes CI to fail but my goodness will this take a long time

unique wren
unique wren
unique wren
#

You can always force a full run to make sure you have everything OR do a verify to try cleanup - We still don't have a good delete story with mirror

#

wonders if I can ask for more notifications from this channel. I don't swing by often

#

Found it, some reason I was only mentions ... don't remember doing it

ember nimbus
ember nimbus
#

Are you the author of bandersnatch?

unique wren
# 
python3 -m venv /path/to/venv
/path/to/venv/bin/pip install -U pip setuptools
/path/to/venv/bin/pip install bandersnatch

I'm the lead maintainer. I ported it from python2 to python3 then moved from requests to asyncio / aiohttp ...

#

I was not the original author

#

and I managed to get notifications working. Discord god.

ember nimbus
#

Now I can hassle you without the risk of you missing my messages. 😉

ember nimbus
unique wren
#

We offer a docker image

#

We don’t have detailed docs tho. But yes, you can bind mount in your existing package directory

#

Will take a PR with details docs for docker

ember nimbus
ember nimbus
unique wren
#

Will take a guthub pull request if you want to detail how to use our docker image better.

vale swan
#

@unique wren we'll just need the following

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
``` for pyproject.toml (you can edit setuptools req to include a lower bound if you'd like to)
#

we should be able to stop installing both wheel and setuptools globally too

unique wren
#

Cool. This is more for me to learn and update my old out of date python skills here. Thanks.

#

👴🏻

vale swan
#

I've been following the newer packaging standards as I packaged my python tools like diff-shades :p

unique wren
#

<old_man_rant>I do dislike all the different files </old_man_rant>

vale swan
#

we could probably move to flit and get rid of setup.cfg too and just use pyproject.toml (although I have no idea what setuptools-only features we're using)

unique wren
#

We need a “the current way to package a python project” QuickStart guide

unique wren
#

But down to refactor / add supper to build etc. If need be

vale swan
#

It's not the end of the world if we don't have a lower bound for setuptools since it'll be installed in an isolated virtual environment for just bandersnatch's build so the chance if it conflicting is very low

#

it's more nice for anyone manually building bandersnatch (or attempting static package metadata analysis)

unique wren
#

@vale swan Decided to just roll with build's hard coded setuptools requirement and if someone opens and issue will care then

#

Think the PR is ready now and added a twine check

#

Will then add this to all my other Open Source projects. Need to see if there is a pypi upload action these days I can move / contribute to rather than copy pasting this action everywhere.

sterile arch
#

might not actually save you much on your PR though

ember nimbus
#

hi,
Ok, so ive been toying with the docker version of bandersnatch and I seem to be encountering some strange errors:

When running it just with --version it works fine:

bandersnatch 5.1.1

but when trying with a conf file this happens:

root@ows-out-vm:~# docker run -v /data/pip:/data/pip pypa/bandersnatch bandersnatch -c /data/pip/bandersnatch.conf --debug mirror
Traceback (most recent call last):
  File "/usr/local/bin/bandersnatch", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/bandersnatch/main.py", line 215, in main
    logging.config.fileConfig(str(Path(config.get("mirror", "log-config"))))
  File "/usr/local/lib/python3.9/logging/config.py", line 71, in fileConfig
    formatters = _create_formatters(cp)
  File "/usr/local/lib/python3.9/logging/config.py", line 104, in _create_formatters
    flist = cp["formatters"]["keys"]
  File "/usr/local/lib/python3.9/configparser.py", line 963, in __getitem__
    raise KeyError(key)
KeyError: 'formatters'
root@ows-out-vm:~#
#

@unique wren do you have any idea what could be causing this?

#

i have python 3.9 on my machine as well:

root@ows-out-vm:~# which python3.9
/usr/bin/python3.9
unique wren
#

What’s in your config? Is it an empty file per chance or invalid format?

ember nimbus
unique wren
#

This comment doesn’t help me try and reproduce it and fix it does it …

#

A paste of it would be great

ember nimbus
# 
[mirror]
directory = /data/pip
json = false
release-files = true
cleanup = false
master = https://pypi.org
timeout = 600
global-timeout = 1800
workers = 10
hash-index = false
stop-on-error = false
storage-backend = filesystem
log-config = /data/pip/bandersnatch-log.conf
verifiers = 3
diff-file = /data/pip/pypi/mirrored-files
diff-append-epoch = false

[plugins]
enabled =
    blocklist_project

[blocklist]
packages =
    tf-nightly-gpu
    tf-nightly-gpu-2.0-preview
    tf-nightly
#

and this is the command I used to run it:
docker run -v /data/pip:/data/pip pypa/bandersnatch bandersnatch -c /data/pip/bandersnatch.conf --debug mirror

ember nimbus
#

@unique wren can you spot anything wrong with my config?

unique wren
#

Seems deleting log-config allows it to run

# 
[mirror]
directory = /tmp/pypi
json = false
release-files = true
cleanup = false
master = https://pypi.org
timeout = 600
global-timeout = 1800
workers = 10
hash-index = false
stop-on-error = false
storage-backend = filesystem
verifiers = 3
diff-file = /tmp/pypi/diff_file
diff-append-epoch = false

[plugins]
enabled =
    blocklist_project

[blocklist]
packages =
    tf-nightly-gpu
    tf-nightly-gpu-2.0-preview
    tf-nightly

I was able to run with /tmp/tb/bin/bandersnatch --debug -c /tmp/bandersnatch.conf mirror 2>&1 | tee /tmp/bandersnatch_run_1

unique wren
#

Definately a bug I need to fix, but with docker, it captures stderr, so use it to create a log for for now if you need it

Otherwise docker logs -f CONTAINER_NAME should do the trick

unique wren
ember nimbus
ember nimbus
unique wren
#

So you just disabled the file logging? I will get the fix out eventually. Just haven’t had time to write the test and bandersnatch is all pytest magic which I don’t write anywhere by bandersnatch

ember nimbus
#

Thank you for your help @unique wren

unique wren
#

Awesome. Great to hear. Logging the log to file in the container isn't that useful 🙂

#

needs to get an s3 docker image built so I can move PyPI to using it one day

vale swan
unique wren
#

Yeah noticed, still got an image

#

So didn't care

vale swan
#

I can PR a change to fix it (just to need to set some stepIDs and repoint the echo command to the relevant step) or we could just remove it since the build step already echos it (although it's buried under 600+ lines of output :p) https://github.com/pypa/bandersnatch/runs/5280366774?check_suite_focus=true#step:7:604

GitHub
Add S3 Docker Image support + uploading (#1092) · pypa/bandersnatch...

A PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/ - Add S3 Docker Image support + uploading (#1092) · pypa/bandersnatch@4886ca9

unique wren
#

Either or works for me

vale swan
#

I'm just going to rm * -r them then

unique wren
#

If you're bored too - I want to remove hard setting the python version from the containers and just use latest and then only hard set if we stop working on released version of python ... Hate that it's yet another manual thing I have to update.

#

If ya don't have time all god

#

*good - I'll get to it one day

vale swan
#

Ideally there would be a way to parametrize steps within a single job but I don't see such a feature 😦

unique wren
#

Going to have one last attempt at updating pypi.org's bandersnatch install this pycon - The hold up has been getting a test instance as I don't want to not test and update the prod one I have access to

vale swan
#

Matrix only works across many jobs which I would rather avoid given PyPA's GHA limits

unique wren
#

We don't build enough to care imo - the 10 mins is fine

vale swan
#

Is PyPI's bandersnatch still on 3.x or 4.x ?

unique wren
#

Good question - pretty sure 4.x.x

#

But we were thinking of moving it to be S3 backed and dockerize it all

#

Save us AWS credits

#

Will try contact ee again

#

Will also look later to see if I still have access to the prod install

#

Trying to workout my taxes for now - Nightmare

#

Hate tax in USA. So much struggle.

vale swan
#

I've heard! Aren't the personal tax filing software companies literally lobbying congress to keep it complicated?

#

Anyway I'm just trying to figure out how docker image labelling works since I have no idea.

unique wren
#

O I can login, Someone updated config to use a blocklist in Jan

# 
[ec2-user@XXX bandersnatch]$ ./bin/bandersnatch --version
bandersnatch 4.4.0
#

Still posix filesystem

/dev/xvdb        12T  7.6T  3.8T  68% /data
unique wren
#

It's also IPv4 only 😦

#

Much sad

#

Also fixed latest for bandersnatch docs to come from 'main' branch - No idea if the webhook works anymore - I had to manually build

#

Poor unloved project.

unique wren
#

@stuck aurora - Any interest in doing this anymore?

ember nimbus
#

😦

#

so how to people mirror the pip repo otherwise

#

for apt i use apt-mirror. However, for pip is seems the only option is to use bandersnatch

#

😕

unique wren
#

It’s the main option if you want a 1:1 mirror … it would be nice if I had another maintainer who actually uses it. I haven’t actually used bandersnatch in years 😦

drowsy topaz
#

I wrote my own mirror for a previous employer, that I believe they still use. (before bandersnatch existed)

ember nimbus
#

i mean, i code a fair amount - but definitely not to your level

ember nimbus
unique wren
#

I’m no coding guru.

#

I’m a network engineer who likes to code

unique wren
#

They’ve contributed a fair amount

unique wren
ember nimbus
drowsy topaz
ember nimbus
#

At least an ex network engineer

unique wren
#

Or recursively pull all the html each run to sync?

drowsy topaz
#

Scrape the source packages from HTML index listings

unique wren
#

That’s a lot of html parsing

ember nimbus
drowsy topaz
#

Nothing insane, it's only mirroring packages requested by engineers, and their dependencies

#

Using setuptools to do the scraping.

unique wren
#

Well that cuts it down. I did ask that first

#

Your second response sounded like you did every package on PyPI

drowsy topaz
#

Sorry about that 🙂

unique wren
#

Was only 100s of gb when I first used bandersnatch

#

Then data science exploded

chrome ore
#

Latest Updates Feed¶
Available at https://pypi.org/rss/updates.xml, this feed provides the latest newly created releases for individual projects on PyPI, including the project name and description, release version, and a link to the release page. Only the latest 100,How do I get full updates?

unique wren
#

Legacy XMLRPC API

torpid pivot
#

Hey guys! We're looking into speeding up package installation for our users and also being nicer to PyPI (and our internet connection 😅). A (partial) PyPI mirror seems like a good idea and bandersnatch appears to be the way to go.

From what I've found it should be possible to configure bandersnatch with an allowlist of our most installed packages to mirror and set PIP_INDEX_URL for our users to point to our local mirror. I'm wondering though what would happen if a package isn't available from our mirror. Can we make pip fallback to official PyPI in those cases?

unique wren
#

Sounds like devpi might be better for you. It’s a look through kinda cache I believe.

torpid pivot
#

Mh. I saw devpi, but this scared me off: https://pypi.org/project/devpi/#history

Looking at it again it seems like the actual projects do receive regular updates 🙂

PyPI
devpi

(deprecated, install devpi-server, devpi-client, devpi-web instead)

snow sparrow
#

devpi is a metapackage that simply depends on a bunch of child packages that provide the actual functionalities; the metapackage naturally don’t need to be updated very often

torpid pivot
#

Yep. It's a shame that it shows up in search results way in front of the actual packages :/

unique wren
#

Merged

narrow grailBOT
unique wren
#

Cool - It worked.

#

Don't get issue updates tho with that webhook ... Must be something else that does that for black

vale swan
#

Did you only setup the webhook for push events?

unique wren
#

Is that on the GitHub side?

vale swan
#

Yup

unique wren
#

"Send me everything" now

#

lol

vale swan
#

I'm not sure how black does it, but I'd assume black has most events enabled, and there's some additional filtering provided by pydis infra

unique wren
#

If this is to noisy we can tune it down

vale swan
#

They have access to a lot of cloud resources :p

narrow grailBOT
vale swan
#

Oh yea, the actions webhooks kinda suck, IIRC Python discord uses their own action (not that black uses it anyway)

#

(just throwing what I do know out there, we don't have to action any of it)

unique wren
#

Yeah, bummer GitHub don't have more filtering options

narrow grailBOT
#
cooperlees
[pypa/bandersnatch] New branch created: latest\_python
unique wren
#

O yeah, much spam.

narrow grailBOT
vale swan
#

I was about to mute this channel temporarily

narrow grailBOT
vale swan
#

Actually, might do that still

narrow grailBOT
vale swan
#

my gosh

unique wren
#

Ya - I'll disable for now. And see what options we have.

narrow grailBOT
vale swan
#

github surprisingly provides a solid amount of options

unknown.png
unique wren
#

Yeah just found that and tuning

narrow grailBOT
unique wren
#

Ok - ticked minimal that made sense

narrow grailBOT
unique wren
#

Yeah no chance of 3.11 working yet without building lxml for 3.11 ... Not down to do that.

#

So giveup there. Wonder if I can get a non C dep since we don't do a huge amount of xml processing

#

Wish I could fix this codecov how we get a false positive every run untill all CI differs. We get more coverage in 1 cpython version vs. others it seems ...

narrow grailBOT
unique wren
#

Bummer we don't get a summary of the comment there

narrow grailBOT
#
pyup-bot
[pypa/bandersnatch] Pull request opened: #1119 Scheduled monthly dependency update for May

Update filelock from 3.4.2 to 3.6.0.

Changelog

3.6.0

-------------------
- Fix pylint warning &quot;Abstract class :class:`WindowsFileLock &lt;filelock.WindowsFileLock&gt;` with abstract methods instantiated&quot;
:pr:`135` - by :user:`vonschultz`
- Fix pylint warning &quot;Abstract class :class:`UnixFileLock &lt;filelock.UnixFileLock&gt;` with abstract methods instantiated&quot;
:pr:`135` - by :user:`vonsch...
unique wren
#

@vale swan Did you ever get a if condition or something to not run docker uploads on forked repos?

#
  • On a different repo I could go copy
unique wren
vale swan
unique wren
#

No worries

#

Might look into it quickly in a bit

#

If easy will get a PR up

vale swan
#

if github.repository == 'pypa/bandersnatch' should be enough?

unique wren
#

Probably

narrow grailBOT
unique wren
#

Cool ^^ That's useful.

narrow grailBOT
unique wren
#

Ok - might look at a few bug reports tomorrow + try see how to get docs to show contents listed out maybe. Don't know if I like the CLI usage as the main thing that comes up

#

Actually it's fine, just wonder if we can get more down the left hand side column rather than just "Contents"

narrow grailBOT
unique wren
#

*629

narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
quaint schooner
#

Aren't you at the airport @unique wren?!

unique wren
#

Ya

quaint schooner
#

:slowclap:

#

:osssalute:

unique wren
#

Have just been waiting for CI to pass with PyPI issues earlier

#

Now it's just GUI clicks

#

Lame - upload failed.

quaint schooner
#

Yea, I think the new security policy that got deployed is borked somehow.

unique wren
#

Manual worked

#

Na, I screwed up the action. My bad here

quaint schooner
#

XD

narrow grailBOT
narrow grailBOT
narrow grailBOT
#
dependabot[bot]
[pypa/bandersnatch] Pull request opened: #1125 Bump docker/build\-push\-action from 2 to 3

Bumps docker/build-push-action from 2 to 3.

Release notes
Sourced from docker/build-push-action's releases.

v3.0.0

Node 16 as default runtime by @​crazy-max (#564)

This requires a minimum Actions Runner version of v2.285.0, which is by default available in GHES 3.4 or later.

Standalone mode support by @​crazy-max (#601 #609)
chore: update dev dependencies and workflow by @​crazy-max (#571)
Bump @​actions/exec from 1.1.0 to 1.1.1 (#573)
Bump...

#
dependabot[bot]
[pypa/bandersnatch] Pull request opened: #1126 Bump docker/setup\-qemu\-action from 1 to 2

Bumps docker/setup-qemu-action from 1 to 2.

Release notes
Sourced from docker/setup-qemu-action's releases.

v2.0.0

Node 16 as default runtime by @​crazy-max (#48)

This requires a minimum Actions Runner version of v2.285.0, which is by default available in GHES 3.4 or later.

chore: update dev dependencies and workflow by @​crazy-max (#43 #47)
Bump @​actions/core from 1.3.0 to 1.6.0 (#37 #39 #41)
Bump @​actions/exec from 1.0.4 to 1.1.1 (#38 #...

#
dependabot[bot]
[pypa/bandersnatch] Pull request opened: #1127 Bump docker/login\-action from 1\.14\.1 to 2\.0\.0

Bumps docker/login-action from 1.14.1 to 2.0.0.

Release notes
Sourced from docker/login-action's releases.

v2.0.0

Node 16 as default runtime by @​crazy-max (#161)

This requires a minimum Actions Runner version of v2.285.0, which is by default available in GHES 3.4 or later.

chore: update dev dependencies and workflow by @​crazy-max (#170)
Bump @​actions/exec from 1.1.0 to 1.1.1 (#167)
Bump @​actions/io from 1.1.1 to 1.1.2 (#168)
Bump minimist fr...

#
dependabot[bot]
[pypa/bandersnatch] Pull request opened: #1128 Bump docker/setup\-buildx\-action from 1 to 2

Bumps docker/setup-buildx-action from 1 to 2.

Release notes
Sourced from docker/setup-buildx-action's releases.

v2.0.0

Node 16 as default runtime by @​crazy-max (#131)

This requires a minimum Actions Runner version of v2.285.0, which is by default available in GHES 3.4 or later.

Full Changelog: https://github.com/docker/setup-buildx-action/compare/v1.7.0...v2.0.0
v1.7.0

Standalone mode by @​crazy-max in (#119)
Update dev dependencies an...

narrow grailBOT
unique wren
#

Yuck. Node.

#

But thanks for beating me Richard 😄

narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
#
dependabot[bot]
[pypa/bandersnatch] Pull request opened: #1132 Bump actions/setup\-python from 3 to 4

Bumps actions/setup-python from 3 to 4.

Release notes
Sourced from actions/setup-python's releases.

v4.0.0
What's Changed

Support for python-version-file input: #336

Example of usage:

  • uses: actions/setup-python@v4
    with:
    python-version-file: '.python-version' # Read python version from a file
  • run: python my_script.py

There is no default python version for this setup-python major version, the action requires to specify either python-vers...

narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
unique wren
# narrow grail

I guess I should filter that - It's useless not showing my comment

narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
#
galesauryan
[pypa/bandersnatch] Issue opened: #1136 TimeoutError for bandersnatch mirror

hi,
the bandersnatch mirror can't work smoothly for almost 2 months. The error shows time out. How to fix it?

[root@Server-sc0r86 Python-3.8.0]# bandersnatch/bin/bandersnatch --version
bandersnatch 5.1.1
[root@Server-sc0r86 Python-3.8.0]# python3 --version
Python 3.8.0
[root@Server-sc0r86 pypi]# ll
-rw-r--r-- 1 10032 10001 349919 Jun 15 14:12 pypi202206141655169932.log
-rw------- 1 10032 10001 8 Apr 21 16:30 status
-rw------- 1 10032 10001 248...

narrow grailBOT
narrow grailBOT
#
whiskerch
[pypa/bandersnatch] Issue opened: #1137 FileNotFoundErrors when using S3 storage

Hi,

I am trying to set up a bandersnatch instance to mirror pypi using S3 as the backend storage.

I have configured the S3 storage as per the guides and am using the pypa/bandersnatch:s3-3.10 to run bandersnatch.

When the docker container is running, it seems to successfully start and populate S3 with a load of folders, however as soon as it tries to sync packages, I get loads of 'FileNotFoundError' errors.

For each package error I can see the json file being created so it s...

narrow grailBOT
#
cooperlees
[pypa/bandersnatch] Issue opened: #1138 Implement PEP 691 JSON Simple Index Support

Add logic for bandersnatch to save both the HTML and JSON simple index files. This will allow people to serve both the HTML and JSON in their mirrors.

  • Happy to introduce a config to select the formats to save
    • Eventually most tools should prefer the JSON we would expect

We should also update docs + give an example way to serve based on request headers (conneg) as outlined in PEP691.

narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
#
dependabot[bot]
[pypa/bandersnatch] Pull request opened: #1143 Bump lxml from 4\.8\.0 to 4\.9\.1

Bumps lxml from 4.8.0 to 4.9.1.

Changelog
Sourced from lxml's changelog.

4.9.1 (2022-07-01)
Bugs fixed

A crash was resolved when using iterwalk() (or canonicalize())
after parsing certain incorrect input. Note that iterwalk() can crash
on valid input parsed with the same parser after failing to parse the
incorrect input.

4.9.0 (2022-06-01)
Bugs fixed

GH#341: The mixin inheritance order in lxml.html was corrected.
Patch by xmo-odoo.

Other changes

Built ...

narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT
unique wren
#

@stuck aurora or anyone - can nginx or Apache respect content type requests - reason im asking is im planning bandersnatch pep691 support

#

Want to workout if I should just start mirroring both for any reason or just one or the other

stuck aurora
#

Apache had conneg for sure

#

lemme work up an example confit

#

Config

unique wren
#

So bandersnatch saving a index.html and index.json would make sense? If so ill support it

#

We can default to html for now until more tooling only use json

#

No need for the config soon but id look yo make a docker container example like I do with nginx today

#

We could make a banderpache equivalent

stuck aurora
#

@unique wren If you have a Dockerfile like this

FROM httpd

RUN echo '\n\
    LoadModule negotiation_module modules/mod_negotiation.so\n\
    \n\
    <Directory "/usr/local/apache2/htdocs">\n\
    AllowOverride All\n\
    </Directory>' >> /usr/local/apache2/conf/httpd.conf

You can drop a .htaccess in the bandersnatch mirror directory like this:

Options -Indexes +Multiviews

DirectoryIndex index

AddType application/vnd.pypi.simple.v1+json v1_json
AddType application/vnd.pypi.simple.v1+html v1_html

and it should work assuming you write 3 files, index.html, index.v1_html, and index.v1_json

#

you could remove the need for the .htaccess and move all of those things into the <Directory> block, which is actually more performant, but I did it this way to make it clear which things are setting up options to allow it to work, and which things are just enabling features in apache

#

@unique wren with the above, Apache will listen to the client preferences (the optional ;q= parameter), and if the client has multiple equal preferences, or doesn't state a preference, it will serve whichever file is smallest.

There's probably a way to set a default Accept header using mod_rewrite, but I'm not sure how off hand

#

You can do something similiar in nginx, but it won't respect the client preferences for (the optional ;q= parameter). That's legally fine to do with conneg, if the client accepts it, it's allowed to send it down, it's just nicer to send the one it prefers

#

in practice, I doubt anyone is going to prefer html over json, so we can probably just assume json is preferred

unique wren
#

Nice. Ill try get some of this into the issue and get to work on it. First step ill just add support to get 1 format or both and save the files.

stuck aurora
#

you might want an option to save them to the same, or seperate directories

unique wren
#

Two index files one in a pkgname/html and pkgname/json?

#

Did you implement the global index as JSON too btw? I haven’t checked …

stuck aurora
#

yes

#

to the second

unique wren
#

Ill have to generate two of them I guess too now

#

I == bandersnatch

stuck aurora
#

what I mean you might want different root directories for json and html, like na option to switch from:

/var/www/bandersnatch/*.{html, v1_html, v1_json}

To

/var/www/bandersnatch-json/*.json
+
/var/www/bandersnatch-html/*html

Because some people might want to setup seperate urls completely for json and html.

The apache example I wrote expects the first option (well it will work in the second too, but it'll only server json or html, not both)

But some people might not want to use conneg, and just configure different urls

stuck aurora
#

@unique wren https://gist.github.com/dstufft/861538ababfafefaba7a1b34fca5a0b7 this should work for banderx.

This technically isn't quite content negotiation. It doesn't actually read the Accept header and understand it, it just does some dumb regexing on it.

How this works:

  1. The map $http_accept $mirror_suffix block assigns a file suffix to the $mirror_suffix var based on a regex match, it bails out at the first match, so if the client mentions json, it prefers that, if the client mentions the custom html, it prefers that, if it mentions text/html it prefers that.
  2. If none of the blocks match, the default suffix of .html is used (this counts for not having an Accept header, or for having an Accept header that doesn't mention any of our content types).
  3. Adds a location block for /simple/ to scope conneg to the simple api.
  4. Inside the simple block, it sets the default index to index$mirror_suffix, so if it's looking for a application/vnd.pypi.simple.v1+json, the $mirror_suffix will be .v1_json, and the index directive will be index index.v1_json.
  5. Overrides our default mimetypes with a custom types that just lists our 3 content types (you could remove this and mutate /etc/nginx/mime.types to add it, but this felt cleaner).
  6. The try_files isn't strictly needed, so you could omit it. It basically exists so that files named something that than index.$EXT will work too... but I think all of bandersnatch's simple output is just index.html (and presumably in the future index.v1_json). In other words, this means you could drop a foo.{html, v1_json, v1_html} inside of /simple/ and going to /simple/foo will do the same logic to look up the right file.
#

I think even though the nginx thing isn't technically conneg, it's close enough that nobody will ever notice the difference unless they're explicitly trying to

#

If 100% of your files under /simple/ will be named index.{html, v1_json, v1_html}, I would probably just delete the try_files thing, it will be faster without it

#

@unique wren I forget if I mentioned it, but I have an idea to write a PEP to allow an optional parameter to /simple/, so that you can do /simple/?since=$CHANGELOG, and it filters the list of projects returned to just ones that have changed since that changelog ID.

Optional because obviously a mirror like bandersnatch can't really support that itself, but it degrades in an "OK" way, if someone pointed a bandersnatch at a bandersnatch, they would just do a full sync every time

#

hopefully that's all useful

unique wren
#

This is all useful. I'll get started on pulling a json or html or both for each package first + configuration

#

Unittest + CI test that

#

Then look at adding serving examples to docs + into the repo

#

So we can hush people who are like this PEP killed static file mirrors ... which it didn't ...

#

Then I need to hassle Ee to finally let me update PyPI's bandersnatch 😛

#

Unless we deem we can decom it - We should use the s3 storing method ...

stuck aurora
#

@unique wren Honestly, it might make sense to just have PyPI's bandersnatch run with release-files = false.

All our files are stored in Google's version of S3, and somehow I doubt that us storing the files ourselves is really a useful use of resources, but the /simple/ pages are still served by Warehouse itself

unique wren
#

Fair enough. Personally I like the warm feeling of all our artifacts living on a second cloud

#

But that could be done in a different way than with bandersnatch I guess too

stuck aurora
#

if we wanted to have dual clouds, we could just have Warehouse write to multiple clouds

#

anyways, I don't feel super strongly, and that's not really related here 🙂 Just a random thing that popped into my head, that our real value of the bandersnatch mirror is as a fallback for /simple/ when Warehouse is down for one reason or another

unique wren
#

And you're totally right, warehouse double writing is the best place.

stuck aurora
#

Awesome

#

We've had PEP691 enabled on PyPI since like a day or two after it got accepted FWIW

#

using effectively the same config as I wrote for nginx (except we fully support conneg, unlike faking it with regex), and I don't think we've had a single complaint or problem stemming from it

#

so you can probably enable it by default?

#

up to you though

unique wren
stuck aurora
#

grab/generate both, and have them both enabled by default

unique wren
#

Yeah, my only worry there is people getting annoyed having to change their “front end” serving proxy/web server config

stuck aurora
#

with the way I suggested, if they don't change it, nothing will change for them

#

they'll just have extra index files

#

that they won't serve

unique wren
#

O really - I missed that. If so, then yeah. Im down to yeet it

stuck aurora
#

yea, the way it works is dynamically setting the index directive in nginx so that it uses different extensions, but text/html is still index.html

#

so the old hard coded index index.html that exists in nginx by default just won't do any conneg

unique wren
#

Might also get people complaining about the double the simple files + more dirs so I might make it configurable …

#

A lot of users are on slow ass links

#

Thus why they mirror

stuck aurora
#

I think you generate the simple html instead of copying it from pypi

#

so you'd just generate html + json

#

stil be more files of course

unique wren
#

O yeah, you’re right we do generate …

#

Will default to both and have option to go back to one format

stuck aurora
#

I happened to just dig into the other day when we were trying to figure out if it's safe to remove some keys from the versioned URLs on PyPI's old json api

#

so it was fresh in my mind

#

in theory you could stop hitting the legacy json api all together

#

and just use the v1 json url on /simple/, but that might be hard since you're also mirrirong /pypi/$project/json

narrow grailBOT
narrow grailBOT
narrow grailBOT
narrow grailBOT