tools-and-devops

[rapid sparrow]

At the same time u leave default git submodule referencing explicit commit, that will help you freezing final result with version locks in order to release it

alternative path would be going away from git sub modules

and actually just using package managers of specific languages u use.

python has dev edit mode for its pip package installations / plus easy to update for CI too

[shut wagon]

theyre all the same language

[rapid sparrow]

public or private repos?

[shut wagon]

currently private but i will open all of them to public

so anyone can work on any component if they want

[rapid sparrow]

okay, than i suggest you just to learn publishing shared types repo to pypi
and installing it as a python pip package into other repos.
For development u could be comfortable using it installed in dev mode that links some folder with having it installed directly

that is simpler than dealing with git submodules

[shut wagon]

sure but I am feeling like a solution that depends on the language supporting it, is not the best software solution in principle

if I go to a different language that has bad monorepo support

what would i do ther?

Im wanting to internalize one solution I can use everywhere, in every language and every software domain

[rapid sparrow]

git sub module is rather crude tool for development. u should enjoy ability to use specific language ecosystem to your advantage if u can 😅
that will also make the repository FAR more friendly for other python devs if they will be joining you.
Because pip usage is a standard
git submodule usage is more difficult thingy people are far less familiar

[rapid sparrow]

it is not really a problem for this specific project 😅

even if u will have additional new languages in shared repo.
They could be having separate CI pipeline to publish itself into their own package manager
Or being just in another repo

[rapid sparrow]

Actually i know a thing that will make you satisfied.

pip has ability to work without pypi

by just referencing another git repository 😅

then u can use inbuilt language package management and being less depended on pypi if u wish

[rapid sparrow]

https://stackoverflow.com/questions/20101834/pip-install-from-git-repo-branch

pip install git+https://github.com/tangentlabs/django-oscar-paypal.git@issue/34/oscar-0.6

here u go.

anyway, those are means INBUILT into language pretty much at this point. U should utilize INBUILT standard libraries of a language to full capacity if u can
They are the most stable stuff, usually good to rely upon

depending on too many third party libs, that is a bad software architecture design usually on another hand.

[rapid sparrow]

i suspect your thinking may be came from using language that has this its package management really under developed.
u should not do this in python because python has it very advanced and comfortable (minus some scratches)
Because if u will not be using python standard, u will pay the price of making repository less friendly for other python developers.

best to use language ecosystem package management standards by default for this reason.

Some language do not have them though or having them in an awful state, but u don't use them anyway

otherwise u are kind of overengineering

u don't need it

[shut wagon]

i plan on making the repositories also usable within the language specific package managers

so it would be both decentralized in terms of git repo, and py packages

i acknowledge that this is engineering to the max

i think it is usually a safer bet to overengineer than under engineer

especially if you can internalize the over engineering, make it muscle memory

that way you are able to overengineer at the same speed as you would be normal-engineering, once you're good enough

i think i will feel more confident when I get my first job, having practiced overengineering

thanks for showing me that though

[rapid sparrow]

it is not good type of overnegineering.
Good type of overnegineering is the one that leads to flexibility of architecture and testability and documentation
Bad type of overnegineering is pretty much everything else usually

in context of package management => it would be good to build CI that automates making releases and unit testing your libraries

So your current change is going to affect

• making more complicated to configure CI unit testing
• making more complicated to configure CI for deployment
• decreasing self documentation for other developers onboarding

i do enjoy concept of language agnostic package management, and thought even making tool for this, but i would not have do that for language that has already fine package management as it is

[unkempt zodiac]

You guys know any software that runs AWS Lambda functions locally, not for testing, but for production?

[astral apex]

Why would you want that?

[rapid sparrow]

it is called using Message Queues. Celery is fine example of such really great made library.

[glacial crystal]

can anyone help with the telegram bot when the telegram bot does not save messages sent to the telegram channel in the database and the console does not get errors?

[vague silo]

is lambda comparable to a message queue? i thought it was more similar to REST APIs

[rapid sparrow]

It is versatile tool that can work as both rest API and message queue like.
With retry options for both choices

[rapid sparrow]

And even as cron job it can work with AWS event bridge

[vague silo]

TIL, thank you

[thorny shell]

what part of lambda is like a queue? I haven't used it in ages, and even then wasn't good at it. But I dimly recall that, if you want a queue with your lambda, you use SQS

does it automatically queue incoming requests for you?

[inner matrix]

How screwed am I?

$ git fsck
error: object file .git/objects/19/c0a39d721aa0340713bc68592bc481906f401d is empty
error: unable to mmap .git/objects/19/c0a39d721aa0340713bc68592bc481906f401d: No such file or directory
error: 19c0a39d721aa0340713bc68592bc481906f401d: object corrupt or missing: .git/objects/19/c0a39d721aa0340713bc68592bc481906f401d
error: object file .git/objects/5a/7af54520947e1adb2502ca8371bb5d8e6b7659 is empty
error: unable to mmap .git/objects/5a/7af54520947e1adb2502ca8371bb5d8e6b7659: No such file or directory
error: 5a7af54520947e1adb2502ca8371bb5d8e6b7659: object corrupt or missing: .git/objects/5a/7af54520947e1adb2502ca8371bb5d8e6b7659
error: object file .git/objects/62/1163f3e3c6e6c141a9507e5fd8a1e99cf84443 is empty
error: object file .git/objects/ab/9bf8814e5cf9f616d6b2c9021411dd96d1d9ad is empty
error: unable to mmap .git/objects/ab/9bf8814e5cf9f616d6b2c9021411dd96d1d9ad: No such file or directory
error: ab9bf8814e5cf9f616d6b2c9021411dd96d1d9ad: object corrupt or missing: .git/objects/ab/9bf8814e5cf9f616d6b2c9021411dd96d1d9ad
error: refs/heads/main: invalid sha1 pointer 6b66d6b2fb30d84eef33178f1699506c053e105e
error: HEAD: invalid sha1 pointer 6b66d6b2fb30d84eef33178f1699506c053e105e
error: object file .git/objects/ab/9bf8814e5cf9f616d6b2c9021411dd96d1d9ad is empty
error: object file .git/objects/ab/9bf8814e5cf9f616d6b2c9021411dd96d1d9ad is empty
fatal: loose object ab9bf8814e5cf9f616d6b2c9021411dd96d1d9ad (stored in .git/objects/ab/9bf8814e5cf9f616d6b2c9021411dd96d1d9ad) is corrupt

[thorny shell]

dunno -- can you checkout, commit, etc?

it does look bad, I'll admit

[tawdry needle]

HEAD being invalid seems bad

[thorny shell]

nice thing about git is that you generally have another clone somewhere

generally

[tawdry needle]

if you can check out another commit your repo might be fine

[inner matrix]

$ git status
fatal: bad object HEAD

$ git commit -m "pls help"
fatal: could not parse HEAD

I asked Chatgpt, it advised I needed to consult a professional, and here I thought I *was *the professional.

[tawdry needle]

if those bad objects are only referenced from commits you aren't using, the rest of your repo will probably be ok

[tawdry needle]

what about checking out a different commit?

also: how did you even get in this situation? it might help elucidate a way out

[thorny shell]

here I thought I was the professional
🤣

[thorny shell]

it probably won't, but it might be a lesson in "don't do that again" 😕

[tawdry needle]

i find that root cause analysis is usually worthwhile for git issues

but that's just my own experience

[inner matrix]

So this is an empty github repository with minor boiler plate type changes on my local. I'll probably just clone again and copy my stuf fover.

[thorny shell]

on the one hand: yes, it'd be ideal to know how you got into this mess.
On the other: I'm a pessimist, and assume you'll never figure it out 😦

[inner matrix]

Honestly I think this happened because my computer is set to go to sleep after 2 minutes of inactivity. Part of an "energy efficiency" easy button given by microsoft. I clicked it 3 days ago. The timeline is

1hr ago:

• making changes, happy coding
• do some local commits
• everything is fine
• "oh I don't have coffee"
• walks away for 3-4 minutes
• computer asleep, so I wake it up
• $ git status
• "oh no"
• severe googling
• chatgpt: "you're not a professional"
• cry

[tawdry needle]

huh

that honestly sounds like filesystem corruption

[inner matrix]

the repository is also inside WSL, so maybe WSL didn't handle the host suddenly going to sleep very well.

[tawdry needle]

almost sounds like windows went to sleep and didn't flush a filesystem buffer

yeah

that or git assumed some linux-ey filesystem behavior that isn't actually true on WSL, and resulted in the git objects not actually being written to the filesystem

or something weird between WSL and the actual filesystem

[inner matrix]

🤷 well time to copy/paste, figured the channel would find some humor in the situation 😁

[tawdry needle]

yeah, sorry that happened. honestly lucky it wasn't something more important. next time close WSL if it's going to go to sleep? or run sync to try to ensure data is written safely

[inner matrix]

what's sync a linux binary or git command?

/usr/bin/sync```

Nevermind, `man sync` is helpful.

[thorny shell]

I cannot beleive that we need to run sync in this, the twenty-first century

I would be geniunely surprised and disappointed if WSL was as flaky as salty suggested above

[inner matrix]

I would be genuinely surprised and disappointed if WSL was as flaky as salty suggested above

Speculation from a junior dev with no evidence or indicator on the health of the physical disk with the only clue given was "he got up to go get coffee".

[thorny shell]

never used WSL so I can't comment.

[inner matrix]

Maybe a Sun Particle paid me a visit? Similar to this story: https://www.thegamer.com/how-ionizing-particle-outer-space-helped-super-mario-64-speedrunner-save-time/

🤣

[thorny shell]

possible I suppose

[hardy remnant]

from python3.11 homebrew stopped shipping python compiled with gnureadline due to licensing issues and libedit is a PITA, is there a way to use gnureadline anyway?

I remember when this news was out I searched for how to do it, but it involved downloading a separate copy of gnureadline, I should've done it then, but didn't and now I don't know what the process was 😓

[brazen forge]

have you tried using rlwrap?

[hardy remnant]

ah missed that discussion

nope, doesn't seem to work :/

it doesn't respect my inputrc

set editing-mode vi
set keymap vi-command

[crimson spruce]

I see the homebrew-installed Python like a system Python — it's used by homebrew packages and shouldn't be messed with.

I recommend installing pyenv via Homebrew and then installing Python via pyenv — that includes GNU readline I believe.

[hardy remnant]

oh?

[brazen forge]

agreed

[hardy remnant]

interesting, I thought since homebrew already nicely versioned its python installations like python3.11 and others, I didn't have to use pyenv

[brazen forge]

pyenv allows you to customize the Python installation, unlike Homebrew, if you want

[hardy remnant]

like?

the install directory and things like that?

[brazen forge]

compile-time and linking-time options
enabling optimizations, for instance

[hardy remnant]

should I brew install pyenv? Or does that have some pitfalls?

[brazen forge]

no, that's okay

pyenv itself isn't used by other formulae

[hardy remnant]

alr!

thanks

Thank you so much, I was loosing my mind using libedit, finally back to sanity

[unique stratus]

can someone help me

please

[flat path]

What's your question?

[unique stratus]

I don't think they have a channel for this topic so thats why im asking here

but my matplotlib graph isn't popping up

[flat path]

You'd probably have a better shot at getting help in #data-science-and-ml

[indigo zenith]

#❓｜how-to-get-help

[rapid sparrow]

reading manifest template 'MANIFEST.in' appears during python3 setup.py bdist_wheel --build-number 123123
I have MANIFEST.in with

global-exclude *_test.py
recursive-exclude *_test.py

but files like
library_name/some_file_test.py do not get excluded.

what am i missing to exclude test files https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html ? 😅

for some reason MANIFEST.in does not getting applied 🤔

[tawdry needle]

i've had so many struggles with those

there have been times when i couldn't get it to work for hours, then the next day it just worked

[rapid sparrow]

def gen_data_files(*dirs: str) -> list:
    results = []

    for src_dir in dirs:
        for root,dirs,files in os.walk(src_dir): # type: ignore
        
            addable_files = []
            for file in files:
                filepath = str(pathlib.Path(root) / file)

                if "__pycache__" in filepath:
                    continue

                if "_test.py" in filepath:
                    continue

                addable_files.append(filepath)
            results.append((root, addable_files))
    return results

setup(
    ...
    data_files=gen_data_files("lib_name"),
)

Solved lazily

if MANIFEST.in is not going to like me, then i am not going to like it too 😅

[tawdry needle]

ironically that's literally what the manifest file is supposed to take care of for you

[thorny shell]

see above under #1185659467988209815 message

[willow pagoda]

@rapid sparrow would this happen to fix your issue with MANIFEST.in?

also i would have thought it would raise an error or something when the syntax is incorrect, recursive-exclude is meant to take a directory and glob pattern

[rapid sparrow]

i have deleting built/dict always during builds as it is. fan of https://taskfile.dev

version: "3"
tasks:
  build:
    desc: To try building lib
    cmds:
      - rm -r build dist flowey.egg-info | true
      - python3 setup.py bdist_wheel --build-number 123123

[willow pagoda]

could it have something to do with invoking setup.py directly?

[rapid sparrow]

i think it is not worthy of time to investigate further. gen_data_files solution does work.

i have fun with trying to hack Django inner working as it is ^_^

[willow pagoda]

:shrug:

fair enoguh

[rapid sparrow]

setup.py file is pretty small

effect from having gen_data_files solution is not really impacting any complexity, so... good enough

[astral apex]

Have you seen https://packaging.python.org/en/latest/discussions/setup-py-deprecated/

This means that commands such as the following MUST NOT be run anymore:

• [...]
• python setup.py bdist_wheel

[rapid sparrow]

Not usable commands for custom pypi anyway.
Zip must be built, which bdist does well

[tawdry needle]

this has been a long time coming

this does however leave kind of a functionality gap. people used to use setup.py to implement custom project scripts

setuptools is basically removing that capability, with no replacement

https://packaging.python.org/en/latest/discussions/setup-py-deprecated/#what-about-custom-commands

Likewise, custom setup.py commands are deprecated. The recommendation is to migrate those custom commands to a task runner tool or any other similar tool. Some examples of such tools are: chuy, make, nox or tox, pydoit, pyinvoke, taskipy, and thx.

i don't blame them for wanting to reduce their "maintenance surface", but that's pretty rough

[astral apex]

That’s not setuptools’ job

[tawdry needle]

no it's not their job, but it's not a small use case either

to be clear: i'm not saying they're wrong for doing it

it's just that the projects most likely to still be depending on that behavior are also the projects least likely to have the resources to migrate to a new system

i'm not talking about internal projects at for-profit leech organizations. i'm talking about esoteric scientific libraries and various research projects over the years

[astral apex]

[…] but it's not a small use case either
I actually didn’t even know that was a thing until I read this page
I’ve never seen anyone use it
Where do you see it used so much?

[astral apex]

[tawdry needle]

i don't see it used much, but i've seen it for things like custom build processes

where i think setuptools has done a great job here is documenting the transition

they're doing it very gradually and clearly documenting what's changing, and how to transition to the new system

that's impressive and hard to do well

[lost dust]

I'm trying to set up proper linters and plugins for my company private repo (both with config files that will be run locally by employees and with github workflows).
I'm wondering does PyPi allow reuploading old package versions, or tinkering with them? Because if I target a specific release I don't ever want anything changed.
I was googling around and seems that PyPi doesn't allow it, just want to double check here.
Also, any additional ideas for security?

[astral apex]

It does not. It will reject it and say "file already uploaded".

[astral apex]

What security?

[lost dust]

Maybe I'm overthinking, but currently I don't want my plugins possibly getting malicious in the future updates if the owner ever gets compromised (very low chance but still).

[flat path]

Would strongly recommend using pip-compile to enforce checksums

There is a way to replace wheels for a release using build numbers

[astral apex]

https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-third-party-actions
See "Pin actions to a full length commit SHA"

And https://pip.pypa.io/en/stable/topics/secure-installs/

Pin to specific versions and check the hashes to make sure that no one has changed them out from under you

[astral apex]

Wait what

[lost dust]

https://stackoverflow.com/a/63944201

looks like that

[flat path]

Three ways:

• Normal version equivalence requirements don't distinguish post releases so 1.0.1 and 1.0.1.post3 would be both matched by ==1.0.1
• Wheels can come with a build number. It's not really part of the distribution version, however it is expressed in the filename.
• You can reupload once by modifying the distributions file extension (I don't remember for which distribution types you can do this)

[astral apex]

That... Probably shouldn't happen

just release a new version

[flat path]

fwiw, even if all of these avenues were disabled in a hyper secure Python ecosystem, simply uploading a new more specific wheel could still own you

you'd have to make cutting a release an atomic, one-time action to make it truly immutable

[astral apex]

I'm down

Let's do that

[alpine horizon]

why was the last job completely skipped, not even an error message?
https://github.com/shner-elmo/flashtext2/actions/runs/7262207751/job/19785089430

[vague silo]

it's not a tag but a regular commit, so this condition isn't true

  release:
    name: Release
    runs-on: ubuntu-latest
    if: "startsWith(github.ref, 'refs/tags/')"

[alpine horizon]

ohhh

I was wondering if it was that

[alpine horizon]

can I add the tag directly from the website?

and rerun just the last job

[vague silo]

hmmmmmmmm

I think you can, over the "releases" tab, hit "new release" and under "select tag", enter the desired name and it should create it

I haven't used that though

[alpine horizon]

I did that and it worked! thanks

[atomic granite]

Can I get a quick rundown on using virtual environments with VSCode?

[vague edge]

has anyone here tried jetbrains fleet?

[indigo zenith]

I googled it for you... There is also a more concise summary on StackExchange that should come right up if you look. https://code.visualstudio.com/docs/python/environments

[atomic granite]

When should I use requirements.txt for specifying dependencies and when should I use environment.yml?

[subtle quarry]

You use conda or pip

[atomic granite]

I know that. I'm using conda. I'm wondering when I'm supposed to use requirements.txt for listing dependencies and when I'm supposed to use environment.yml. Or if it even matters.

[gleaming gate]

the req.txt can be used by any python installation

unless you dont have any packages that are only avail at conda i would always take the req.txt instead of env.yaml

[tawdry needle]

the latter is for conda and the former is for pip

[hardy remnant]

Then how do you decide what goes where?
Say you're using some python package globally does that go in the pyenv installed python? And what if that globally installed package is also used by some other package, in that case it should probably go in the homebrew installed python.

And what do you let extrenal programs use, say your editor, homebrew installed python or your own pyenv python?

[indigo zenith]

Do whatever works for you. Just avoid as much as possible to be messing with your OS or homebrew packages... When in doubt, use a venv.

By default each venv inherits whatever packages your OS has, but then in a venv you can install or uninstall whatever version of whatever package

And you can have as many venvs as you need. They're disposable. You can freeze them to a requirements.txt and recreate it as needed

[hardy remnant]

Just avoid as much as possible to be messing with your OS or homebrew packages

What does messing mean?

[tawdry needle]

each project gets its own venv.

for "global" python apps that aren't available through a package manager like homebrew, i use pipx to create a separate venv for every tool.

the result is that there are no "global libraries" that you install by yourself -- either it's needed for my project, so it goes in the venv, or it's needed for an app, so it goes in the package manager or pipx.

[stark raft]

import gc, threading, time, atexit, traceback

class Connection:
    open: bool

    def __init__(self):
        self.open = True
        spot_abandoned_connections(self)

    def close(self):
        self.open = False

def spot_abandoned_connections(conn: Connection):
    stack = traceback.extract_stack()
    calling_frame = stack[-2]

    def detect_hanging_connection():
        if conn.open and get_instance_count(conn) == 1:
            atexit.unregister(detect_hanging_connection)

            file = f"{calling_frame.filename}:{calling_frame.lineno}"
            print(f"Warning: Connection left open in {file}")

    def thread_body():
        while conn.open:
            time.sleep(0.01)
            detect_hanging_connection()

    threading.Thread(target=thread_body, daemon=True).start()
    atexit.register(detect_hanging_connection)

def get_instance_count(obj) -> int:
    references = gc.get_referrers(type(obj))
    instances = [ref for ref in references if ref is obj]
    return len(instances)


conn = Connection()
conn.close()

Made some proof of concept code to detect when a connection is left open.

[hardy remnant]

oh that definitely sounds less head-ache-y

I just looked up what does homebrew do to can handle potential conflicts, and it also uses venv's lmao

Applications should be installed into a Python virtualenv environment rooted in libexec. This prevents the app’s Python modules from contaminating the system site-packages and vice versa.

[heavy knot]

hey im looking for somebody that can code me a linkvertise checker what i need it to do

login to acc
check balence
how many invalid/valid

[flat path]

!rule 9 - This sounds like a freelance gig offer, unfortunately we do not allow paid work of any kind here.

[rancid schooner]

Rules

9. Do not offer or ask for paid work of any kind.

[flat path]

We also aren't a code writing service. We can help you write your own code, however.

[atomic granite]

When you are doing a project, you have your virtual environment. You put the packages you need in environment.yml. You create the environment using that environment file. But let's say later on in the project, you find out that you need another package and/or one of the packages you thought you needed ended up not being needed. What do you do? Do you create a new environment? Do you install the new package in the current environment? Do you do something else?

[crimson spruce]

I don't use any Python packages globally.

[indigo zenith]

I don't use Conda but based on a skim of the docs it looks like you can either a) install whatever you want and then conda env export > environment.yml to update the file or b) update the file first and reapply it

[hushed orbit]

Does anyone have any experience installing a private organization python package with pipenv? I've tried just doing the jolly ol' pipenv install "git+https://${{GITHUB_PAT}}@github.com/org/pack#egg=pack" but that seems to fail with 403. So my assumption is that I need to add my PAT to the organization, but I can't seem to find where to do that

[hardy remnant]

Not even ruff or mypy?

[willow pagoda]

personally i've only used deploy keys (SSH) to clone a specific repo, but after some testing, i think you need to:

1. Enable (or have an admin enable) Git access in the organization settings for classic/fine-grained PATs (Third-party Access > Personal access tokens)
2. For fine-grained PATs, set the resource owner to the organization during generation
   Also you may need administrator approval before the fine-grained PAT becomes valid if you aren't one

[hushed orbit]

Ye i couldn't get any of that to work

I think

Did have some pipenv version messups, but got it working with ssh

[cyan whale]

Is better PyCharm or VSCode?

[rapid sparrow]

Vscode is universal tool for a lot of languages. Lightweight to launch dozens of insurances. It has features u will not find on community edition of pycharm

Pycharm is having content behind pay wall, works for single language, and super heavy and slow

[indigo zenith]

Fight it out in #editors-ides

[cunning geode]

Is there a way to mirror python pip packages between package manger like gitea or nexuss?

[atomic granite]

Why doesn't VSCode show a * next to the active environment name?
Here is my terminal output in Visual Studio Code.

PS D:\Python_Projects\idol-sentiment-analysis> C:/Users/Owner/miniconda3/Scripts/activate
PS D:\Python_Projects\idol-sentiment-analysis> conda activate idol-sentiment-analysis
PS D:\Python_Projects\idol-sentiment-analysis> conda info --env
# conda environments:
#
base                     C:\Users\Owner\miniconda3
idol-sentiment-analysis     C:\Users\Owner\miniconda3\envs\idol-sentiment-analysis

I tried typing conda activate idol-sentiment-analysis. Then I typed conda info --env. I expected to see

# conda environments:
#
base                     C:\Users\Owner\miniconda3
idol-sentiment-analysis  *  C:\Users\Owner\miniconda3\envs\idol-sentiment-analysis

I expected to see the * after idol-sentiment-analysis because that is the environment I just activated. I also expected to see the * because other users of VSC see the * next to their activated environment.
Here is an image of another user that sees the *.
visual studio code screen
What actually happened was that I saw

# conda environments:
#
base                     C:\Users\Owner\miniconda3
idol-sentiment-analysis     C:\Users\Owner\miniconda3\envs\idol-sentiment-analysis

[crimson spruce]

Nope. I don't use mypy, and ruff gets installed into most new venvs.

[subtle quarry]

Not sure but try deactivate first. I know it's not active but try it. I remember I had some issue like that with conda.

[placid nacelle]

The answer is in the file you show,
Terminal.integrated.shellarg activates klick environment

It takes the default pythonpath from the click Environment

Jn vscode it also Shows your anaconda envs, whilst your sentiment analysis is a miniconda env

[lusty nymph]

is import os

enough to figure out os system time ram, gpu, cpu and more?

[thorny shell]

I guess. Of course you'll have to run functions in that module, but you knew that. And it might not provide all the information you want

"and more" is pretty open-ended 🙂

better to ask specific questions, like "how can I find all the RSS for all processes"

[molten bluff]

does anyone have a recommendation for doing error handling for a large scale project with multiple functions instead of just wrapping each function with it's own try/except block

[thorny shell]

only use try/except when you know how to recover from the exception, and use as few of them as possible (i.e., maybe just a single one that wraps your "main" loop)

[or maybe none at all; it depends on the project]

[molten bluff]

can you explain what you mean by recover from the exception?

[tawdry needle]

handle only the errors you "expect" to encounter. ignore all others. and only handle errors at clearly defined application boundaries. don't try to catch all errors everywhere

[thorny shell]

"recover from the exception" means "the code you put in the 'except' part". Like, say the exception is a 500 (internal error) from a web server. What's the right way to deal with that? ||I generally have the code sleep a bit and then retry||

On the other hand, a 404 basically means your code messed up by using a URL that points nowhere, so there's no point retrying.

You have to think about each case, and having an exception handler doesn't always make sense.

[atomic granite]

PS D:\Python_Projects\idol-sentiment-analysis> conda deactivate
PS D:\Python_Projects\idol-sentiment-analysis> conda info --env
# conda environments:
#
base                     C:\Users\Owner\miniconda3
idol-sentiment-analysis     C:\Users\Owner\miniconda3\envs\idol-sentiment-analysis

PS D:\Python_Projects\idol-sentiment-analysis> conda activate idol-sentiment-analysis
PS D:\Python_Projects\idol-sentiment-analysis> conda info --env
# conda environments:
#
base                     C:\Users\Owner\miniconda3
idol-sentiment-analysis     C:\Users\Owner\miniconda3\envs\idol-sentiment-analysis

Is this what you meant? It didn't seem to work.

[atomic granite]

I'm not really sure what you're trying to say with your answer. Could you please rephrase it or expand on it?

[hazy echo]

@atomic granite , if you are using OS X, @fast pendant wrote a handy summary of the best ways to set up your python interpreter:
https://blog.glyph.im/2023/08/get-your-mac-python-from-python-dot-org.html

[atomic granite]

Hi, thanks for the reply. I am not using OS X.

[subtle quarry]

Yep. That's it. I think this was work around for tmux if I remember correctly.

Does it help when you start vscode from command line where this conda env is activated? code .

[placid nacelle]

On terminal start you are activating another environment. It also looks like you have a miniconda + anaconda installation on your pc. Your sentiment analysis is created in miniconda, but the klick env that starts as default comes from anaconda.

[thorny shell]

I have been following that, just because glyph is a smart guy :-). Can't say it's made my life a lot better, but it seems OK.

[tawdry needle]

back up. vs code is just showing you a regular shell here. so if it doesn't work in your system command prompt, it won't work here

so, does it work if you just open a command prompt?

it's very likely that the activate command is just failing

on windows it's just activate, not source activate

if you run the latter command, you should see an error

[tidal marsh]

hey

[vague edge]

ok i'm looking at vim here

ctrl + w closes my terminal window 💀

[thorny shell]

keyboard shortcuts are always a mess

see if you can tell your terminal not to "steal" that keystroke.

[hardy remnant]

If you're using kitty then you can map it to no_op

map ctrl+w no_op

https://sw.kovidgoyal.net/kitty/actions/#action-no_op

[vague edge]

uh i'm using tilix

and turns out shortcuts are customizable

so just ctrl w -> ctrl shift w

[atomic granite]

I just have miniconda, not anaconda.

[atomic granite]

Like I said in my original post, the image (with the klick environment) coems from another user. It is not my screenshot. I do not have the klick environment.

[atomic granite]

Yes, I can confirm that it works in a command prompt.

C:\Users\Owner>conda activate idol-sentiment-analysis

(idol-sentiment-analysis) C:\Users\Owner>

Doing so without using conda in the command also works in command prompt.

C:\Users\Owner>activate idol-sentiment-analysis

C:\Users\Owner>conda.bat activate idol-sentiment-analysis

(idol-sentiment-analysis) C:\Users\Owner>

[atomic granite]

I don't see an error when I run conda activate by itself (without specifying the environment to activate).

[tawdry needle]

i mean source activate, the source command isn't on windows

[tawdry needle]

so this all works fine in a normal shell, but not in vs code?

is the vs code terminal using powershell, or cmd?

[atomic granite]

Why would I try to use the source command if it's not on Windows?

[atomic granite]

If by "normal shell", you mean the one that opens why I type "cmd" in the Windows search bar, then yes, it works in the normal shell and not in vs code.

[atomic granite]

I'm not sure. I'd guess that it's powershell because of the preceding "PS" on each line.

[thorny shell]

ya "PS" is the start of the default powershell prompt, for just that reason

[indigo zenith]

Why would you not?

[atomic granite]

Because it's not on Windows.

[indigo zenith]

If that's the issue you can tell VS code to use default cmd instead. But it seems more likely you're in the wrong folder or something else

[willow pagoda]

i think salt lamp means that the message you had replied to, "if you run the latter command, you should see an error", is supposed to be about source activate rather than conda activate which succeeded as expected

[atomic granite]

Why do you think it would make a difference whether I use powershell as the terminal vs. cmd?

[indigo zenith]

Because you showed that you confirmed it works in a normal cmd prompt

[indigo zenith]

Certain commands do work differently in PS, but again, it's more likely that's not the issue

[atomic granite]

I guess I'm confused about why they would bring up source activate when it doesn't work on Windows.

[willow pagoda]

ah i remember encountering this issue myself well i encountered a related issue with vscode activating miniconda, but not an issue with conda activate itself #1134189979472502865 message
never really bothered to look for a fix because i don't use conda (although later in that thread i realized miniconda doesn't have a .ps1 activation script), but yeah, it works in command prompt but not in powershell

[atomic granite]

I can activate the conda environment now in vscode using cmd for the terminal instead of powershell.
I have a follow-up question. Am I supposed to have a .conda folder in my project folder, beacuse I don't have one? I think I remember other people having a .conda folder. Maybe when I created the environment, the folder saved somewhere else?

[willow pagoda]

i think that's meant to be in your %USERPROFILE% (home) directory, i still have mine there as an artifact that the uninstaller didn't remove

[tawdry needle]

because i thought you said above that you tried it. i must have misread

[tawdry needle]

they're completely different programs and have completely different configurations

you need to set up powershell to use conda activate separately from whatever you did with cmd

[tawdry needle]

i personally use conda env create -p .conda to create my project env in .conda, but it's not a requirement and i don't know anyone else who does it that way

and yes, there is usually a centrally-located .conda dir that stores your base environment and all the envs you create with -n

[atomic granite]

Now I'd like to ask, what's the right way to do it, if there is one?

[tawdry needle]

you mean choosing between -n and -p? there's no right way, it's your choice

[heavy knot]

dd

[sage holly]

Hi. Does anyone know libraries for python that can send a signal to the phone to set the alarm?

[astral apex]

That’s probably not possible

[sage holly]

Something like I installed an app on my phone, it's linked to a python library, when something happens the alarm goes off

How to link python to telegram but only alarm clock

[astral apex]

An app can have push notifications that it can handle with alarm sounds, sure

Or you could make your own alarm clock app

But you can’t interface with the native clock app

[sage holly]

The fact that with native I can not understand, I just wanted to clarify whether there are similar ready-made solutions, honestly do not really want to do my own alarm clock, I'm the second day in python and already mobile development?💀

I can understand*

[astral apex]

Uh….
I don’t think so
There’s lots of alarm clock apps
But I don’t think any of them offer a way to set alarms externally
There might be some that have their own accounts system, but you wouldn’t be able to access that
This really sounds like something you’re going to have to build yourself

[sage holly]

Sad, I'll try to look for something similar tomorrow and see if I get lucky. Thanks for your help

[indigo zenith]

If you want to schedule a Telegram message using Python, that's pretty simple

If you want an actual alarm clock app for mobile that you can configure using Python, I'm pretty there's not, and why would there?

[thorny shell]

There used to be a really slick clock app for Android, that would sync your alarms across all your devices. [Google bought them, and afaik that technology has vanished from the Earth. https://www.theverge.com/2014/1/4/5273474/google-acquires-bitspin-timely] Presumably they could have an API, if they'd chosen to.

https://www.cultofandroid.com/39970/timely-beautiful-clock-app-with-device-sync/ iirc.

[atomic granite]

Alright. Thanks.

[crimson spruce]

Automate can set alarms in your alarm clock app (whichever is the active one). That's Android-only though.

[pure wedge]

hey , can someone help me figure out why github is being weird about commit signing

when I click on the unverified tag I get this , but it does not make sense because i've already added the key to github

[vague silo]

do you sign your commits with SSH or with GPG? i thought git only supports GPG

[astral apex]

What does it say when you click on the unverified banner?

[astral apex]

https://github.blog/changelog/2022-08-23-ssh-commit-verification-now-supported/

But you have to explicitly add your key for signing

[pure wedge]

it's asking me to upload my public signing ssh key to verify my signature

[astral apex]

Yeah
You have to go through the process of adding a new key to GitHub again, but this time change the drop-down from authentication to signing
The change log entry above has a screenshot

[pure wedge]

OH

I see

yeah that was it

thanks a ton

[astral apex]

[versed gulch]

Don't know if anyone knows a quick way of improving pytesseract ocr?
Fractions always seem to come out as "%". I've tried increasing the image size to see if it will pick up the characters easier.

[sharp ibex]

Are you doing any image pre processing ?

[versed gulch]

I'm not, but the images are coming from pdf pages that are just plain black and white text.

Example:

[sharp ibex]

I see, so i can list down few options based on my previous experience:

1. Still apply an opencv image grayscale to ensure its really black and white
2. Playing around with binary thresholding (opencv also)
3. Cropping only the part you need in the image
4. tesseract provides several psm (which page format to expect) and oem (which engine to use) options, you can find more information about them here: https://ai-facets.org/tesseract-ocr-best-practices/. You can just try a combination of these, personally psm 6 and 11 with oem 0 gave the best results in reading text.
5. There is a character whitelist argument you can pass to tesseract to only detect certain characters, you can try passing there alphanumeric and the special characters you allow

[versed gulch]

Awesome, this is super helpful. I was starting to look at the psm and oem options, but wasn't quite sure what ones were good to use.
I'll make sure I run the images through a grayscale. 👍

[versed gulch]

Still not much luck, but I'll play around more. Now it's just not recognizing the fractions at all.

pytesseract.pytesseract.tesseract_cmd = r'C:\Program Files\Tesseract-OCR\tesseract.exe'
tessdata_dir_config = '--psm 11 --oem 1 -c tessedit_char_whitelist=" ½⅓⅕⅙⅛⅔⅖⅚⅜¾⅗⅝⅞⅘¼⅐⅑⅒@-/.0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz  " tessedit_char_blacklist="%"'

images = os.listdir('files')
images_from_path = [i for i in images if i.endswith(".jpg")]
for i, imageName in enumerate(images_from_path):
  image = numpy.array(Image.open('files/' + imageName))
  with open('files/page_' + str(i+1) + '.txt', 'w') as f:
    text = pytesseract.pytesseract.image_to_string(image, lang="eng", config=tessdata_dir_config)
    f.write(text)

[sharp ibex]

Have you tried cropping only that part, with and without the whitelist

[versed gulch]

I have not. I still need to figure out how to crop. I'm also not sure if there's an easier way of doing what I'm trying to accomplish. I figured converting to an image and then using tesseract would be easier. But maybe I can find a way to extract the text and images from the pdf and only use ocr for the images.

[sharp ibex]

https://pypi.org/project/ocrmypdf/ this can also be helpful, also definitely you should only use ocr if its an image page, and extract the text using other tools if its not. https://pypi.org/project/pdfplumber/ and https://pypi.org/project/pdftotext/ have given some good results for me, cropping is easy in opencv as well

[dim blade]

Does anyone know of a good vs code extension for codebase navigation and search ?

[short peak]

built-in explorer is good enough for me. What kind of features are you looking for?

[placid nacelle]

Vim extension? Can easily jump to functions definition/occurence with pressing gd while over function.
Dont know if theres a way within normal editor.

[dim blade]

Like semantic search, code documentation .. I am building a similar tool (codemuse). And want to have an idea of what's out there ! Sort of like copilot or codeium but rather focused on code understanding and search rather than auto completion

[sage holly]

Hi. What are some ways to link one python script on PC and one on phone? That is, from the PC I will get a signal to the phone in the python application, let's say by turning on the alarm clock

[azure crater]

I don't think that's related here

[rapid sparrow]

#web-development problems pretty much.
As a simple choice I could imagine having some simple backend application
Which is used to query states on PC and phone in a periodic loop
U can put new records to same rest API too

Check Django Ninja for this, and use at least its default sqlite3 thorough Django orm. It should work fine

[sage holly]

Okay, thank you very much

[rapid sparrow]

Alternatively u could be making more complicated solution with sockets 😉 may be web sockets. Django Channels can be friend here

For more real time communications between your apps

Rest API is simpler though

[versed haven]

does anyone know the API name of the Google OAuth 2.0? I mean to apply the oauth 2.0 in my application

[native scroll]

Hello im trying building a docker image here's my Dockerfile

FROM python:3.10.10-slim

# Path: /app
WORKDIR /app

COPY requirements.txt /app/requirements.txt

RUN pip install -r requirements.txt

COPY . /app

ENTRYPOINT ["python","src/models/predict_model.py"]

but im getting error

(from -r requirements.txt (line 2)) does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

[flat path]

can you provide the full error?

clearly one of the dependencies listed in requirements.txt is not valid but your message doesn't say

[native scroll]

when i changed it to

FROM python:3.10.10-slim

# Path: /app
WORKDIR /app

COPY . /app

COPY requirements.txt /app/requirements.txt

RUN pip install -r /app/requirements.txt

ENTRYPOINT ["python","src/models/predict_model.py"]

it works now and downloading the dependecies

[flat path]

lol fair enough

[native scroll]

it needed to see setup.py

[flat path]

it does

pip can't install a project without knowing how it's packaged (which is defined by setup.py or pyproject.toml)

[native scroll]

cool , i didnt know that , thanks ^_^

[flat path]

https://packaging.python.org/en/latest/tutorials/packaging-projects/ may be a good read! ^^

[shut ruin]

Hi everyone ! I’m 25M, just started learning python last year, currently working on an IT automation course. I’ve been working alone and think I could benefit from some collaboration now. If anyone is willing to mentor me or work on projects together let me know. My goal is to treat this like I’m a junior engineer and do similar tasks/work flows

[native scroll]

sorry for ping but why is my dependecies getting installed twice , like im installing torch from requirements.txt and downloads it twice

[flat path]

can you show the pip install log?

[native scroll]

sadly , i accidently closed the terminal 😦 , when i run it again , I will send u the logs

[slim maple]

(Unrelated) You should change it to

FROM python:3.10.10-slim

# Path: /app
WORKDIR /app

COPY requirements.txt ./
RUN pip install -r requirements.txt

COPY . /app # I think you can also do COPY . . since you're using workdir

ENTRYPOINT ["python", "src/models/predict_model.py"]

So your dependencies are not installed each time you change anything in your source files

[native scroll]

it gives me the error that it doesn't see the setup.py

RUN pip install -r requirements.txt:
5.174 Obtaining file:///app (from -r requirements.txt (line 2))
5.175 ERROR: file:///app (from -r requirements.txt (line 2)) does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.
5.794
5.794 [notice] A new release of pip available: 22.3.1 -> 23.3.2
5.794 [notice] To update, run: pip install --upgrade pip
------
WARNING: current commit information was not captured by the build: git was not found in the system: exec: "git.exe": executable file not found in %PATH%
Dockerfile:8
--------------------
   6 |     COPY requirements.txt ./
   7 |
   8 | >>> RUN pip install -r requirements.txt
   9 |
  10 |     COPY . .
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install -r requirements.txt" did not complete successfully: exit code: 1

idk why though i saw someone dockerizing a python project who have same Dockerfile as me on youtube

[slim maple]

What's file:///app?

And why it's in your requirements?

[slow hare]

Hey guys, trying to use a webhook to send a notification to a discord server when a user submits a perforce change.

Got the code for the webhook in Python, works OK when called from cmd and powershell.

But when it's called from a perforce trigger, it throws a ModuleMissing error. The module is installed. Kinda lost. Any idea how to fix this?

[Re-uploading easier to read image]

[thorny shell]

sounds like the usual "I pip-installed package but python cannot find it" problem

see if this helps https://teensy.info/weiXMUuqpq

[slow hare]

Thanks for the detailed writeup!

I've had a read, and inserted the 'sys.executable' line, but it doesn't seem to be printing in the error log... Any idea how I can get it to print to see what it's trying to use?

Once again, prints and works fine in cmd and powershell

[thorny shell]

sorry, I can't tell what you're doing, so I don't know why it's not printing

I'd guess you're somehow invoking your program in a way that "captures" standard output, so it doesn't wind up on the screen where you can see it

I don't really know what "a perforce trigger" is (although I can guess; I know what perforce is). Perhaps it logs all output to a file or something

[thorn portal]

what you have added already gave you a pretty big hint what seems to be happening.

you are running your script with one python installation (green arrow with 1) in cmd/powershell
but then after that you are running your script with another python installation (green arrow with 2) in perforce.

what you probably need to do is to make sure you have the necessary package installed for python installation green arrow'd with 2 prior to running in perforce.

[slow hare]

Thanks buddy, really helpful!

I changed the trigger to use the python where I have the packages loaded onto. Looks like I'm running into environment errors... neverending fun!

[merry coral]

how do you use virtual env with a command without requiring the user to activate it explicitly?

[willow pagoda]

what is the context for your question? if it's related to installing applications through pip, users can use pipx which isolates each application in their own venv

[merry coral]

the goal is to have the user type a command (x.py), and the x.py command in a virtual environment runs in the virtual env.

[crimson spruce]

If x.py starts with a shebang referencing the venv's python, it'll use that.

[merry coral]

and that will trigger a change to the venv environment?

[crimson spruce]

The script will run inside that venv then, yes.

[dusty pond]

I have to use an outdated version of Python through Spyder/Anaconda on my work PC. I do hobby python stuff on my personal PC using VS Code with 311. I can figure out packages pretty easily since it's only one location for my Python. I cannot understand how this IDE brings in packages, everything I've looked up says to use Conda or %pip but I can't get it to work. I think I am missing something. I also tried just putting the package folder in my Anaconda.

[robust dagger]

Hi. I wanted to make a script that loops through all the directories in the working directory and inside those directories to do a regex to find the file to unzip. My idea is below

work_dir = r"working:\directory\path
for dir in os.walk(work_dir):
# go inside each directory and search for *_temp.zip and unzip them in there directory

struggling on the comment part

[earnest plank]

Hello I have a question does anyone know what I'm doing wrong I'm trying to make a groceries list maker for shopping so I don't lose track of what I bought as I like to try to code it
Where I have a buy list
Then I can move it to a bought list
With a adding feature for adding to the list
Transfer info to list two
Or change a specific part of the list.

Having a trouble I don't know how time transferring to another list and if I put a pen to edit something else gives me an error so I'm not going to try to remove until I can figure it out

[brazen forge]

you can use the zipfile module to unpack… zip files

[astral apex]

from pathlib import Path
from zipfile import ZipFile

for file in Path(".").glob("**/*.zip"):
  with zipfile.ZipFile(file) as zf:
    zf.extractall()

[willow pagoda]

im having fun with hatch :)

somehow i managed to stumble on this weird behaviour where the mere presence of my js/node_modules/ directory causes hatch to completely miss the files in js/packages/ even if it's included in .gitignore, and after some hours of debugging, i suspect it has something to do with their safe_walk() function: ```py
def safe_walk(path: str) -> Iterable[tuple[str, list[str], list[str]]]:
seen = set()
for root, dirs, files in os.walk(path, followlinks=True):
stat = os.stat(root)
identifier = stat.st_dev, stat.st_ino
if identifier in seen:
del dirs[:]
continue

    seen.add(identifier)
    yield root, dirs, files``` when i add a print statement showing what identifiers were repeated: ```

(581629201, 9570149208442212) already seen by C:\Users\home\Documents\GitHub\reactpy\src\py\reactpy\js\node_modules@reactpy\client
(581629201, 11821949022129021) already seen by C:\Users\home\Documents\GitHub\reactpy\src\py\reactpy\js\node_modules\event-to-object``` turns out that npm created directory junctions for node_modules/@reactpy/client and node_modules/event-to-object to their source code, so safe_walk() thought it already saw the contents of js/packages/ and therefore hatch ignored them

[willow pagoda]

pypa/hatch#1197 for anyone interested in following the issue

[signal cloak]

GitHub

[hatch] #1197 Junctions/symbolic links can cause directories to be missing from source distributions

[rare anvil]

ive got a system that's two frontends (an admin dashboard and mobile app) connected to the same backend. what's the strat in making the git repo.

right now ive got all three in separated git repos, but ill make a change in the backend for the admin dashboard then ill find that it might break sth in the mobile app and so keeping track of everything is a bit wack.

should i just keep all three projects in one repo or whats the strat usually hers?

[brazen forge]

one way to do this (takes a bit initial effort) is to have a client library that all clients use to interact with the backend

another (not mutually exclusive) way is to have an OpenAPI schema for the backend, which you then use to generate clients for the appropriate languages/platforms

[rapid sparrow]

if i would be approaching this problem, i would have prefered client library approach.
We develop backend and its clients in necessary languages in same repository.
CI automatically runs tests for all clients to confirm their working correctly with current backend

in its turn all is left to upgrade client library dependency to latest in order to see all miss matches in front/mobile.
CI could be running to validate it for you automatically if necessary (on new backend/clients releases or smth)

This approach kind of scaling. regardless of in which languages u have your fronts

client library approach will work especially perfect if u adopt using Static typing / Structs in all languages
(Strict mypy/pyright + Pydantic for python, Typescript for js and etc)

although using OpenAPI to generate client is not harming to benefit this way. it would cut some corners for me in development of this client

[unborn harbor]

Hi! I was wondering if I can get some help with github. I'm trying to create a repo w/ for my portfolio. I thought I knew how to do pull requests and push changes but clearly not seeing how much I'm struggling. If anyone's free, I'd really appreciate some help. I can get on vc and show you my screen.

[stark hamlet]

what exactly is the issue?

[unborn harbor]

i want to push two folders from my local computers onto the repo. It showed that I had to a pull request first and I tried doing that and got stuck.

I created a new repo and pushed some files but they're not in a folder like i wanted them organized

idk if i can create the folder within the repo or not

[stark hamlet]

yeah u can

some_main_folder/
    - folder 1/
    - folder 2/

assuming this structure u would open some_main_folder in the terminal and run the following commands

git init
git remote add origin <repo_link_here>
git add .
git commit -m "feat: Base commit"
git push origin master

[unborn harbor]

i see! lemme try those

[stark hamlet]

keep in mind that folder 1 and 2 need to be inside a common main folder which will be considered the root directory and ur local respository

[unborn harbor]

will it work since I already initialized the folders and worked from them individually

yeah i kept them separate, i'll try this now

[stark hamlet]

initialized as in?

u ran git init inside the folders?

[unborn harbor]

yeah!

was i not supposed to?

[stark hamlet]

yeah u don't need to

u can simply go inside the folders respectively and delete the .git folder inside them

[unborn harbor]

this is what i got when i tried the git add .

[stark hamlet]

.

[unborn harbor]

but there's no git folders inside them!

it's just the files i wanted

[stark hamlet]

.git go to view and enable show hidden items

its a hidden folder

[unborn harbor]

hmm lemme see,, i dont see the hidden items options

can i do it on the command line?

[stark hamlet]

see how to show hidden folders on windows

u just need to open file explorer and go to view settings

[unborn harbor]

im on mac,, maybe that's why. i'll look it up and see if it helps

foudn the culprit lol

[stark hamlet]

also run the command they are telling to remove git rm --cavh...

[unborn harbor]

i got it now! it looks pretty close to what I wanted. thanks a lot!

[brazen forge]

how is this, at all, relevant?

[steady ibex]

How do I fix this am trying to install ultralytics

Keep getting this error

[astral apex]

!paste
You cut off part of the red line on the right
Also the actual error is at the top, above that list
Put the entire output, including the command you entered, into a paste

[rancid schooner]

Pasting large amounts of code

If your code is too long to fit in a codeblock in Discord, you can paste your code here:
https://paste.pythondiscord.com/

After pasting your code, save it by clicking the Paste! button in the bottom left, or by pressing CTRL + S. After doing that, you will be navigated to the new paste's page. Copy the URL and post it here so others can see it.

[placid nacelle]

What did you run in your console?
I just received an issue lately on my own package, where someone faced the same problem with a pipx Install.

I was not able to reproduce it, and he hasnt answered yet. So I am interested what command you where running, also what python version do you use?

Best regards

[hazy echo]

I'm looking for examples of projects that get built (or tested) for multiple version of Python and multiple architectures.
Why?
I'm glad you asked. I have been learning about dagger.io and I want to use it to build complex projects, like the ones in this example:
https://docs.dagger.io/cookbook/#perform-matrix-build

[thorny shell]

any serious library -- like requests

[dusty pond]

Can I upgrade Anaconda Navigator/Conda using pip? I am on my work PC and it will take a while to get an admin to do it. I have tried a few different solutions but I keep getting an error. Is it even possible?

[high monolith]

does anyone know how to fix this?

when i try to install the discord package it wont, this error pops up

[astral apex]

!paste
We can’t read that
Copy and paste the entire en outpout into a pastebin

[rancid schooner]

Pasting large amounts of code

If your code is too long to fit in a codeblock in Discord, you can paste your code here:
https://paste.pythondiscord.com/

After pasting your code, save it by clicking the Paste! button in the bottom left, or by pressing CTRL + S. After doing that, you will be navigated to the new paste's page. Copy the URL and post it here so others can see it.

[viral terrace]

How to a app for python language

[azure crater]

Any library that uses PDM?

[willow pagoda]

In the context of Python, what ways are there to handle dependencies between packages in the same repository? I'm not that familiar with monorepos so I'm having trouble proposing a simple solution for reactpy's non-functional source distributions (discussion). To summarize, this is the project structure: r src/ js/ ... package.json py/ reactpy/ ... pyproject.toml # hatchling backend src/py/reactpy/ depends on src/js/ being built, however since it is above pyproject.toml, it's difficult to make the sdist preserve the file structure for src/js/'s build commands to be automated correctly. More specifically, the sdist archive root starts at pyproject.toml and can't go beyond that root to reach ../../js, so it needs to include src/js/ under a different path like ./js/. hatch-build-scripts only allows one working directory so it's hard to tell it which is the correct one in the current context. There were further alternatives mentioned like symlinking, hardlink cloning, and nesting js/ inside reactpy/, but they all have their own issues mentioned in the discussion/PR. I also thought of making the JS client an installable Python package so it could be included as a dependency in pyproject.toml, but that requires maintaining two packages on PyPI and the potential version conflicts makes it harder to practise "living at HEAD". Is this something where we have to make a separate script or use some monorepo tool that handles linking src/js/ to src/py/reactpy/js/ before invoking the command to build reactpy? Or would it be better to ditch the monorepo layout and make reactpy the only package at the repository root, that being: r js/ reactpy/ pyproject.toml

[tawdry needle]

Can you use a DAG/task runner tool like Make to sequence everything properly? Or just a shell script even

maybe use hatch as far as you can, and then go up a level to a Makefile to orchestrate across components

(or equivalent if you hate Make)

[willow pagoda]

the repo has an invoke script suitable for that purpose so i'll suggest it as another solution too

rmorshea also suggested not publishing the sdist which might be the easiest path, but I'll respond with a couple concerns about it based on what the packaging guide says about the usefulness of source distributions

For context, in the github discussion I mention:

This isn't a major issue since reactpy's wheels aren't dependent on system architecture, but it would be nice to have this part of the build process fixed (or at least documented).

[zinc umbra]

I couldn't find an .env file parser that was appropriate without including an external library and pydotenv didn't handle all syntax edge cases so I wrote my own:

https://gist.github.com/loopyd/60c21f09ca3e2007d00681e5554e3103

Specifically I needed to serialize and deserialize to pydantic models to write .env files on the fly and docker-on-whales didn't include this tooling in the docker compose cli wrapper so I wrote automated .env file serialization and deserialization myself. It handles code comments and blank lines and comments on the same line too, something that pydotenv doesn't do and also has proper bash type hinting so the .env files are sourceable.

I am still looking into pydantic api to figure out how to do this as a serialization / deserialization classmethod. As I'd like to avoid inheritence but it works for now.

If I want a BaseModel that is capable of being loaded from an .env file this was my solution.

It also provides most primitive types and is capable of inferring what type the environment variable should serialize to/from by checking field annotations.

[vocal pewter]

An interesting idea. Folks do like their pydantic models. Have you considered allowing to define the .env name? Projects I work with can often have two or more for dev environments.

[zinc umbra]

Yes env name is passed as the env_file argument to load_env and save_env so any file can be used.

I thought of that. 🙂

[vocal pewter]

Ah, I overlooked that. This seems like good library material.

[zinc umbra]

Yes ever since I found pydantic I love it is my go to for type hinting

[vocal pewter]

I generally treat environs as strings but I can see the value of the flexibility.

[zinc umbra]

I have to dig into pydantic's API for doing the adding extension method things (for classmethods). They had some way of doing it with email fields (I had seen that example of an extension) but its not something that's openly documented.

Just so one could install such a thing as a package in the same manner to have .env file serialization or not.

I have thusfar found it for __slots__ and ThirdPartyTypeAdapter for Field and Annotated but not classmethods that 'plug' functionality into the BaseModel without introducing class inheritence.

https://github.com/pydantic/pydantic/discussions/7008#discussioncomment-6759608

Oooh, I think I just found it in this __polymorphic__ implementation in their git issues. Time to get cracking.

[thorny shell]

git kraken

[tawdry needle]

dare i ask why you couldn't just use json?

[zinc umbra]

docker.

they utilize a fixed .env file to allow variable expansion in compose projects. that's an .env file and not a json file (sadly, i could model.dump_json directly if that was the case

short answer: use case doesn't permit it, so I had to implement my own serializer/deserializer.

docker docs - the .env file in docker-compose

since using python on whales CLI wrapper to invoke docker.compose.up()

[zinc umbra]

also my friend makes hand made Himalayan Salt lamps. Nice name. Very close to home, he imports from Pakistan. 🙂

this used to be totally easier, but they changed the api completely for custom validation/serialization from pydantic V1 to V2, honestly its one use case and i don't see to make my implementation any more robust. done spending time on it next feature. v2 has some very effy dunder method stuff that is stated as experimental in the docs. oof.

[tawdry needle]

bind mount? or embed the json in the env data, with proper escaping and quoting (which python and pydantic should mostly do by default) its safe to use json within python-dotenv

understood of course, but that sounds like far from my first choice for a solution

[tawdry needle]

cool! he imports chunks of salt and carves them by hand?

[zinc umbra]

He imports the blocks and does all the carving. Got a lathe in the workshop and such.

Well I have it working so I moved on into refactoring into modules.

[bold herald]

https://github.com/alexanderthegreat96/nadeshot-watcher

Maybe it will help someone.

It's basically py-watcher, but as a standalone executable. That works for both Linux, docker and windows.

Let me know your thoughts on this.

[warm blade]

looks like a watchdog alternative except it's made in go and i can't use programatically

[bold herald]

Define programatically.

It's supposed to keep rebooting your app entry point

Everytime you make changes to the file system.

It's what I use to build my discord bots.

[warm blade]

as in it can be used from py program

[bold herald]

So instead of running: py main.py, this thing will run the app for you.

[warm blade]

but now it sounds like

!pypi pymon

[rancid schooner]

pymon v1.1

Application monitoring in Python

[crimson spruce]

Any advantage over entr?

[bold herald]

Yeah, something like that.

[warm blade]

!pypi entr

[rancid schooner]

entr v0.0.1

entr Python Library

[queen lynx]

@bold herald That's pretty cool. How do you stop execution without exiting the program? Say you want to be able to stop at a specific line of code and then rerun. Is that possible?

[warm blade]

sounds like you want a debugger

[crimson spruce]

It's not a Python lib

[queen lynx]

No

[bold herald]

It will do it automatically. Once a file system change is detected, it will stop the previous thread.

[warm blade]

[crimson spruce]

https://eradman.com/entrproject/

[bold herald]

And start a new one.

[warm blade]

can i pass args to it?

[bold herald]

That's a good question still. If you actually make a mistake or something. It will keep re-running the app and stop there. Optionally, I could add a keyboard shortcut to have it stop somewhere.

[warm blade]

what if my file does python main.py --hello --world etc

[bold herald]

Good question. I could implement the use of additional python args

Can you open a ticket on issues in git for that?

[warm blade]

i can do that tomorrow i guess

[bold herald]

I just shared the app so I never really used that in this scenario. I used it for discord bot development.

Since I needed something to be outside of the discord loop

It was also a good exercise for me for using threading / go routines

[queen lynx]

I mean more like what if your program isn't running continously (like a discord bot would). Is this the wrong tool then?

What I mean is if you have, let's say, a program that makes a request to an API and then uploads said data to a SQL table. If I wanted to change that request on the go without having to stop and run again, would I be able to sort of set a "breakpoint" with this program?

My scenario sounds a lot like a debugger type scenario and this may be an incorrect use of your program but I'm curious still. Does it make sense?

[bold herald]

Yeah it does make sense. I could implement that, since it's not hard. Open an issue on the repo.

And we can make it better.

[queen lynx]

I'm unsure of how to eloquently describe it lol

But I'll give it a shot

[bold herald]

Thanks. We can figure it out together. No worries.

[queen lynx]

Done!

[bold herald]

Also I am not sure if python would support this. Now that I think of it. Since ,it would need to resume a previous state.

Maybe create a buffer

That holds the previous code in memory and then binds it to the new one to execute.

Not even C does this, if memory serves.

Yes, with debuggers that deal with memory instructions, it can be done. But python wise, don't think so.

But will give this a thought.

[queen lynx]

Cool! 👍

[severe spruce]

Hi all. Rather new to Python. I've a small library and I'm trying to figure out how to autogenerate the README.rst file from my emuobect/Emu.py file. Is there an easy way to do this?

Or is there a better channel to ask this in?

[rapid sparrow]

Jinja2 templating library is an option

[severe spruce]

I'm hoping to have something more automatic. In Perl, I can do pod2markdown path/to/file.pm > README.md and I'm done. Is there an equivalent in Python?

[tawdry needle]

Unfortunately no, but there are HTML docs generators Sphinx, PyDoc, MkDocs

Err not pydoc

That's another thing. Trying to find the name

[severe spruce]

Yeah, sadly that's what I'm seeing. On github, the HTML doesn't automatically render. Manually updating the README.rst is fraught with error.

[tawdry needle]

Pydoc is useful though because you can dump a docstring to plain text

Actually if you write rst or markdown in your docstrings that would work

https://docs.python.org/3/library/pydoc.html

Pdoc is what i was thinking of. Although weirdly i think it was subject to a hostile fork and I don't know which version is "correct"

[tawdry needle]

python -m pydoc emuobect.Emu > README.rst will at least get you the module docstring

[severe spruce]

Better 'n nothing 😦

So in Python, people just manually write the README.rst file?

[tawdry needle]

Yes but usually people don't try to render the api docs as a readme, it's not a use case i have even considered having a tool for

[severe spruce]

OK, thank you 🙂

[bold herald]

just write it by hand.

[compact badge]

anyone used: https://www.prefect.io/ ? any reviews ... I have got a bunch of python script that I run to download and format and upload data. I was looking for soem tool that i can use for monitering, schedualling tasks kind of ETL stuff ... any recomendations so some really simple stright forward tool

[bold herald]

I never used it

[severe thistle]

I like it a lot! Been my orchestrator of choice since like 2020

[tawdry needle]

what makes you like it more than airflow? i've only ever used airflow but i'm curious how the other "next gen" orchestration systems differ from it / improve on it

[severe thistle]

Well one thing was just how fast it is to stand up, which was relevant for some consulting gigs where I had to stety from scratch. I also liked its integration with Dask!

There were a lot more differences when I got into it, I've think Airflow has absorbed a lot of the criticisms that Prefect addressed since then? But idk

I also like their community and customer service, even when I was on a free tier you'd have Prefect employees answering questions in their slack reliably (almost always within 24 hours, often in under an hour)

[tawdry needle]

i see, that makes some sense

Airflow has absorbed a lot of the criticisms that Prefect addressed since then?

i've heard that as well. i'm new to airflow as of the last two years and only in the last few months i really had to learn it

[granite lantern]

hi guys, I'm self teaching myself devops slowly and on one of the courses they're asking to install K9s for managing kubernetes clusters. I'd like to know if this is a must have tool or not in devops world, as well as if it'd be better to learn first without it then maybe consider using it (maybe the fact that it's more difficult on vanilia makes writing commands more frequent and thus buils up the muscle memory or such things?).

so is this yet another tool to learn worth it?

[thorny shell]

I used kubernetes some in my last job, and I've never heard of k9s, so I doubt it's "must have"

might be useful, though

[brazen forge]

I'd count it as must-have, if you're gonna be using/managing Kubernetes daily

you'll still have to know how to use kubectl, though, as k9s doesn't support a lot of the operations supported by kubectl

creation of resources, for instance

[granite lantern]

See? I had a strong feeling that told me "you should start messing around with kubectl first, then move on to a tool to make it easier". So you recommend I wait a while before using K9s?

[thorny shell]

fwiw I found https://k8slens.dev/ to be super-useful, mostly because I could never remember the various commands; wheras the gui is easy to "explore"

[rapid sparrow]

gave overlook, it looks like monitoring tool mostly + some small management stuff to execute.
Looks cool, going to try it too.

[rapid sparrow]

it is nice to know CLI, and u need always first to change infra look via code
but there is no shame to do monitoring via some GUI tools. that's just easier to do this way ^_^

having it as TUI is extremely cool tbh.

we will always have some monitoring at the end for all clusters
Having some solution easily reusable across all clusters without attachements to having smth deployed... is nice, because it is experience that u can learn once and reuse across all your clusters

it is like.. sure you can code in Notepad, but using IDE and visual debug from vscode is better you know in order to be more productive

k9s in this regard is this visual debug and code navigation features from vscode so you could be better aware about your application state

[rapid sparrow]

@granite lantern as for creating resources you should not use GUI or CLI for that.
Infra resources should be created as infrastructure as a code only. if u do in any other way... then you preferably should avoid having kubernetes cluster in the first place and go back to using Windows/Linux servers in a raw way without any instruments ^_^

of course for debugging or learning purposes it is still valid way

[granite lantern]

Wowow easy haha I don't understand any of these still, I'm a little baby in devops, networking, and sys admin :(
But I'll keep these notes in mind and thanks a lot for your insights they're VERY helpful. One can't ask for a better help really, big thanks ❤️

[rapid sparrow]

they have funny beginning btw " Who Let The Pods Out?"
https://youtu.be/Qkuu0Lwb5EM

K9 = Canine

[granite lantern]

yeah I like this kind of funny and silly things in our world of CS lol
I'm a monthy python fan and it's one of the biggest reasons why I picked up python :P

or things like yaml or yacc for yet another thingie that ends up becoming universally used

also another trivial thing is that, Python Flask library has been intented to be an easter egg and ended up being one of the most used backend frameworks XD

[hexed fiber]

hey there, is out there any lib that reads screen and tracks stuff on it like words?

[untold arrow]

does anyone have original mega man 8-bit sprites (orginal size aswell, not scaled up)
(mainly just the idle)

[haughty badge]

I used to just use the autocomplete that you set in your bashrc but this UI is pretty cool

[thorny shell]

autocomplete only works for me sometimes -- I probably don't set it up properly

[warm blade]

does anyone know of any good guides on sphinx to MKdocs migration?

[astral apex]

Banned for leaving Sphinx

[rapid sparrow]

why would u leave Sphinx?

[tawdry needle]

Because rst is a pain in the rear

[astral apex]

Would you accept Sphinx with the Markdown extension?

[willow pagoda]

ive only used sphinx for a few projects and had to deal with a few issues in the process, but they were more so related to sphinx rather than the reST language (i'm still not fully used to the syntax, but i can usually figure out what i need with sphinx's primer and python domain reference)

i did find that overall the initial learning curve for sphinx+reST was a bit higher than i would have liked, but perhaps that's just me having no prior experience with doc generation

[rapid sparrow]

I just use Sphinx with markdown https://www.sphinx-doc.org/en/master/usage/markdown.html @tawdry needle 😊

and in cases where markdown Sphinx is not enough, there is fallback to regular Sphinx syntax for usage in special blocks

[vague edge]

" https://superuser.com/a/760272
map <C-a> <esc>ggVG<CR>
```despite the `<esc>`, i still can't seem to ctrl-a to highlight everything in vim
anyone know what the problem might be

[crimson spruce]

Works for me, perhaps there's something else later in your config that overwrites it. What's the <cr> supposed to do?

[vague edge]

no clue, i just copied it from that link above lol

[vague edge]

oh wait i did not say that i can't highlight everything in insert mode

[crimson spruce]

What do you mean by that? You want stuff "highlighted" while staying in insert mode?

[vague edge]

the command is supposed to take me back to normal mode (or visual?) with everything highlighted

[crimson spruce]

Ah, so you're fine with it landing you in visual mode, but the map doesn't work in insert mode?

[vague edge]

yup

[crimson spruce]

Change the map to inoremap.

The important part is the i, so it works in insert mode, the nore just makes sure the keys it maps to aren't recursively looked up as maps.

[vague edge]

inoremap <C-a> <esc>ggVG<CR>
nnoremap <C-a> <esc>ggVG<CR>

alright these two seem to fix it

[crimson spruce]

Yes. Although you don't need the <esc> for the nnoremap, since you're already in normal mode.

And the <cr> are pointless in both cases

[vague edge]

what's that thing mean anyways

[warm blade]

Looks ugly

[astral apex]

Not nearly as ugly as mkdocs

[tawdry needle]

it's just enter, "carriage return"

[tawdry needle]

i always forget that. i wonder how it works with roles and directives

i really wish it supported asciidoc, which is basically rst but done right

[tawdry needle]

ahh it's myst

right

iirc that's more like asciidoc than plain markdown

unfortunately adoc is like rst, kind of locked away behind a reference implementation and lack of a spec

[tropic hound]

it's the return key, which i guess extends the selection to the start of next line. However since capital V (visual-line mode) is used the entire last line would already be selected, hence it's useless here

I'm actually surprised gg goes to BOL of first line but G doesnt go to EOL of the last line

[warm blade]

hmm, have you looked at the textual documentation / website

[astral apex]

yeah

[warm blade]

mkdocs 😛

[astral apex]

Ruff docs too

[warm blade]

and you think they're ugly?

[astral apex]

Nah

I hate the stock theme

Material is okay

[warm blade]

lol

[brazen forge]

lol, the Material theme is the most popular by a large margin

https://github.com/mkdocs/catalog#-theming

[crimson spruce]

(I know, my point was: What is it supposed to do here?)

[tawdry needle]

not sure

[novel sinew]

Can I install Poetry in first step in Bitbucket pipelines and then use it in next step? I'm having trouble finding a way to do that

[astral apex]

BitBucket? 🤨

Should be fine

What have you tried?

[novel sinew]

definitions/caches/pip and pypoetry at the top level and then caches: ~/.cache/pip and ~/.cache/pypoetry in the first step (where I upgrade pip and then install Poetry using pip). In the next step I also do 'caches:', but Poetry command is not found

[left girder]

If each step is in a separate container, might not have access to what happened in the previous one @novel sinew

You might have to tar up the install, and mark it as an artifact or something

Or just install and run in one step

[stoic tiger]

guys, do you know if it's possible to separate package creation from package upload in setuptools ? aka run python setup.py sdist, and later python setup.py upload <repo>. granted the ./dist directory contains the packages

apparently setuptools upload is deprecated .. maybe i should just skip to use twine

[astral apex]

setup.py sdist is deprecated too

[stoic tiger]

so build + twine is the way

time to rewrite ci

[astral apex]

See here for the deprecation: https://packaging.python.org/en/latest/discussions/setup-py-deprecated/

[astral apex]

Oh you're already in CI?
The brand new way is "Trusted Publishing": https://docs.pypi.org/trusted-publishers/
Which means that you get to to use the same action but skip twine and auth entirely

Here's an example of one of my workflows: https://github.com/darbiadev/hermes/blob/main/.github/workflows/python-publish-pypi.yaml

[stoic tiger]

thanks @astral apex

[astral apex]

[rapid sparrow]

we build with
python setup.py bdist_wheel --build-number ${{ github.run_number }} command ^_^

oh, and i also froze to pip install wheel setuptools==59.6.0

in order to avoid for now problem of deprecations 😅

it will be problem of future me.

[fallow smelt]

I wonder if it would be possible in vscode to render docstring markup inline. Like you write your docstring, and as soon as your cursor is not in the docstring it renders it in the same place. And if you click on the rendered version it is replaced with the source. Maybe with some kind of diagraming so you can make state charts.

[compact badge]

https://www.mage.ai/

this is perfect etl tool

[spring notch]

Hi,
Is it possible to run django scripts on google cloud functions?

[indigo zenith]

Possible? Probably.

Advisable? Probably not, unless you're sure you don't care about preserving state and don't mind the limitations on length of run time, etc.

[full cipher]

anybody here who has used instaloader tool extensively?

[rare anvil]

whats a django script? you mean a management command?

[inner eagle]

Hello, is anyone knowledgable on software defect detection?

[vague silo]

yes

ask your question

[inner eagle]

thanks, im going to be working on my first major project for my university and i chose to develop a software defect detection application. Simply put, im very lost on how i should approach developing it with the various types of algorithms needed

my supervisor has advised i create at least four or five algorithms in which i can use for comparison in my paper which i have somewhat of an idea on which ones to choose

just getting the code puts me at a standstill

[remote sand]

any tools that help map out source code flow (data or control flow) purely from source files?

[heavy knot]

Is there a good video or course that more or less teaches DevOps at a higher level than specific implementations? Not for becoming a DevOps engineer but more or less understanding the value proposition/common terminology

[rapid sparrow]

https://sre.google/books/
U could read those books, and probably Site Reliability Engineering one is the one u seek.
It describes different situations? Human interactions included? It tells every day mini stories and how Google approached DevOps

There is also a very fun book describing what is DevOps supposed to mean originally, culture, what is its original idea https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592
It is way less close to practical, more philosophical. From where it all began. Since u formulated your question like this, may be it makes sense to read u too and may be even first. It is literally novel book fun to read

[inner eagle]

@vague silo

[idle depot]

not sure if this is the right place to ask. So please guide me otherwise.

I have a FastAPI app inside a docker container in an AWS EC2 server.
And another API set in AWS API Gateway+Lambda(s)+DynamoDB.
I want to reach a certain/multiple endpoints from the fastapi app -> aws api gateway
The AWS API Gateway has cors set to true in all endpoints + a JWT Cognito authorizer (username+password is requred).

**Is there a way to let the FastAPI app access the AWS API Gateway without they outh - based on it being a local service? **

[rapid sparrow]

don't create same msgs in multiple channels, that's really bad behavior

[rapid sparrow]

common sense says that it is possible through usage of IAM roles (usually all services allow directly calling stuff as long as u have correct IAMs to call it)
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_function_url
Yeah, lambda function url to call it directly from EC2 should work
Authorization type AWS IAM is available ( authorization_type = "AWS_IAM")
I think this should be the correct resource to go and security right way in general (having restricted calling only to services that have the right and very narrow down IAM role specified)

[rapid sparrow]

your EC2 machine should be having assigned IAM task role
and it should be having mentioned something like

Statement
Allow
{
"Lambda calling permission" (u will see exact IAM permission in docs, or during first attempt to use it, lambda endpoint should return correct error what is missing)
for resources {
"arn: lambda bla bla your lambda arn"
}
}

This would be working way if u just wished to call your lambda (Although to be honest u could call it directly through Boto aws cdk library too)
Not sure how your API gateway thingy may affect it
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/api_gateway_integration
it looks like bendable to similar stuff as function url according to observed options

[rapid sparrow]

https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html
Quick scan over docs show that API Gateway thing is for publishing APIs in insecure networks (public internet)
Since u are requesting from AWS EC2 to Lambda, i think u don't need it and u can utilize a simple Lambda Function URL (with turned on IAM auth) resource or calling by Boto CDK direclty

[idle depot]

when saying channels, you mean sessions? if not, can you explain what do you mean by channels?

[idle depot]

Didn't think of pinging the lambda directly. funny because i thought of this before writing the api.
If i will decide to apply this change, i would probably go with the AWS CDK kinda way and using the lambda directly in my code.

I just know it's common for microservices to communicate via APIs with each other. Am I wrong? Is that a possible "anti-pattern"?

[rapid sparrow]

We are in #tools-and-devops channel
U wrote your msg also in #software-architecture channel

[idle depot]

I saw your msg there. i removed mine.. *

[rapid sparrow]

I just know it's common for microservices to communicate via APIs with each other. Am I wrong? Is that a possible "anti-pattern"?

It is common and fine. The difference only that in CDK way u have extra overhead in using Boto library (but u gain kind of more code flexibility), and in function url way u have extra overhead in infrastructure code.

The important part here

1. writing code in unit testable way (both are testable ways)
2. use pydantic to parse input in lambda and generate JSON data for its call. It will make automated code validity to interact between API and lambda code (at a typing level and runtime validation)
3. consider using strict mypy/pyright for extra quality

[idle depot]

Thanks for your help dude. This helped me a lot.

or dudette 🤷‍♂️ *

[plucky vale]

how i can make like this i mean tkinter can do like this?

any programmer?

[rapid sparrow]

Small warning btw, be very cautious with using lambdas 😅

[tropic mantle]

Is there a tool that given a package will return it's "public interface"? Classes, method, etc. in modules that don't start with _.

[willow pagoda]

like, for generating documentation materials? or for personal inspection of a module?

!e if its the latter, the help() output might be sufficient for that: py help("heapq")

[rancid schooner]

@willow pagoda :white_check_mark: Your 3.12 eval job has completed with return code 0.

001 | Help on module heapq:
002 | 
003 | NAME
004 |     heapq - Heap queue algorithm (a.k.a. priority queue).
005 | 
006 | MODULE REFERENCE
007 |     https://docs.python.org/3.12/library/heapq.html
008 | 
009 |     The following documentation is automatically generated from the Python
010 |     source files.  It may be incomplete, incorrect or include features that
011 |     are considered implementation detail and may vary between Python
... (truncated - too many lines)

Full output: https://paste.pythondiscord.com/6NYCGYJ6XBCEW72Z6XJ4AVA4EQ

[willow pagoda]

or alternatively importing the module and doing some introspection on it: ```py
import importlib, sys

module = importlib.import_module(sys.argv[1])
for name, value in sorted(vars(module).items()):
if name.startswith("_"):
continue
print(name)``` but if you mean automatic generation of documentation for a package that you're developing, i'd consider setting up sphinx and using their autodoc extension

[tropic mantle]

Well I'm looking for something that would work for an entire package. It would be useful for checking that everything "public" is documented (I understand that pydocstyle does this), but also I'd like to see a list of everything I'm providing to users to clamp down on things I may have not intended to be public interfaces. The project I'm working on has been really bad about both points: documenting public interfaces and being careless about what is public. We are working on a major rev and I want to take advantage of the chance to fix this.

[hazy echo]

Just in case anyone is curious about dagger, I have started a migration guide and I have a simple Hello world exercise at https://github.com/blaisep/pytest-bdd-ng/blob/1be3ad57cc899327143b54d0be90395a69d0ceaa/docs/developer.rst#prepare-your-virtual-environment
with more example in progress. Eventually I'll implement the whole testing matrix to run locally, independent of github actions.

[willow pagoda]

im not aware of a tool for this purpose, but i ended up spending the last few hours writing one just to see how it would go
https://github.com/thegamecracks/documember c $ documember asyreader asyreader (undocumented) reader (undocumented) AsyncReader close() read() start() Readable (undocumented) close() (undocumented) read() (undocumented) AsyncReader close() read() start() Readable (undocumented) close() (undocumented) read() (undocumented) though as of now it doesn't check inheritance for docstrings (oh wait it does, thanks inspect.getdoc()!) and it can't tell if global variables or class attributes are documented since those get erased at runtime, for that i'd parse the AST if i knew how to

as a sidenote ruff also implements pydocstyle rules too https://docs.astral.sh/ruff/rules/#pydocstyle-d

[tropic mantle]

@willow pagoda Cool, I'll check it out! And yeah we are using ruff. Getting "D" passing is a process, lol. I wish we could have that set of checks warn but not error right now, since we are running ruff in pre-commit and in CI. Seems there's been no effort towards implementing warnings since it was initially discussed.

[heavy brook]

Hey, i'm sure i set it up correctly, but github is not verifying that the commit is by me

[hazy echo]

Sphinx is your friend.
It's a little hard to get started (long story), so I suggest getting the initial project set up by someone who has done it before.

[rapid sparrow]

agreed with suggested sphinx.
Utilize module Autodoc, it has option: Try to parse for docs everything present and form docs from them

[placid nacelle]

I created a package which supports managing .env files, since I have not found anything like that around.

Is that something to post here?
Or does anyone know about a tool that does that already?

[rapid sparrow]

people usually use for .env files dotenv https://pypi.org/project/python-dotenv/
otherwise in backend development, it is some common to load .env through docker-compose
that is more than enough for dev env.

for deployment .env files aren't needed. as it is kind of against best practices ( https://12factor.net/ )

[placid nacelle]

Its like a central location store with the option to export paths/shell strings/ or copy to a local .env file.
To kinda work alongside python-dotenv.

https://github.com/Zaloog/dotenvhub

[rapid sparrow]

+1 for TUI. what a cool thing to make (no matter what it does 😅 )

the idea is cool in general as well. secret manager pretty much

[rapid sparrow]

Interface looks incredibly sexy 😊 so much alluring

TUI... fully fledgly looking like GUI 😅

Also +1 point for Linux friendliness / instruction for Linux

[placid nacelle]

Thanks for the friendly words :)

[spring beacon]

A quick Q, related to pyproject.toml spec:

• [project.dependencies] specifies dev deps
• [build-system.requires] specifies build deps
• you can also specify (under build tool) what packages to build and where to find them

? How to specify exclusions of some dev deps that aren't required for install, like pyright, black, pytest, etc?

[willow pagoda]

i would more accurately describe project.dependencies as installation dependencies rather than specifically for development, but regardless you can use the [project.optional-dependencies] table to declare "extra" dependencies the user can install, for example: ```toml
[project.optional-dependencies]
dev = ["black>=23.12.1", "pyright>=1.1.348", "pytest>=7.4.4"]

Install with "pip install .[dev]"```

[spring beacon]

that works. thanks @willow pagoda !

[willow pagoda]

oh and depending on what build system you use, it may provide a tool-specific table for development dependencies which is hidden from pip and only available to their frontend CLI
PDM: https://pdm-project.org/latest/usage/dependency/#add-development-only-dependencies
Poetry: https://python-poetry.org/docs/managing-dependencies/#dependency-groups

[spring beacon]

@willow pagoda anything liek this for setuptools?

Spent hours trying to find something like this in setuptools docs before asking here

[willow pagoda]

afaik setuptools doesn't even have a frontend, other than the setup.py script which running directly has since been deprecated (ref), so i just use the dev extra for my setuptools-based projects

[astral apex]

Pretty much, yeah

[fierce pulsar]

where can I ask question regrading setting up

like vscode sort of questions

[placid nacelle]

#editors-ides i guess?

[orchid ore]

Hey there, i'm having an issue with installing a whl directly from a download link, which PEP508 specifies as possible.

the issue seems to be that pip relies on the URL itself to feature the correct extension .whl, which gitea registry URLs don't.

requirements.txt

my_package @ https://git.company.com/org/-/packages/pypi/my_package/0.1.0/files/21

a pip install on this fails with Cannot unpack file .../pip-unpack.../21 (downloaded from: ....); cannot detect archive format

Is this a limitation of pip, requiring the extension to be there, or am i misreading pep508 and the packaging.python.org docs that this should be possible?

of course, i can install from git using the git+https:// syntax without issues but that means i miss out on the advantages of pre-built wheels

[thorny shell]

if it were me, I'd do some experiments: create a .whl, make sure I can install it from a file. Then rename it to remove the .whl extension, and again try to install it.

[orchid ore]

good call, will try now

[orchid ore]

aight so a file:///package.whl works, if i remove the .whl it tries to read it as a folder and errors out with does not appear to be a python project: neither setup.py or pyproject.toml found.

seems to confirm that pip cares about file extension, eugh

[thorny shell]

yeah but it'd be nice if you got the same error that you'd gotten before, with the https url. 😐

I bet someone else has had this problem, if gitea URLs are always like that. Maybe it's time to do some googlin'

[orchid ore]

i already tried. sadly, nothing conclusive. gitea themselves say to use the --extra-index-url but that dont work in pyproject.toml

i filed an issue with gitea to include filename in their download URLs for now: https://github.com/go-gitea/gitea/28916

[vocal gorge]

So I just installed miniconda since it is light weight compared to the full anaconda distribution, but now it's causing my shell to load like 10 times slower. Any tips to not do that? I'm using fish as my shell btw

[hazy echo]

I'm going to be working on this:
https://github.com/blaisep/pytest-bdd-ng/issues/4
in case anyone wants to join me. I don't think I'll livestream on this Discord because one of my partners isn't on here. I can share a link if interested.

[rapid sparrow]

is it possible running Dagger for free / self hosted?

[hazy echo]

That's exactly what I do

There's a commercial cloud offering but that's more for reporting and analytics.

[rapid sparrow]

Excellent idea 😄

that's one of my dreams. Having static typed full programming language controllable CI for free

[hazy echo]

Oh! yes, that is why I have been hanging out with them for the past few months

It's a really diverse community because theree are SDKs for so many languages.

You may remember that I'm a big self-host/opensource advocate.

[hazy echo]

you're welcome to drop in:
https://jitsi.pio-p.io/daggerize-pytest
I'm the only one here so far.

[rapid sparrow]

https://blog.gitea.com/creating-go-actions/ - Gitea
https://github.com/sethvargo/go-githubactions - Github Actions

https://go.dev/ref/mod#go-mod-vendor

[warm blade]

i have an use case for coordinating tasks or actions within three or more windows VMs,
i had read up on https://firecracker-microvm.github.io/, is there anything like this but for windows?

[brazen forge]

Coordinating tasks/actions with machines sounds like Ansible's job

[astral apex]

Doesn’t Ansible explicitly refuse to run on Windows?

[brazen forge]

No?

https://docs.ansible.com/ansible/latest/os_guide/windows_usage.html

[astral apex]

https://docs.ansible.com/ansible/latest/installation_guide/installation_distros.html#installing-ansible-on-windows

https://docs.ansible.com/ansible/latest/os_guide/windows_faq.html#windows-faq-ansible

Huh

I watched a video once where the host said “[…] will not work on Windows, at all”, and I’ve just been running around telling people that Ansible is Linux-only since then

[brazen forge]

Trust, but verify

[astral apex]

Well I had like 5 people all saying the same thing

Guess I coulda poked at the docs

[warm blade]

then you should've trusted but verified 5x, smh, doggos

[wary relic]

This is quite a long read, but I need some help.
Python script 👇

#!/usr/bin/python3

"""
a script that starts a Flask web application
"""

from flask import Flask

app = Flask(__name__)


@app.route("/airbnb-onepage/", strict_slashes=False)
def hello_hbnb():
    """
    displays Hello HBNB when the application runs
    """
    return "Hello HBNB!"


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

ubuntu@289442-web-01:~$ gunicorn --bind 0.0.0.0:5000 web_flask.0-hello_route:app

[2024-01-25 08:44:05 +0000] [3145087] [INFO] Starting gunicorn 21.2.0
[2024-01-25 08:44:05 +0000] [3145087] [INFO] Listening at: http://0.0.0.0:5000 (3145087)
[2024-01-25 08:44:05 +0000] [3145087] [INFO] Using worker: sync
[2024-01-25 08:44:05 +0000] [3145089] [INFO] Booting worker with pid: 3145089

ubuntu@289442-web-01:~$ sudo vi /etc/nginx/sites-available/hbnb

# Nginx configuration to server /airbnb-onepage
server {
        listen 80;
        listen [::]:80;

        root /home/ubuntu/AirBnB_clone_v2/;

        server_name 18.204.13.162;

        location / {
                try_files $uri $uri/ =404;
        }
}

ubuntu@289442-web-01:~$ curl 127.0.0.1:5000/airbnb-onepage/

Hello HBNB!

Now trying to make Nginx serve pages or responses on port 80
ubuntu@289442-web-01:~$ curl 127.0.0.1/airbnb-onepage/

Error 404 - Page not found

What am I doing wrong?

[rapid sparrow]

i would want to say first... say, you are doing excellent job of describing your problem
Many people have communicational very great problems doing that.

[rapid sparrow]

U a using Nginx in the mod to serve Static assets / js/css/whatever files.
But u seek to use its Reverse proxy first, to proxy its requests to gunicorn + optionally using static assets serving in addition.
A moment will give u code example

[wary relic]

Alright @rapid sparrow
I should also add that I created a symbolic link for .../sites-available/hbnb in /etc/nginx/sites-enabled/

[rapid sparrow]

    upstream gunicorn {
      server localhost:8000;
    }
   server {
      listen 80;
      listen [::]:80;

      server_tokens off;

      location / {
        proxy_pass http://gunicorn;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;

      }
    }

this will work as reverse proxy.
U can add some extra location like location /static inside this instruction, to serve its static files too if wishing

[rapid sparrow]

I should also add that I created a symbolic link for .../sites-available/hbnb in /etc/nginx/sites-enabled/
as far as i remember it was supposed to be having some kind of command to turn on and off stuff. smth enable smth
never used though due to using more optimal docker version of nginx, which easier to configure

[rapid sparrow]

there is a free nginx cook book. (free!) from o'reilly
U could be wishing to explore it
https://www.nginx.com/resources/library/complete-nginx-cookbook/

[wary relic]

Thanks. I will try these in a moment.

Thank you again.

[rapid sparrow]

https://github.com/darklab8/darklab_examples/blob/master/nginx/reverse_proxy/default.conf
here is example just in case how to configure it working with ssl certs too in a future for reverse proxy
ssl certs are attached to 443 port (self signed they are, not exactly valid)
and 80 port is redirected to 443
443 port is redirecting to our reverse proxy location accordingly

Hehe, this folder nginx/reverse_proxy is exactly example on flask with nginx

[wary relic]

You've been very helpful.

[warm blade]

Well, it’s more like I want to have an API, I make the request to the API, one of the VMs does the work and then give back response to the API

[astral apex]

Kubernetes?

Or containers or whatever

Does it need to be a specific VM?

[brazen forge]

Something like snekbox?

[rapid sparrow]

Oh, u just want celery + simple backend API.
U wish windows machines to run Celery workers

That is assuming u want just to execute some CPU smth jobs

[rapid sparrow]

Made reading about snexboxes. Definitely Celery + Django Ninja for API can recommend https://django-ninja.dev/

Celery will distribute tasks between machines, will execute and will put result to some storage

[rapid sparrow]

Celery ecosystem offers nice monitoring called flower

For tasks and workers health monitoring

It will show helpful errors and travelogs if u will use python in a good enough way

[brazen forge]

If it's about isolating the compute (and network, storage, etc.), configuring the isolation will be more of a challenge than a standard setup like Celery + Django

[rapid sparrow]

for isolating i can offer using rootless Docker with non root user 😅 not a foolproof method though. but very lightweight
running in VM like vagrant for easy as a code launch should be may be more reliable for security purposes. VMs do make more reliable isolation after all.
docker without usage of root user, and may be even using rootless engine is more easier as a code alternative though

Linuxes also offer chroot... but chroot is pain in the ass to configure. to hell with chroot

[warm blade]

yes, windows.

[warm blade]

yes, they should all run separately hence VMs

[warm blade]

security isn't a concern.

[dull lion]

dm me if you know how to code in assembly

[brisk lake]

got 3 volumes defined in docker-compose, they're all created and mounted and whatnot with docker-compose build
but, in my dockerfile if i do RUN ls /vol i only see one of them
any ideas or keywords to look up, google is failing me

[rapid sparrow]

got 3 volumes defined in docker-compose, they're all created and mounted and whatnot with docker-compose build
compose/docker build operation is not attaching volumes i think
Volumes are attached at runtime for compose/docker during up/run command

[rapid sparrow]

but, in my dockerfile if i do RUN ls /vol i only see one of them
during build time u are supposed to use COPY/ADD operations

[rapid sparrow]

Volumes are a thing meant for containers.
which u can see as docker container ls. For launched instances! They get attached volumes.
Images which u see with docker image ls, which u build with docker build, aren't tied to anything. They are usable thing to create containers/instances

============
Funny enough we can for some reason define VOLUME at the level of Dockerfile...
... i have no idea why they decided to add it.
https://docs.docker.com/engine/reference/builder/#volume
U can see from instruction that they are created only at run operation regardless according to documentation

[brisk lake]

ok this might be an XY problem

so project structure is

Dockerfile
docker-compose.yml
project/
  - dir1
  - dir2
  - js_nonsense/
    - package.json
    - src/
    - node_modules/

the 3 volumes are for ., an external lib lib and js_nonsense

in the dockerfile they only do COPY . <some dir>

nothing is installed in the js dir, thats what i was trying to do

basically RUN cd /vol/js_nonsense && npm i

[rapid sparrow]

definitely XY problem then. u aren't supposed to install stuff with volumes.
docker builds image with COPY/ADD instructions

define

WORKDIR working_directory_inside_container. Will be used for COPY commands at relative path
COPY local_path_at_your_machine path_in_container
COPY local_path_at_your_machine path_in_container
COPY local_path_at_your_machine path_in_container
for your all three folders

[brisk lake]

any ideas why they chose to do this then?

is there some other benefit maybe of defining vols

[rapid sparrow]

Volumes are useful for... mm... using actively docker/docker-compose for development from within container. For having feature hot reloading. (npm start dev server is fine to run for containers with forwarded volumes)
changing files in container and having stuff autosaved to dev machine / volume automatically syncronize stuff between your dev machine and container

together with plugin vscode for remote containers, u can more or less develop from within containers

Alternatively volumes are useful for dynamic data accumulated during run of application in deployment. Accumulated logs / loaded by user pictures and etc

[brisk lake]

bro we are so backwards in terms of tech, we're basically in the stone ages

no chance for dev containers

anyway cheers for advice, will report back

[rapid sparrow]

Words like that makes me appreaciated my company more and make me scared ever leaving it.
Sure i disagree a bit in terms of code quality standards a bit...
...but at least we are doing unit testing for all the code and they don't mind me doing my extra on top.
Or we have... more or less modern deployment systems too.
Sure code is yaml ton of garbage, but it does work

[brisk lake]

first this issue, then getting code cov up, then time to start applying lmao

[shut hedge]

im getting an error that it doesnt find python while instaling pyinstaller

[floral smelt]

I have a Docker-related question when it comes to COPY because I've seen multiple Dockerfiles like that so I've been wondering:

# syntax=docker/dockerfile:1.4
FROM --platform=$BUILDPLATFORM python:3.10-alpine AS builder

WORKDIR /src
COPY requirements.txt /src
RUN --mount=type=cache,target=/root/.cache/pip \
    pip3 install -r requirements.txt

COPY . .

CMD ["python3", "server.py"]

FROM builder as dev-envs

RUN <<EOF
apk update
apk add git
EOF

RUN <<EOF
addgroup -S docker
adduser -S --shell /bin/bash --ingroup docker vscode
EOF

# install Docker tools (cli, buildx, compose)
COPY --from=gloursdocker/docker / /

CMD ["python3", "server.py"]

Why do we only and only copy requirements.txt first instead of that file and server.py? Is there any particular reason? Wouldn't that waste layers?

[vague silo]

It's to help with caching. You change the server file more often than the requirements, so when you only change the server file you don't need to reinstall all dependencies

[floral smelt]

Oooh I see

[brazen forge]

To add to this, there's also COPY --link which creates an entirely separate layer that's cached on it's own.

With that, the layer ordering doesn't affect it.

[floral smelt]

I'm not too familiar with BuildKit's syntax I admit

[rapid sparrow]

Exactly. it will help with caching.
Saves time to rebuild previous commands since it will be needed only on text of requirements.txt changing.
I highly recommend mentioning without version only main dependencies in requirements.txt and locking pip freeze into constraints.txt
then installing them as pip install -r requirements.txt -c constraints.txt btw
It will make easier management of dependenies and easily deleting no longer needed ones

Instruction COPY . . is not cachable really
Since will change on anything.
Also recommendation, don't use COPY . .
It is bad command because makes u copying stuff u did not intend to do. (Secrets / .git repo / and etc)
use COPY exact_folder exact_folder
COPY file1 file2 file 3 ./
Also add .dockerignore against copying unwanted stuff like __pycache__ in addition

[floral smelt]

Noted. In a stage though I tried COPY manage.py src/* ./ and it copied the contents of src right next to manage.py. Is there a way to align it on a single line instead of duplicating COPY for each?

[rapid sparrow]

u can align multiple files to be copied in a single command
u need to copy folders on separate lines (folders are big anyway)
not aware about command to copy multiple folders in command (it cam be made though)

[floral smelt]

Wouldn't that weigh much more on the final image by doing that? I read some bits and pieces from Poulton's book on Docker and COPY, FROM and RUN are all layer-generating commands.

[rapid sparrow]

no. exactly for a reason of Layering it won't
docker image is summ of its layers, which are like git diffs, they just add some extra content on top of previous layer. Layer remains just a small difference, files on top that are added
So... resulting image is... multiple layers arrangement.

Since copying two folders to image, and copying one folder and then another folder, equal to same size... there is no difference.
Difference appears only if u in single command for example made RUN apt update && apt install -y smth && apt cleared its cache. Then total result of the added layer will remain only added app, with skipped apt cache from saving

also difference appears if u do Multi stage building. Built in one, copied result to freshly clean image

[floral smelt]

My favorite technique so far. Only inconvenient is the cache disappearing when i'm doing docker build in CI/CD pipelines

[rapid sparrow]

nothing is perfect 😅 that's why there is need to cache more.
Usually longest docker image stage is installing libraries though
And this one is easy to cache
And all long stuff other stuff of installations easy to cache too.

Only the operation of copying the code and building it is not cachable for obvious reasons
Although even this is somewhat cachable depending on language... features (as far as i know in java u can build plugins separately and provide already built stuff to other code)

[versed gulch]

Can anyone help me out with getting CI CD set up for bundling Python FastAPI to upload to an AWS Lambda?
I've got a CI set up, but it doesn't seem to want to install the correct Python version.

name: FastAPI CI/CD

on:
  # Trigger the workflow on push
  push:
    branches: 
      # Push events on main branch
      - main 

jobs:

  CI:
    # Define the runner used in the workflow
    runs-on: ubuntu-latest
    steps:   
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12.1'

      - name: Install Python Virtual ENV
        run: pip install virtualenv

      - name:  Virtual ENV
        uses: actions/cache@v2
        id: cache-venv # name for referring later
        with:
          path: venv
          # The cache key depends on requirements.txt
          key: ${{ runner.os }}-venv-${{ hashFiles('**/requirements*.txt') }}
          restore-keys: |
            ${{ runner.os }}-venv-

      - name: Activate Virtual ENV
        run: python -m venv venv && source venv/bin/activate && pip3 install -r requirements.txt
        if: steps.cache-venv.outputs.cache-hit != 'true'

      - name: Run Tests   
        run: . venv/bin/activate && pytest
        # check what the venv/lib python path is
      - name: Check Python Path
        run: ls $PWD/venv/lib
        
      - name: Create archive of dependencies
        run: |
          cd ./venv/lib/python3.12/site-packages
          zip -r9 ../../../../api.zip .
      - name: Add API files to Zip file
        run: cd ./api && zip -g ../api.zip -r .
      - name: Upload zip file artifact
        uses: actions/upload-artifact@v4
        with:
          name: api
          path: api.zip

[floral smelt]

Oh and I also wanted to add: when installing packages through pip in the build stage, do you set up the wheel folder? Or is there a more efficient way to build packages?

The command I entered was

RUN pip install -U pip && pip wheel --wheel-dir=/home/app/wheels -r requirements.txt && pip install --no-cache-dir --no-index --find-links=/home/app/wheels -r requirements.txt

[rapid sparrow]

wait... building installed package?

what is it and what for? 😅

somehow never encountered it

[rapid sparrow]

RUN pip install -U pip && pip wheel --wheel-dir=/home/app/wheels -r requirements.txt && pip install --no-cache-dir --no-index --find-links=/home/app/wheels -r requirements.txt

what is the usage of wheel here?

🤔 i mean i compile wheel as distribution format for my lib, but never discovered or needed details, it worked at is

https://peps.python.org/pep-0427/
Okay, there is a hint that pip wheels are OS/CPU platform depended. never encountered issues from it
Supposedly platform depended if they have C extensions inside, but since i don't develop stuff with C extensions inside, i never encountered issues from it I think

So I use pip wheels only pretty much as zip archives to upload my files into our pip server

[floral smelt]

Haha, I dug a bit in testdriven.io's articles: https://testdriven.io/blog/

Needed to find a flask deployment example ASAP

https://github.com/testdrivenio/flask-docker-swarm/blob/main/services/web/Dockerfile

It's this dockerfile I was basing myself on

[rapid sparrow]

😅 if I read stuff correctly pip wheels aren't really for flask deployments at all.

As I see pip wheels are meant for library distributions and pip installable applications like Ansible or smh

[floral smelt]

Another solution to the problem is to build all the packages as wheels—binary packages with the compiled code—and then copy the wheel files over to the build stage and install them there. This works, but doesn’t really add much.
https://pythonspeed.com/articles/multi-stage-docker-python/
Looks like it's also a solution for multi-stage builds

[rapid sparrow]

Interesting. That is a good usage case. Abusing wheels for multi stage building to make smaller final image

May be it will work

[rapid sparrow]

It will be interesting to try but I have some doubts that it will make any significant difference without drawbacks

[versed gulch]

Is this the right place to ask these questions? I was referred here by someone in python-discussion.
I think I've partially figured things out. It seemed to be caching the virtual environment, which is why it wasn't installing the 3.12.1 one.

[rapid sparrow]

So... A thing to experiment 🧪 🥼😉

Probably it will be thing I will be forever lazy to try. Not really a big advantage in decreasing image sizes

[willow pagoda]

ive done that for one of my dockerized discord bots too, downloading/compiling wheels in a separate stage rather than copying over a venv
https://github.com/thegamecracks/thestarboard/blob/main/Dockerfile#L10-L11

[rancid schooner]

Dockerfile lines 10 to 11

RUN --mount=type=cache,target=/root/.cache \
    pip wheel --wheel-dir pip-deps -r requirements.txt```

[floral smelt]

So it doesn't look like a bad strategy!

[rapid sparrow]

looks like very easy to do 😅

i have to try and see if there is image size saving

a moment

okay, without wheels

flowey-api-app                                                 latest          b81b35ae81d4   7 seconds ago   1.36GB

very terrible 1.36gb size. lets try with wheels
Keeping same python image for clean experiment

[floral smelt]

By the way I heard from pythonspeed that Alpine images weren't recommended for Python

Because of the C compiler

[rapid sparrow]

flowey-api-app                                                 latest          48aae15054ea   6 seconds ago    1.3GB

compiled project accroding to your method 😅

one library is missing not was fully installed
Anyway... 60mb saving... that's pitiful for 1 minute of efforts to compile stuff

build time with wheels 57.2, making image without wheels in the mean time takes same time.
okay at least build time more or less not affected

[willow pagoda]

sad

[agile stag]

Is welcome-to-docker container suppose to remain in your repository for Docker?

[willow pagoda]

this article right? https://pythonspeed.com/articles/alpine-docker-python/
hm, how do i inspect what occurred while an image was created? i believe i didnt have any issues using C-based wheels for me

[floral smelt]

docker history?

[rapid sparrow]

using slim image i can save more space, but it is building in slim too 🤔

flowey-api-app                                                 latest          79ef2f2baa98   5 seconds ago        418MB

ergh... a lot of effort for small gain. to hell with it

it is not increasing dev comfort / or even deployment comfort

it is interesting concept from the point of prehistoric times though, before Docker was invented we had no other choice to freeze libraries locally in some easy movable way between machines

[floral smelt]

@rapid sparrow About awesome-compose'd Dockerfile for Flask: https://github.com/docker/awesome-compose/blob/master/flask/app/Dockerfile#L15

[rancid schooner]

flask/app/Dockerfile line 15

FROM builder as dev-envs```

[floral smelt]

Why is there an extra stage here when the CMD command is above?

[rapid sparrow]

i have no idea whey they added extra stage... perhaps for pure documenting purposes to separate Build section from dev running section
That's the only explanation i see

[rapid sparrow]

CMD command can be defined at any level of the same image. it does not have to be at the end (it is just common practice to do it 😅 )

as long as it was not reset by using different base FROM, it will remain

[floral smelt]

In the end it will still execute once the layers have finished compiling?

[rapid sparrow]

ENTRYPOINT and CMD only get activated on container run command

for image building they do not participate at all, except defining default ENTRYPOINT/CMD (overridable values)

[floral smelt]

Alright. What about USER? Is it different from dropping Linux capabilities (using underprivileged user role instead of root)?

[willow pagoda]

hmm, i couldnt find a log of the stdout or even how long each layer took, so i tested with a simpler dockerfile instead: dockerfile FROM python:3.11.5-alpine RUN pip install numpy ENTRYPOINT ["python3"] as i watched the output, i caught the line Downloading numpy-1.26.3-cp311-cp311-musllinux_1_1_aarch64.whl which does indicate that alpine can only take advantage of musl-specific wheels for C extensions, but otherwise it still skipped the compilation process which would have otherwise been horrendously long (prob 30+ min) for my raspberry pi

although if hadn't flashed a 64-bit OS image, my docker image would have needed to build for armv7l regardless of it being alpine, and even cibuildwheel doesn't target that architecture

other than that, cibuildwheel generates musllinux wheels by default so whatever packages have it in their CI/CD (like numpy) will very likely be pre-compiled for alpine

[floral smelt]

No funny side effects or extra bloat?

[willow pagoda]

if i rewrite it with multi-stage build: ```dockerfile
FROM python:3.11.5-alpine AS deps
RUN pip wheel --wheel-dir wheelhouse numpy
FROM python:3.11.5-alpine AS project
COPY --from=deps wheelhouse/ wheelhouse/
RUN pip install --no-index wheelhouse/* && rm -r wheelhouse/
ENTRYPOINT ["python3"]

though i had accidentally written it like this beforehand: ```dockerfile
FROM python:3.11.5-alpine AS deps
RUN pip wheel --wheel-dir wheelhouse numpy
FROM deps AS project  # Didn't restart from base image
RUN pip install --no-index wheelhouse/* && rm -r wheelhouse/
ENTRYPOINT ["python3"]``` and that produced a 156MB image

[lost light]

Did anyone worked on Gitlab

Need help to create pipeline in Gitlab

[brazen forge]

Not if you're not using it

[brazen forge]

IIRC, that article was published before the introduction of musllinux wheels

So all packages (with C extensions or not) needed to be built from source when using Alpine

These days, packages publishing wheels (without looking at any stats, I think most packages do this) will have a musllinux wheel because of cibuildwheel support like thegamecracks mentioned above

I just copy over a venv from a build stage, not caring about what was wheeled in or built from source ¯_(ツ)_/¯

Though for some packages you do need to care

For example, confluent-kafka-python has a runtime dependency on librdkafka (a C-based library), so you have to install via the distro package manager in the final stage as well

[floral smelt]

True. Now I'm wondering what's the best image for my apps in production

I use slim debian

[brazen forge]

Same.

[arctic rose]

Hi I have created an elasticache cluster for work and I have setup its security group and I have madesure all the VPC and subnets are properly configured.

when I connect to work VPN and then connect to the elastic cache using redis-cli from local, I am able to establish the connection but I when I run any commands it gives me the error

Error: Server closed the connection
(1.00s)

when I connect to the elastic cache cluster through node I get the error:

Error:  read ECONNRESET
at TLSWrap.onStreamRead (node:internal .... ){
  errno: -4077,
  code: 'ECONNRESET',
  syscall: 'read'
}

and the error keeps repeating

I am not sure why it is not able to run any commands, I am not sure if it is connection issue IDK T_T, please help why am I not able to connect to redis elastic cache is ther any steps I need to be doing? how can I go about troubleshooting this issue.

[brazen forge]

Do you have encryption in transit enabled? If so, are you connecting via a secure tunnel?

[arctic rose]

No I don't have encryption transit enabled

It is encrypted at rest tho

[brazen forge]

!rule 5 9

[rancid schooner]

Rules

5. Do not provide or request help on projects that may violate terms of service, or that may be deemed inappropriate, malicious, or illegal.

9. Do not offer or ask for paid work of any kind.

[brazen forge]

Hmmm, have you tried connecting to the Elasticache instance from a machine in the VPC?

As in,
Not You -> VPN -> Elasticache
But You -> VPN -> Machine -> Elasticache

[arctic rose]

No I haven't I have used the VPC Path Analyzer tool tho and it said that the redis instance was reachable

Also it seems the redis instance is trying to close the connection

I tried from python also and it gave me similar error

But why is redis trying to close the instance I also tried upgrading to medium cache cluster and trying that dint work as well

Anything I should maybe change in the parameter group?

Is there any where I can check logs of redis or something

[brazen forge]

You need to configure log delivery for Elasticache

Do you see anything odd in the CurrConnections metric for the Elasticache instance?

[arctic rose]

Let me check

[brazen forge]

That's a connection reset, just more descriptive than the node one

[arctic rose]

Yeah 😭😭

I have an engine log group and slow log group

Idk what this means

[arctic rose]

This is from the engine log group

[brazen forge]

Do you have a read replica?

[arctic rose]

I do not know

Mine is cluster on encrypted only at rest

[brazen forge]

How many nodes do you see?

[arctic rose]

2shards 3 nodes each

[brazen forge]

Did you try connecting to a basic, one node, non-clustered Redis?

[arctic rose]

No

I will try that

That dint cross my mind

[brazen forge]

Isolating the problem is the way :D

Perhaps it's a clustered vs. non-clustered mode Redis problem

[arctic rose]

After lunch

Right

[brazen forge]

I've run into that before

[arctic rose]

I see

Thanks

[brazen forge]

Np

[arctic rose]

Just wanted to verify the configs once before creating it

[brazen forge]

LGTM

[arctic rose]

I created this, and it's still giving the same error

[arctic rose]

Few more logs

[brazen forge]

That seems very weird considering the new Elasticache doesn't even have a replica

[brazen forge]

Did you try connecting to it like I mentioned [here](#tools-and-devops message)?

[thorny shell]

Hmm

pipx upgrade-all ...
upgraded package pytest from 7.4.4 to 8.0.0 ...

and yet https://docs.pytest.org/en/ redirects to https://docs.pytest.org/en/7.4.x/

ah, https://docs.pytest.org/en/latest/changelog.html#pytest-8-0-0-2024-01-27

[arctic rose]

Not yet it just seemed a bit expensive but I'll try it out.

[arctic rose]

@brazen forge it worked from an instance within the VPC

[brazen forge]

Feels like it might be a VPN issue.

Is the VPN configured to block any ports?

Also, is the SG(s) for Elasticache instances configured to allow connections from VPN SG?

[crimson spruce]

!rule ads

[rancid schooner]

Rules

6. Do not post unapproved advertising.

[plain axle]

anyway I can get people to see if they would like to sign up?

[crimson spruce]

Nope

[plain axle]

🙂

[torn sparrow]

Greetings guys

Hope all are well.

Can someone help me publish my package to PyPi?

[visual oxide]

What trouble are you running into?

[torn sparrow]

Not a trouble really, just never done it.

I tried the user/password one but apparently that is no longer supported.

I have made a pypi account now, but not sure how to publish the package.

[visual oxide]

You're publishing with twine?

[torn sparrow]

Yes

PS C:\Users\A.C.EA\Documents\GitHub\QRandom> twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Enter your username: A.C.E07
Enter your password: 
Uploading qrandom-1.0.0-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.4/68.4 kB • 00:01 • ?
WARNING  Error during upload. Retry with the --verbose option for more details.
ERROR    HTTPError: 403 Forbidden from https://upload.pypi.org/legacy/
         Username/Password authentication is no longer supported. Migrate to API Tokens or Trusted Publishers instead. See https://pypi.org/help/#apitoken and
         https://pypi.org/help/#trusted-publishers

[visual oxide]

For the username use __token__

[torn sparrow]

Ok, one moment...

[torn sparrow]

Where do I get this?

The pypi account?

[visual oxide]

And for the password use the token starting pypi-

https://pypi.org/manage/account/#api-tokens

[torn sparrow]

What should I use for the scope?

I can only choose entire account.

[visual oxide]

When first publishing you need the entire account afaik

[torn sparrow]

I see

one moment..

[visual oxide]

Don't paste the token here btw

It's as valuable as your password

[torn sparrow]

I won't

It gave this error now

PS C:\Users\A.C.EA\Documents\GitHub\QRandom> twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Enter your username: __token__
Enter your password: 
WARNING  Your password contains control characters. Did you enter it correctly?
WARNING  See https://twine.readthedocs.io/#entering-credentials for more information.
Uploading qrandom-1.0.0-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.4/68.4 kB • 00:01 • ?
WARNING  Error during upload. Retry with the --verbose option for more details.
ERROR    HTTPError: 403 Forbidden from https://upload.pypi.org/legacy/
         Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for more information.

Did I do sth stupid? Hehehe

[visual oxide]

https://discuss.python.org/t/pypi-api-upload-token-isnt-working/20754

There's a warning about control characters

https://twine.readthedocs.io/en/stable/#entering-credentials Ctrl+V doesn't work on windows

You need to use Edit > Paste

@torn sparrow ^

[torn sparrow]

I am using VSCode's terminal

[visual oxide]

Maybe use a different terminal app?

[torn sparrow]

Ok now this :

ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
         Invalid value for classifiers. Error: Classifier 'License :: OSI Approved :: GPL License' is not a valid classifier.

I think I got past the error.

I am using GPL 3.0 license

[visual oxide]

You probably want

License :: OSI Approved :: GNU General Public License v3 (GPLv3)

Or

License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)

Copy it from here https://pypi.org/classifiers/

[torn sparrow]

Where do I paste this? Can't find where it is reading it from.

So sorry for the bad questions hehe.

[visual oxide]

Paste your pyproject.toml

!paste

[rancid schooner]

Pasting large amounts of code

If your code is too long to fit in a codeblock in Discord, you can paste your code here:
https://paste.pythondiscord.com/

After pasting your code, save it by clicking the Paste! button in the bottom left, or by pressing CTRL + S. After doing that, you will be navigated to the new paste's page. Copy the URL and post it here so others can see it.

[torn sparrow]

Ohh one moment...

Ok found it hehe

Sorry I had accidentally deleted the file.

# pyproject.toml

[build-system]
requires      = ["setuptools>=61.0.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "qrandom"
version = "1.0.0"
description = "Quantum Random Number Generator."
readme = "README.md"
authors = [{ name = "Amir Ali Malekani Nezhad", email = "[e-mail]" }]
license = { file = "LICENSE" }
classifiers = [
    "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3",
]
keywords = ["quantum computing", "random", "random generator"]
dependencies = [
    "numpy",
    "qiskit",
]
requires-python = ">=3.12"

[project.optional-dependencies]
dev = ["black", "bumpver", "isort", "pip-tools", "pytest"]

[project.urls]
Homepage = "https://github.com/ACE07-Sev/QRandom"

Good?

[visual oxide]

You'll need to rebuild your project

[torn sparrow]

I see, one moment...

PS C:\Users\A.C.EA\Documents\GitHub\QRandom> twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Enter your username: __token__
Enter your password: 
Uploading qrandom-1.0.0-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.4/68.4 kB • 00:00 • ?
WARNING  Error during upload. Retry with the --verbose option for more details.
ERROR    HTTPError: 403 Forbidden from https://upload.pypi.org/legacy/
         The user 'A.C.E07' isn't allowed to upload to project 'qrandom'. See https://pypi.org/help/#project-name for more information.

Did I do sth wrong?

[visual oxide]

Someone already uploaded https://pypi.org/project/qrandom/ back in 2021

You'll need to pick a unique name for your project

[torn sparrow]

Ohh ok.

Any suggestions?

Hehehehe

It's quite literally one method, I wish I could contest the name.

nevermind. So it's a quantum version of the random package. I have all the core methods, so you can use is just like random.

Can I later rename it?

[visual oxide]

You can rename it by uploading to a new project URL

[torn sparrow]

I see.

Ok, so I need to rename the repo right now?

[visual oxide]

Then deprecating the old one

Your repo name and project name don't need to match

But it's nice if they do match

[torn sparrow]

Ooh ok.