#tools-and-devops

1 messages · Page 22 of 1

potent cedar
#

I can probably figure it out, but I assumed some other project has already solved this problem before lol

rapid sparrow
#

So your choice to do cross compiling, or raising self hosted runners with different arch

#

I don't remember python supporting cross compiling

#

That means self hosted runners is the only choice may be

rapid sparrow
#

Gh should act as trigger/status reporter at most

potent cedar
#

just need to figure out how to get multiple python versions installed in there...

#

maybe I'll run my multi-python-version tests (on native arch) and my multi-arch-tests (on whatever the default alpine python version is) separately

#

and not bother trying to do many-pythons and many-archs at the same time

potent cedar
kindred halo
#

What do you guys think about competitors of Windows Terminal? Recently Ghostty became stable. Do you think somebody should make a term emulator for Windows that rivals wt?

rapid sparrow
vocal pewter
#

Hard to disagree with that any more than I do.

As far as competitors, it's just another form of "X editor is better". Use the one that provides the interface and features you need.

short peak
#

before Windows Terminal I needed a terminal for git, another for k8s, another for WSL (V1), another for PowerShell, etc

gentle solstice
tawdry needle
gray obsidian
#

hey

kindred halo
sterile nova
#

I have an API in fastapi that I'm sending to a VPS and using Docker to start, but I'm having a problem that is reaching (100%), I didn't do anything to reach such a large %, when I used htop I came across this

/usr/bin/python3 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=11, pipe_handle=*) -- multiprocessing-fork

it creates several processes

molten brook
civic ridge
#

hey all, have a quick question. I'm looking to send a few thousand emails (over the course of a few weeks) from my company account to potential partners. Code seems easy enough but I'm looking for some insights on whether this is going to work or is even a good idea (I don't want to screw anything up for my company). Basically have to decide 1) whether to use SMTP or 2) Microsoft Graph or 3) some sort of paid service (outside of python).

Main goal is to have this be coming from my email, and for their inboxes to not detect anything 'spammy' (Currently, I've been doing this email outreach manually, but it's time consuming and I'd like to reach more people). IDK maybe this is a better question for an IT Discord? Grateful for any help or insights

rapid sparrow
# civic ridge hey all, have a quick question. I'm looking to send a few thousand emails (over ...

SMTP
rate limited against programmatic spam. few emails rarely it could handle, but few thousand is already a question, with timeframew over 2 weeks may be it will be able. Question of reliability. May be it will be able to handle few thousand emails. May be u will get banned potentially

Microsoft Graph
not aware about it

some sort of paid service (outside of python).
Email sending is known to be the most difficult thing to do (Very strictly regulated area 😅), so... things like paid service Sendgrid, or AWS SES are recommended

civic ridge
#

Okay, thank you!

smoky compass
#

Does anyone know if it's possible to include the only the source code of a method somehow using mkdocstrings?

gentle solstice
#

you can customize the options to exclude everything but the source code.

onyx pine
#

Ha, didn't even think of that myself, but it's true, you can hide the heading, the signature, the class bases, and all docstring sections, leaving only the source

vale wedge
#

im wondering - anyone aware of a tool that allows to develop on the most modern syntax and generate older syntax versions of the code for distribution

thorny shell
#

never heard of one

crimson spruce
#

Like the opposite of modernize?

#

I naively think it should be fairly easy if you don't need to go below 3.10.

vale wedge
#

i make excessive use of generis, new style type aliases & co - its a pain to use them in the python 3.9 compatible forms - in particular when one cant use the typing future as well

rapid sparrow
rapid sparrow
vale wedge
rapid sparrow
#

Obviously it will be not as good as just stripping of types or making python in general compatible to lower version 😅

#

all compilings are highly likely CPU/OS sensitive

rapid sparrow
#

that could make compatible to lower python version too

#

at a cost of a somewhat nasty result of lesser library self documentation for those older python versions

vale wedge
#

Actively unhelpful suggestions are kinda rude

thorny shell
#

I didn't take that as "actively unhelpful"; I took it as "you probably don't want to go to this much effort, but perhaps you do, so here's another idea"

vale wedge
#

The first suggestion was a c implemented library

The next suggestion was practically just to break the code

Zero effort and zero background knowledge

Just dumping broken suggestions i have to throw out directly after researching

visual oxide
#

It is weird something like Babel doesn't exist for python

vale wedge
#

There's less ecosystem pressure for something like it

Most preexisting libraries just keep up with the minimum

Im currently starting a experiment to replace the config system of pytest and is like to use 3.12 syntax a lot

visual oxide
#

And there's no pyupgrade for it either

visual oxide
royal silo
#

I am building a chat app and I am using Docker to make it. I have three services in my Docker Compose file: client, server, and caddy.

client is a React app that generates static files under dist/ in the container. And server is a Python server that is the backend of the chat app, so it's a websocket server. And caddy is a web server. I would like to share the generated dist/ folder from the client container with my caddy container so that I can serve it. How do I do that?

Or is my way of doing a chat app just wrong or smth? In that case, I'd like to learn the proper way of doing it

I guess I should mention that it's just a site-project that I'm doing to learn the basics; so it's not supposed to be actually a thing where I need to care about production use

thorny shell
#

I suspect you want to use docker "volumes". Can you share your code via github?

umbral jacinth
#

Anyone here proficient enough to help walk me through adding another piece of code to an existing lambda script so I don't mess up what's already in it haha. I want to learn but I'm new and honestly can't understand how to do what's needed.

royal silo
# thorny shell I suspect you want to use docker "volumes". Can you share your code via github?

https://github.com/eeriemyxi/rorica

I pushed some bare-bones code (I started working on it today). Everything works fine when running export RORICA_TARGET=develop && docker compose up --build --watch but if the target is production, it won't work, since I don't know how to serve the generated dist/ files in another container.

I tried looking at the documentation for Docker's volumes and I found it pretty complicated, like IDK what's subpath, source, target and how to adapt it to my case. Could you have share some examples?

thorny shell
#

ooh! Only saw this now; will look

thorny shell
#

Here's a toy example ```
volumes:
postgres_data: {}

services:
bash:
image: bash:latest
command: "bash -c 'while true; do date; sleep 1; done'"
volumes:
- postgres_data:/postgres/data

postgres:
image: postgres:17
# only for initializing the db
environment:
POSTGRES_DB: woteva
POSTGRES_PASSWORD: postgres
volumes:
- postgres_data:/var/lib/postgresql/data

#

if you put that into docker-compose.yml, then type docker-compose up, it'll start running. Then in a second window, type docker compose exec -it bash bash which gets you a shell in the running bash container.

#

Now type ls /postgres/data/ in that shell, and you'll see the database files from the other container.

rapid sparrow
royal silo
royal silo
#

but if it just copies things at build time, then how does it decide which container's volume to copy to other containers?

thorny shell
#

it's a mount point.

#

try it and see

supple phoenix
#

using pytest for the first time, saw in a conference talk that it shows the values of variables being compared in assert expressions

    def test_auth_login():
        a = 1
        b = 2
>       assert a == b
E       AssertionError

tests\test_auth.py:37: AssertionError

do I have to do something to enable that ?

#

ahh found it

#

-l, --showlocals Show locals in tracebacks (disabled by default)

#

lmao it also has a
--pastebin=mode Send failed|all info to bpaste.net pastebin service
so convenient

short peak
#

I remember django having that feature on error pages, blew my mind ages ago

gentle solstice
#

Typer (terminal command framework) enables this feature by default and I hate it.

#

I don't like my entire stack being printed to my terminal when there's an error.

#

It wouldn't be so bad if typer automatically detected any package installed in site-packages as internal and hid the variables

#

"your code crashed, but since it crashed inside sqlalchemy and it printed every single variable in scope, your code's traceback has flown off the buffer and is unavailable."

rapid sparrow
# supple phoenix using pytest for the first time, saw in a conference talk that it shows the valu... 
$ pytest 
============================================================= test session starts ==============================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/naa/repos/pet_projects/experiments/pytest_test
configfile: pytest.ini
collected 1 item                                                                                                                               

tests.py F                                                                                                                               [100%]

=================================================================== FAILURES ===================================================================
_______________________________________________________________ test_auth_login ________________________________________________________________

    def test_auth_login():
        a = 1
        b = 2
>       assert a == b
E       assert 1 == 2

tests.py:4: AssertionError
=========================================================== short test summary info ============================================================
FAILED tests.py::test_auth_login - assert 1 == 2
============================================================== 1 failed in 0.01s =============================================================

As far as i am aware it works by default (and having confirmed on experiment)

#

just have pytest.ini

[pytest]
python_files = tests.py test_*.py *_tests.py *_test.py

to find test files

#

and run command pytest

visual oxide
supple phoenix
#

I have been running it like this
python -m pytest --disable-warnings -k "auth_login" -v --no-header

supple phoenix
#

🤔

rapid sparrow
#

at linux

supple phoenix
severe inlet
#

rate my requirements installation

j9GTyST.png
astral apex
#

5/10, who's blood is that??

#

What's wrong with just putting a pip install -r requirements.txt in your instructions?

severe inlet
#

its just an annoying step

#

also venv creation

#

i found out pip install will stop you on linux if you are outside a venv

astral apex
#

Not by default

#

It's a setting

brazen forge
gentle solstice
visual oxide
#

But also the displayed code here won't help

twilit forum
severe inlet
gentle solstice
severe inlet
gentle solstice
severe inlet
#

oh this looks nice

gentle solstice
#

Something that's useful is using a shebang #!/usr/bin/env -S pipx run

earnest scarab
#

if a python script is converted to exe does the machine it will be executed on need to have python installed ?

thorny shell
#

no, that's the whole point

thorny shell
#

vaguely similar to Tailscale

compact epoch
#

anybody use localstack for setup local development for their team project?

night frost
#

SyntaxError: invalid syntax

KeyboardInterrupt

/opt/anaconda3/bin/python /Users/ravi/RAVISBOTS/main.py
File "<stdin>", line 1
/opt/anaconda3/bin/python /Users/ravi/RAVISBOTS/main.py
^
SyntaxError: invalid syntax

#

what does this mean

thorny shell
#

it means you typed a file name into the python repl

#

and python doesn't know what to do with it

#

if you're trying to run that main.py file, do these things:

  • exit the python repl by typing exit
  • type /opt/anaconda3/bin/python /Users/ravi/RAVISBOTS/main.py at the shell prompt
#

shells know how to run progams; python doesn't

thorny shell
brazen forge
compact epoch
brazen forge
glass ember
#

Hey y'all, can someone kindly point me to the appropriate channel to ask about UV?

crimson spruce
glass ember
#

@crimson spruce ah ok cool, I thought so, but I wanted to make sure first. I have a basic github action script in place for downloading deps and running mypy on push to main, but I've been searching the internet for other examples of an actions using uv. Do you or anyone have any examples you could send?

#

I've already looked through their docs

thorny shell
glass ember
compact epoch
rapid sparrow
#

despite being very AWS centric

#

i am hesitatnt to think about using aws localstack for dev env, encountered it before

#

idea of getting overly extra attached to AWS is not really welcome i think

#

we try to do the opposite, of moving more towards cloud agnostic direction

#

and for this reason move from AWS ECS to Kubernetes

compact epoch
# rapid sparrow we are completely satisfied with just using Docker Compose for dev env 🤔

Thanks for your sharing, @rapid sparrow .
The only issue I have with AWS is that we're so attached to AWS infrastructure that everything goes through a private subnet.
My first approach is to create an environment with AWS public subnets for everything, such as ECS, API Gateway, and Lambda.
My second thought is about using LocalStack.
Otherwise, we can't even develop further with the production infrastructure.

#

Good to know that you overcome any issue with k8s.

rapid sparrow
#

that ensures we have some env to develop besides production 😄

#

Since Staging infra is maintained by infra devs, average skill level needed to interact with it is very low for devs and non devs

compact epoch
#

If the production environment requires everything to be in a private subnet except for some services,
would you replicate the same private subnet setup for UAT/staging, or would you make everything public for development purposes?

rapid sparrow
# compact epoch If the production environment requires everything to be in a private subnet exce...

Production and Staging both run in private subnets.
In addition, Production runs on AWS WAF by default all allowed, and some black list of stuff is forbidden to be accessed
while Staging has AWS WAF by default all forbidden, and whitelist of Office Ips, our wireguard VPN is allowed to access it.

In order to access Private Subnet infrastructure (both in Prod and staging), we use Bastion approach / having EC2 instance raised that is publicly accessable that can tunnel forward us to allowed elements of private infrastructure (u need to specify which objects are accessable there somewhere at its networking stuff)
in order to get access to this Bastion service, u need to add your SSH key into Terraform code of its creation.
For security reasons we eventually were able to configure were you can use this EC2 instance only to forward tunnel traffic, but not able to just SSH and walk around in it itself

#

Bastion approach is default thing recommended for AWS infra, and some books recommend it as well to access AWS ECS/EC2 and other private infra.
We use it to access at least AWS ECS,RDS and redises

compact epoch
#

Your wise knowledge enlightens me, Darkwind.
I now know what to learn and what to do for my projects. You have saved me many days.

#

I'll try harder to become more skillful and contribute to this community like you.

golden ice
#

Hi Guys, i work with GENAI and LLM's to build assistants and chatbots. Now my manager is asking me to provide some good automation ideas using GENAI. I have already implemented my ideas at work which benefitted the team but it seems they need more, could anyone of you please tell me if you have any ideas or how did u automate ur work like deploying dockers, github cicd or code automation, anything....Thanks in advance!

thorny shell
#

please don't post the same message in multiple channels

snow mica
#

Guys , I need to build a tool for resetting passwords and change mails from outlook of discord accounts

gentle solstice
#

are you resetting passwords on outlook via discord, or discord via outlook?

worldly creek
#

Is there a Pylint rule for accessing a non-existant attribute on an instance? Like if my class does not define self.blah anywhere, and i use instance.blah somewhere, it warns me?

short peak
#

or W0201

worldly creek
#

hmm those are specific to assigning a non-existant attribute, which is also useful, but I'm looking for just accessing a non-existant attribute in my current case

short peak
# worldly creek hmm those are specific to assigning a non-existant attribute, which is also usef... 
Traceback (most recent call last):
  File "/home/cherny/test.py", line 9, in <module>
    main()
  File "/home/cherny/test.py", line 6, in main
    test.invalid
AttributeError: 'Test' object has no attribute 'invalid'
➜  ~ uvx pylint test.py 
************* Module test
test.py:1:0: C0114: Missing module docstring (missing-module-docstring)
test.py:1:0: C0115: Missing class docstring (missing-class-docstring)
test.py:1:0: R0903: Too few public methods (0/2) (too-few-public-methods)
test.py:4:0: C0116: Missing function or method docstring (missing-function-docstring)
test.py:6:4: W0104: Statement seems to have no effect (pointless-statement)
test.py:6:4: E1101: Instance of 'Test' has no 'invalid' member (no-member)

-----------------------------------
Your code has been rated at 0.00/10

➜  ~ cat test.py 
class Test:
    pass

def main():
    test = Test()
    test.invalid

if __name__ == "__main__":
    main()```
#

E1101

worldly creek
#

heh yeah I just found it, I guess it was harder to find because it uses "member" instead of "attribute"

#

thank you very much!

fleet cloak
#

which tools are yall using for version management? changelog entries, bumping pyproject.toml, creating a github release and all the accompanying jazz?

rapid sparrow
#

bumping version => using Env var from GH in setup.py

rapid sparrow
analog kettle
#

@rugged chasm did you figure out the rest of your docker thing?

rugged chasm
#

I tried a different approach and can't get it to work

#

it has some permissions errors that I don't understand

#

I figured out how to start a container correctly with the normal endstone/endstone but now I want to use a different work dir and a different bedrock_server (-s) dir but it says it has some permission errors

#

this is my code:

    def start_container(self, playerxuid):
        folder_path = f"/home/skyblock/server_instances/{playerxuid}"
        print(f"Starting container '{folder_path}'")
        self.logger.info(f"Starting container '{folder_path}'")
        client = docker.from_env()
        container = client.containers.run(
            "endstone/endstone",
            detach=True,
            name=f"sbi_{playerxuid}",
            ports={"19150/udp": None},
            volumes={
                folder_path: {"bind": f"/server_instances/{playerxuid}/", "mode": "rw"}
            },
            working_dir=f"/home/skyblock/server_instances/{playerxuid}",
            command=f"endstone -y -s /home/skyblock/server_instances/{playerxuid}/"
        )
#

[2025-01-18 19:21:33.057 INFO] [LinuxBootstrap] Loading index from the remote server... [2025-01-18 19:21:33.082 ERROR] [endstone._internal.bootstrap] [Errno 13] Permission denied: '/home/skyblock/server_instances/(my name)/tmp9ilyjrfp' Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/__init__.py", line 26, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/__init__.py", line 73, in cli exit_code = bootstrap.run() ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/base.py", line 199, in run self._install() File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/base.py", line 162, in _install self._download(self.server_path) File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/base.py", line 79, in _download with tempfile.TemporaryFile(dir=dst) as f: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/tempfile.py", line 677, in TemporaryFile file = _io.open(dir, mode, buffering=buffering, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/tempfile.py", line 670, in opener fd, name = _mkstemp_inner(dir, prefix, suffix, flags, output_type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/tempfile.py", line 256, in _mkstemp_inner fd = _os.open(file, flags, 0o600) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PermissionError: [Errno 13] Permission denied: '/home/skyblock/server_instances/(my name)/tmp9ilyjrfp'

#

and that's the error

analog kettle
# 
FROM endstone/endstone
COPY bedrock_server /home/endstone/bedrock_server
USER root
RUN chown -R endstone /home/endstone/bedrock_server
USER endstone

looks like the endstone user in the container lacks permissions for /home/skyblock

#

@rugged chasm

rugged chasm
#

i think

analog kettle
rancid schoonerBOT
#

Dockerfile lines 87 to 88

# Switch to non-root user
USER endstone```
analog kettle
#

the endstone user doesn't have the permissions it needs for /home/skyblock/server_instances/(my name)/tmp9ilyjrfp

rugged chasm
analog kettle
rugged chasm
analog kettle
rugged chasm
#

you mean USER root?

analog kettle
# rugged chasm you mean USER root?

right.
in a linux system, the root user is basically god
and when you have a docker container, everything is done by the root user by default, unless you use a USER statement to switch to a different one.

#

and when you have a USER statement, all subsequent statements are by that user, unless you switch again with another USER statement

#

and whichever USER it is last is the one that does whatever the image does when you run it

rugged chasm
#

ah okay

#

so just instead of USER endstone

#

I do USER root?

analog kettle
rugged chasm
#

wut

analog kettle
rugged chasm
#

how do I do that?

#

using root before FROM endstone/endstone and then back to USER endstone?

analog kettle
rugged chasm
#

😭

analog kettle
#

I'm saying that because I believe in you and because you'll learn more if you make the final connection

#

not to be confused with the rainbow connection ohnokermit

analog kettle
#

@rugged chasm how's it coming along?

rugged chasm
analog kettle
#

nice
I like being in forests

placid nacelle
rugged chasm
#

why isn't ```docker
FROM endstone/endstone

USER root

COPY bedrock_server /home/endstone/bedrock_server

RUN chown -R endstone /home/endstone/bedrock_server

USER endstone

#

it still says permission denied for that work folder

analog kettle
rugged chasm
#

Okay give me a sec

#

[2025-01-18 23:02:20.711 INFO] [LinuxBootstrap] Loading index from the remote server... [2025-01-18 23:02:21.061 ERROR] [endstone._internal.bootstrap] [Errno 13] Permission denied: '/home/skyblock/server_instances/VictorJu4751/tmpnijkjx3o' Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/__init__.py", line 26, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/__init__.py", line 73, in cli exit_code = bootstrap.run() ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/base.py", line 199, in run self._install() File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/base.py", line 162, in _install self._download(self.server_path) File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/base.py", line 79, in _download with tempfile.TemporaryFile(dir=dst) as f: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/tempfile.py", line 677, in TemporaryFile file = _io.open(dir, mode, buffering=buffering, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/tempfile.py", line 670, in opener fd, name = _mkstemp_inner(dir, prefix, suffix, flags, output_type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/tempfile.py", line 256, in _mkstemp_inner fd = _os.open(file, flags, 0o600) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PermissionError: [Errno 13] Permission denied: '/home/skyblock/server_instances/VictorJu4751/tmpnijkjx3o'

analog kettle
#

@rugged chasm how do you think you can modify the Dockerfile to grant the required permissions for '/home/skyblock/server_instances/redacted/tmpnijkjx3o'?

rugged chasm
#

Uhm

#

Changing the permissions for that folder?

#

Using root?

analog kettle
#

yes

rugged chasm
#

Root?

analog kettle
#

what do you think RUN chown -R endstone /home/endstone/bedrock_server does?

rugged chasm
#

Makes user endstone to the owner of that folder?

analog kettle
#

yes

#

so what do you need to add to the Dockerfile?

rugged chasm
#

So you are telling me if I do RUN chown -R endstone /home/skyblock/server_instances it'll work?

analog kettle
#

try it.

#

show me the updated Dockerfile before you run it.

rugged chasm
#

Ok i am turning my pc on just for this again

analog kettle
#

turning your PC?

#

ah yes

#

surely turning on your PC is a small price to pay for free help. (and it doesn't go to me.)

rugged chasm
analog kettle
rugged chasm
#

No

analog kettle
#

what OS is your computer?

rugged chasm
#

debian

#

Not my pc but where I am running this

analog kettle
#

do you get the error while trying to build the image, or when you run it?

rugged chasm
#

When running it

analog kettle
#

what command do you use to run it?

rugged chasm
#

endstone -y -s /home/skyblock/server_instances/{playername}/

analog kettle
#

that can't be it. it has to start with docker run

rugged chasm
#

I'm starting the container with python

# 
    def start_container(self, playerxuid):
        folder_path = f"/home/skyblock/server_instances/{playerxuid}"
        print(f"Starting container '{folder_path}'")
        self.logger.info(f"Starting container '{folder_path}'")
        client = docker.from_env()
        container = client.containers.run(
            "my-endstone",
            detach=True,
            name=f"sbi_{playerxuid}",
            ports={"19150/udp": None},
            volumes={
                folder_path: {"bind": f"/server_instances/{playerxuid}/", "mode": "rw"}
            },
            working_dir=f"/home/skyblock/server_instances/{playerxuid}",
            command=f"endstone -y -s /home/skyblock/server_instances/{playerxuid}/"
        )
analog kettle
#

okay, so the "command that starts the container" is the whole thing client.containers.run( ), not just the command= part.

rugged chasm
#

yep

analog kettle
#

can you show the updated Dockerfile?

rugged chasm
#

I didn't try with the updated one

#

wait let me try

analog kettle
#

Please show the updated Dockerfile before trying to build it.

rugged chasm
# 
FROM endstone/endstone

USER root

COPY bedrock_server /home/endstone/bedrock_server

RUN chown -R endstone /home/endstone/bedrock_server

RUN chown -R endstone /home/skyblock/server_instances

USER endstone
analog kettle
#

okay, go ahead and try building that image and running it.

rugged chasm
#

ok

#

oh

analog kettle
#

oh?

rugged chasm
#

` => ERROR [4/4] RUN chown -R endstone /home/skyblock/server_instances 0.7s

[4/4] RUN chown -R endstone /home/skyblock/server_instances:
0.164 chown: cannot access '/home/skyblock/server_instances': No such file or directory

Dockerfile:9

7 | RUN chown -R endstone /home/endstone/bedrock_server
8 |
9 | >>> RUN chown -R endstone /home/skyblock/server_instances
10 |
11 | USER endstone

ERROR: failed to solve: process "/bin/sh -c chown -R endstone /home/skyblock/server_instances" did not complete successfully: exit code: 1`

analog kettle
#

maybe just do RUN chown -R endstone /home/

#

if /home/skyblock gets created when the container starts, hopefully endstone will already have perms for it

rugged chasm
analog kettle
#

build that image and try running your python code to run the image; if you get an error at either point, show the whole entire error message.

rugged chasm
#

okay thanks

#

error

#

` => ERROR [3/3] RUN setfacl -R -d -m u:endstone:rwx /home 0.4s

[3/3] RUN setfacl -R -d -m u:endstone:rwx /home:
0.159 /bin/sh: 1: setfacl: not found

Dockerfile:4

2 | USER root
3 | COPY bedrock_server /home/endstone/bedrock_server
4 | >>> RUN setfacl -R -d -m u:endstone:rwx /home
5 | USER endstone

ERROR: failed to solve: process "/bin/sh -c setfacl -R -d -m u:endstone:rwx /home" did not complete successfully: exit code: 127`

#

ry said something about entrypoints

analog kettle
#

hold on.

rugged chasm
#

im holding on very tight

analog kettle
#

first do docker run --rm -it --entrypoint /bin/bash my-endstone and see if you can run these commands

apt install acl
setfacl -R -d -m u:endstone:rwx /home
rugged chasm
#

ok

#

endstone@a4208adb8de0:~$ apt install acl setfacl -R -d -m u:endstone:rwx /home E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied) E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root? bash: setfacl: command not found

analog kettle
#

oh okay. get out of the container and do docker run --rm -it --entrypoint /bin/bash --user root my-endstone

#

it's the same but with --user root

#

then do the two commands again

rugged chasm
#

root@f398c2149c78:/home/endstone# apt install acl setfacl -R -d -m u:endstone:rwx /home Reading package lists... Done Building dependency tree... Done Reading state information... Done E: Unable to locate package acl bash: setfacl: command not found

analog kettle
# 
apt update
apt install -y acl
setfacl -R -d -m u:endstone:rwx /home
#

try that

rugged chasm
#

ok worked

#

no error

analog kettle
#

all three?

rugged chasm
#

yep

analog kettle
#

okay. don't leave the container yet; try su endstone

#

it should say endstone@f398c2149c78:/home/endstone#

rugged chasm
#

yes

analog kettle
#

try mkdir spamspam

rugged chasm
#

did

analog kettle
#

and the spamspam directory is there?

rugged chasm
#

yes

#

now

#

it wasn't there before

analog kettle
rugged chasm
#

what's that doing?

analog kettle
# 
FROM endstone/endstone
USER root
COPY bedrock_server /home/endstone/bedrock_server
RUN apt update && \
    apt install -y acl && \
    setfacl -R -d -m u:endstone:rwx /home
USER endstone
#

so then rebuild the image and try running the python program

#

this is something I do a lot: enter the container with --entrypoint /bin/bash, figure out the bash commands that fix the problem, and put them in the Dockerfile.

rugged chasm
#

ok ok ok

analog kettle
#

yes yes yes

rugged chasm
#

I'll try running it

#

nah that's crazy it doesn't work

#

[2025-01-19 00:07:58.582 ERROR] [endstone._internal.bootstrap] [Errno 13] Permission denied: '/home/skyblock/server_instances/VictorJu4751/tmpchimt1fn' Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/__init__.py", line 26, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/__init__.py", line 73, in cli exit_code = bootstrap.run() ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/base.py", line 199, in run self._install() File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/base.py", line 162, in _install self._download(self.server_path) File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/base.py", line 79, in _download with tempfile.TemporaryFile(dir=dst) as f: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/tempfile.py", line 677, in TemporaryFile file = _io.open(dir, mode, buffering=buffering, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/tempfile.py", line 670, in opener fd, name = _mkstemp_inner(dir, prefix, suffix, flags, output_type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/tempfile.py", line 256, in _mkstemp_inner fd = _os.open(file, flags, 0o600) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PermissionError: [Errno 13] Permission denied: '/home/skyblock/server_instances/VictorJu4751/tmpchimt1fn'

analog kettle
#

wtf

rugged chasm
#

What type of degree do you need to understand this program it's insane why isn't it just working

analog kettle
#

@rugged chasm try this command

docker run -it --rm \
    -v /home/skyblock/server_instances/player42:/server_instances/player42/:rw \
    -w /home/skyblock/server_instances/player42 \
    --entrypoint /bin/bash \
    my-endstone
rugged chasm
#

and then endstone? It asks me if I want to download the bedrock server and if I say yes it says the exact same thing

analog kettle
#

see if you can mess with files in /home/skyblock/server_instances/player42

rugged chasm
#

it says permission denied

analog kettle
#

I wonder if the problem is that Docker doesn't have permission to write to the file system at the mount point

rugged chasm
#

c:

analog kettle
# 
docker run -it --rm \
    -w /home/skyblock/server_instances/player42 \
    --entrypoint /bin/bash \
    my-endstone
#

see if you can run endstone -y -s /home/skyblock/server_instances/player42/ in there

rugged chasm
#

endstone@67cca49361b4:/home/skyblock/server_instances/player42$ endstone -y -s /home/skyblock/server_instances/player42/ [2025-01-19 00:18:24.722 INFO] [LinuxBootstrap] Loading index from the remote server... [2025-01-19 00:18:24.897 ERROR] [endstone._internal.bootstrap] [Errno 13] Permission denied: '/home/skyblock/server_instances/player42/tmptyxcvh9o' Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/__init__.py", line 26, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/__init__.py", line 73, in cli exit_code = bootstrap.run() ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/base.py", line 199, in run self._install() File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/base.py", line 162, in _install self._download(self.server_path) File "/usr/local/lib/python3.12/site-packages/endstone/_internal/bootstrap/base.py", line 79, in _download with tempfile.TemporaryFile(dir=dst) as f: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/tempfile.py", line 677, in TemporaryFile file = _io.open(dir, mode, buffering=buffering, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/tempfile.py", line 670, in opener fd, name = _mkstemp_inner(dir, prefix, suffix, flags, output_type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/tempfile.py", line 256, in _mkstemp_inner fd = _os.open(file, flags, 0o600) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PermissionError: [Errno 13] Permission denied: '/home/skyblock/server_instances/player42/tmptyxcvh9o'

#

I bet making facebook was easier >:c

analog kettle
#

try this

docker run -it --rm \
    -w /home/skyblock/server_instances/player42 \
    --user root \
    --entrypoint /bin/bash \
    my-endstone
rugged chasm
#

WORKED!!!!!!!!!!!!!!!!!!!!!!!!

analog kettle
#

okay
so the problem is with the endstone user in the container

#

there might be a security risk with running it as root in the container

rugged chasm
#

why security risk what could happen

analog kettle
#

idk, I do AI, not cybersecurity

rugged chasm
#

damn

analog kettle
#

I think if you just add user='root' to the client.contaner.run function in python, it will work
you can instead delete the last USER endstone from the dockerfile and rebuild it.

#

hopefully, someone will come along and tell you why running it in the container as root is a bad idea, and what the solution is.

rugged chasm
#

...

rugged chasm
analog kettle
rugged chasm
#

the folder /home/skyblock/server_instances/player42

#

it's empty

analog kettle
#

can you show the updated python code?

rugged chasm
analog kettle
#

that's not a final solution.
that command removes the mounted volume.

rugged chasm
#

ok I'll try with user="root"

#

it works

#

but the port is 19132

#

and still nothing in player42

#

I can't take this anymore I am gonna sleep now

#

good night and thank you for your help

#

I am coming back tomorrow

thorny pasture
#

I am developing a CLI application in Python and I am struggling with an issue I don't fully understand. The CLI is installed via pip via a console_scripts entry point declared in setup.py. The CLI works as expected when used in a command prompt (Windows), but when it's being called by another application (in my case Unreal Engine), it keeps on showing a Window for a brief moment. I can share the CPP code that is responsible for running the process in the host application if that helps, but I was wondering if anyone knows what's up. Also let me know if there is a better channel to post this question.

rapid sparrow
rapid sparrow
#

It is not python specific issue at all I believe

rapid sparrow
# thorny pasture I am developing a CLI application in Python and I am struggling with an issue I ...

https://stackoverflow.com/questions/12010103/how-do-i-launch-a-program-from-command-line-without-opening-a-new-cmd-window
This stack overflow I believe contains the answers

Stack Overflow
How do I launch a program from command line without opening a new c...

I'm trying to programmatically execute an external file from cmd using this command:

START "filepath"
Where "filepath" is the path of my file. It opens fine but it also open a new command prompt ...

#

Flag /b looks like a solution

#

Or optionally design your program to work through network calls instead of cli calls

thorny pasture
hearty storm
#

Hey everyone!
I'm trying to understand why when I run the following make command on a build of a Docker Image I always get the error of poetry not installed event though poetry is installed correclty.

This is my makefile section trying to execute:

    @echo "Installing dependencies with Poetry..."
    @source "$(VENV)/bin/activate" && poetry install || (echo "Error installing dependencies. Ensure Poetry is installed and a pyproject.toml file exists." && exit 1)
    @echo "Dependencies installed."
    @pip install "fastapi[standard]"
    # @deactivate```

Dockefile:
`RUN  make install`

And this is the error:

------
> [8/8] RUN make clean && make venv && make install:
0.191 Cleaning...
0.483 Removed .VENV
0.487 Creating VENV
3.333 VENV created
3.337 Installing dependencies with Poetry...
3.339 /bin/bash: line 1: poetry: command not found
3.340 Error installing dependencies. Ensure Poetry is installed and a pyproject.toml file exists.
3.340 make: *** [makefile:25: install] Error 1
------
abws_api.Dockerfile:15
--------------------
  13 |     # TESTING GROUND  -----
  14 |
  15 | >>> RUN make clean && make venv && make install
  16 |
  17 |     # ENTRYPOINT [ "poetry shell", "fastapi dev", "./abwsapi/main.py", "--host", "0.0.0.0" ]
--------------------
ERROR: failed to solve: process "/bin/sh -c make clean && make venv && make install" did not complete successfully: exit code: 2

I'm using Python3.12 , Poetry 2.0.0, Dockerfile

Any guidance would be great please !!
thorny shell
#

"source" doesn't affect the lines that follow it

astral apex
#

Did they even install Poetry?

astral apex
thorny shell
#

just change poetry to $(VENV)/bin/poetry and similarly with pip

hearty storm
#

Here is the entire dockerfile @astral apex :

`FROM fedora:39
ENV PS1='\e[1;36m[API Con]\e[0;32m[\u:\w]$\e[m '
WORKDIR /app

RUN dnf update -y
RUN dnf install make python3 python3-pip curl -y
RUN ln -s python3 /usr/bin/python
RUN curl -sSL https://install.python-poetry.org | python3 -

COPY . /app

ENTRYPOINT [ "poetry shell", "fastapi dev", "./abwsapi/main.py", "--host", "0.0.0.0" ]`

hearty storm
thorny shell
#

you've got a Makefile, yes?

hearty storm
#

Yes

thorny shell
#

one of the rules has more than one command line

#

one of those command lines is "source"

#

that "source" command does not affect the lines that follow it

#

actually

#

I'm trying this simple makefile ya: @source .venv/bin/activate type -a python3 and when I run it I see ```
make ya
make: source: No such file or directory
make: *** [ya] Error 1

hearty storm
#

When I run the following interactively I dont get any error
made dev

My understanding is that when make executes each line, each line runs in the different shell. So by unifying them into a single line they would run under the same shell. Like its done on:
@source "$(VENV)/bin/activate" && poetry install ...

image.png
thorny shell
#

I still think you should ditch "source" and just explicitly spell out the path to the executables. It's one less moving part to worry about.

#

I cannot explain why your makefile works at all. Are you using an unusual shell?

hearty storm
#

Gotcha, ok will try that.

hearty storm
thorny shell
#

nothing unusual about that

#

I have this vague feeling that no project started after the year 2010 should use "make" at all

astral apex
astral apex
thorny shell
#

same reason as no project started after the year 2010 should be written in assembly language

#

I mean, sure, if you're used to it and it fills the need, go ahead. But if you're "In Industry", and you expect teammates to help maintain it ... they'll probably have never encountered it.

hearty storm
#

I work for an org that has decades old apps running and every project has its makefile 🫠

thorny shell
#

Gotta hire old dudes with beards then 🤣

fallow iron
#

Has anyone used one of these paid Django solutions that bake in all the boring but needed stuff (e.g. team accounts, payment plans, etc)? General thoughts on them?

crimson spruce
#

Anybody use AWS Timestream and can help me understand their billing system?
If I make a 10 TCU query for 30 seconds, it just costs me 1/12th of a TCU-hour, right? Or is "Each TCU is billed on an hourly basis" supposed to mean "if you use less than an hour, we round up"?

brazen forge
rough quiver
#

Would THIS be the correct place to discuss coding agents?

thorny shell
#

🤷

#

can't hurt to ask

twilit forum
#

What is a coding agent? 🤔

thorny shell
#

surely it's a spy who is writing up a coded message

visual oxide
#

It's a chemical having a heart attack

echo grail
#

sorry, i can't disclose this information

eager rose
#

if I tell you, I'd have to kill you

heavy knot
#

django,flask and a python script for a Dockerfile

glass ember
#

Good day y'all, are people using black or ruff for linting these days?

#

I understand it's a bit of a loaded question, people may be on legacy code bases and not able to make changes to newer tooling

#

but the reason I ask is that I don't typically work in the Python ecosystem and my knowledge of the tooling is dated

#

I currently use Black as a formatter and Pyright as a linter

#

but looking at the UV documentation for ruff it appears to be a much faster tool for both formatting and linting and I am having trouble determining if it's worth changing over

rapid sparrow
glass ember
rapid sparrow
#

I don't remember ruff being same, it should be Flake8 alternative. Language best practices linter I think

#

Nah it can replace black too

#

U still need pyright/ type checker

glass ember
rapid sparrow
#

Yes. The Ruff linter is compatible with Black out-of-the-box, as long as the line-length setting is consistent between the two.

#

huh. out of the box Black replacement 🤔

#

Ruff can be used as a drop-in replacement for Flake8 when used (1) without or with a small number of plugins, (2) alongside Black, and (3) on Python 3 code.
and drop in replacement for Flake8

#

may be its adoptation would not be as hard as i imagine

glass ember
#

Ok interesting

#

so instead of Black + isort + Pyright it would be ruff format + ruff lint + pyright?

rapid sparrow
glass ember
#

damn, okay cool, thanks very much

#

I'm not familiar with flake8 at all

rapid sparrow
#

ruff check --fix i like that they offer some errors to fix automatically

#

this feature i don't remember being present in other python linters

#

(i hate fixing issues manually 😅 )

rich summit
wary lance
#

Hi! I’m new to this server and posting on the most relevant channel.

I’m trying to learn KerrGeoPy, a library to simulate orbits in Kerr geometry. There are no tutorials online and I’m having a hard time understanding the documentation. Wizards please help…

glass ember
#

@rapid sparrow I replaced isort and black with ruff, now its pyright + ruff format + ruff lint

#

it's nice

#

I do ruff as pre-commit and pyright as a github action

rapid sparrow
hidden kayak
glass ember
hidden kayak
glass ember
# hidden kayak Opinion on mypy vs pyright?

I was originally using mypy, but then my neovim had the pyright lsp installed which led me to research the difference.

Mypy is the og type checker for python - it’s much older. Pyright is a Microsoft product, it’s built for speed.

#

After integrating pyright, by GitHub action was twice as fast

hidden kayak
hidden kayak
umbral jacinth
#

Hi

abstract fox
#

I'd like to integrate kafka into my Python django project. Which library do you recommend? Which is the most reliable?

https://github.com/confluentinc/confluent-kafka-python OR https://github.com/wbarnha/kafka-python-ng

GitHub
GitHub - confluentinc/confluent-kafka-python: Confluent's Kafka Pyt...

Confluent's Kafka Python Client. Contribute to confluentinc/confluent-kafka-python development by creating an account on GitHub.

GitHub
GitHub - wbarnha/kafka-python-ng: Fork for Python client for Apache...

Fork for Python client for Apache Kafka. Contribute to wbarnha/kafka-python-ng development by creating an account on GitHub.

tawny ledge
#

need python helper some one

twilit forum
tawny ledge
#

@twilit forum brother can u help please i request u

twilit forum
#

Ask your question

tawny ledge
#

i have a github files is this possbile can you update

#

@twilit forum ?

#

@twilit forum are u alive?

twilit forum
#

I don't see a question yet

frank fractal
glass ember
#

Does anyone know if nh3 is the only alternative to bleach for input sanitization?

visual oxide
#

For input sanitation isn't really the right terminology

#

or the right technology, instead you have a separation of markup and content with a template engine, and escape all content regardless if it's input or not

glass ember
visual oxide
#

Don't do that, just escape it

glass ember
#

if I have an empty array of white listed elements, meaning no html is allowed, then it should return the contents without any markup

visual oxide
#

If it's in a JSON endpoint, insert it into a DOM node as text

#

endpoints shouldn't do any escaping

#

Do it at the last possible moment before making html

glass ember
#

why not?

visual oxide
#

Because otherwise it's ambiguous if a field value needs inserting with or without escaping

glass ember
#

well from the users perspective, nothing changes

#

if a bad actor attempts to store markup in the db, it's stripped prior to

visual oxide
#

It's not markup it's plaintext

glass ember
#

right, but it will come back as markup when fetched no?

#

wont the browser just render it as markup if it recognizes valid tags?

visual oxide
#

What do you mean when you say endpoint

glass ember
#

there's a the server and the client. The server has a series of endpoints exposed. One of which allows the client to upload job notes, which should only be plain text. There's an additional endpoint to get that job note by id, or all job notes by user id

visual oxide
#

And you send back what exactly?

#

A JSON object with a string field?

#

And how do you render it on the client? This is a web app?

visual oxide
#

Or insert into an element as text

void eagle
#

Is there any Python package for serving Git repositories? I know at the most simplistic I can use Apache plus the git-http-backend to do this, but I just wondered if there's any Python project to do similar.

golden ice
#

Hi Guys, I work on a project where we use LLM's like gpt-4o models, and use AWS as cloud provider. I would like to explore GraphRAG and implement it in our project, could you please let me know some good resources to study about it, or how have you implemented it.

Thanks in advance!!

thorny shell
gentle solstice
rapid sparrow
#

Gitea now comes even with its own CI runners 😏 which support even actions written in go

void eagle
#

Thanks. I had just wondered. My use case is serving Git repos from a (say) Flask application. I'd like some routes to hit git repos and all other routes should get processed by Flask functions.

It seems that the two simple ways to do that are to either have a web server (e.g. Apache) sit in front and redirect Git requests to CGI git-http-backend, or alternatively from the Flask app, invoke the git-http-backend CGI directly.

#

Gitea looks nice, but if too nice for what I'm implementing. 🙂

rapid sparrow
# void eagle Thanks. I had just wondered. My use case is serving Git repos from a (say) Flask...

just put in front of Flask, Nginx (or Caddy) web server
and you could make with it redirecting to different services. Two ways of common ways to redirect rqeuest is supported

  • Domain based. gitea.example.com to gitea, api.example.com to api for example . Your cookies/authorizations can still easy travel between subdomains of same domain
  • Path based, create some rule that example.com/api to api, the rest to gitea or smth else

https://www.digitalocean.com/community/tutorials/how-to-configure-nginx-as-a-reverse-proxy-on-ubuntu-22-04
https://caddyserver.com/docs/quick-starts/reverse-proxy

How To Configure Nginx as a Reverse Proxy on Ubuntu 22.04 | Digital...

This tutorial will demonstrate how to set up a reverse proxy using Nginx, a popular web server and reverse proxy solution. You will install Nginx, configure …

Reverse proxy quick-start - Caddy Documentation

Caddy is a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go

#

Technically you can make probably reverse proxy such features implemeted at the level of Flask if u need that

rapid sparrow
void eagle
#

These are all good suggestions thanks. The reason I'm thinking of having the Flask application itself make the decisions about whether to divert a route off to some (CGI) git backend is that in my scenario git repos could appear at any path (e.g. https://example.com/team/user/project/blah/repo1/) so the app knows it's a repo and not a Flask page to render and so invokes the CGI. Doing this in web server (Caddy/Nginx/Apache) seems infeasible to me.

rapid sparrow
rapid sparrow
#

if the site does not forbid it

#

in this way your interface can still be present at outside

void eagle
#

No, becuase the client could be the Git command line. Iframe isn't much use if you're wanting to clone a repo, no?

rapid sparrow
void eagle
#

hmm, don't think that's right. If I use git-http-backend then it will be clonable.

#

Anyway, thanks for your help! I'll see how I get on.

rapid sparrow
#

http is used only by.. students/novices and people wishing to clone public repo they don't have internal access to

void eagle
#

You don't know my use case. Thanks for your help though.

hasty drum
#

Unsure if this is a good place to ask. But does anyone have any experience with setting up jupyter-notebook docker images and adding custom kernels?

#

I'm working on a project where a basic lambda calculus kernel has been made, and we want to add this to out deployed jupyter server

late root
#

Hello does anyone have a custom code that integrates Marauder and CC1101 on an M5StickC Plus2? Im looking to run both attacks from the same device. Thanks!

thick scroll
#

I would like advice on virtual environments for developing and deploying Python web apps. I got started with conda due to an interest in data science, and it's been fine to start with, but I'd like to avoid gotchas down the road e.g. I'm unsure of the consequences of combining pip and conda installs. (Mac OS for dev, Ubuntu Linux for deploy)

marsh acorn
#

Are you still working mainly in data science?

#

If it ain't broke don't fix it I would say. But if you're curious start with plain old pip and venv. Once you understand their limitations you can make informed decisions on 3rd party tools.

thick scroll
#

I wasn't working in data science, just getting familiar with the field. I primarily develop web apps in Racket, and will likely be switching to Python. No need for a virtual environment in Racket. I used a couple in Ruby ages ago.

#

Is there any issue using conda for one project and venv/pip for another?

marsh acorn
#

Cool! No problem with switching between conda and pip. Other than it might be a little harder to go deep on both.

#

I use conda for my data projects often, because all the resources online do. I just find myself having to look up the commands.

thorny shell
#

wow, racket ... you must be impervious to pain 🤣

thick scroll
# thorny shell you mean https://racket-lang.org/, right?

Yes, that Racket. I've been developing with it full time for about 5 years, and have enjoyed every minute of it :) It's a long story re: switching, but one of the main factors is simply Python's ecosystem, especially since we'll be adding data science and machine learning. And, I prefer being an application developer instead of a web framework developer :)

#

re: my previous question, Poetry is looking pretty good for virtual environments.

thorny shell
#

Poetry is what I use. But my head is being turned by "uv"

thick scroll
#

It's really a pity the Racket leadership is so focused on research and education, the runtime is one of the most solid I've ever used - processes simply do not crash nor leak memory, and the threading concurrency is top notch (requires one process per core), but recently starting to develop yet another library that already exists in Python was too much :)

thorny shell
#

I seem to recall being frustrated at the lack of handy libraries, which drove me to python

thick scroll
#

Just installed Poetry, and I'm a bit confused about how to install a specific Python version in the virtual environment. My MacOS comes with Python 3.9.0. The Poetry docs state:

Unlike with other packages, Poetry will not automatically install a python interpreter for you. If you want to run Python files in your package like a script or application, you must bring your own python interpreter to run them.
With conda, I would specify the Python version when creating a new virtual env. How is this done with Poetry?

#

I thought it was a best practice to leave the OS installed Python alone, and install the desired Python version in the virtual env.

gentle solstice
thick scroll
#

The Poetry docs also seemed to indicate you could use conda, but that just seems very odd.

gentle solstice
#

!pip pbs-installer is another option (requires python)

rancid schoonerBOT
gentle solstice
#

Its what pdm uses

gentle solstice
thick scroll
#

search for "conda activate"

gentle solstice
#

That's for using poetry to install packages into a conda managed venv

thick scroll
#

I know it's premature, but at this point, I give the lead to conda :) It just makes sense to specify the Python version when instantiating a new virtual env to me.

gentle solstice
#

It only uses conda as an example. It's really for any tool that automatically activates your venv

thick scroll
#

I thought Poetry was the tool to activate my virtual env.

gentle solstice
#

Do poetry env use 3.12

#

https://python-poetry.org/docs/managing-environments/#switching-between-environments

Managing environments | Documentation | Poetry - Python dependency...

Managing environments
Poetry makes project environment isolation one of its core features.
What this means is that it will always work isolated from your global Python installation.
To achieve this, it will first check if it’s currently running inside a virtual environment.
If it is, it will use it directly without creating a new one. But if it’...

gentle solstice
#

It wants to hide away the concept of venv activation

thick scroll
#

Sorry if I'm being dense, but consider the following conda workflow:

conda create -n foo python=3.11
conda activate foo
# install stuff and run apps in the virtual env
conda deactivate

What is the equivalent workflow with Poetry?

gentle solstice
# 
poetry env use python3.11
poetry install

poetry shell
python x.py
exit
# or
poetry run python x.py
thick scroll
#

Interesting - thanks! The Poetry startup doc had me start with poetry new demo, and then told me I'm on my own for Python :)

#

Maybe they should've started with poetry env use python3.11 instead ?

#

Your first line failed unfortunately.

# 
% poetry env use python3.11                

Poetry could not find a pyproject.toml file in /Users/badkins/python-projects or its parents
gentle solstice
#

You need to run new or init first

thick scroll
#

Won't that cause Poetry to set the Python version to whatever my system version is then?

gentle solstice
#

Check the --help text

thick scroll
#

I did new, and then I get:

% poetry env use python3.11

Could not find the python executable python3.11
gentle solstice
#

You need to have python3.11 installed

thick scroll
#

I may just stick with conda, Poetry needs some serious work on their onboarding.

#

Maybe my thinking has been influenced by conda being my first experience, but it does make sense to first create the virtual environment, and then install the desired Python version into that env.

gentle solstice
#

You need to install the python you need

thick scroll
#

Yes, I understand that, I just don't know why Poetry forces that.

gentle solstice
#

Because platforms are different

thick scroll
#

why not, poetry new demo version=3.11 or something similar

gentle solstice
#

The Python versions UV and pdm install have caveats compared to the system versions

#

Check new --help

thick scroll
#

Which is the best method of installing a Python version that works the same on MacOS (my dev) and Linux (my deploy) ? i.e. brew is MacOS specific.

gentle solstice
#

What distro?

thick scroll
#

Ubuntu

gentle solstice
#

Use apt

#

Combined with the deadsnakes ppa

#

You can also use pyenv

#

That builds python locally and installs it in your home dir

thick scroll
#

I'll give that a shot. Just seems weird to have pyenv manage what seems to be a virtual env of sorts (just with Python alone), and then Poetry manage separate virtual environments, that are depended on the pyenv environment for a specific Python version.

gentle solstice
#

It builds python itself

thick scroll
#

yes, but also allows multiple versions of Python on the same machine, right?

gentle solstice
#

You need to install a Python version before you can use it to make a venv

#

Yes

thick scroll
#

And Poetry knows the internals of Pyenv enough to be able to link different Python versions to different venv's ?

gentle solstice
#

Pyenv adds shims to your PATH, so poetry should see it

thick scroll
#

Please don't tell me I need to first activate a specific Python version before activating a specific Poetry venv!

gentle solstice
#

Um...

#

You can activate multiple Python versions at once

thick scroll
#

Oh, it gives different names I suppose.

gentle solstice
#

pyenv global 3.{9..13}

#

Run that once

#

Its bash shorthand for pyenv global 3.9 3.10 3.11 3.12 3.13

thick scroll
#

But once I get all this setup , when I activate a Poetry venv, I just run python, and the right version is invoked, right?

gentle solstice
#

Yes

thick scroll
#

ok, cool. Thanks for your patience - much appreciated!

#

<sigh> I really should've chose the Linux method for pyenv even when installing on MacOS. brew just installed dozens of packages as part of the pyenv install, so I'm pretty sure it screwed up my postgres for the Nth time. My fault for forgetting how screwed up homebrew is :(

thorny shell
#

I thought pyenv was just a single shell script

#

and I struggled with postgres on MacOS until I decided to always just run it in docker -- now my life is great, and I'm sitting on a tropical beach with a cold drink 😁

thick scroll
#

I was fortunate, my postgres is ok. Probably because the last time brew "did its thing", it upgraded to a new enough version.

thick scroll
#

poetry shell doesn't work with Poetry 2.0.1. There is poetry env use but that requires specifying a Python version vs. activating the one associated with the project. There is also poetry env activate, but that just displays a command to activate the env. I'm wondering what Poetry buys me beyond venv

thorny shell
#

poetry run $SHELL is probably an OK substitute

thick scroll
#

poetry env info seems to indicate I've already activated a virtual env, but I have no idea how I did that. Either new did it, or the command to display the command to activate a venv actually activated it instead of just showing the command. I'll continue to keep an open mind about Poetry, but it's one of the least intuitive tools I can recall using.

#

Some web research seems to indicate a lot of people favor Poetry, and conda has terrible code, etc., but I got up and running w/ conda in seconds, and it always worked the way I expected activate -> work in the env -> deactivate I can see why it appeals to academics and non-software engineers.

#

Any idea how to deactivate a Poetry virtual env?

gentle solstice
thick scroll
#

None of the commands listed with poetry list seem like candidates.

gentle solstice
#

If you prefer conda, keep using it.

thick scroll
#

To be frank @gentle solstice having to redirect the output of a command to the shell seems utterly ridiculous - why would a virtual env manager not have a command to activate the virtual env?

gentle solstice
thick scroll
#

re: keep using conda, my impression is I may regret that down the road

gentle solstice
#

To make shell activation work with a command, it needs to be a bash function.

#

Poetry is not written in bash

thick scroll
#

So maybe conda needs to be more intrusive to allow easier activation?

gentle solstice
#

Maybe it could provide an init function you can source in your zshrc, but I don't think so.

thick scroll
#

Do you know how to deactivate?

gentle solstice
#

In conda?

#

Poetry?

thick scroll
#

lol - in Poetry. With conda it's unsurprisingly conda deactivate

gentle solstice
#

Poetry uses virtualenv

#

If you activated the venv, it will have the name in the prompt. Then you can run the deactivate function.

#

!pip virtualenvwrapper this might interest you

rancid schoonerBOT
thick scroll
# 
badkins@create web-demo % deactivate
zsh: command not found: deactivate
gentle solstice
#

Its bash functions to manage venvs

gentle solstice
thick scroll
#

it had the name in the prompt, and poetry env info seems to indicate a virtualenv is active

gentle solstice
thick scroll
#

Ah! Yes, that threw me off. sourcing the command does then allow me to deactivate

gentle solstice
thick scroll
#

So pyenv for managing multiple Python versions on the box, venv for the virtual environments, and Poetry is just for package dependencies?

gentle solstice
#

More or less

thick scroll
#

Appreciate the tip on the additional wrapper, but I think I'm understanding things ok now, and would rather not introduce another layer. I don't expect to have too many different environments. I had 5 Racket apps in production with a single environment w/o issues, so I think I can keep my Python envs to a minimum.

gentle solstice
#

Feel free to write some zsh functions to help

thick scroll
#

Yeah, I'll automate the tedious stuff later as I refine my workflow.

marsh acorn
# thick scroll Sorry if I'm being dense, but consider the following conda workflow: ``` conda c...

This is that same workflow in uv:

# this command installs the requested python if it does not already exist on your machine
➜  uv venv --python 3.9
Using CPython 3.9.21
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate
➜  source .venv/bin/activate
(scratch) ➜  uv pip install requests
Resolved 5 packages in 1.90s
Prepared 2 packages in 195ms
Installed 5 packages in 30ms
 + certifi==2025.1.31
 + charset-normalizer==3.4.1
 + idna==3.10
 + requests==2.32.3
 + urllib3==2.3.0
(scratch) ➜  deactivate
lethal pond
#

Modern python tooling seems to be moving away from "activating" venvs, which is a good thing in my eyes.

#

Activating venvs is nothing but a source of problems for new users

#

Node is held in much better light for nearly the same system, simply because it's a standard there to do $tool run app that automatically uses the correct node_modules (I know it's not the only factor, but certainly one of the factors)

#

with poetry, uv, hatch, pdm, etc you're supposed to use $tool run app to run your app with the correct venv. It's simple and cross platform.

marsh acorn
#

That's valid. I just have to remember it when I'm running linters and formatters so I don't mess up everything with another version. blobsweat

lethal pond
#

That's only a concern if you have a global version installed too.

#

Otherwise black/ruff . will just fail without $tool run

marsh acorn
#

I usually do 🙂

#

So I can lint scripts that aren't in a venv

lethal pond
#

Makes sense

marsh acorn
#

Not a big deal, just a new habit I have to pickup

lethal pond
#

I think pipx and uv tool install let you global install with custom prefixes

#

Which would let you have a global version with a different name

marsh acorn
#

I think I agree that it's the way to go, using $tool run ...

reef zealot
#

What's the best way to setup a monorepo that publishes multiple packages to PyPI?

Additionally, some packages depend on others. These would be the packages:

packages/
  inngest/
  inngest-middleware-encryption/
  inngest-middleware-sentry/

Where the 2 middleware packages depend on inngest. So when I'm running tests, they'd use the source code in inngest instead of actually installing inngest from PyPI

gentle solstice
#

What's your test runner? tox? nox? github-actions?

reef zealot
#

GitHub Actions + pytest

cloud bough
#

hey anyone currently free to explain about github thru vc or chats? pls dm

visual oxide
#

It's also better to ask your question rather than ask to ask. And we don't help in dms

cloud bough
#

i didnt know mb

stark hamlet
#

do any of you guys run some cron jobs to reclaim disk space (docker) every once in a while if so what all do u prune
i have a script setup for building and deploying on my self hosted runner (on push/merge to main branch) which leads to build cache accumulating
also i do check if some critical files were changed before rebuilding and deploying again

heavy knot
#

runner platform? some offer cache pruning regularly

stark hamlet
heavy knot
#

i run a scheduled cache prune

#

but its provided by ci

astral apex
#

I prune immediately after changing containers

gentle solstice
#

I do docker system prune every month

#

Ot was it system reset?

astral apex
#

New computer every month?

#

Nice

#

I'm 🤏 close to being able to do that

gentle solstice
#

as long as nobody deploys at midnight, it'll be fine

sudden igloo
#

What's everyone's thoughts on continuous deployment where you deploy to production automatically on each new push of a feature VS creating a release of the software to be downloaded and deployed elsewhere manually on VPSs and the like?. I personally feel like the first option requires a level of discipline I am not comfortable with and would prefer to manage the exact resources used on a budget and not have all the details of infrastructure mixed into a pipeline with the code. I feel like it would be more comfortable to tag commits and push whenever a release is desired then take that release and deploy it either manually or with iac like terraform.

thorny shell
#

up to you, really

#

I think people set up automated pipelines when they get tired of doing it manually, or after they've made some bad mistakes

rapid sparrow
# sudden igloo What's everyone's thoughts on continuous deployment where you deploy to producti...

DevOps engineer here.

Manual is not the way. My memory is too much human to remember all details.

Automated is the way.
To staging can be fully automated.
To production must be having one verification. I deploy currently pet projects by just pushing git tags.

Tag like v1.32 triggers deployment to production
v1.32-a1 deploys to staging (to avoid triggering some rate limits had to make deployment to staging with verification too)

Before you deploy to production at work
Important to open git diff of made changes and review what you deploy for sure. Check with people that git committed it, if they made QA for all features

#

--+
Frequent automated deployment mean that chances for Smth bad present is quite small.
And if it is present, easier to discover

Automated deployment to staging ensures people have easier time to validate their stuff

sudden igloo
# rapid sparrow DevOps engineer here. Manual is not the way. My memory is too much human to rem...

I completely agree to an extent that manual isn't the way in general, it's a lot of error prone work and each change or addition to a server is something to keep track of or something to go wrong, and a massive amount of labor in the long run. But I think just having unit/integration/acceptance tests pass would tell you the software you built is fit for use, so like if I build the software, and if I wasn't the one managing/deploying it, then those tests would probably be considered enough for other people to be using it and not expect any bugs that mess with existing features in drastic ways in production. I personally do not want to manually manage upgrading servers software or adding new instances (as in the setup work required, management, etc) but I was thinking maybe I could just do the same quality assurances of continuous deployment, release it to a docker registry and then treat it like trusted third party software and provision infrastructure and make infrastructure as code scripts to deploy it, like from DNS records, to installing docker, setting up stuff for the private repository for downloading it, user accounts, firewall rules etc, and just provision and instance with automatic setup or add more that way, destruct existing ones and bring new ones up. Like so that instead of managing instances and all the software and keeping track, you just delete one and make a new one with the software all done for you. And then you only manage more permanent stuff like databases, message queues, redis, etc.

rapid sparrow
# sudden igloo I completely agree to an extent that manual isn't the way in general, it's a lot...

But I think just having unit/integration/acceptance tests pass would tell you the software you built is fit for use, so like if I build the software, and if I wasn't the one managing/deploying it, then those tests would probably be considered enough for other people to be using it and not expect any bugs that mess with existing features in drastic ways in production.
First, auto tests can't reveal problems with infra bugs, unless u write End to end testing, which is painful and expensive to maintain

Secondly, important to automate infrastructure is keeping Infrastructure as a Code. Git versioned together with the rest of your app code. It makes documentation as a code how your application is build, released, tested and deployed

I personally do not want to manually manage upgrading servers software or adding new instances (as in the setup work required, management, etc)
modern infra devs do not want that too. That's why Container Scheduling Abstractions are very prefered... we would prefer to deploy application to AWS ECS, or docker swarm, or as last resort to Kubernetes, but not to deal with Linux machines under the hood.
Because procedurally updating linux machines is very... dirty experience. Declarative infrastructure is way faster to apply itself
It is good when Linux Machines are built as server images, u don't get attached to them and can easily swap one version to another one without any problems

#

Ideally you don't see thos server machines at all, at least if u made app for work

#

u just interact with container scheduling abstraction, and all the state is kept in external databases

#

Thus your apps run stateless, and server machines can be easily destroyed/recreated at any time 😏

#

Thus=> you can even easily have turned on autoscaling of those machines, and not care about manually scaling them

rapid sparrow
# sudden igloo I completely agree to an extent that manual isn't the way in general, it's a lot...

but I was thinking maybe I could just do the same quality assurances of continuous deployment, release it to a docker registry
Saving to docker registry is the way. In this way u can relatively easily rollback to previous version.
Also... this choice is universally compatible with all container scheduling systems.

and then treat it like trusted third party software and provision infrastructure and make infrastructure as code scripts to deploy it, like from DNS records, to installing docker, setting up stuff for the private repository for downloading it, user accounts, firewall rules etc, and just provision and instance with automatic setup or add more that way, destruct existing ones and bring new ones up. Like so that instead of managing instances and all the software and keeping track, you just delete one and make a new one with the software all done for you. And then you only manage more permanent stuff like databases, message queues, redis, etc.
Basically i have already at my pet projects situation like that
my CI just builds image, saves, marks as production deployment one, and in one comand app is rotated to new deployed container 😏
docker service update --image darkwind8/darkstat:production production-darkstat-app
https://github.com/darklab8/fl-darkstat/blob/23348746dd4a1126f1b4e8c6b1cc57d96ae57fdd/.github/workflows/docker-production.yml#L126

All the configuration of infrastructure, DNS, environment variables are done through terraform for me
https://github.com/darklab8/fl-darkstat/blob/master/tf/production/main.tf
i like terraform for its declarative nature, and ability to review what WILL be changed in comparison to new code, what will be appleid changed?
I like that if i delete terraform code => It auto deletes resoruces
I don't need to write deletion scripts 😏 state is kept clean by terraform nature, no longer present infra code deletes its resources on next terraform apply

rancid schoonerBOT
#

.github/workflows/docker-production.yml line 126

docker service update --image darkwind8/darkstat:production production-darkstat-app```
sudden igloo
# rapid sparrow Thus=> you can even easily have turned on autoscaling of those machines, and not...

I am building servers for a game with rooms and I don't think I can do something server less to auto scale them. The design also relies on people being routed to specific server instances with other players for chatting, rooms and queries, so more servers with their caches, and own room instances = more players and they need to be up all the time. It's possible that the design could be changed so that the API and query stuff could be standard and server less but the game stuff would still need servers and I think need more provisioned manually or at least with some machines explicitly. But I am opting to vps style stuff and getting the most out of my money.

rapid sparrow
rapid sparrow
#

we just can autoscale them freely because we keep no state on them

#

Stateless apps, is the secret ingridient in average for good infra

#

Keeping state in some remotely accessable storage

sudden igloo
#

I can sort of do that in this design to an extent. There is no state aside from caches on them, and all business logic will use the one database atomically, and when they start they'll load and stream in everything they need. So one can be destructed and players routed to a different one when it happens if I can make it shut down gracefully. Everything that is stateful to the server doesn't matter is only local.

sudden igloo
# rapid sparrow > But I think just having unit/integration/acceptance tests pass would tell you ...

I feel like infra bugs are a different issue. Like I feel like I can guarantee that given docker is installed and an image is run that nothing will go wrong, but that anything to that point and maybe on the server like in configuring it with the iac scripts could be wrong. But at that point my software works and the issues in infrastructure should be minimal, I am pretty sure there's ways to test that some of the infrastructure will be deployed correctly too. I remember seeing stuff on acceptance testing terraform stuff.

rapid sparrow
# sudden igloo I feel like infra bugs are a different issue. Like I feel like I can guarantee t...

Infra bugs...

  • Is correctly configured firewall at server side and cloud provider side
  • correct env vars are inputted for access to infra resources RabbitMQ/Redis/SQS and etc, credentials to every kind of them
  • is web server correctly configured for its reverse proxy working, and able to actually proxy pass traffic to your http/grcp whatever ports
  • is TSL actually working, and automated renewal does work
  • does some app has memory leak and breaks other apps by taking over the server because u haven't configured memory hard limits?
  • does some app breaks because consumes suddently far more ram and gets killed because of hard limits
  • did you configure directly access from public to private network
  • did u configure load balancer correctly working and it is able to pass traffic to relevant application
  • did u correctly configure IAM permissions
  • does networking configured correctly and the app in question is able to access by network necessary resources

The thing with infra bugs, that anything can be wrong there, and u just need to handle Infrastructure as a code for sanity, who changed what, when it was last working and what changed?

#

It is not really different from main application code management

#

Also u need good monitoring systems for complex applications/infra in order to manage it sanely, preferably with automated alerts

#

It is a question of basically size of app/infra

#

Things can be fairly simple for a single app, that has small amount of users

#

Things become different when u have dozens of developers, independently releasing their own applications for working with million of users lets say

#

Because i like simplicity, i enjoy a fair share of my infra code even in my pet projects though 😄 Which have just thousands users.
It is still nice not remembering how to manage them in terms of deployment/infra code, everything being done automatically for all my apps

#

If you don't have recorded infrastructure as code, then u keep naturally things in your head how to manage it. And head can only fit so much

rapid sparrow
# sudden igloo I feel like infra bugs are a different issue. Like I feel like I can guarantee t...

Like I feel like I can guarantee that given docker is installed and an image is run that nothing will go wrong
problem with this approach. that it works only if your app is not depended on any external infra resoruces to function.
And not requiring any extra infrea features on top for its functioning.

server infra intended for large amount of users grows in ... many extra infra resources/features to function correctly, and everything should be correct, otherwise will be downtime

sudden igloo
# rapid sparrow Infra bugs... - Is correctly configured firewall at server side and cloud provid...

I agree that all of those are very important, some depend. Firewall configuration can be tested independently of application code, as the concern is weather the code in docker works as expected imo. So you could start it on your infrastructure for a test and see if it is accessible I think (I am aiming for using pulumi so I think I can) so that's easily testable I think. Correct env vars could be messed up easily for sure, I do imagine it happening not initially but if changed in the future though, maybe a smoke test to verify everything is up and configured correctly would address that? And tests for the iac code like that it will pass in the correct values from the correct resources. Is the web server configured correctly, I was kind of hoping not to have to set one up or pay for hardware to do load balancing and use DNS instead of possible but I will if I have to, but I think if I had one it would be easy enough to test. I feel like a memory leak might still be a concern with testing the application.

rapid sparrow
sudden igloo
rapid sparrow
sudden igloo
frigid jetty
#

I just worked a bit more on a linter I was writing (for Suricata rules, not for Python, but written in Python) and it made me wonder: What distinguishes a good linter from a bad linter? Anyone got suggestions to improve my linter?

crimson spruce
#

Oh, and the ability to automatically fix the fixable issues, when directed to by the user.

frigid jetty
#

So, basically it shouldn't raise issues for which the user thinks "don't bother me with this"

frigid jetty
rapid sparrow
crimson spruce
#

That's what I would measure a good linter with. But less of "don't bother me" and more of "you're wrong/I disagree".

#

Yeah, ability to disable rules globally/per file/function/line is a must.

frigid jetty
#

Or well, the per-line part. Already had global

crimson spruce
#

I personally would also love a stateful linter, I'm not aware of any that exist: Where it asks me and I can say "no, don't ask me for this piece of code and this rule again", without having to put a magic comment in the code.

#

(Same for formatters)

frigid jetty
crimson spruce
#

It could store the information externally in some other file(s) that you commit.

frigid jetty
#

I think there's a PHP linter that allows you to pass a baseline of pre-existing errors to ignore, stored in a seperate file

#

psalm I think its called

#

Really great for working with legacy code bases

crimson spruce
#

Thanks for the hint, I'll have to look into whether that does what I mean. Unfortunately/thankfully I don't write PHP these days.. :D

frigid jetty
#

Haha, me neither anymore

#

Thanks for your thoughts

twilit forum
crimson spruce
twilit forum
#

Yeah, this is very difficult/impossible to get right, so the fallback should probably be to drop the waiver when it's unclear if it's a new error or not.

Manually written waiver files is common within hardware development.

last tartan
#

hey there, i've always dev on pycharm, since it's what one of my friend showed me. The thing is i find it in most use case way too complicated for me and to be fair it's too much. It gets really messy and since my little dev thing is starting to be big. I want to change and setup something properly and better understand. Same for the version of my project, i would like to organize everything properly. Does anyone got any tutorial somewhere, course or review i should follow/read ?

wet cliff
#

Hey devs I want to archive this type of Subtitles with python or any language . I'm using whisper to transcribe. But don't know how to style, add animation, Blending diffrance transperant type effect on it . Can anyone help me guide me ?

opal verge
astral apex
#

It was me

#

I said it

eager rose
thorny shell
#

eww sounds tricky

#

IME most dot directories were not designed to be put into git

crimson spruce
#

I've seen enough people do it with that one specifically 🙄
In this case, I'd obviously would want a solution that's editor-independent. Ideally even linter-independent.

eager rose
gentle solstice
slim maple
# crimson spruce I personally would also love a stateful linter, I'm not aware of any that exist:...

Honestly I don't think using codes is such a big problem?

ignore = [
    "RUF012", # Mutable class attributes should be annotated with `typing.ClassVar`
    "E501", # Line Length
    "A003", # Class attribute ... is shadowing a python builtin
    "A005", # Module shadows a built-in. We shouldn't care since almost all of our modules are namespaced
    "D", # pydocstyle
    "PD", # pandas-vet
    "ISC001", # Ruff Format
    "COM812", # Ruff Format
]

With code though - yeah, but it's usually a select few codes

crimson spruce
slim maple
#

mypy is a bit better when it comes to ignoring errors, e.g. ignore[arg-type]

crimson spruce
crimson spruce
slim maple
#

Honestly you should just ignore some of the errors that are bugging you a bit too much IMO

#

I don't care about line length, in most cases it's some kind of giant string, if it would bother me I'd just break it down later 🤷‍♂️

#

And I certainly don't want to mark all my functions, methods, classes and modules with noqa: DWhateverNumber to ignore the mandatory docstrings, right?

#

Even in the codebase that I'm currently working on I only have around 70 noqas per 32k loc, for me this isn't a big problem

crimson spruce
slim maple
crimson spruce
#

(E.g. I recently ran across this with "don't assign a lambda to a name". I generally agree, but for symmetry reasons in this case it looked better.)

slim maple
foggy ocean
#

Hello everyone,

I’d like to share an open-source tool we’ve been working on called Makim: https://osl-incubator.github.io/makim/

Makim is a task automation tool, similar to Make/Makefile and Ansible, but with a more practical approach. Many people use Make for task automation, but since it was originally designed as a build system, its usage for general automation can be cumbersome. In many ways, Makim is similar to Ansible but is designed to work with a single file. In the future, we may add support for splitting .makim.yaml into multiple files, similar to how Ansible handles configuration.

If anyone is interested in trying it out, feedback would be highly appreciated.

Here’s an example in Google Colab that allows you to test it online without installing anything locally:

https://colab.research.google.com/drive/1m131pqi6Bjq8Hp8YN_Cbuyezn61f19lB?usp=sharing

I’ll be happy to hear any thoughts or suggestions.

Thanks for your time, and apologies if this isn’t the best place to ask for feedback.

Google Colab
rapid sparrow
foggy ocean
# rapid sparrow nice. Going further into writing Self hosted + provider agnostic locally executa...

I guess it could be done integrated with docker. we've been developing another tool that was built on top of docker-compose, called containers-sugar (or just sugar): https://github.com/osl-incubator/sugar ... we are planning to add support for docker swarm soon.

GitHub
GitHub - osl-incubator/sugar: Simplify the usage of containers

Simplify the usage of containers. Contribute to osl-incubator/sugar development by creating an account on GitHub.

rapid sparrow
#

for dev env, i would prefer keeping things with Compose only. Extra abstraction is not justifying extra upkeep

#

Terraform/Pulumi so far implement best choice for deployment management due to ability to remember state/having typing features/ability to review plans

#

and dev env is too much simple to justify wrapper around

foggy ocean
#

Hi @rapid sparrow thanks for your feedback! Really appreciate that. I will review the link you shared, thanks! I will also review about pulumi, thanks!!

With sugar, basically we have the same interface/commands for dev/staging/ci/prod etc .. so the team doesn't need to know different technologies.

Once a variable in the .env file specifies which env (ci, testing, dev, prod, etc) should be used, the user doesn't need to worry selecting any env, which avoid human errors.

It can also allow hooks, so before running the given sugar command, it can execute other commands .. for example of you need to prepare something in the host before executing the containers.

rapid sparrow
# foggy ocean Hi <@370435997974134785> thanks for your feedback! Really appreciate that. I wil...

dev/staging/ci/prod etc .. so the team doesn't need to know different technologies.
compose alone is not good enough for staging/production running because of not having advantages that Terraform/Pulumi have.
You can't as i mentioned review what changed from previous deployment. (what on next apply will be deleted/created/recreated)
Once a variable in the .env file specifies which env (ci, testing, dev, prod, etc) should be used, the user doesn't need to worry selecting any env, which avoid human errors.
Even worse, you don't have ability that Terraform/Pulumi provide to pull secrets dynamically from remote storages and injecting them into terraform
Solving deployment at the level of Terraform/Pulumi is perfect because we can define all infra resources and input infra variables directly as a code into our containers. Human mistake is pretty much becomes smallest possible.
As well as Terraform module abstractions help to simplify with infra code due to making reusable code that have well defined input/output typed stuff

#

So what u suggest, using for staging/production such yaml wrapper as sugar, is the same as going to... more prehistoric times and using Cloudformation as yaml to manage infra for example.

#

such approach does not work well (because has no state remembering, insufficient secret management capabilities, too much yaml)

#

Or at least it will be hardly accepted for people who prefer to see less Yaml in their infrastructure code 😏

#

TLDR: in my opinion with sugar tool you concentrated on a wrong direction? for usage cases that other tools have more perfect solution

#

Your original Makim tool was more interesting in its idea

  • since you made pretty much Makefile in yaml (upgrade from confusing original Makefile in my opinion)
  • and ability to execute pretty much array of CI code locally through it. (restrictions of Github/Gitlab needing to be executed only remotely is a restriction, and your tool does bring liberation from that problem)
#

Sugar on another hand so far is too much closely tied to Docker features, and i would be hesitant to bring it
Just because it exchanges one array of yaml for me to another array of yaml for me
While closely being tied to docker features. I see fragility here of not justified abstraction

  • its features may fall apart because of incompatibility between itself and docker
  • and all the features it offers can be done by other tools better (terraform/pulumi for staging/production)
  • or not using the tool itself brings me more stability of interacting with docker/compose directly
  • i don't see pretty much enough value Sugar brings in its features to justify its addition
#

P.S. to be fair i also question sanity of bringing infra tool made not in Golang. (or at least made without compiling to a single binary file to download and run as executable)

  • If u used golang u could have used its Docker bindings without extra lib wrapper.
  • And your tool would have been installable into Docker image/CI as a single binary file, without needing to bring python and its libs, which can be very difficult to install in another docker image. Python installing from sources is very nasty complicated, any mistake in C compiler missconfiguration/library dependencies/Smth OS level and not even possible to compile it with all its features
    • For those reasons highly likely i will continue using only Taskfile (for yaml makefile) https://taskfile.dev/usage/ , since it is easy to bring into any image / easy for devs to install in a cross language fashion. I discovered actually that non python devs have very big problems to install anything python related locally (while static binaries made by go are easy to download by any devs). So your choice to utilize Python for infra tool, makes it pretty much restricted to python devs only, unless u will provide some options to install it through apt/and etc stuff
foggy ocean
#

hi @rapid sparrow , thank you so much for your feedback, I really appreciate that.
I will re-read it again to ensure I am not missing any part of what you said.

but I would like to hightlight some points:

  • I am not against terraform, and although I am not an expert, it is a great tool, and from our community we also hosted a talk about that.
  • about secrets, I am very aware about that, I am checking some options for integrating that in the pipeline in the way that it would select the proper secrets for the given environment name.
  • about the ability to review what was changed from previous deployment, tbh I never needed that, because everything is on github, so all the history is there, but if you have any reference/link about that, I would love to learn more about that.
  • and again about terraform, I really think it is great tool, but for a small team, with small budgets with mainly non experienced developers, using terraform is not a feasible option.
foggy ocean
#

@rapid sparrow , and about your last comment, that is a very interesting point.

  • so using python, it helps me in terms of productivity, as I am the maintainer of many projects, I prefer to pick the one I can implement a feature or fix a bug as fast as possible.
  • if I would forced to change the programming language, for performance issues for example, I would prefer rust over golang, but maybe for personal reasons
  • usually I don't have makim installed inside a container, usually I have it outside in the host, and because most of my projects are in python, we always have python environment very well defined. so it is not a big deal for the projects I am maintaining. but anyway we have plans to package makim to other package managers like apt

thank you so much all your inputs, I will keep that in mind! feedback from people with different setups help me to understand more other needs and workflow. appreciate that.

rapid sparrow
#

Docker compose 1.0 was made in python too before they migrated to go 🙂

foggy ocean
#

that makes sense.

because I also work with data stack, and a lot of libs are moving their cores from c++ to rust, I have this personal bias. also the linux kernel is doing this move as well.

Docker compose 1.0 was made in python too before they migrated to go

that is right XD

we needed to create a rebundle for the v2 to be installed easily with python https://pypi.org/project/compose-go/

#

PS: we need to fix the CI that cut new releases XD

rapid sparrow
# foggy ocean hi <@370435997974134785> , thank you so much for your feedback, I really appreci...

about the ability to review what was changed from previous deployment, tbh I never needed that, because everything is on github, so all the history is there, but if you have any reference/link about that, I would love to learn more about that.
Example.
https://github.com/darklab8/fl-darkstat/blob/6d4c7aa8c0a26f0f995a3368b9d89ea42cdcbb65/tf/modules/darkstat/dns.tf#L14
all infra code is stored in git too.
And lets say i have environment Production
https://github.com/darklab8/fl-darkstat/blob/master/tf/production/main.tf
It was applied before, currently state is synced to code.

module "dns" {
  source = "../../../../infra/tf/modules/cloudflare_dns"
  zone   = var.zone
  dns_records = concat([{
    type    = "A"
    value   = var.ipv4_address
    name    = var.stat_prefix
    proxied = false
    }
    ],
    var.rpc_prefix != null ? [{
      type    = "A"
      value   = "${var.ipv4_address}-smth"
      name    = var.rpc_prefix
      proxied = false
    }] : [],

Now lets imagine i changed value of DNS A records to include -smth suffix and run tofu apply again?

https://gist.github.com/dd84ai/614eb2edb304bc00c702adc084b1f326

module.darkstat.module.dns.cloudflare_record.record["darkgrpc"]: Refreshing state... [id=e28fee810fb6b7a8aba8267013c6f006]
module.discovery_dev.data.external.disco_dev_webhook: Read complete after 5s [id=-]
module.darkstat.module.dns.cloudflare_record.record["darkstat"]: Refreshing state... [id=e6e9473271f2f056e6f6a152290fea64]
module.darkstat.module.dns.cloudflare_record.record["darkrelay"]: Refreshing state... [id=04142eea253fe9200102c28e39b31675]
module.darkstat.module.dns.cloudflare_record.record["apigateway"]: Refreshing state... [id=a39498b88ea3b2089e2330edbec09066]

You can see my tool refreshing state, what is already synced and it writes me precisely what will be CHANGED, from previous state to new apply?

rancid schoonerBOT
#

tf/modules/darkstat/dns.tf line 14

name    = var.rpc_prefix```
rapid sparrow
# 
  # module.darkstat.module.dns.cloudflare_record.record["darkgrpc"] will be updated in-place
  ~ resource "cloudflare_record" "record" {
        id              = "e28fee810fb6b7a8aba8267013c6f006"
        name            = "darkgrpc"
        tags            = []
      ~ value           = "37.27.207.42" -> "37.27.207.42-smth"
        # (10 unchanged attributes hidden)
    }

  # module.darkstat_vanilla.module.dns.cloudflare_record.record["darkgrpc-vanilla"] will be updated in-place
  ~ resource "cloudflare_record" "record" {
        id              = "465b9216e33034c9ad4267eba653a55e"
        name            = "darkgrpc-vanilla"
        tags            = []
      ~ value           = "37.27.207.42" -> "37.27.207.42-smth"
        # (10 unchanged attributes hidden)
    }

Plan: 0 to add, 3 to change, 0 to destroy.
#

Now i could delete some infra code entirely.

#

tofu apply command will show that next apply will be expecting to DESTROY no longer present in code resources

# 
# module.darkstat_vanilla.module.dns.cloudflare_record.record["apigateway-vanilla"] will be destroyed
  # (because cloudflare_record.record is not in configuration)
  - resource "cloudflare_record" "record" {
      - allow_overwrite = false -> null
      - created_on      = "2025-02-09T19:42:27.490505Z" -> null
      - hostname        = "apigateway-vanilla.dd84ai.com" -> null
      - id              = "ed6b2647f4b0d5b818e98b31da51b357" -> null
      - metadata        = {} -> null
      - modified_on     = "2025-02-09T19:42:27.490505Z" -> null
      - name            = "apigateway-vanilla" -> null
      - proxiable       = true -> null
      - proxied         = false -> null
      - tags            = [] -> null
      - ttl             = 60 -> null
      - type            = "A" -> null
      - value           = "37.27.207.42" -> null
      - zone_id         = "b637e66e10cbec997625a2b78c7170fc" -> null
    }

all related resources will be destroyed

#

i have a powerful feature at my disposal where i apply in new deployment only what is Changed / going to be NEW created / or Destroyed from no longer present code

#

i have ability to REVIEW

#

what changed from previous deployment apply

#

if i delete infra code i do not need to write deletion of infra resources, they are all cleaned up after themselves on their own on next apply

#

that helps to keep very clean infrastructure, that is ensured in being sync with infra code

#

and it helps Reviewing that changes are good actually? as u give extra check that applied stuff to production is good to apply

#

This Plan Review feature basically helps to have human step ensuring extra quality assurance check.
Plus the ability to remember state itself (with refreshing for it to be up to date) makes way cleaner running of infra in staging/production

foggy ocean
#

got it! thanks for the explanation, I really appreciate that. I will play a bit with that this week to learn more about that.

rapid sparrow
# foggy ocean hi <@370435997974134785> , thank you so much for your feedback, I really appreci...

I am not against terraform, and although I am not an expert, it is a great tool, and from our community we also hosted a talk about that.

so using python, it helps me in terms of productivity, as I am the maintainer of many projects, I prefer to pick the one I can implement a feature or fix a bug as fast as possible.
Some extra note, as mentioned infra tools are overly tied to Golang in some cases. Terraform Providers are for example writable in golang only. Option of writing them in any other language is not even present at all as far as i am aware. No any library/binding thingy to make it happen in smth else.

foggy ocean
#

got it! good to know! I thanks for the note!

rapid sparrow
# foggy ocean <@370435997974134785> , and about your last comment, that is a very interesting ...

usually I don't have makim installed inside a container, usually I have it outside in the host, and because most of my projects are in python, we always have python environment very well defined. so it is not a big deal for the projects I am maintaining. but anyway we have plans to package makim to other package managers like apt
i would imagine a good feature of makin is supposed to be ability for easy installation in any Docker image or any CI instrument to reuse its commands in CI or running app container
Some CI tools like Gitlab basically by default run by Docker containers its jobs too.
So.. same problems in CI with installations as to installations to some Docker container
Technically Github Actions has ability to run its jobs in Docker containers too, so even in GH can be faced same issue
For apps that prefer to build and run themselves purely in docker (pretty much all modern web apps), this issue in CI (with difficulty to install smth related to Python) will be encountered even for CI Shell executor running way

foggy ocean
#

got it .. so maybe offering an option to install it from a apt-ish package manager, and maybe adding makim via github action markplace, maybe it could be a good start to address the issues you pointed

rapid sparrow
# foggy ocean got it .. so maybe offering an option to install it from a apt-ish package manag...

if packaging to apt, it will always remain not always easy installable (as in comparison to static binary from Golang/Rust)
Some docker images can be using Alpine (that has different manager apk)
Some can be using different other distros (Centos based stuff with Yum packager)
It will be extra plentiful of different effort to have it somehow running for all those cases
Thing compiled for specific distro like debian can be not working for alpine due to it having entirely different C dependencies and etc.
Some distros can be not having package managers at all

At least with golang (and probably rust too) there is ability just to turn off CGO features, and making same binary file equally runnable on all Linux distros, regardless which C libs it has installed.
That pretty much simplifies distribution of a tool, to a same curl https://install.sh or wget https://binary_file can be offered equally for all OSes

#

So a thing to be aware about

#

Easy installability is important feature simplifying adoptation of a makefile like tools (or infra tools in general) 😏

foggy ocean
#

right, you mentioned that before. makes sense. i will think a little bit more about that.

#

thank you for all your comments. really appreciate that!

foggy ocean
rapid sparrow
sand meteor
#

Hi there,

I having an issue where git
Consider there are 2 branches
B1 and B2
I have created PR from B2 to B1 and reverted it.
Again I made changes in B2 and created PR
From B2 to B1
Some of the files and var are not moving and there is an conflict

#

There are several files changed but it's not shown as change

foggy ocean
rapid sparrow
#

https://taskfile.dev/
Those guys ship their stuff for all OSes with embedded Linux shell and with basic Linux programs/commands that make same command running even on Windows

Home | Task

Task is a task runner / build tool that aims to be simpler and easier to use

#

When u make Smth in go, u can cross compile from Linux to Windows/macos without visiting another OS
It is not guarantee it will work though, as it can be still having screwed file paths or smth

#

You use python... Supposedly cross OS language. We could have expected cross OS functioning from even that choice too when u ship it in Pip

foggy ocean
#

For now makim works on linux and macos ... the executable, for now, works just on linux. We had plans to support windows .. but for now we use a lib called sh, internally, to run all the commands .. and it does't work on windows. If anyone wants to contribute to add support to windows, it wil be more than welcome 🙂

#

If we are accepted by gsoc this year, we can also add this project idea

twilit forum
hoary cipher
#

Folks, I need some feedback regarding my project setup.
I need sagemath as a library, so I'm using pixi to get it from conda. I also need some other libraries from pypi. I've never used tox but it seems to be what people use, so I tried to set it up but the tox environments don't contain sage.
Is there a way to fix this or am I on a wrong path?

#

This is my pyproject.toml

[project]
name = "my_project"
requires-python = "~= 3.12"
version = "2.0.0"

[build-system]
build-backend = "hatchling.build"
requires = ["hatchling"]

[tool.pixi.project]
channels = ["conda-forge"]
platforms = ["linux-64"]

[tool.pixi.pypi-dependencies]
my_project = { path = ".", editable = true }

[tool.pixi.tasks]
lint = "ruff check --fix"
fmt = "ruff format"
style = { depends-on = ["fmt", "lint"] }

[tool.pixi.dependencies]
sage = ">=10.5,<11"
pixi-pycharm = ">=0.0.8,<0.0.9"
cryptography = ">=44.0.1,<45"
attrs = ">=25.1.0,<26"

[tool.pixi.environments]
default = { solve-group = "default" }
dev = { features = ["dev", "test", "lint"], solve-group = "default" }

[tool.pixi.host-dependencies]
hatchling = ">=1.27.0,<2"
sage = ">=10.5,<11"

[dependency-groups]
test = ["hypothesis", "pytest>=8.3.4,<9", "tox>=4.24.1,<5"]
lint = ["ruff", "pyright", "validate-pyproject[all]"]
dev = ["pre-commit"]

[tool.ruff]
preview = true
line-length = 100

[tool.ruff.lint]
select = [
    # ...
]

[tool.pyright]
include = ["src"]
pythonVersion = "3.12"

[tool.tox]
requires = ["tox>=4.24.1"]
env_list = ["3.12", "lint", "type"]

[tool.tox.env_run_base]
description = "run tests under {base_python}"
dependency_groups = ["test"]
commands = [["pytest"]]

[tool.tox.env.type]
description = "run type check on code base"
dependency_groups = ["lint"]
commands = [
    ["pyright", "src", "tests"],
]

[tool.tox.env.lint]
description = "run linting on code base"
dependency_groups = ["lint"]
commands = [
    ["ruff", "check", "src", "tests"],
    ["ruff", "format", "--check", "src", "tests"],
    ["validate-pyproject", "pyproject.toml"],
]
#

The error:

type: commands[0]> pyright src tests
/home/user/PycharmProjects/my_project/src/my_project/main.py:7:6 - error: Import "sage.all" could not be resolved (reportMissingImports)
#

Please ping me if you have a suggestion

hoary cipher
#

I decided to remove tox

slate bramble
#

Can anyone suggest best resource to learn docker

rapid sparrow
#

originally i learned by another book, but its quality seriously degraded in recent years 🤔

#

as far as i checked this is the only rich and up to date enough alternative

#

with correct given advices

slate bramble
gentle solstice
#

!pep 751 will make it even better

rancid schoonerBOT
hoary cipher
#

I have a conda dependency, that seems to be too much

gentle solstice
#

tox-dev has a ton of useful packages

hoary cipher
#

Nah, it’s dead

#

Doesn’t even work with tox >=4

gentle solstice
#

pip package is archived too

#

what conda package?

hoary cipher
#

sage lemon_holding_back_tears

marsh acorn
hoary cipher
#

Im extremely happy with pixi so far

gentle solstice
#

is that another test runner?

hoary cipher
#

No, another dependency management thing

gentle solstice
hoary cipher
#

But it lets me combine conda and pip packages pretty easily

#

Yeah

gentle solstice
#

you could just include conda as a command

hoary cipher
#

Replaced uv with pixi because it doesn’t support conda

hoary cipher
gentle solstice
#

seems like a good alternative that supports conda

hoary cipher
#

I wonder whether I’ll miss any tox features. The main thing is running tests in different environments, right? Seems like pixi can do that

rapid sparrow
#

i wish to have my local environmen running tests fast for a single env

drowsy field
#

dumb question here, how do you guys go about testing a (.py) server in docker image launched by ec2 instance to verify if port is actually working?

rapid sparrow
# drowsy field dumb question here, how do you guys go about testing a (.py) server in docker im...

Monitoring systems

😏 Healthcheck services for dumb minimal solution
https://github.com/ivbeg/awesome-status-pages
Like Gatus for example https://github.com/TwinProduction/gatus

Distributed monitoring systems like Datadog/Grafana for advanced solutions (they have features for logging/tracing/metrics/profiling)
(Synthethic test runs in Datadog at least)
(Some metric prometheus scrapping in grafana)
+Configured alerts obviously in both solutions

hoary cipher
#

Folks, what is the correct way to configure logging? I have multiple modules and some third-party modules that I want to have on different levels. It seems wrong to set it in a dictConfig because I'll have to edit code anytime I need a change in verbosity. Do I put a json file in the repository? Do I set up a bunch of env vars?

rapid sparrow
#

enriron.get("library_name","warn")

#

for your own web app at least

#

if we have situation that u develop a library on your own... then it is a bit more tricky.
needing just to adhere to std lib logging stuff, and initializing loggers as __name__ of a current file

#

it will open access for apps to adjust logging based on import path to specific sections of app

#

in its turn if u develop cli/desktop app, it is not unheard of configuring logging via actually file 😏

hoary cipher
#

I'm developing a library and using it

full jay
#

anyone know the reason for conda installed in a WSL2 not letting install packages in new env but only in base?

rigid girder
#

Hi everyone,
Has anyone managed to set up OAuth2 with Casdoor in Dify? I'd really appreciate it if you could share your experience or any guidance. Thanks in advance!

glass ember
#

Is the logger module that is built into python adequate for production applications? Or do you guys reach for something specific like structlog?

rapid sparrow
# glass ember Is the logger module that is built into python adequate for production applicati...

Is the logger module that is built into python adequate for production applications?
yes and no
In average case yes, it is standard to which we adhere.

but not enough for serious production applications. For them we need JSON logger transoformer on top, because all modern logging systems are able to parse json logs and provide full features to logs navigations through that in the simpliest way
https://pypi.org/project/python-json-logger/
Because json logging is compatible with ANY modern distributed monitoring system, which are used for serious bleeding edge production

#

JSON logging compatible with any modern logging system is the way

#

Or do you guys reach for something specific like structlog?
it is confusing to see with which modern system it is compatible. i am highly suspicious of it

#

Otherwise Structured logging / Json logging is the way / logging enriched with plentiful of context regarding its fields. std logging powered by json stuff is pretty much already providing it

glass ember
#

I've only briefly read through structlog's docs, but it seemed like a structured logging module

rapid sparrow
glass ember
#

but you're recommending python-json-logger?

glass ember
rapid sparrow
# glass ember right right, or datadog?

we use datadog. With datadog too, it is able to parse all json logging and gives very easy quering by all json params
Though we migrate from datadog to grafana for money reasons 😏

#

anyway, json logging gives very smooth experience to datadog log quering

glass ember
#

ok cool

rapid sparrow
# glass ember Is the logger module that is built into python adequate for production applicati...

Out of curiosity did read through structlog. It has nothing shared with Structured Logging i was speaking about
It is interface dedicated for human readable console log, so not scalable at all
plus vendor lock in into its library how to manage stuff, instead of working on default logging compatibility
I would not use it ever (Just because i have access to distributed logging systems at least / will configure them even for pet projects )

glass ember
sterile ruin
thorny shell
#

in tuh rest ing

rapid sparrow
# glass ember Is the logger module that is built into python adequate for production applicati...

https://youtu.be/1X3dV3D5EJg We run this stuff as conglomerate of helm charts in kubernetes cluster/AWS ECS
But there exists simplified docker image to run it all https://github.com/grafana/docker-otel-lgtm

YouTube
Fireship
Grafana is the goat... Let's deploy the LGTM stack

Get up to 67% off VPS at Hostinger. Use code FIRESHIP for an extra discount at https://hostinger.com/fireship

#linux #programming #softwareengineer

💬 Chat with Me on Discord

https://discord.gg/fireship

🔗 Resources

Docker 101 https://youtu.be/rIrNIzy6U_g
Full Deno Course https://fireship.io/courses/deno
Grafana on GitHub https://github.com/...

▶ Play video
GitHub
GitHub - grafana/docker-otel-lgtm: OpenTelemetry backend in a Docke...

OpenTelemetry backend in a Docker image. Contribute to grafana/docker-otel-lgtm development by creating an account on GitHub.

#

Going to try if it will be simple option for my pet projects to run this way 😏

short peak
short peak
#

it is not the right channel, I think there are general purpose help channels for these kind of questions. About your problem, it will loop until it reach the "break" statement. Instead of having a while True consider having a condition like while j != "n"

rapid sparrow
#

for 5-8 euro u can get a lot of hardware resources

Screenshot_20241223_201158.png
rapid sparrow
#

needing to use another deployment solution -_-

#

not really big issue though

sterile ruin
#

isnt Hetzner in Germany or something?

#

Suppose it depends where you are, but doesn't seem great for latency (I'm in the US)

crimson spruce
sterile ruin
#

ahhh. fair enough

thorny shell
#

Mine's in "Hillsboro OR" iirc

heavy knot
#

hmm thats fun

twilit forum
#

How to setup python-parts of project in a git repo that has python code organized like this:

tools/tool_a/main.py
tools/tool_a/tool_a/foo.py
tools/tool_a/tool_a/bar.py
tools/tool_a/test/test_foo.py
tools/tool_a/test/test_bar.py

Where currently

  1. main.py is an executable usually run when standing in the root of the repo.
  2. foo uses imported such as
    from bar import barium
# or
tool_a. from bar import barium

This is just a developer tool, not packaged and release to any customers. These are more such tools in the repo tool_b, tool_c...

Any suggestions for handling file structure and imports so that tools such as pylsp, mypy and pylint can understand where the python code is and be invoked as simply as possible.

visual oxide
#

Run run python -m tool_a

#

Then use relative imports only throughout the project

twilit forum
# visual oxide Then use relative imports only throughout the project

Thanks.

Relative import inside tool_a/tool_a/*.py? For example from .bar import barium?

I don't want to change the command to execute the tool unless there is no other way (as it would be a breaking change). How can I achieve that after moving it into __main__.py? Symlink from old filename?

visual oxide
twilit forum
#

But I also need to specify that tools/tool_a is a module for this to work. How do I do that?

twilit forum
rocky rapids
#

hey all! I just open sourced a python module that automates Unit and E2E testing with just a few liens of YAML. Would be interesting for the community?

visual oxide
#

You can also add __init__.py

twilit forum
# visual oxide It's a module by default

Standing in the repo root , how would ```
python -m tool_a

find `tools/tool_a` without specifying `tools/`?
Do you have verbatim example of the `sys.executable` suggestion?
#

Oh, if I start it as tools/tool_a/main.py --options from command line, it would "just work"?

visual oxide
twilit forum
#

Modifying PYTHONPATH in the source code is something I would like to avoid if possible. It seems like having a symlink might be simpler than this suggestion.

Is there someway to setup pyproject such that the developer can activate the developer python environment which automatically modifies pythonpath or finds it anyway through "project structure"?

rapid sparrow
#

Flat python packaging structure is the best

#

everything works out of the box, without any hacks

#

Flat structure assumes adding __init__.py in every python folder Except REPOSITORY ROOT.

#

u can use then relative paths

#

and absolute path which always start from the ROOT of repository

#

executing files takes python3 main.py in the root of repository (usually the most comfy choice)
And if u wish executing some nested ones in subfolders, then u execute them as python3 -m some_lib.smth, going through module paths like this (executing in such way will help absolute path being still recognized from the root of repo)

twilit forum
# rapid sparrow executing files takes `python3 main.py` in the root of repository (usually the m...

All of this sounds nice, but we have a project where python is just 20% of all the code. Sprinkling __init__.py everywhere sounds hacky and polluty. Are those files really necessary for this behavior to work?

Running the tools as python -m tool_a is probably not beneficial for us. Apart from being non-backwards compatible we also don't want to care if the tool is written in python or any other language.

#

If we decide to change the implementation of some tool from python to c++, the command line to invoke that tool should not change.

rapid sparrow
rapid sparrow
#

makes easy to write unit tests for related python code in such way

rapid sparrow
# twilit forum If we decide to change the implementation of some tool from python to c++, the c...

If we decide to change the implementation of some tool from python to c++, the command line to invoke that tool should not change.
u can wrap python code into thing like

your_exe (named in any way at linux) (even without any extensions)

#!/usr/bin/env python 

import subprocess
import argparse
import os

# u will be needing hacking sys.path, if those files will not be at repo root
# i think it is  forgivable in my point for entry point files having some sys.path hacks

some python code

if __name__=="__main__":
   entrypoint to code
#

it will be executable just as ./your_exe (relative or abs path)

twilit forum
twilit forum
odd roost
#

has anyone ever used a library called scheduleduty before? my team and i are looking to programmatically import a csv file into our pagerduty account that holds information on users and creates a PD schedule based on the csv data. I found this library but it seems pretty outdated. seems to only work with python versions 3.9 and lower, and once i got it imported i wasn't able to do a typical from abc import abc statement in my python file as it didn't seem to pick up the library somewhere.

Am I missing something?

rapid sparrow
#

it looks abandoned

#

Usually always a good idea to check Repositories of what u install to see if it is still developed or abandoned

#

discovering software abandoned 10 years ago is not a infrequent

trail cedar
#

Is requirements.txt being replaced by pyproject.toml ?

#

Or is it too early to state that

marsh acorn
#

No, they serve different purposes. But it might be replaced by pylock.toml.

#

!pep 751

rancid schoonerBOT
twilit forum
#

Pylint question. I want to support relative import in source code folders without __init__.py.

Currently when using relative imports pylint reports error E0402. [2]

This section of documentation tries to explain what to do. What does it mean? And exactly what should I do? If a new source code folder appears, would I need to update my pylint cmd?

[1] https://pylint.readthedocs.io/en/latest/user_guide/usage/run.html#with-implicit-namespace-packages

[2] https://pylint.readthedocs.io/en/latest/user_guide/messages/error/relative-beyond-top-level.html

twilit forum
dawn jolt
#

is there a reasonable python checker that checks imports - a broken import passes
py_compile and pyflakes checks ...
this code has no warnings:

# 
async def foo():
    from os import thing_that_doesnt_exist as dothing
    result = dothing()
    return result 
foo()
short peak
stray musk
#

Can someone help with Building a Solana meme trading bot

visual oxide
twilit forum
#

Maybe nothing, but we removed a lot of __init__.py files earlier because we didn't see any need for them.

I thought pep 420 was about making init-files obsolete, but I realize there might be more to it skimming the first page of it. https://peps.python.org/pep-0420/

Python Enhancement Proposals (PEPs)
PEP 420 – Implicit Namespace Packages | peps.python.org

Namespace packages are a mechanism for splitting a single Python package across multiple directories on disk. In current Python versions, an algorithm to compute the packages path must be formulated. With the enhancement proposed here, the import ...

twilit forum
# visual oxide What's wrong with `__init__.py`

Also it is a bit strange to have to put ~200 empty files all over the place. Is it the best way to solve this? It means that as soon as someone does a new folder with python code inside, they have to add an init file there (and in parent folder as well?!).

visual oxide
twilit forum
visual oxide
warm iris
#

what are some good tools for security?

sharp ibex
rapid sparrow
# warm iris what are some good tools for security?

Not giving access to production database 😏
And not giving access for everyone to servers.
Configure proper automated CI CD instead.
Set always not standard (or at least some) password to every infra resource (at least at the level of basic auth or smth)
And don't bind docker ports to host unless u intend exposing them, firewall will not cover you
Configure cloud provider level firewall to server (in case u still exposed smth u did not mean to anyway, or forgot to set password to smth)
Using your SQL library parameter binding to avoid SQL injections 😏

stable hatch
#

Is this the right place to ask a Docker/Compose question?

I've started running a Flask app in Docker. And I use Docker Compose to load a Postgres database as well. It works fine when I first run it. The problem is:

  • docker compose will build the app if it finds it, calling the image "mydirectory:servicename"
  • on subsequent runs, compose always runs with what's in the cache, so it won't pick up my code changes
  • if I manually rebuild the app directly to get new code into an image, it gets a "sha:whatever" image name, which compose doesn't use (presumably because the name doesn't match)

How can I get this process to work, ideally so that compose spots when a rebuild is needed and does it itself, or as a next best option, so that I can manually rebuild and have compose pick it up properly?
I can force a rebuild via compose if I destroy the image that it made previously, but that seems to be working around the problem rather than solving it.
I know there are also workarounds with volumes and bind mounts and file watches etc, and while I expect to use these going forward, I want to resolve the underlying problem first.

short peak
#

does anyone know an alternative to Kubernetes CronJob? I need that feature but I'd like to avoid having to setup a cluster just for that

#

(beside Celery and plain cronjob)

sharp ibex
astral apex
#

Azure Functions?

stable hatch
sharp ibex
stable hatch
#

Not at my machine right now, but it’s a pretty standard file, generated by docker init. I don’t think there’s a problem with the caching, just that the names of the images default to different things

sharp ibex
terse cove
#

Shilling myself as always, but chancy is a viable alternative with minimal requirements if you're already using postgres in your stack.

thorny solar
#

this is my programe , and about every 5 min alarm@@

import time
import sys

def notify_every_five_minutes():
while True:
print("Reminder: This is your 5-minute notification!")
sys.stdout.flush() # Ensure output is immediately visible
time.sleep(300) # Sleep for 5 minutes (300 seconds)

if name == "main":
notify_every_five_minutes()

rancid schoonerBOT
#
Star / Wildcard imports

Wildcard imports are import statements in the form from <module_name> import *. What imports like these do is that they import everything [1] from the module into the current module's namespace [2]. This allows you to use names defined in the imported module without prefixing the module's name.

Example:

>>> from math import *
>>> sin(pi / 2)
1.0

This is discouraged, for various reasons:

Example:

>>> from custom_sin import sin
>>> from math import *
>>> sin(pi / 2)  # uses sin from math rather than your custom sin
  • Potential namespace collision. Names defined from a previous import might get shadowed by a wildcard import.
  • Causes ambiguity. From the example, it is unclear which sin function is actually being used. From the Zen of Python [3]: Explicit is better than implicit.
  • Makes import order significant, which they shouldn't. Certain IDE's sort import functionality may end up breaking code due to namespace collision.

How should you import?

  • Import the module under the module's namespace (Only import the name of the module, and names defined in the module can be used by prefixing the module's name)
>>> import math
>>> math.sin(math.pi / 2)
  • Explicitly import certain names from the module
>>> from math import sin, pi
>>> sin(pi / 2)

Conclusion: Namespaces are one honking great idea -- let's do more of those! [3]

[1] If the module defines the variable __all__, the names defined in __all__ will get imported by the wildcard import, otherwise all the names in the module get imported (except for names with a leading underscore)
[2] Namespaces and scopes
[3] Zen of Python

thorny solar
rancid schoonerBOT
# thorny solar

Please react with ✅ to upload your file(s) to our paste bin, which is more accessible for some users.

short peak
terse cove
#

0.20.0 will be out today with the built-in metrics collectors and dashboard improvements.

terse cove
dry vale
#

How would I test a pre-commit hook without publishing it to a tag? Use a hash rev?

brazen forge
iron folio
#

Hi.
Anyone got experience with caching uv packages between Podman image builds (assuming dependency list does not change)?
I have painfully slow to download packages and rebuilding the image takes forever. Can't get the caching to work

I tried this (https://medium.com/@shaliamekh/python-package-management-with-uv-for-dockerized-environments-f3d727795044):

...

ENV APP_DIR=/app
WORKDIR $APP_DIR

...

RUN --mount=type=cache,target=/home/$USER/.cache/uv \
    --mount=type=bind,source=uv.lock,target=uv.lock \
    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
    uv sync --frozen --no-install-project

COPY src/ $APP_DIR
COPY pyproject.toml $APP_DIR
COPY uv.lock $APP_DIR

RUN --mount=type=cache,target=/home/$USER/.cache/uv \
    uv sync --frozen

...

but it never actually does cache anything 😦

brazen forge
iron folio
dry vale
gentle solstice
#

$HOME/.cache/uv should work too

#

Not sure if mounts will interpolate vars

iron folio
solar spindle
#

Is possible to upload video on youtube without authorization bcs im using vps

stiff grove
#

Is there any tool to check between charged?

heavy knot
#

hello

#

would anybody like to help me

#

dm me if interested

gentle solstice
#

@heavy knot Keep help requests on-server.

heavy knot
#

this ap

#

p

#

detects if you are entering a password

#

on which site

#

and uses your face

#

to automatically enter the password

#

its like faceid

#

but for windows

#

supported multiple passwords

#

@gentle solstice

#

if ur in let me know

gentle solstice
#

Now if someone is interested, they'll help.

heavy knot
#

are you?

gentle solstice
#

no.

heavy knot
#

..

peak flint
#

I am using airflow docker. I am not able to run simple dag file, it's just stuck in running status.
Can someone help?
Code is pretty simple:

`from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def hello_world():
print("Hello, Airflow!")

default_args = {
'owner': 'airflow',
'start_date': datetime(2024, 2, 17),
}

dag = DAG(
dag_id='hello_airflow', # this defines the name of the DAG in Airflow UI
default_args=default_args,
schedule_interval ='@daily',
catchup=False # Prevents running historical DAG runs
)

task = PythonOperator(
task_id='print_hello',
python_callable=hello_world,
dag=dag
)
`

gentle pulsar
#

hi does anyone know which vm box to use for the m1 chip mac, seems like bandit145/centos-stream9-arm is not found:

Screenshot_2025-03-21_at_11.40.13.png
slate sail
#

hello every

#

I programmed a tools about Process Memory Dumper

image.png
rapid sparrow
#

and licensing

mental hawk
clear warren
#

what kind of websites should i crawl

thorny shell
#

https://thepretenders.com/ -- they have a "learning to crawl" section

🤣

PRETENDERS
Latin America Tour 2025 - PRETENDERS

CONTINUE TO SITE FOR VIDEOS AND DISCOGRAPHY MAY 07 Mexico City, MX ON SALE NOW  TICKETS  MAY 10 Santiago, CL ON SALE NOW  TICKETS  MAY 13 Montevideo, UR ON SALE NOW  TICKETS  MAY 15 Buenos Aires, AR ON SALE NOW  TICKETS  MAY 18 Porto Alegre, BR ON SALE NOW  TICKETS  MAY 20 Curitiba, BR ON […]

steel coyote
#

Hey anyone here who has experience in kuberenetes nvidia runtime
I have been running into a few errors for the past two days

Okay so let us consider I have a master 1 which is a cpu only vm
and worker 1 which is a gpu+cpu only vm
Now initially when i had gotten the machine there were 4 pods running each was a different service. One of the pod used 570 drivers and the remaining had their own required driver version ie 550. I wanted to check how much GPU each of the pod was consuming so I wrote nvidia-smi in the worker vm and it said nvidia was absent. But if i exec to the pod and write it would show me but wouldnt tell how much each of the process would be consuming ukwim.
So i installed nvidia 550 drivers thinking the container have the runtime of their own and it wouldnt effect it and rebooted the machine. I could see the consumption of each pod and but the 570 using pod stopped using gpu ie the program was failing whenever it got an input. So installed 570 cuz it was gving me driver mismatch and still the program gives cuda device not found error.
Now how do I revert back do i delete the nvidia drivers and reboot or do i install the GPU operator. In the case of reinstalltion of the operator what should I do.
Thank you and sorry if my question isnt clear

I figured the issue and i cannot see the nvidiadriverdaemonset running belonging to the nvidia gpu operator

any clue on what am i supposed to do

#

@surreal ridge
Here

surreal ridge
steel coyote
surreal ridge
#

It's hard to identify the problem. Try to redeploy and check pod logs to see what's actually missing

#

At some point restarting from scratch is an option

clear warren
#

no worries hopefully next time

steel coyote
#

i feel good now

wet barn
#

hi everyone!
Am not that active here on this server but i would like to introduce my Desktop application that is Tool Made In python

Its OR Code generator tool that is pretty simple to use
I would like to you guys check out and support me and thank u yall
https://github.com/noureldeen216/QRCode-generator-Tool

GitHub
GitHub - noureldeen216/QRCode-generator-Tool: You can now generatin...

You can now generating your own QRCodes! just by this simple Tool - GitHub - noureldeen216/QRCode-generator-Tool: You can now generating your own QRCodes! just by this simple Tool

#

Leave a ping That Just to know that anybody replied!! :D

surreal ridge
#

How do you expect people to review anything?

#

It almost sounds like a scam/hack attempt

wet barn
#

What?

surreal ridge
#

I am not downloading or executing that .exe rofl

wet barn
#

I don't understand anything may you explain to me

surreal ridge
#

Your repository contains 2 files

#

a readme