tools-and-devops

[grand vector]

how do I use the python kubernetes lib to use "kind"?

[warped charm]

kind being the local kubeneretes tool, or the kind part of the spec?

[proud dragon]

Hello

Do you know how to get rid of this annoying output in Jupyter Lab?

i mean "show at cmd5.sc:1" and so on

[brazen forge]

that question would be better suited in #web-development, but do you mean data URI images?

[heavy knot]

Already posted it, thanks!
BTW, I meant something like this
<img alt="My Image" src="data:image/png;base64,iVBORwA<MoreBase64SringHere>" />

[topaz aspen]

I'm using the python:3.10.2-slim-bullseye and want to install some system programmes for something - but i'm not sure what i can / can not install in this image.

Looking at the Dockerfile : https://github.com/docker-library/python/blob/56cea612ab370f3d05b29e97466d418a0f07e463/3.10/bullseye/Dockerfile it seems that apt is used, so should I be able to assume that anything that's installable with apt is installable within the 3.10.2-slim-bullseye image ?

[tawny temple]

Yes

It uses apt, and apt installs packages from sources defined in /etc/apt/sources.list

I believe by default it contains all the sources you'd expect in bullseye

[vocal pewter]

!rule 9 We do allow for this here. Please to not ask again.

[rancid schooner]

Rules

9. Do not offer or ask for paid work of any kind.

[short peak]

what's the de facto practice to install pyenv in ubuntu?

[warped charm]

Probably the install script, since it's basically just a single command and you're done

[short peak]

https://github.com/pyenv/pyenv-installer

that one?

[warped charm]

Yeah

[short peak]

thanks, just wanted to confirm. I've been using linux for over 2 decades, python for over a decade and I still do these questions 😅

[warped charm]

Hey it's a fair question

[heavy knot]

Guys, I know a little bit of docker

What is the best to learn? Still docker?

I've a old laptop 4gb ram, so, I can't run lots of stuff, but I can work with it

Docker seems good so far, but maybe it's not the best?

I'm curious about your opinions, what would be better in the long run and not too hard atm (if it's kubernetes, maybe it won't run on my laptop?)

I'm mostly curious about your thoughts, it's not like I'm going betting all in for this

Openshift seems interesting too, and apparently it's safer

[thorny shell]

my thoughts: kubernetes is a pain in the ass. You'd have to have a really nasty problem before kubernetes becomes a worthwhile solution.

[deep estuary]

Kubernetes wraps existing container concepts, so you should wrap your head around those first, Docker is the standard for containers more or less so definitely worth familiarising yourself with that.

OpenShift is a different thing to Docker, it actually builds on top of Kubernetes and is another form of container orchestration, to get the max out of those tools you need to understand Docker and containers first (and as offby1 says, you don't need Kubernetes or an orchestration tool from the get-go, there becomes a point where orchestrating things yourself becomes tricky and at that point you bring in something like Kubernetes/OpenShift/Nomad/etc. to do the brunt work, until then, you can orchestrate just using Docker)

[heavy knot]

Thank you a lot guys!

I appreciate it, a lot

[gentle solstice]

Is there a tool like heroku or digitalocean app platforms, but for on prem solutions via k8s?

[thorny shell]

not sure what you mean. Google Cloud and Azure and AWS all let you run kubernetes clusters.

[gentle solstice]

I mean deploying applications similarly to in app platform via k8s

And gcp/aws/azure is the opposite of on prem

I'm just trying to avoid having to write a helm chart for every application I deploy.

Maybe automate the build process on start (that part is mostly for cron jobs)

I already have a couple ideas on how to do it for cron jobs. Since they run periodically instead of continuously, I can run the build script each time git pull changes the HEAD.

[thorny shell]

funny, where I work most of our k8s things are also cron jobs

I know very little about k8s so I might be blathering

but the only time I've encountered helm is when we needed to install "aad-pod-identity" (that's a thing from Azure); installing our stuff doesn't involve it afaict

[gentle solstice]

I'm actually writing a helm template that will do what I want

It already is capable of running scripts located on a volume mounted on the host (/x/eng), updating the path, including and env vars, etc

I just need to add a git clone/pull and build step

I might end up writing a frontend for it if I'm dissatisfied with rancher

[thorny shell]

you know at least five times as much about k8s as I do

godspeed

[gentle solstice]

The point will be that someone who doesn't know k8s will be able to create a job

[thorny shell]

well I can do that 🙂 kubectl create job --from cronjobs/whatever iirc

[gentle solstice]

But can you create a cronjob from a script you just wrote?

And have it update whenever you push to git?

[thorny shell]

maaaaybe but it wouldn't be easy if so

[rapid sparrow]

U a supposed to write CI CD pipeline for deployment. The most nice/comfortable to use Gitlab CI here for first time users.
For example u could write that pipeline activates itself on push commit to Master branch
It would be running terraform script to get Managed Kubernetes from DigitalOcean, then in next step authentificating into it, and deploying your Helm chart. Then optionally it could be having script awaiting for IP address assigned to Load Balancer, and then running terraform scripts again to adjust DNS settings automatically.

Besides regular on push CI CD instruments, there is also pure kubernetes CD tool like argo CD, which allows more flexibility and deployment features meant for kubernetes only.

[gentle solstice]

argo cd looks promising

[rapid sparrow]

it is better be learnt only after Gitlab CI

Argo CD meant for deployment only (CD tool only)

you still need a regular CI tool to build docker images

[rapid sparrow]

I really like to use one day Tekton+Argo CD

Tekton is CI platform agnostic CI instrument

u write your CI pipelines once, and then u can reuse being activated from any other cloud CI tool, which u use only as frontend-trigger-for-gitops operation

Tekton is purely kubernetes only CI tool too

technically we could go for lightweight Drone CI for self hosted solution of CI, but whatever

anyway, better just to go for Gitlab Ci for starters. It has everything out of the box of high quality. It is a regular GitOps on push tool though

better be still combined with ArgoCD I think

[gentle solstice]

I can't really use gitlab ci because the runner's docker repo isn't part of our docker mirror

[rapid sparrow]

what did u say? 🤔

[gentle solstice]

I've tried to use it before (in our offline environment, we have mirrors for certain projects), but the gitlab-runner image isn't on docker hub, quay.io, or other public registries. It's on a gitlab specific registry.

[rapid sparrow]

gitlab-runner is on some public registry though? https://docs.gitlab.com/runner/install/kubernetes.html u can use helm chart to deploy its runner to yourself?

[gentle solstice]

plus, we have a mixed usage of gitlab and bitbucket

[rapid sparrow]

like what really prevents u from loading gitlab-runner from their docker public registry?

[gentle solstice]

What part of "image not available in our environment" is hard to understand?

[rapid sparrow]

what stops u from loading it from their docker registry and reloading to yours?

[gentle solstice]

our mirror doesn't include the gitlab registry images

[rapid sparrow]

https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/main/values.yaml u can still even use their helm chart for runner deployment

just adjust input value to use your registry

(at the top of uncommented values)

[gentle solstice]

I can probably just ask the guy in charge of the mirror to add gitlab

honestly, our environment is just a pain to work with. We have access to some registries via a mirror, but the mirror is shared for all registries. i.e. we have to remove the domain portion for most images we want to use.

This is the reason I really love podman's registry v2 configuration. You can configure mirrors on a per-registry basis

I can just use podman pull quay.io/ubi8/python and it will pull from the configured quay.io mirror

too bad our k8s clusters are still using docker 1.13

that's from 2017

[eternal wave]

someone can help me when yes please dm

pls

[gentle solstice]

@eternal wave we can help you here, no dm.

[eternal wave]

ok i have a script and i will make a that ask me in terminal how many time it repeat the script

[gentle solstice]

ok?

[unborn void]

This the place to ask about poetry?

[thorny shell]

that'd be #arts-and-literature

🤣

[unborn void]

:D

[thorny shell]

srsly, yes

[unborn void]

So I'm in the process of switching from pip/ requirements.txt to poetry

My project is a CLI tool.

It's called via a bash script/batch file depending on OS

(the bash script is just python main.py)

How do I achieve similar stuff with poetry considering it uses virtual envs and stuff?

[thorny shell]

well nobody else has answered, so I'll start talkin' -- despite never having used poetry

iiuc, poetry is just a way to make a package for your code; it has little or nothing to do with how you run it as a standalone

[unborn void]

Sure, but it does so in a virtual env

I'm fairly sure it installs dependencies in that virtual env

In the case poetry isn't a good build system, what would you recommend instead?

[thorny shell]

that's what I'd expect, too

but I don't think you get a standalone executable as output. Maybe you can; I'm skimming the docs now

[unborn void]

I've read the docs and it seems that isn't an use case

[eternal wave]

ok i have a script and i will make a that ask me in terminal how many time it repeat the script

[unborn void]

Can I use pyproject.toml without poetry? That's pretty much the reason I'm using it.

[civic grove]

poetry run python main.py?

im pretty sure poetry has some way to write/use predefined scripts in pyproject.toml but cant remember how

found an example
https://dev.to/bowmanjd/build-command-line-tools-with-python-poetry-4mnc

[gentle solstice]

Yes. You can use it with setuptools and hatch

https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html

[unborn void]

I'm trying to make a poetry script but it doesn't... run.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/twilight/code/sifas_card_downloader/src/main.py", line 7, in <module>
    import config
ModuleNotFoundError: No module named 'config'

poetry run python src/main.py works fine though.

I'm doing:

packages = [
    {include = "src"}
]

[tool.poetry.scripts]
run = "src.main:main"

(Shown relevant parts only)

Both main.py and config.py are in the src/ folder.

[dusty maple]

@unborn void The script that poetry generates assumes that src is the package, so something like this.

from src.main import main

if __name__ == '__main__':
    main()

main should then import from src.config instead.

The reason why poetry run python src/main.py works is because it sets the import "root" inside of src, which allows config to be seen as is

[unborn void]

More poetry woes

Running poetry gives me:

Traceback (most recent call last):
  File "/home/twilight/.local/bin/poetry", line 5, in <module>
    from poetry.console import main
  File "/home/twilight/.local/lib/python3.10/site-packages/poetry/console/__init__.py", line 1, in <module>
    from .application import Application
  File "/home/twilight/.local/lib/python3.10/site-packages/poetry/console/application.py", line 7, in <module>
    from .commands.about import AboutCommand
  File "/home/twilight/.local/lib/python3.10/site-packages/poetry/console/commands/__init__.py", line 4, in <module>
    from .check import CheckCommand
  File "/home/twilight/.local/lib/python3.10/site-packages/poetry/console/commands/check.py", line 2, in <module>
    from poetry.factory import Factory
  File "/home/twilight/.local/lib/python3.10/site-packages/poetry/factory.py", line 18, in <module>
    from .repositories.pypi_repository import PyPiRepository
  File "/home/twilight/.local/lib/python3.10/site-packages/poetry/repositories/pypi_repository.py", line 11, in <module>
    from cachecontrol import CacheControl
ModuleNotFoundError: No module named 'cachecontrol'

Started happening after I switched from Arch's pacman pip to the one ensurepip gives you.

Also added .local/bin to $PATH.

[thorny shell]

maybe you switched python installations, and are now using a version of python into which you haven't installed the requirements

[unborn void]

Nah, pip install cachecontrol says Requirement already satisfied: cachecontrol

AFAIK I only have the one Python installation.

[thorny shell]

well

if one python program (namely pip) says it's there, and another python program (namely poetry) says it isn't there, then you have two python installations.

Maybe one of them is a virtualenv, so it doesn't feel like a separate installation.

[unborn void]

How do I check/fix that?

[thorny shell]

well "poetry" is clearly looking in /home/twilight/.local/lib/python3.10/site-packages; do pip --version or something to try to figure out where pip is looking

[unborn void]

pip 22.2 from /home/twilight/.local/lib/python3.10/site-packages/pip (python 3.10)

[thorny shell]

🤔

ok I'm stumped

[unborn void]

My Arch install is over 5 years old and it's a ridiculous mishmash of things installed with pip (no virtualenvs until yesterday, my bad :S) and things installed with pacman.

So it's a bit understandable that things... are starting to get weird.

Is there a way to completely wipe a Python installation and start over?

[thorny shell]

probably but I can't tell you exactly how

just hunt down and kill everything remotely related to python, I guess 🙂

the big risk, though, is that important bits of your operating system might depend on it

[unborn void]

Yeah Arch does need Python

[thorny shell]

you could always vaporize the entire disk 🙂

[unborn void]

Can't, very specific setup that I couldn't dream of remembering how to replicate. :D

[thorny shell]

ugh

well

if you intend on using this system for many more months, you're going to be forced to do this at some point

[unborn void]

I think my "hope" is to reinstall "pacman pip" and never ever touch pip outside of a virtualenv

[thorny shell]

hardware will fail, some awful security vulnerability will pop up

[unborn void]

Hardware is pretty new, I just clone the drives

[thorny shell]

well anyway: I don't have good advice for you right now, except perhaps for "delete everything associated with python that isn't owned by root"; that should be safe.

And "in the future, ensure you can recover from disaster more easily"

[unborn void]

Ah, fixed it.

But...

poetry breaks pip? That's what I was trying to fix in the first place when I got into this mess.

ImportError: cannot import name 'PythonVersion' from 'pip._vendor.packaging.tags' (/home/twilight/.local/lib/python3.10/site-packages/packaging/tags.py)
When poetry is installed.

[thorny shell]

🤷 never used poetry

[ember glade]

Hey folks, is anyone interested in working on a conceptual new framework for python code quality? I have a POC but it's not easy enough to use for mass release yet. It can do a lot of things to enforce code standards from specific file locations, AST structures, naming convention, import graphs, metrics about code quality, etc. It's just a POC but I used it on about 10 of my own projects.

[unborn void]

Should tools like black, flake8, etc... be dev dependencies?

[thorny shell]

Yes

[unborn void]

How can I "force" them to run independently of code editor?

[thorny shell]

Run from command line I guess

[unborn void]

Sure, but that's... not enforced?

I can always forget to do so.

I want to be able to commit/build/etc only if those tools have been run and have no "complaints".

Is that possibile?

[brazen forge]

you can use git hooks for that

!pypi pre-commit

[rancid schooner]

pre-commit v2.20.0

A framework for managing and maintaining multi-language pre-commit hooks.

[thorny shell]

I have that sort of stuff in my ADO build steps

Nothing gets "published" unless it passes all those checks

And thus merges to mainline cannot happen either

[buoyant marsh]

Hello, it's me again, sorry for asking this and interrupting your time.
I have a bug.
I installed all packages, but this always happens.
[CRITICAL] [Camera ] Unable to find any valuable Camera provider.
And my screen is always black.
If someone could be generous, can someone give me an explanation? In advance, thank you very much.
I tried finding solution on internet, but it didn't help.

[ember glade]

What do you mean "all packages"?

Did you try https://stackoverflow.com/questions/17606705/kivy-unable-to-import-camera ?

[rapid sparrow]

is it possible to have celery tasks executed unit tests without problems? 🤔

any minimalistic example perhaps

[ember glade]

Imho if you define the task decorator around the function definition instead of function reference it makes it hard to test because you are forced to run celery instead of the code it wraps

A lot of people do this by adding a layer of indirection, for example a service facade exposes a method, the method is just called by a celery task definition

import facade from .foo
@task
def foo_task():
    facade.service_method()

Then just unit test service_method and forget about celery being in the way

But I guess that's probably going to be an integration test given the amount of components under test. So maybe consider not calling it a unit test if it runs a whole bunch of deeper logic

[ember glade]

This is also annoying because 90% of python frameworks love getting in the way with decorators, there's no way to get at the underlying logic without fighting whatever frameworks get in your way

[rapid sparrow]

yeah, i guess it is integration test. i just use as dev env setup docker-compose with running vscode inside for development, so what's easy to run test for me is quite extended

[rapid sparrow]

@shared_task
def mul(x, y):
    return x * y

@pytest.fixture(scope="session")
def celery_config():
    return {"broker_url": "memory://", "result_backend": "redis://redis:6379/0"}

@pytest.mark.usefixtures("celery_session_app")
@pytest.mark.usefixtures("celery_session_worker")
def test_try_testing_celery():
    task_handle = mul.delay(2, 3)
    assert task_handle.get() == 6

this thing managed to work

yeah, of course I would leave only abstracted Action.run inside a celery task, but still i wish to get... 🤔 in case celery tasks run as byproduct of something, so they would be surely considered in tests

[ember glade]

Do you want to test celery?

[rapid sparrow]

fair point. i don't want to test celery

[ember glade]

You might want to depending on different situations

[rapid sparrow]

it is better being able to test celery, than not being able 🙂

[ember glade]

But I think mul can be an easy one if you can remove celery from the testing equation .. assuming you do not need to test how other functions use the side -effects of mul in celery specifically

[rapid sparrow]

some of my abstracted logic could be calling celery task inside of it. I would wish it being performed during test

[ember glade]

I see

You want all side-effects

[rapid sparrow]

mm yeah.

[ember glade]

Your approach for sure handles that. You might encounter issues with flakey test in the future if there's ever a race condition.

[rapid sparrow]

troublesome 🤔

[ember glade]

My mental model for this is the more complex the code you test, the more complex your test will become over time if you want to prove "it multiples" plus "it can handle delay" plus "it can handle celery" plus anything else. That complexity will either come out as code, challenges understanding the test, or sneaky defects.

If you can find a way to reduce the amount of complexity under test, you can reduce the size of the problem. In this case if it was my projects i would unit test multiply by itself, and add another test that can mock out results from celery. If I had edge cases it might be easier to handle with a mock response from the call to a task.

This is also more work, so the trade-off depends a lot on your project goals and needs. Your test approach might be fine for the requirements of your project. If it's for a major e-commerce framework that needs to be maintained for the next 6 years i would go for a different way however.

[thorny shell]

sounds like someone who does this for a living 🙂

[rapid sparrow]

I go with https://github.com/Mahmoudz/Porto for now. I like the idea of organizing Actions / Subtasks in the way it is proposed there

with tests my main concern only about their time to run though 😉

So I would use only once function.delay stuff to interact with celery

for all other tests involving function.delay, i would be satisfied with testing my wrapped logic into Action

To be honest if I am concerned with tests run speed, I should mock database interactions too 🤔

but i kind of wishing to test it 🤔

without database interactions tests are going to be without important core stone. Well.. Now speed issue is definitely not standing, i'll think of it later

[stuck badger]

https://github.com/YashIndane/pypi_uploader

[topaz aspen]

I'm not sure what best (or even "good") practice is with regards to requirements files and packaging.

If I have setup.py and within this is install_requires - which lists the packages my package depends on.

Then I also have requirements files requirements.txt, requirements-test.txt, requirements-dev.txt - where requirements.txt mirrors the packages that are in install_requires argument in setup.py, and the others are in order to have packages to run tests and some niceties for developing/working on things.

So - I have requirements*.txt files to install things into a venv, and install_requires to tell pip which packages are required for mine.

There's obviously some duplication - and I'm wondering what's typically done here in order to manage the requirements files and install_requires properly. If I was using pip-tools then I could convert my requirements*.txt files to requirements*.in files and create hashes in the txt at least I guess.

It just feels a bit vague and as though there are a million ways to go about it - and am curious (given the above) if perhaps there's one obvious (or suggested) way to go about it.

[thorny shell]

I've never really understood this myself

The best I can come up with is: the stuff in setup.py is vague -- you don't list version numbers, or if you do they're pretty open-ended like >=21.

Whereas the stuff in requirements.txt is as precisely-specified as you can make it. It shows what you've tested with.

I'm not sure this is correct, but it's the best I can do

https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/ might help some

[somber lion]

The main difference is that install_requires should specify all version numbers your package is allowed to be installed with - IE if it doesn't match, you know it will break. So you know old versions won't work because they're missing features, sometimes you might know a specific version has a bug, etc.

requirements.txt on the other hand is for developing your library, it's instructions to recreate your environment exactly, for testing, deployment etc.

[valid sonnet]

Hi. I'm trying to create an Nginx load balancer for a simple flask app.
The flask app should print the local IP.
The requirement is to scale to 3 instances and use docker compose to run everything.

What I'm trying to achieve is to browse to localhost:9090 and on each page refresh, have the loadbalancer cycle the IP address of each flask app instance.

The problem is that I'm only getting the IP address of nginx server instead of the flask apps.

[rapid sparrow]

where is located flask application (which cloud provider u a using?)
where is located 3 instances of flask app (there are on different servers?)
where is located Nginx load balancer (is it application on some server, or cloud balancer? or some another form of it?)

[valid sonnet]

hi mate, thanks for helping out.
It's all running locally.

That's my folder structure.

app.py:

from flask import Flask, request, jsonify


app = Flask(__name__)

@app.route("/", methods=["GET"])
def get_my_ip():
   ip_addr = jsonify({'Your IP address is ': request.remote_addr}), 200
   return ip_addr

app.run(host="0.0.0.0", port=5000, debug=True)

nginx.conf:

http {
    # Define the group of servers available
    upstream backend {
        server app1:5000;
        server app2:5000;
        server app3:5000;

    }

    server {
        # Server group will respond to port 80
        listen 80;
        location / {
            proxy_pass http://backend;
        }
    }
}```

docker-compose.yml:

version: '3.7'

services:
  # Build the app services
  app1:
    build: app
  app2:
    build: app
  app3:
    build: app
  nginx:
    container_name: nginx
    build: nginx
    # Bind the port 80 of container to machine port 9090
    ports:
      - 9090:80
    # Make app as nginx dependency service
    depends_on:
      - app 1
      - app 2
      - app 3

[rapid sparrow]

version: '3.7'

services:
  # Build the app services
  app1:
    build: app
    ports:
      - "5001:5000"
  app2:
    build: app
    ports:
      - "5003:5000"
  app3:
    build: app
    ports:
      - "5003:5000"
  nginx:
    container_name: nginx
    build: nginx
    # Bind the port 80 of container to machine port 9090
    ports:
      - 9090:80
    # Make app as nginx dependency service
    depends_on:
      - app 1
      - app 2
      - app 3

events {}



http {
    # Define the group of servers available
    upstream backend {
        server app1:5001;
        server app2:5002;
        server app3:5003;

    }

    server {
        # Server group will respond to port 80
        listen 80;
        location / {
            proxy_pass http://backend;
        }
    }
}

I am not really good with networking, but i think it will not work until u open ports in your default Bridge network

[valid sonnet]

I'm not good either 😄

do you happen to know how does one open ports in default bridge network?

[rapid sparrow]

i just showed above

[valid sonnet]

this? ports: - 9090:80

[rapid sparrow]

i added ports opened to each docker-compose application setion

  app1:
    ports:
      - "5001:5000"

and changed your nginx config to point to 5001-5003 ports

[valid sonnet]

The app.py Dockerfile has this EXPOSE 5000

[rapid sparrow]

remove it

[valid sonnet]

ok

[rapid sparrow]

I think ports should be exposed only in higher level tools above dockerfile

[valid sonnet]

like this?

    build: app
    ports:
      - 5001:5000
  app2:
    build: app
    ports:
      - 5001:5000```

[rapid sparrow]

yeah

[valid sonnet]

this and changed your nginx config to point to 5001-5003 ports

do you refer to this part?

        server app1:5000;
        server app2:5000;
        server app3:5000;

so actually I need to change the .yml file to point to a different port number from 5001-5003 and not as I did above, right?

doesn't work. All I'm getting is the flask app printing the IP address of nginx server

[vocal quiver]

Why does this appear when I run git pull origin main?

Weird, I didn't get this popup when I ran this command in the past

[brazen forge]

you might have some changes that get automerged when you're pulling

[vocal quiver]

what does automerged mean?

[brazen forge]

git pull is basically git fetch followed by git merge

automerging is when the merge conflicts get auto resolved

[rapid sparrow]

i mean, your flask app would be still thinking it is host at 5000 port inside of the container

i just need to have it directed at docker-compose level being exposed in range of ports from 5001 to 5003

and then in Nginx container, we can utilize current networking situation to point to our flask application like this

        server app1:5000;
        server app2:5000;
        server app3:5000;

considering that you exposed nginx with redirections too

      - 9090:80

end result should be available at 9090 port

I just wish to add though that freemium Nginx serves as not really good load balancer choice to be honest.
Haproxy could work better. More free features for loadbalancing (healthchecks!)

[valid sonnet]

thanks. this is just a home assignment, so it's not critical

[valid sonnet]

this is what I have configured in my case, but all I get is the flask app printing the ip of nginx server and not of the 3 apps.
When I refresh the page, I expect the IP to change based on round-robin LB.

and that doesn't happen

[rapid sparrow]

As reliable way to check
Add to each flask app environment same variable with different value at docker compose level
And print it in return to requests

app1:
  environment:
    my_var: app1

import os
print(os.environ["my_var"])

It should serve as reliable indicator

[valid sonnet]

using semicolon throws an error. I think it should look like:

    container_name: app1
    environment:
      - my_ver=app1```

[rapid sparrow]

-, u added -

[valid sonnet]

oh got it

[valid sonnet]

what package do I need to install to get os ?

[rapid sparrow]

None. It is default python lib

[valid sonnet]

ok

[valid sonnet]

ok, I ran docker compose

@rapid sparrow that's what I see

that's the code:

from flask import Flask, request, jsonify
import os

app = Flask(__name__)

@app.route("/", methods=["GET"])
def get_my_ip():
   ip_addr = jsonify({'Your IP address is ': request.remote_addr}), 200
   return ip_addr

app.run(host="0.0.0.0", port=5000, debug=True)

print(os.os.environ["my_var"])

[rapid sparrow]

Return my_var instead of IP in your URL endpoint

[valid sonnet]

ok

[rapid sparrow]

It is more reliable alternative to check into which container you got load balanced.

[valid sonnet]

AttributeError

AttributeError: module 'os' has no attribute 'os'

[rapid sparrow]

Because you return os.os.environ

[valid sonnet]

oh crap

sorry

[wet wadi]

Does anyone have any idea how they made this tool? https://www.englishtools.org/en/convert-english-words-into-ipa-phonetic-alphabet

Or how to make one

I wanna make one for american english but with cambridge dictionary ipa

[valid sonnet]

KeyError

KeyError: 'my_var'

from flask import Flask, request, jsonify
import os

app = Flask(__name__)

@app.route("/", methods=["GET"])
def get_my_ip():
   #ip_addr = jsonify({'Your IP address is ': request.remote_addr}), 200   
   return os.environ["my_var"]

app.run(host="0.0.0.0", port=5000, debug=True)

[warped charm]

Did you call it my_ver perhaps?

[valid sonnet]

:/ yeah... smh

thanks

[gray scaffold]

hi guys

[valid sonnet]

ok, it works... so each page refresh, I get cycled of my_var from app1 to app3

so why it doesn't work with printing the ip address of each app?

[rapid sparrow]

because... we aren't qualified in networking stuff enough.
networks can immitate for end networks, that requested came from same network
and all of that networking magic. When request comes to Flask app, it can scan multiple IP addresses from headers/meta imformation of request and it can contain multiple IP addreses of different origins.

With just having docker-compose, u had (within my level of understanding), at least 7 networks in some sort of relationship.
Local network for each container (app1, app2, app3, nginx). They were all united with bridge network for easy communications between them through internal DNS (using app1 as DNS naim resolvable from nginx)
and at same time all of those container networks were exposed to local network of a host (making available those applications by addresses like localhost:5001, localhost:9090)
And at same time u made it available to public network outside of host by default. That made possible asking external IP address of a server and getting redirected into any of the containers

i know there is such technology as NAT, that allows for request coming from internal network to some external resource, being imitated like it came from public endpoint of a local network, and then receiving answers to it, while it would be still resolved all working well for this internal PC

And at the same time docker-compose allows at least 3-4 different network strategies or more to configure all this stuff

[rapid sparrow]

So, short story told, you returned wrong IP address (incoming request has multiple types of IP in it, including IP of real origin / from which it was forwarded depending on... proxy servers on a way), or you could not see right IP address due to networking magic that transformed IP addresses when crossing different networks involved on a way.

that's why i recommended as returning Environment variable as more reliable and easy way. Less complexity involved xD

my explanation could be a bit of messy, i am not good in networking stuff

There should be read a book about Networking to get it, may be some kind of Docker Networking book exists 🤔 Technically this is generic networking stuff though (Docker involvement here minimalistic, it just boilerplates them). So some kind of a book about Networking / types of Networks and configurations should be read for that.

[valid sonnet]

thanks for the help

the problem is that I have to submit this as part of an interview exercise :/

from what I gather, I don't need to request remote_address but local_address of the container

and it doesn't work with variables

[rapid sparrow]

#networks #tools-and-devops #web-development
in conclusion this is a subject
on 80% about knowing networking fundamentals, on 15% how it matches to boilerplating in docker (at some point docker introduces internal DNS resolving? 🤔 ), and 5% how it matches to Flask.

probably in #networks people could help more

[valid sonnet]

thank you very much. I'll ask there

[lusty forge]

I'm running a Discord bot that I'm soon to release, though I'd like to begin with good practises. Currently I'll be using Docker compose for deployment, orchestrating the db and itself.
I've heard about swarm, and it seems to be a step up from compose, but considerably less powerful than k8s. What would the benefits of switching to swarm / k8s be?

[silver gale]

I'm going to be merging 2 branches together, is using the --no-ff option usually not a good practice? I want that so I can have a single commit to revert if I don't like the results of my merge, as opposed to doing fast forward "procedural" commits

[thorny shell]

I think --no-ff is exactly what you want then

another idea is to make a "squash rebase" but that's two steps

[brazen forge]

if you don't want the commits from the branch you're merging from, use squash merge

which is git merge --squash

[thorny shell]

maybe "squash merge" is what I meant 🙂

[silver gale]

Ah okay thanks I'll check that out! I'm shying away from rebases these days, that led to a nasty bug being brought in and it was quite hard to find out the commits it originated from -- I had to revert using git reflog on my local repo and revert before I rewrote my branch history with rebase

[thorny shell]

it's a subtle thing

[silver gale]

Follow up question -- I have 2 feature branches. Branch A and Branch B. Branch B branched off of Branch A a bit ago, but I needed to make some other changes to Branch A -- I want to bring these changes into Branch B using this merge strategy. When I eventually merge Branch B back into Branch A, will this lead to some conflicts? And if so, is the only way to avoid that by allowing it to use the fast-forward method of merging now?

[brazen forge]

if you merge A into B, then later merge B into A, you won't get conflicts the second time (provided there are none after the changes from the last merge)

[thorny shell]

and I probably wouldn't use squash merges anywhere but the "main" branch

let the feature branches be a bit chaotic, assuming they're ultimately destined to be merged back into "main".

[If they are meant to live forever, though, then I guess not]

do people still have more than one release branch?

that seems like a style that would have died with shrink-wrapped software

[silver gale]

I will be the first to admit - my team is very behind the times on best version control practices

[thorny shell]

well at least you're using git!

[silver gale]

People have been rebasing things a whole bunch, and we have a lot of branches that have been "living" for a good while. I'm trying to learn how to do things better, but that stuff is being learnt outside of my work lol

True, it could be worse. But what say you about the use case I mentioned earlier? I only wanted to "squash" the merge so I could easily revert if an issue arose

[thorny shell]

well you can revert regular merges too.

https://mirrors.edge.kernel.org/pub/software/scm/git/docs/howto/revert-a-faulty-merge.txt

[thorny shell]

Probably wise now that I have done some reading 🙂

[silver gale]

I'm looking over that link and wow it is very confusing lol. I was checking out some videos/atlassian articles to see if I could get a better picture of the idea

[thorny shell]

my takeaway is: if you do revert a regular merge, fix the branch, and want to re-apply that branch's changes, you should re-create the branch first.

[silver gale]

I'm uncertain what my takeaways are right now. A lot of things I'm reading about involve using my local repo's reflog history to fix a bad merge, as opposed to working with the branch history itself. I suppose it shouldn't be much of a concern since I am the one doing this stuff, but it doesn't seem "scalable" in the sense that I would like to know a way to do it without having access to my own local repo's history

They're basically saying:
git reflog show branchB (after I merge branchA into branchB)
git reset --keep branchB@{1} which would be the point in time right before @{0} where presumably my fast-forward merge took place in my local repo

[loud bear]

How to hide our dataclass members from the ToC on the left in mkdocstrings?

My dataclass members are mostly undocumented, as docstrings would be redundant

Also I would like to limit the annotations to name and type only, I don't want the default value / field declaration to be shown

Nvm got it

Just need to set heading_level, 4 seems comfortable enough

[rapid sparrow]

How to write python package in the way that it registers CLI commands when installed? (like django-admin)?

[short peak]

don't django commands require python manage.py command to run?

otherwise you'd need to create alias, symlinks or alter PATH envvar

[rapid sparrow]

$ python3 manage.py shell
Python 3.10.5 (main, Jun 11 2022, 16:53:24) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> exit()
$ django-admin shell
Python 3.10.5 (main, Jun 11 2022, 16:53:24) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> exit()

both commands are equally valid

when Django is installed, it allows being launched as python manage.py or with autoregistered alias django-admin

[blissful quarry]

So I am still quite a beginner with learning Python as it is my first programming language too. Can anyone recommend any devop tools I should create that would help in the real world?

[short peak]

wondering how you installed django and how your PATH looks like

anyway. I'm not sure if just installing a package ensures that behavior. Poetry and pyenv have installation scripts

[rapid sparrow]

python3 -m venv venv
source venv/bin/activate
pip install Django

echo $PATH

/home/naa/repos/parent_folder/current_folder/venv/bin:/home/naa/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin

essentially we can probably read details in Django repo

:doc:`django-admin </ref/django-admin>` should be on your system path if you
installed Django via ``pip``. If it's not in your path, ensure you have your
virtual environment activated and you can try running the equivalent command
``python -m django``.

hmm 🤔

i somewhere also encountered some python packagers allow to define such commands i think

[short peak]

ahm but you activated the environment. That's a script. Compare your path with the environment deactivated

i meant your PATH envvar

[rapid sparrow]

https://github.com/django/django/blob/main/setup.cfg

[options.entry_points]
console_scripts =
    django-admin = django.core.management:execute_from_command_line

defined in setup.cfg

[rapid sparrow]

easier answer
I just checked bin of venv

$ ls
activate  activate.csh  activate.fish  Activate.ps1  django-admin  pip  pip3  pip3.10  python  python3  python3.10  sqlformat

here it is

[short peak]

cool

[civic grove]

i recommend doing basic stuff with github actions such as running tests and linters

also good to learn about docker

[blissful quarry]

Yeah i am learning about docker now, created some containers using Docker Compose etc. I think it is knowing what sort of things to script in python

[balmy basin]

Is there on windows any way to check what files some process has opened? I think there was some module with .proc or .process method and it took pid as an argument but I think it was only on linux and cant recall the module name

[brazen forge]

have you checked the psutil package?

@balmy basin

[balmy basin]

This is the one I was looking for! Seems to work on windows too, thanks!

[lusty forge]

How does Docker swarm differ from Docker compose / k8s, is it worth using Docker swarm for a single Discord bot (that likely will need to scale in the future) on a single VPS?

[rapid sparrow]

I think for single VPS docker compose is sufficient solution.

Docker swarm is docker compose with augmentations for cross server deployments.

Kubernetes is more feature rich alternative to docker swarm, superior in any way, ecosystem included

I use kubernetes for my discord bot though

Because a lot of ready out of the box solutions, learning experience most relevant to jobs

Just comfortable once u dive into it.

Kube is pleasant to work with and seen as default option for everything once u get into it

[lusty forge]

Ahh, I see, in that case if I were to upgrade, I could skip swarm and go to k8s?

[rapid sparrow]

Yes, I would recommend going straight to k8s after docker compose. At current times learning swarm is waste of time for alternative no longer used tool

[lusty forge]

Alrighty then, should it be difficult to get my bot running on k8s now, despite it not really needing the power of it? I'd like to give it a shot, so in the future I wouldn't have to scramble to quickly learn things when I inevitably will need to cluster.

[rapid sparrow]

As long as u keep your solution fully in docker containers / nothing is installed to host for main work. It is relatively easy to upgrade to k8s

K8s is just more superior docker compose alternative to launch same containers

In still yaml syntax and way more features

[lusty forge]

Sounds good, would you be able to point me to a good guide for it / how to get started with it?

[rapid sparrow]

This book was awesome jumpstart for me

[rapid sparrow]

Obviously.. main advantages of k8s, it scales better in code growth.
Allows multi server launch and still working with one kube cluster (easy prebuilt HA / load balancing things)
Allows to use ecosystem of premade kubernetes solutions (logging/monitoring and etc) which are just meant to work for containers launched in kube first

[rapid sparrow]

For tiny projects I just love its internal DNS system / namespaces having utilized. Docker compose has dns too, but kube one is way more feature rich

[lusty forge]

Got it, thank you so much!

[rapid sparrow]

U a welcome. My most love with kube, that it can be fire and forget type of system if user base is not growing at least.
Your containers can be having CPU/SSD limitations, once reach it auto relaunches stuff.
Zero maintaince system for lazy people.
Everything auto heals itself u can stack as many applications as hardware allows into one machine, and still not feeling discomfort

In production it is not the same obviously, but still u get an idea

[wooden ibex]

That’s what Google wants you to think.

[thick wren]

hey, working on a cloud platform specifically for python, and would really LOVE some brutal, honest feedback. anyone up to it? 🥺

[wooden ibex]

Sure, why should this exist at all?

[thick wren]

well the thesis is that building web UIs is labor-intensive, requires extra learning and extends development time by a lot. so the proposed solution is an open source lib with one line commands like read() and display() + easy hosting

[ionic ridge]

You might be interested in how they've tackled this problem at https://anvil.works

Runtime engine is open source: https://github.com/anvil-works/anvil-runtime

(sorry, can't remove embeds on my phone)

[thick wren]

yes! great benchmark. our take is slightly different in that you don't even need to drag and drop uis, the ui is autogenerated which makes development quite a bit faster

you can also use libraries, packages, files, etc like you would locally

[ionic ridge]

You don't need the drag and drop with anvil either. You can also just create ui objects from (Python) code

(or, manually edit the yaml file that the drag and drop populates)

[narrow slate]

Does anyone know a way to check if a variable exists in bash without exiting the script while it make use of set -u ?

[wooden ibex]

As always, the problem is outside tiny projects, Python is bad at front end and meh at backend.

So it's always a tiny market you are gunning for. If you want to do it as hobbyist project, go for it but I don't see it massive business opportunity.

[thick wren]

that's a great point. I'd argue that a multitude of tiny projects could perhaps make up a large enough market, fragmented across users, projects and over time. but I do see what you mean. also, i feel like there might be an opportunity for a suite for small projects, like python anywhere does for cron jobs. what do you think?

[thick wren]

i didn't know that! thanks, checking this out now. have you used anvil in prod before?

[ionic ridge]

I'm a freelancer. I have several clients with anvil based apps in production.

[boreal vortex]

Hello, I got this error with postgres volumes in docker desktop for windows:

data directory "/var/lib/postgresql/data" has group or world access
scrap_balance-db-1       | 2022-07-27 20:29:50.010 UTC [1] DETAIL:  Permissions should be u=rwx (0700)

The only solution I found in the internet was to use named volumes, but I want to use regular ones.
The most interesting thing is that everything used to work.
I've already tried:

• delete all containers and images
• reinstall docker
• reinstall WSL

[thin oriole]

Did you chmod the data directory on your host system that was mounted into the container?

[boreal vortex]

I only remember that I cleaned everything completely, because docker started throwing exceptions.
After that, this appeared.

[indigo zenith]

Check the permissions on the relevant folder with ls and change them if needed with chmod

Ah, and you said Windows. Especially if this is a bind mount maybe you need to run as admin or something

Isn't that message just a warning anyway? Seems like it shouldn't fail to run just because of extra permissions

[thorny shell]

fwiw, ssh refuses to run if the permissions on your ~/.ssh directory are too lax

that refusal is meant to incentivize you to tighten up your security

works, too 🙂

[boreal vortex]

full logs

scrap_balance-db-1       |
scrap_balance-db-1       | PostgreSQL Database directory appears to contain a database; Skipping initialization
scrap_balance-db-1       |
scrap_balance-db-1       | 2022-07-28 13:48:23.673 UTC [1] FATAL:  data directory "/var/lib/postgresql/data" has group or world access
scrap_balance-db-1       | 2022-07-28 13:48:23.673 UTC [1] DETAIL:  Permissions should be u=rwx (0700).
scrap_balance-db-1 exited with code 1

[indigo zenith]

Maybe worth sharing your Dockerfile and/or yml

[boreal vortex]

version: "3"
services:
  db:
    image: postgres:10.13
    restart: always
    ports:
      - 5432:5432
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_NAME}
    volumes:
      - ./data/postgres-data:/var/lib/postgresql/data
  redis:
    image: redis
    restart: always
    volumes:
      - ./data/redis-data:/data
  pgadmin:
    image: dpage/pgadmin4:6.4
    ports:
      - "9002:80"
    environment:
      PGADMIN_DEFAULT_EMAIL: [e-mail]
      PGADMIN_DEFAULT_PASSWORD: ${PGADMIN_DEFAULT_PASSWORD}
    restart: always
    depends_on:
      - db

It worked on my firend's PC

[thick wren]

would you be willing to give us a spin?

[ionic ridge]

sure. post a link

[indigo zenith]

Does ./data/postgres-data exist? Is your friend also on Windows?

[boreal vortex]

1. It creates automatically
2. yes

[indigo zenith]

Maybe try deleting and creating it manually and rebuild. Or ask your friend what they did differently

[tropic olive]

so, i made a Dockerfile, how do i build an image from it and run it in a container?

[wooden ibex]

docker build -f dockerfile .

[viral marlin]

So it's been a year a hot second, since our DevOps Hour with Joe event. Well, I've finally finished a small page containing the video, the miro board, and a compilation of some of the resources mentioned. Check it out and feel free to let me know if I'm missing a resource!
https://pythondiscord.notion.site/DevOps-Hour-w-Joe-Resources-2c17da9a5cc94de58c82b676f339ba7f

[brazen forge]

docker build -t image-name .
docker run image-name

[tropic olive]

if i got a folder with subfolders and files? (C:\Users\me\folder) how would i copy that same folder to my images file system, i've tried COPY C:\Users\me\folder /

[heavy knot]

cp -R /path/to/folder /path/to/images/folder

[night sparrow]

Hi all how can i install requests o Qpython,please?

[granite lantern]

Hi people !
I code in python and i'd like to develop a mobile app.
However I need some clues around the approach of it. I know that Kivy library lets you develop mobile apps with python but is it the best approach? Wouldn't it be better to create a website (html css js) and like the mobile app would be a python browser?
Should I just learn Kotlin instead and use it?

I'd like your feedbacks and i'd like to know if some big companies uses python/Kivy for their mobile apps ! :D
(I saw spotify, google, netflix does but, i don't know if they're talking about websites or mobile apps)

[rapid sparrow]

as far as I know people traditionally use Java/Kotlin as best choice to Android mobile dev.
using html/css/js for mobile dev is possible, and companies can choose it. This is interesting to explain choices 🤔 https://dev.to/ionic/the-easiest-way-for-web-developers-to-build-mobile-apps-1ih8 kind of liking a bit idea of Capacitor
Flutter / Xamarin devs are around present too

[rapid sparrow]

i am quite certain that python would not be chosen at all, unless they have python devs only 🤔

[granite lantern]

alright so the only option i have with python would be to just make a little python browser and display a html/css website that would be the said app in it

[granite lantern]

alright i'll take all these into consideration, thanks a lot !

[rapid sparrow]

xD as next ultimate development environment
We can develop not just in docker-compose container, but straight going into developing from inside kubernetes cluster.

kind of obvious idea, and technically easy to make 🤔 a bit of minikube/microk8s with binding correctly volume onto our project folder, and it should work 🤔

plus i heard about already out of the box solution exactly for this purpose (DevSpace?) it would be interesting to check what it offers

probably silly way to go though. Docker-compose is already sufficient dev env solution. (or not? 🤔 )

[tepid spruce]

Hello, was not sure where exactly to post this question so please forgive me if this is the wrong channel, but I am trying to develop a program where I can input keywords and it searches a specific website for products i.e I input 10G SFP Cisco and it will pull up any related products to that keyword. Can someone point me in the right direction to how I can make this happen ?

[thorny shell]

@tepid spruce this is a good question for one of the help channels. But in the meantime: sounds like you want https://docs.python.org/3/library/webbrowser.html

[tepid spruce]

Sorry new to all of this lol, I want to get more into coding and feel like this is a good project to start on, thank you for your input. Very much appreciated.

[heavy knot]

Hi, sorry for interrupting but would like to ask a question, I am planning coding some ML models using pytorch and also thinking of some flask project. whats the best license that I should use for github repositories?

[rapid sparrow]

MIT if u wish it being available freely for everyone (it says: do anything u want, i am just not responsible for anything)
some LGPLv2.1 if u wish that it could be used only as unchangable package inside of other propritary software. And in open source it can be changed, but it will be stuck with same or higher grade GPL license (depending on pressence of +)
GPL3 if u wish it being open source only, and not possible being used in properitary at all
Affero GPLv3 if u wish being especially restrictive, and locked from propritary even in closed server usage

[heavy knot]

thank you so much for the answer. so would MIT be best? I am only having it on github to impress others but might expand on it later as a business or research project

[rapid sparrow]

in general if u wish being super nice, yes

[heavy knot]

alright, i'll just pull the trigger and go with MIT

thanks again Darkwind

[wanton shale]

What does "included support for web frameworks" mean with respect to PyCharm's professional edition

[civic grove]

i'm guessing support for these type of stuff when creating a new project and probably some other features idk about

probably stuff like debugging django templates

[wanton shale]

So parsing, autocompletion, error detection

Analysis, but which does beyond the language and is catered more thoroughly to the API

[civic grove]

i remember when i was writing fastapi, it also had an HTTP client for debugging endpoints

[wary oasis]

This is a noob question, but what does does everything inside the parenthesis at the top of the log mean?

[heady plinth]

The (HEAD -> master, origin/master, origin/HEAD)?

[wary oasis]

yeah

[heady plinth]

It tells you the current location of the branches of your repository

[wary oasis]

HEAD is the most recent commit right?

[heady plinth]

the HEAD of a repository is the current point your repo is at, and the origin/master is the location of the HEAD in that remote

Yeah, I think most recent commit

[wary oasis]

So head is at master in origin (github remote). Then what does "origin/HEAD" represent?

[heady plinth]

The HEAD of origin, just like the HEAD on your computer

[wary oasis]

Alright thank you!

[heady plinth]

@wary oasis here's an example with multiple branches. origin/comp is a branch named comp on the github, which is behind the other branches

[thorny shell]

HEAD is where your next commit will go, and it's often the most recent commit, but not always.

git checkout wat # HEAD is now wat
git add something; git commit -m yeah baby # HEAD is still wat, your most recent commit
git checkout old-stuff # HEAD is now old-stuff, *not* your most recent commit

[wary oasis]

So this is saying that the HEAD is the main branch, the main branch at origin was updated. But what about origin/HEAD that is still a bt confusing to me

[thorny shell]

that's where HEAD is pointing on the "origin" machine, wherever that is

[wary oasis]

And it being in the parenthesis just means that it was affected by the push/commit ?

[thorny shell]

nah I think the parens show a list of the various "refs" for that commit

https://eagain.net/articles/git-for-computer-scientists/ might help

[wary oasis]

Alright thanks, I'll give that a read

[rapid sparrow]

HEAD, means like where cursor position, where currently reading/writing into repo u are.

So, we can read that u a currently at the end of master branch on your local computer. (Head -> master)
At same time end of local master branch looks like matching end of current last synchronised version in remote/cloud master (origin/master)
And last one Head -> origin/head I have no idea

[thorny shell]

<@&831776746206265384> ^^

[cosmic sparrow]

Is there some kind of tool that solves this type of merge conflicts?

[rapid sparrow]

U a already in this tool

U a showing us interface of git merge conflict solving tool

Just choose version incoming or current or custom for all conflicts
Stage and commit. Conflict is solved

[cosmic sparrow]

I mean fair enough but this particular conflict seems a bit more straightforwards to solve than other common conflicts, so maybe there was something that did it automatically

[rapid sparrow]

mm yeah, i used tools that solve it automatically in the past.

I am lazy to configure them now. VScode solver right here

[rapid sparrow]

Kdiff3 i previous configured. It solves most of problem automatically

[split radish]

Does anyone how to solve the error of error: subprocess-exited-with -error and error: metadata-generation-failed in the terminal?

[indigo zenith]

What exactly are you trying to install or do? This looks like a conflict between package versions or something

[split radish]

OMG!

[split radish]

I'm installing a model called MIDIDDSP

https://github.com/magenta/midi-ddsp

[tawdry maple]

hey, how can I use a specific commit of a git dependency in poetry?

[gentle solstice]

You can use the commit via poetry add git+https://github.com/user/repo.git@commithash

You can also use a branch or tag name

[unborn void]

Any chance of making Nuitka+Poetry work together?

[finite fulcrum]

the two shouldn't really interact with each other?

[fast grail]

Hi DemonicSavage. Are you trying to add a script to poetry to run nuitka using it?

[unborn void]

Ya

Think I found a bug in Nuitka tho
https://github.com/Nuitka/Nuitka/issues/1713

[sturdy tide]

Hey All, having some issues with moviepy.
I wonder if anyone recognizes the issue below and could help?

[ERROR] OSError: [Errno 8] Exec format error: '/var/task/ffmpeg'
Traceback (most recent call last):
  File "/var/task/CreateThumbnail.py", line 101, in handler
    create_video_thumbnail(bucket, key, file_name, extension)
  File "/var/task/CreateThumbnail.py", line 74, in create_video_thumbnail
    resize_video(thumb_download_path, thumb_upload_path)
  File "/var/task/CreateThumbnail.py", line 32, in resize_video
    with VideoFileClip(video_path) as video:
  File "/var/task/moviepy/video/io/VideoFileClip.py", line 88, in init
    self.reader = FFMPEG_VideoReader(filename, pix_fmt=pix_fmt,
  File "/var/task/moviepy/video/io/ffmpeg_reader.py", line 35, in init
    infos = ffmpeg_parse_infos(filename, print_infos, check_duration,
  File "/var/task/moviepy/video/io/ffmpeg_reader.py", line 257, in ffmpeg_parse_infos
    proc = sp.Popen(cmd, **popen_params)
  File "/var/lang/lib/python3.9/subprocess.py", line 951, in init
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/var/lang/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)

Additional bits of info:

Value for proc: [‘/Users/Tom_Hanks/Documents/CreateThumbnail/ffmpeg', '-i', '/tmp/-866f0413-cb25-4f7d-8916-f8104c03fb5cunicorndrops-public-drops---sample-video.mp4']

executable binary value: os.environ["IMAGEIO_FFMPEG_EXE"] = f"{CWD}/ffmpeg"

[wicked shard]

Hello! I'm terribly confused with git. I have a git repo, with two commits, say commit 0 and commit 1. Inbetween these two commits, I believe I worked on a file in a folder, sandbox/filename. When it came time to committing, I thought "this sandbox folder really shouldn't be synced with remote" so I added "sandbox/" to my .gitignore, and (I believe I git add * and git commit -m "message" and git push origin main), seems to have deleted the sandbox/ folder from local.

1. Is there any chance of recovering the file? I've moved past the stage of denial and I'm in grief now, I'm pretty sure no 😢

2. is what I'm describing a consistent history of events? I might have been misremembering how it all went down

for what its worth, this file was a .ipynb, and the only proof I have that it still existed is that I actually have the jupyter kernel running in terminal, and in the running notebooks it actually shows its still running
but when I try to click it it just tells me the file is missing

[wicked shard]

I HAD FORGOTTEN THAT ACTUALLY I HAD COMMITED THE CHANGES, but had git rebased to squash some commits together, and holy moly git reflog came to the rescue and I git reset --hard to the commit that actually had the changes

crisis averted

[tropic olive]

In my Dockerfile i wanna copy the directory /Users/USR/Desktop/my-folder how can i copy this exact structure over to my image?

when i try COPY /Users/USER/Desktop/my-folder / it gives me failed to compute cache key: failed to walk /var/lib/docker/tmp/buildkit-mount1981927382/Users/USER/Desktop: lstat /var/lib/docker/tmp/buildkit-mount1981927382/Users/USER/Desktop: no such file or directory

[tawny temple]

The source path for a copy is always relative to the build context. It's not possible to specify an absolute path. You wouldn't want this away since it'd make it more difficult to build the image on a different machine.

[tropic olive]

nvm i did it

i got a new problem

in CMD, i wanna run this command python -m uvicorn server:app, however i cannot

[tawny temple]

Why not?

[tropic olive]

🥲

[thorny shell]

I will guess you messed up a uvicorn config file somehow

[tropic olive]

uvicorn config?

does uvicorn have that?

[thorny shell]

probably

[tropic olive]

or gunicorn?

i dunno where i gotta put it

[thorny shell]

is that docker image something that you created yourself?

[tropic olive]

yes

[thorny shell]

post the Dockerfile

(as text, not a screenshot)

[tawny temple]

I think what's happening is your dockerfile already has python -m as an entrypoint.

So your CMD is getting interpreted as a module name

FIX: set entrypoint rather than cmd

[tropic olive]

kk

[tawny temple]

But it's weird that something seems to start running. Yes, share your Dockerfile if that doesn't work.

Maybe gunicorn is the entrypoint. I dunno

[tropic olive]

well, it's working.

[thorny shell]

glad we could help!

NEXT

[tropic olive]

🥲 thanks

[tropic olive]

dockerfile docker FROM ghcr.io/python-discord/snekbox:latest COPY heroic-hydra heroic-hydra ENTRYPOINT [ "python" ] RUN python -m pip install -r /snekbox/heroic-hydra/dev-requirements.txt CMD ["heroic-hydra/main.py"] EXPOSE 8000 main.py py import uvicorn import os os.chdir('/snekbox/heroic-hydra/src/server/') uvicorn.run('server:app')

i get this ERROR: Error loading ASGI app. Could not import module "server"

it might be a perms issue 😬

[tawny temple]

You're probably in the wrong directory.

[tropic olive]

nope

when i do

print(open('server.py').read()) it prints the contents

only when running it on uvicorn i get the error

[tawny temple]

Oh so you have server/server.py

[tropic olive]

yes

oh lets go

i got it running

[tawny temple]

Ah well. The current directory isn't necessarily always on the import path (sys.path)

[tropic olive]

i just did py import os os.chdir('/snekbox/heroic-hydra/src/server') os.system('python -m uvicorn server:app')

[tawny temple]

Changing directories to import a module is hacky

That's also pretty hacky

[tropic olive]

well im gonna go get my phd in devops now

i'll mention you there

[tawny temple]

Why don't you just change directories in your dockerfile?

[tropic olive]

oh yeah

[tawny temple]

`WORKDIR

[tropic olive]

i forgot WORDIR

dammit

[tawny temple]

Then you don't even need this intermediary python script

[topaz aspen]

I'm able to do pip install git+ssh://[e-mail]/<projec>/<repo>

but am unable to do : pip install git+ssh://[e-mail]/<projec>/<repo>/<branch>, which I thought should work (?) I get the error:

remote: ERROR: The project you were looking for could not be found or you don't have permission to view it.

I clearly have permission to some degree, as I can install from the main branch, i just can't seem to pip install from a different branch

[topaz aspen]

ping me if you reply please

ah i just needed to use pip install ... gitlab.com/proj/repo.git@commit-id 👌

[tawdry needle]

you can do @branch or @tag too

[topaz aspen]

using @thing where the branch name is thing didn't work for me

syntax pip install git+ssh://gitlab.com/<proj>/<repo>.git@<branch-name>

[tropic olive]

so, i got a Dockerfile and image up and running, it starts up server.py (my server) on http://127.0.0.1:80 (i'm using fastapi and uvicorn), this has only one websocket endpoint, the root (/). however, when i connect to it i get a ConnectionRefusedError: [WinError 1225] The remote computer refused the network connection.
Yes i have done EXPOSE 80. Any ideas?

and this is what my client is connecting to ws://127.0.0.1:80/

[thorny shell]

the IP address you want to connect to is almost certainly not 127.0.0.1

there's some docker command that will tell you what it is

[tropic olive]

but im running python -m uvicorn server:app --host 127.0.0.1

[thorny shell]

if that command is running inside the container, then my comment stands

[tropic olive]

sadge

[thorny shell]

containers have a different "space" of IP addresses than your host Windows machine does

🙂 [2022-08-03T07:31:50-0700][teensy] % docker ps
CONTAINER ID   IMAGE                  COMMAND      CREATED      STATUS                  PORTS                                                                NAMES
4c368e332fe9   pihole/pihole:latest   "/s6-init"   3 days ago   Up 3 days (unhealthy)   0.0.0.0:53->53/tcp, 0.0.0.0:80->80/tcp, 0.0.0.0:53->53/udp, 67/udp   pihole
``` in my case, those `0.0.0.0` numbers are the relevant IP address

if you see similar numbers or if you see 127.0.0.1 then what you're doing should work, and I can't explain why it doesn't.

[tropic olive]

well, i did py os.system(f'echo my ip is {socket.gethostbyname(socket.gethostname())}') which gave me the ip, and i connected to there. now, it just says timeout error

nvm i might've gotten it

nvm i didn't

docker connections 🥲

[tropic olive]

i got it working!

and i've also decided that devops is not for me

[rapid sparrow]

U just started from a wrong foot 🦶 don't give up, it is awesome https://youtu.be/LmW3H-EXYS0

[lusty forge]

How would I have Grafana tie in to my Postgres db to monitor queries made to graph out the time it took for each one?

[rapid sparrow]

it depends on what your deployment target system
but in any case you would like to have Loki as log aggregator in this recipe. And some agents installed for log collection: https://grafana.com/docs/loki/latest/clients/
and pointing in your Grafana access to Loki

[tropic olive]

rename this channel to #tols-and-devoops

[rapid sparrow]

too many dictionary definitions, requesting explanatory brigade

[lusty forge]

Ah, gotcha. Say I use this for the Loki logger: yml grafana: image: grafana/grafana container_name: overseer_grafana logging: driver: loki options: loki-url: "https://<user_id>:<password>@logs-prod-us-central1.grafana.net/loki/api/v1/push"How would I have it monitoring my Postgres db for any query any applications sends, and getting the metrics of that query?

• any other db related stuff (if possible)

[rapid sparrow]

U a asking a bit too much from this tool.
Loki just collects in a centralised and persistent way logs of application, in your case postgresql.
U can scrap out of it metrics per query, if they are seen in logs

So u wish to have metrics of database? Use Prometheus with its pgsql agents.

Do u wish to have metrics per query already parsed and detailed for you? I don't know such open source tools. I know SaaS system datadog is capable to do it.
Considering how it is installed and collects those per query metrics... U would need a tool that is similarly installed then
(A tool that requires installed SQL functions inside postgresql in addition to its agent)
In any case detailed metrics per query I think is beyond of standard tools available in grafana

Prometheus allows only aggregated metrics gathered

U can make custom agent to your app exposing metrics and collecting them

But they would be only aggregated as far as I know, if I remembered correctly

Django has already ready to user Prometheus agent integration upon its orm

Any way u can install paid tool Datadog, it will scrap for u all possible things regarding database, monitoring and logging, and query metrics included

Or to try finding free alternative
Loki will cover logging
Prometheus aggregated metrics

[lusty forge]

Alrighty, will look into that tomorrow

[tawdry needle]

it should, at least i have definitely done it before

[topaz aspen]

Idk, I tried 🤔 didn't work

No idea if this is platform specific, I'd have thought not though I guess

[tender current]

What does pyright mean by No configuration file found.? What is it looking for?
I'm assuming it's looking for pyrightconfig.json but I'm handling that directly in pyproject.toml's [tool.pyright]
Please ping me if you reply

[languid helm]

according to the documentation, it's either of those, but if you have both it uses the json

Is your pyproject.toml in the project root?

[tender current]

yes it is

And I don't have pyrightconfig.json, just pyproject.toml, so it should be fine with it being either of those 😕

. <--- project root
├── ./__pycache__
├── ./b.py
├── ./env
├── ./e.py
├── ./f.py
├── ./m.py
├── ./parameters.json
├── ./pyproject.toml <--- Here
├── ./requirements.txt
└── ./s.py

Right at project root

[languid helm]

Have you restarted pyright since adding the options?

[tender current]

Uuh how do you do that?

[languid helm]

easiest way is to restart vscode

[tender current]

Done...still getting it

No configuration file found.
pyproject.toml file found at /path/to/proj.
Loading pyproject.toml file at /path/to/proj
Assuming Python version 3.10
Auto-excluding **/node_modules
Auto-excluding **/__pycache__
Auto-excluding **/.*
Searching for source files
Found 1 source file
pyright 1.1.265
0 errors, 0 warnings, 0 informations
Completed in 0.707sec

[languid helm]

It does say it's loading your toml.

What does it look like and what options do you think it's ignoring?

I added this to mine for testing, and it worked after restarting.

[tool.pyright]
reportGeneralTypeIssues = false```

[tender current]

[project]
name = "project_name"
version = "0.1.0"
requires-python = ">=3.10.5"

[project.urls]
homepage = "project_url"

[tool.pyright]
stubPath = ""
reportPrivateImportUsage = false

[tool.mypy]
plugins = ["deal.mypy"]
ignore_missing_imports = true

[tool.pylint]
disable= ["missing-module-docstring",
          "missing-class-docstring",
          "missing-function-docstring",
          "invalid-name"]

[languid helm]

false is not a valid value for reportPrivateImportUsage.

it's "none" or "error"

hmm, but they also say that for reportGeneralTypeIssues

no, boolean should work

[tender current]

Oh ok 👍

[languid helm]

No clue what could cause it, it's really the reportPrivateImportUsage that's failing?

[tender current]

No, that one's working, since I'm not getting warnings for it (as intended)

The unexplainable No configuration file found. is a little annoying line but I suppose I can live with it 😅

[languid helm]

It's telling you it didn't find the .json config.

Then on the next line it's telling you it did find a .toml config

[tender current]

🥲

[echo cypress]

hello. in order to create a client with the docker package I need sudo privileges. Is there a way to only escalate privileges during the execution of a portion of the code? for example, create a docker client and do client.run(...)

[languid helm]

Is your account in the docker group?

[echo cypress]

no. I didn't do that just because of the warning described here https://docs.docker.com/engine/install/linux-postinstall/

[rigid lark]

you have os.setgid() and os.setuid() but you need to run script as root anyway to use it so I miss the point

[gentle solstice]

if you're using linux, I suggest using podman instead

I also suggest using podman if you use windows

Use docker only if you don't want to configure WSL manually.

[deep estuary]

i've had some weird stuff with docker compose through podman, otherwise i'd go 100% podman, sadge

[dense warren]

How can i print out all in "token"?

[topaz aspen]

I'm wondering how to pip install a python package from a private gitlab repo within a docker container.

I've done this without the container, but when I try within the container I get the error :

#6 4.153   Please make sure you have the correct access rights
#6 4.153   and the repository exists.
#6 4.159   error: subprocess-exited-with-error

I tried passing the ssh key I used with this project as :

docker build --tag <name> --ssh github_ssh_key=<path to key> -f <docker file> .

but it doesn't seem to pick it up / work. I get the same error without the --ssh flag, so I think i'm doing something incorrect there. I'm confident it's the right key

[gentle solstice]

Sadly, I always get iptables errors when I use podman-compose

[indigo zenith]

Are you copying the key into the container? If not, do that and make sure it's really there

[topaz aspen]

not too sure - looked through a few different approaches, maybe copying to test would be fine tho just whilst getting things running

is the "multistage" usually used to wipe files or whatever in the previous build, like keys etc ?

[indigo zenith]

Docker has a "secret" function, that's probably the ideal approach https://medium.com/trabe/use-your-local-ssh-keys-inside-a-docker-container-ea1d117515dc

[topaz aspen]

i can't seem to do COPY /Users/name/.ssh/key /root/.ssh/id_rsa anyway for some reason

[topaz aspen]

i'll have a look at this thanks

[topaz aspen]

 > [2/3] RUN useradd -m user:
#9 0.227 useradd: Permission denied.
#9 0.228 useradd: cannot lock /etc/passwd; try again later.

:S

i don't really understand which images can do what, i guess jupyter/datascience-notebook is completely different to ubuntu or whatever

maybe i'll try that guide exactly and then see if it ports over to that jupyter image - is that something a multistage build would do ?

ssh-copy-id didn't work ah, going to bed 😦

[cursive nova]

hi guys, im making a discord bot and am trying to host my bot using heroku. everything is working except for when I try to run my bot token, i get

2022-08-04T02:28:36.236205+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.10/site-packages/boto/s3/connection.py", line 188, in __init__
2022-08-04T02:28:36.236309+00:00 app[worker.1]:     super(S3Connection, self).__init__(host,
2022-08-04T02:28:36.236311+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.10/site-packages/boto/connection.py", line 568, in __init__
2022-08-04T02:28:36.236464+00:00 app[worker.1]:     self._auth_handler = auth.get_auth_handler(
2022-08-04T02:28:36.236466+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.10/site-packages/boto/auth.py", line 1018, in get_auth_handler
2022-08-04T02:28:36.236754+00:00 app[worker.1]:     raise boto.exception.NoAuthHandlerFound(
2022-08-04T02:28:36.236800+00:00 app[worker.1]: boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials```

im using configure vars in heroku
here is the line ``aclient.run(S3Connection(os.environ['TOKEN']))``
i am using boto ``from boto.s3.connection import S3Connection``
any ideas why im getting this and how to fix it?
i have boto installed using pip install boto as well

[balmy veldt]

Some one guide me to learn devops (some one be my mentor please)

[mighty breach]

I’m trying to code my own black ops 1 mod menu on multiplayer I want it to be online what’s the best way to do this?

[cursive garden]

For python logging - if we are working with classes/objects, should the logger go in at the class level or the module level. For example should it be:

log = logging.getLogger(__name__)```

or 

```import logging

class MyClass():
  def __init__(self):
    self.log = logging.getLogger(__name__)```

Or is there no practical difference other than style?

[short peak]

I use both ways. First when I need it in several components (classes, functions, etc), usually standalone scripts. Second one mostly applied in medium/large apps where you want to be consistence and probably set some logger configuration at higher levels

[indigo zenith]

I would start by including sudo per https://adamtheautomator.com/docker-permission-denied/

[indigo zenith]

There's a dedicated SRE / DevOps server, you may have better luck there.

Asking "be my mentor" out of the blue rarely works... Ask specific questions and take an interest in people's individual career experiences to start building relationships

Check this out if you haven't yet: roadmap.sh/devops

[placid glen]

I need help with windows cmd. The console correctly display unicode (éèö) but when I redirect a command output, the format is wrong. for example:
icacls C:/some/dir > C:/output.txt -> output.txt contains malformed chars

how can I handle that ?

[tawdry needle]

maybe you need to specify #egg=? what error did you get?

[tawdry needle]

windows cmd is probably not using utf-8 encoding and so is misinterpreting the utf-8-encoded output

possibly relevant https://stackoverflow.com/a/57134096/2954547

[thorny shell]

it also depends on how you're viewing that C:/output.txt file

[tawdry needle]

__name__ will always be the name of the current module so it definitely should be global in the module (imo)

[topaz aspen]

What's the full syntax for that?

[tawdry needle]

pip install 'git+https://github.com/foo/quux@v1.3.2#egg=PyQuux3

see https://stackoverflow.com/a/55526482/2954547

[topaz aspen]

Is any of that a tag? (I don't have any tags atm)

[tawdry needle]

v1.3.2 would probably be a tag in this case, yeah. although sometimes people put version numbers as branch names, e.g. if they use the git-flow model and they are working on a release branch

@<ref> should work with any ref: branch, tag, commit hash

[topaz aspen]

Ok thanks I'll have a look when I'm at my desk, I'm guessing PyQuux3 is the name of the package that'd be imported

[tawdry needle]

no, PyQuux3 is the name of the distribution if it were uploaded to pypi

[topaz aspen]

Hmmm

[tawdry needle]

remember that the names of the import modules/package(s) contained in a distribution package are unrelated to the name of the distribution package

e.g. the distribution called "scikit-learn" installs the sklearn module

(where an "import package" is a module that contains other modules)

[topaz aspen]

Right hmm

[tawdry needle]

perhaps a better syntax would have been pip install pyquxx3:git+https://github.com/foo/quux, putting the distribution/"egg" name first. but we currently have #egg= and that's just the way it is

[topaz aspen]

Ha all good, I need to read the so post properly

[tawdry needle]

do you have a clear understanding of distributions vs. modules? that i think is the key here

[placid glen]

@tawdry needlethe output in the terminal itself is good, it's the result of the redirection

I managed to find a work around : I now use powershell instead of cmd

took me 2h to figure out the proper commandline in powershell and adapt my program

so now I hate powershell

[tawdry needle]

powershell is actually pretty cool, but it's very different from conventional shells

[placid glen]

yeah "very different"

"very special"

[thorny shell]

seems about the same to me

[placid glen]

I like the syntaxe where you have to put "& 'your_path'" to actually have spaces in path taken into account

that was a brilliant idea

[thorny shell]

I've noticed my IDE does that sometimes, but never knew why

[tender current]

I ran pip freeze > requirements.txt but I also have a pyproject.toml which allows specifying required dependencies. Do these have the same use cases?

[topaz aspen]

erm - I don't think i do really... At least - i get that I can have a base directory called Mug and the package could be tea or something, and then pip installing Mug provides package tea from Mug... i think i get that at least, never really considered it. Does that read like i'm making things up as i go or as though i have a vague idea 🙃

[tropic olive]

Just run everything as root user

[tawdry needle]

that's not far off.

a "package" or "import package" is a module that contains other modules; it's what you import.

a "distribution" or "distribution package" is a collection of one or more modules and/or import packages, bundled up together with metadata like dependencies, a version number, et alia; it's what you install.

python itself only knows about the former. pip mostly only deals with the latter.

by convention, a distribution usually includes exactly one top-level module or import package, corresponding to the name of the distribution. but not always!

[topaz aspen]

Oh,I didn't know there could be multiple top level directories

[tawdry needle]

yep

iirc setuptools does this, it installs a pkg_resources module as well as setuptools

this is actually a nasty supply chain attack vector in python packaging. you can publish a distribution with some arbitrary name that installs its own malicious numpy or requests module

(or monkeypatches malicious code into such a module, but that's a different attack vector)

[warped latch]

I wonder if someone would look at my dockerized app and give me an opinion.

celery-dashboard
    0.0.0.0:5555->5555/tcp,
    8000/tcp  
nginx
    0.0.0.0:443->443/tcp,
    80/tcp,  
    0.0.0.0:8000->8000/tcp
celery-worker
    8000/tcp
celery-schedule
    8000/tcp
django  
    8000/tcp 
postgres:10   
    0.0.0.0:1383->5432/tcp
redis:6       
    0.0.0.0:1382->6379/tcp

django is a web app

celery can control django with a taskqueue and through background processes.

postgres is a db for users.

redis is currently using databases 0 and 1, the first for celery, and the second for caching. (both on same ip:port)

nginx exposes 80 so that it can route traffic to https.The reason it exposes 8000 is so it can receive the django app. (I know its probably exposing things it doesn't need it atm)

The very top one, celery-dashboard (flower) exposes debug settings on 5555. It is not behind nginx or authentication of any kind. The only reason it exposes 8000 is because it's connected to the celery-worker which is connected to the django app by volumes/networks.

I saw a guide that was like, "How to put flower into production," and I'm thinking that that is a terrible idea. But in truth, without flower, I don't have a good way of knowing that my celery app is doing anything at all.

I wonder if there is anyway to access it (some basic auth at least would be needed) without exposing it to the live web.

**If you do not have input about this specific tool, perhaps you could talk about how you access frontend dashboards in prod that are not protected ** Obviously you have to protect but how and where? Can basic auth with htaccess on 2nd nginx container be trusted? Do you need a different domain? Do you make it available only through vpn? And if so... how do you do that.

[indigo zenith]

Maybe look into Nginx Proxy Manager. It's really easy to use

Port 80 would be publicly exposed for that proxy container only, allowing you to expose what you want how you want with a fairly intuitive interface

If you need more control and/or automation you can do the same with Nginx directly, but it requires more knowledge

[rapid sparrow]

I was single Dev, so I went for lazy solution, reverse proxy nginx in front of flower/monitoring systems with enabled basic Auth

In serious organisation, u would wish to make sure u have dynamic web interface to add and remove users to access to objects of choice (similarly to how it happens in Django admin for users)
I would probably make simple custom Auth with Django if I would have found how to have reverse proxy reading JWT tokens with my set level of permissions for a user

U know, I realized I remember that squid proxy allows customisation with python scripts out of the box
It would be gloriously simple and useful
It is only under question if squid can work as reverse proxy. I used it only as forward proxy

[warped latch]

It seems like you can on that last note.

[warped latch]

I will look into it.

but like... anyone who scanned the domain or did subdomain enumeration could figure out that you have a flower server and know that because you implemented auth yourself there is a good chance you botched it.

I would think you would want a different domain for monitoring tools. But maybe that is just like... a false sense of protection anyway.

But wouldn't you at least wnawt two different nginx servers from the ones you serve django from?

or is nginx perfectly capable of safely routing multiple sites in isolation?

[indigo zenith]

I don't have experience configuring professional level stuff myself but I think ideally anything that doesn't really need to be public should require a VPN. Otherwise, yes, using an Nginx reverse proxy to expose your main app without exposing the database and other stuff it's connected to seems to be pretty standard.

[warped latch]

Yeah, to be honest if it's not professional, I would implement a way to put it up and take it down when you're not actively looking at, because... if you're not than most likely no one should be haha

Mine is not professional but I imagine it would be one day and I'm not really interested in shortcuts as much as knowing the right way to do it.

But I suppose the right way to do it does depend on your models.

business and threat models i mean

https://stackoverflow.com/questions/19689510/celery-flower-security-in-production

but also https://github.com/mher/flower/issues/1217 lol

But they merged his PR

I'm gonna have to make sure i'm pulling this version.

[indigo zenith]

Until the next vulnerability pops up ;) So don't expose any service you don't really need to

[warped latch]

That is a very good point. The reality is that I'm so new to celery that I kind of do need it in staging at least. Its possible maybe one day I will understand better and not need monitoring tools for it. But at the same time... I would have to have some way of knowing what celery was doing because you otherwise do not know.

I've come to not care for OAuth very much anyway, I probably would not implement it unless a company forced me to.

But thats kind of getting into #cybersecurity

[indigo zenith]

You'll always need monitoring tools, but you hopefully never need to expose those tools directly to the entire internet.

The details on how to make sure that's it the case depend on the deployment platform etc

[warped latch]

no definitely not. Its just that the tool available for monitoring makes you have to think about htis.

right

Well, im not to that point yet. But Im trying to think ahead

Most likely this app would be.... cut down from the dockerized staging version and made to go onto to heroku for as little money as possible. Ideally 0 money. If the app doesn't make any money I dont understand why you would pay for it and I dont know if it will. It probably wont lol

That is one of the reasons Im using the redis for broker/responses, and caching. Because they have a free tier redis haha

I know that that isn't exactly ideal for a professional app that you know will be successful

[indigo zenith]

I see... If you're trying to host this at zero cost then yes I suppose you'll have to cut some corners on security. But if you can afford to deploy this as containers, you could add a VPN to the mix and get to Flower that way quite safely

[azure crater]

How can I include a DLL for my python package?
I want my python package to be multi-platform, but I need the DLL to be compatible with other platforms. How can I do this?

[tawdry needle]

DLLs are windows-only, but there are different equivalents on other platforms. you will need to build and publish a "wheel" (a binary package) for each platform

i'd start by just getting the packaging and distribution setup to work on one platform first, then figure out the others

@azure crater
see here for instructions on including "non-python" files in your distribution using: https://setuptools.pypa.io/en/latest/userguide/datafiles.html

see here for accessing non-python files in packages from your code: https://docs.python.org/3/library/importlib.html#module-importlib.resources

see here for building wheels using the build tool: https://pypa-build.readthedocs.io/en/latest/

ultimately you would upload both your source tarball and your built wheels to pypi.

[river cairn]

Hi, anyone using Visual Studio 2022 python edition... How can intellisense detect all my packages from custom python environment? Because I can't get my intellisense to work. Thanks

[warped latch]

I wouldn't cut corners. I would just treat the app like a test run and eventually bring it down if it had no profit model. I would not cut corners on paying customers tho or people I had otherwise committed to.

Unfortunately.. I have to stop working on it a bit. Two of my computers are infected with virulent USBs that replicate partitions...

Sigh

Wish me luck

They target both linux and windows

Im screwed

I meant to say rootkit. Not sure where virulent cane from

[echo cypress]

hello. is there someone that runs lavalink in a container? I have this Dockerfile to cpy the config

FROM fredboat/lavalink:master-v3.4

COPY application.yml ./

When I run the program inside the container (after doing docker run -ti --entrypoint=/bin/bash lavalink, just for testing) there is no logs folder and I don't know why

[heavy knot]

brew isnt in my path variables so i cant use it... any idea why this is happening?

is there a way i could add brew to my path variables

[thorny shell]

have you installed brew?

'cuz iirc it puts itself on your PATH

you could do something like PATH=$PATH:/opt/homebrew/bin, assuming that directory is indeed where brew is.

That won't persist past the current shell process, though

[heavy knot]

yea brew is installed even reinstalled it

its probably bc of the new M1 macs

[thorny shell]

well I have an M1 and it's /opt/homebrew/bin there

[heavy knot]

yup same

but how do i add it to my path variables though for all sessions

[thorny shell]

here's what I have in ~/.zshrc

if command -v brew > /dev/null
then
    eval "$(brew shellenv)"

    HELPDIR=$(command brew --prefix)/share/zsh/help
fi

pretty sure the installer did that for me

[heavy knot]

hmmm

what would echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bash_profile do?

[thorny shell]

try it and see

[heavy knot]

i did it, doesnt work not even for the current session

[thorny shell]

well

/usr/local/bin is the wrong directory

it's /opt/homebrew/bin like I said above

and there's a 50% chance that bash is the wrong shell, too 🙂

Macs have you use zsh by default now

https://www.youtube.com/watch?v=YozC8yFrZKI

[heavy knot]

wow ur right lol

i reinstalled brew and just did these two commands lol

i wonder what echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bash_profile did though

[thorny shell]

it put that text at the end of that file

>> means "append"

doesn't hurt anything

[heavy knot]

yea

[thorny shell]

looks like you did it twice 🙂

[heavy knot]

yea lol but it was the wrong path

u were right thanks

[thorny shell]

💐

[heavy knot]

well this is weird

doesnt zprofile set the environment for the logged in user?

[rapid sparrow]

import logging

# create logger
logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)

# create console handler and set level to debug
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)

# create formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# add formatter to ch
ch.setFormatter(formatter)

# add ch to logger
logger.addHandler(ch)

# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warning('warn message')
logger.error('error message')
logger.critical('critical message')

when i try using code example like this. I expected when writing using it to see only my debug msgs
but this thing
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)

also made sure that i see debugging messages from all third party dependencies (even from inbuilt asyncio i get debugging msgs)
is there a way to configure it in a way i don't see third party msgs but still see console output?

[rapid sparrow]

Found!

logger = logging.getLogger('simple_example').getChild("custom")

this made sure to use a child in hirearchy of logging system

which made sure i set my rules for child instead of messing up with root logger that is responsible for everything

[midnight rover]

Hi! Does anyone know a good resource on logging in python? I know how to use logging, I don't know when to use logging. Like, I can identify some bits that would be useful to know what's going on, but I'm finding it hard to draw the line where to stop, where to do it, etc

[rapid sparrow]

it is explained in official docs
https://docs.python.org/3/howto/logging.html#when-to-use-logging

[rapid sparrow]

the point of logging to increase observability of a system, think of it as debugging in staging and production infrastructures through seeing stuff that happens in logs.
People use different log aggregating system like Loki Grafana, Logstash Elastic Kibana, or even SaaS like datadog in order to find quickly logging msgs about some events / happening in the right or in the wrong way

it is a completely complimentary system which you should only use after you already had your system tested xD

more sophisticated print xD

due to existing logging aggregators, i really need only StreamHandler out of this python logging

although it is no doubt cool that logging library allows different funny other handler options to output logs

coming to think of it... its HTTPHandler is perfect.... to make easy quickly error output into discord channel 😁

[tawdry needle]

you might be interested in the Eliot logging framework, i finally tried using it after years of putting it off and it's been very interesting so far

[tawdry needle]

for sure, cloud logging has made it less important to juggle log handlers

[thorny shell]

I feel like I spend my professional life groveling logs

I hate it. Programs always either log too much or too little

[tawdry needle]

i spent way too much time groveling logs at my last job specifically because my org refused to take logging seriously as a part of application design

no attempt to even generate a request id and attach it to the log metadata. no attempt to separate tornado log noise from our own app logs. etc

something like eliot would have made it a lot easier to debug problems too

[rapid sparrow]

you could be wishing solution like datadog then
it is SaaS log aggregator, that has nice GUI with log tracing to any error

[thorny shell]

we're starting to use datadog

[tawdry needle]

datadog is great. the web interface is a bit weird but the tool itself is great in terms of what it can do.

[thorny shell]

log tracing aint free, though; you gotta instrument your code to generate or pass along the traces iiuc

[tawdry needle]

datadog's tracer is pretty magical, you activate it at the top of your app and then just ignore it

[rapid sparrow]

ergh. Due to log tracing being not free, in my pet projects I will live just on Loki Grafana solution xD + AlertManager stuff

kind of wishing to try eventually ELK solution and see how much it is different perhaps I always used only PLAG so far

[thorny shell]

ELK is better than nothing but compared to what I hope datadaog is, it's pretty primitive

[rapid sparrow]

sure. Datadog is really good thing. It has stuff like enabling query explain/tracing on each query in sql databases xD

with its graphical interface it makes really nice analysis of db

datadog is like... really mltipurpose solution, it has parts that are monitoring like Prometheus

parts that are logging

parts that are tracing stuff (I haven't treid stuff Jaeger/New relic stuff may they are similar?)

datadog is surely... one solution for everything in terms of observability xD i have no idea about the costs though

wow, nice Jaeger is open source

i definitely should try and see what is does

curiously New Relic is open source as well, curious

they are both from a category of tools named Application Monitoring. i haven't tried even one out of it. definitely must try them in action in order to realize what this level of tool category does

[midnight rover]

That's what I am trying to avoid

[thorny shell]

good luck 😐

[mystic void]

You should!
One of the benefit is the tight integration between these parts.
It means they can link logs, exceptions, traces and all sorts of data together

[river cairn]

Has anyone had any issues with Visual Studio 2022 IDE not showing intellisense for any pip package? Thanks!

I can only see intellisense suggestions from my code, not from environments packages

[elder beacon]

guys is it possible to use python to read for example a exe details like version etc..

if yes can you tel me how

i've been search but i cant find it

[rapid sparrow]

google search: python reading windwos file properties
answer: https://stackoverflow.com/questions/67734678/reading-windows-file-properties-in-python
https://mhammond.github.io/pywin32/

I was able to find this piece of code
https://github.com/mhammond/pywin32/blob/70ddf693927fa1635f15e9ef41eb1aea37fdf32a/win32/Demos/cerapi.py

        for fileAttr in wincerapi.CeFindFiles(os.path.join(startMenu, "*")):
            fileName = fileAttr[8]
            fullPath = os.path.join(startMenu, str(fileName))

library is purely documented, you need to read its code to find necessary features

[vernal trench]

For the easiest and stupidest way, you can use GUI automation tools like Clicknium to simulate your operation to get the file property. But if you have a lot of files to be processed, this is not an effective way. You can follow the doc and have a try: https://github.com/clicknium/clicknium-docs

[loud bear]

I think this lib might be of help

!pypi lief

[rancid schooner]

lief v0.12.1

Library to instrument executable formats

[loud bear]

That info is added by the linker so its in some standardized pe section

Seems like it can also be done with pefile

!pypi pefile

[rancid schooner]

pefile v2022.5.30

Python PE parsing module

[trail vapor]

hi, does anyone know of a repo or just anything that can reformat proxies with auth from ip:port:name:pass to name:pass@:ip:port?

[rapid sparrow]

lets have it clear, you expect a user to come at hostname address ip:port:name:pass
and being reirected to name:pass@:ip:port?
it is impossible. u are breaking internet protocol/addressing rules

you can make only something like

user is coming at ip:port with headers that contain name:pass
and then redirected to name:pass@:ip:port

[rapid sparrow]

although lets also be clear, it is for HTTP requests
you need to be clear what is your resource target, if its TCP, i think it it will not work

[trail vapor]

no, im just trying to set up proxies upon a requests using the 'requests' module. No one is coming to the addresses. Im using the adderess to go somewhere. I jus need a reformater lol

cuz i have over 1000 and i cant reformat them by hand

[rapid sparrow]

ah, so you just need tool that replaces all strings from one string to another string in file?

[trail vapor]

yes

[rapid sparrow]

that's easy with python at least. it can be done in regular linux tools but i don't how in them

[trail vapor]

this whole splicing and reformatting stuff is outta my skillset apparently

[rapid sparrow]

let me write you simple python script for that

[trail vapor]

i thought it would be too

thank you 🙏

[rapid sparrow]

can i asssume no more than one ip:port:name:pass per line in file?

[trail vapor]

yuh

[trail vapor]

and if it could be like that in the 2nd file

[rapid sparrow]

import re
import argparse
import logging
logging.basicConfig(encoding='utf-8', level=logging.INFO)

parser = argparse.ArgumentParser(description='addreess auto formater')
parser.add_argument('--input', type=str, default="input.txt")
parser.add_argument('--output', type=str, default="output.txt")
args = parser.parse_args()

with open(args.input, "r") as file_:
    data = file_.readlines()

any_word = r"[^\s]*" # without spaces

with open(args.output, "w") as file_:
    for line in data:
        found = re.search(rf"({any_word}):({any_word}):({any_word}):({any_word})", line)

        if found:
            def from_address(ip, port, name, password):
                return fr"{ip}:{port}:{name}:{password}"

            def to_address(ip, port, name, password):
                return fr"{name}:{password}@{ip}:{port}"

            parsed = found.groups()
            file_.write(
                line.replace(from_address(*parsed),to_address(*parsed))
            )
            logging.info(f"parsed={parsed}, replaced={from_address(*parsed)}, to={to_address(*parsed)}")

        if not found:
            file_.write(line)

your thing is made

tested on

sdfsdf
192.157:5000:login:password
abcbcbcb 192.157:5000:login:password 123213
abc
sdsdfdsfsd

to make

sdfsdf
login:password@192.157:5000
abcbcbcb login:password@192.157:5000 123213
abc
sdsdfdsfsd

by default opens input.txt file and outputs to output.txt, you can replace it with CLI arguments

python3 formatter.py --input=input.txt --output=output.txt

[trail vapor]

this is great, thank you!

[rapid sparrow]

u a welcome. removed code repeats

[rapid sparrow]

oh and i forgot to fix regex to \s* in line begining

fixed

kind of does not make sense to have such regex though

[rapid sparrow]

fixed to more universal regex

[heavy knot]

What are devops?

[rapid sparrow]

a cultural approach to software development, about having developers caring about Operations support of the product: its infrastructure / reliability / scaling / security / maintanance
having developers writing the mentioned aspects as a code in a product, and deployed in a transparent way to everyone what is changing from previous release to next release.

main goals are usually automating any Deployment and Maintanance support of a product. Everything should be as automated/autonomous as possible.

[rapid sparrow]

usually from more practical point of view it assumes having someone to write CI CD pipeline of automated testing and deployment of a product through different kind of invoked infrastrcture as a code tools

it allows faster to integrate code between multiple developers into one master branch

and faster seeing feedback from auto redeployed version to staging infra

removes human factor from deployment

from a practical point of view, devops usually involves multiple involved technologies to know. DevOps engineer is usually really quick learner
and ideally a person with previous software developer experience + experience to maintain products

[heavy knot]

Ah

Thx

[rapid sparrow]

also from a practical point of view, Software Architect / Cloud Architect Engineer knows all about different ways to have infrastructure and designs a product accordingly. So it is great responsibility on the one who designs the product before distributing as tasks what to do to other devs

u a welcome

[lament ore]

hey

i have a vps server which i’d like to use to run python scripts

i’m currently using vscode’s ssh remote development tool to run one python script at a time

i was wondering if there was any way of “queueing” python script jobs in a web UI so these can be ran while i’m not sshing into it

and then seeing the output via the web UI

[rapid sparrow]

Everything is possible in many different ways, but I can't recommend solution exactly without more clearly understanding your situation

[rapid sparrow]

Describe in detail, what are you wishing to have / why are wishing, what is your problem (not what u wish to implement)

And provide code examples

[lament ore]

i ended up fixing it!

https://github.com/jhuckaby/Cronicle

i just want to schedule/run tasks (python code) on a web UI and come back to see the output later

[rapid sparrow]

Not enough information what to recommend. I still did not get reason why u wish to do it

[lament ore]

i wish to do it to prevent waiting 5h for a python script

they process a lot of data so they can take a good amount of time

by a lot i mean 8 million rows of data

[rapid sparrow]

Will be code changing for those tasks, or input variables?

So you expect different code versions of python to run?

[lament ore]

no, just one version for now

i thought about just using a docker container as well

[rapid sparrow]

This thought crossed my mind as well to recommend

[lament ore]

im not super experienced with docker containers and i’d like to track the cpu usage too

so i think the cronicle tool is best for my specific scenario

[rapid sparrow]

Do u have multiple third party dependencies to your script?

[lament ore]

like other python modules?

i do yes

i’m using pandas, numpy and yfinance

i’m also storing everything on a db

[rapid sparrow]

That is important detail

Which database

[lament ore]

sqlite3

i might use something like postgre or an actual sql server

not being able to easily view the db is pretty annoying

its a 2GB db

[rapid sparrow]

Do u expect in web UI to see logging info in order to check its status?

[lament ore]

Cronicle should output the print()s from my script so yeah

just in case something goes wrong

is there a significant performance impact if i use postgre?

instead of sqlite

[wooden ibex]

You have to run a Postgres server so overhead of running a server

[lament ore]

true, would that eat a lot of ram?

[rapid sparrow]

Is logging information secure sensitive?

[rapid sparrow]

At least 0.5 ram or more

[lament ore]

no, not really. secure sensitive info would go to the db directly

[rapid sparrow]

New script runs will have to write to same dB, or they can write to fresh new one?

[lament ore]

the same dB

[rapid sparrow]

Is script needed automatically relaunched periodically of the same version?

Or only when new code version is available

Or u wish explicitly manual relaunches

[rapid sparrow]

nevermind. I think you are better of using this Cronicle i guess. In order to repeat same with standard software... would be doing much more actions

it could be repeated with Django + Celery + PLG stack, but it looks like easier to use your Cronicle, as it has already perfect UI for the current task

[tender current]

@gentle solstice Poetry seems to have horrible chemistry with torch sigh Switching back to venv

[gentle solstice]

How so?

[lament ore]

thank you! i might look into making a docker container with portainer once it becomes a production server to do it “properly”

[gentle solstice]

?

[lament ore]

so i could have a portainer that just runs that same python script in a volume

another one with the postgres sql db

etc

[tender current]

https://github.com/python-poetry/poetry/issues/4231

[rapid sparrow]

yeah, consider your postgresql database and chronicle running in containers

you could be also raising pgadmin container web UI for your postgres

[tender current]

 poetry add torchvision==0.13.0+cu116

Updating dependencies
Resolving dependencies... (1.2s)

  SolverProblemError

  Because torchvision (0.13.0+cu116) depends on torch (1.12.0)
   and lds depends on torch (1.12.0+cu116), torchvision is forbidden.
  So, because lds depends on torchvision (0.13.0+cu116), version solving failed.

Such a headache. This is insane. I just wanted to move files in a folder lol

[gentle solstice]

What if you just add the latest version and let the dependency resolver handle everything?

poetry add pytorch torchvision

[tender current]

Is there some poetry uninstall pytorch command I should run first?

[gentle solstice]

remove

[tender current]

poetry add torch torchvision torchaudio
Using version ^1.12.1 for torch
Using version ^0.13.1 for torchvision
Using version ^0.12.1 for torchaudio

Updating dependencies
Resolving dependencies... (1.5s)

  RepositoryError

  403 Client Error: Forbidden for url: https://download.pytorch.org/whl/cu116/pillow/

  at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/repositories/legacy_repository.py:393 in _get
      389│             if response.status_code == 404:
      390│                 return
      391│             response.raise_for_status()
      392│         except requests.HTTPError as e:
    → 393│             raise RepositoryError(e)
      394│
      395│         if response.status_code in (401, 403):
      396│             self._log(
      397│                 "Authorization error accessing {url}".format(url=response.url),

Anyone tried to run poetry with torch?

[gentle solstice]

Do you have g++ installed?

[sturdy tide]

What is the difference between sam build and docker build? I would like to test a local container with a lambda base image in a lambda environment(includes /var/task/.) Will sam build solve that problem?

[tender current]

 g++ --version
g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0

[gentle solstice]

What you can do is exclude pytorch from poetry and install it manually afterwards.

[short peak]

not sure about sam build, but testing those bases using docker build is not friendly. https://docs.aws.amazon.com/lambda/latest/dg/images-test.html#images-test-AWSbase

[tender current]

Install it manually with pip within a poetry shell environment?

[gentle solstice]

yes.

wow, pytorch is huge. 700mb

[tender current]

Yeah...

[gentle solstice]

You can probably download the whl file manually and add the file with poetry add

Unfortunately, that will lock you into the python version

➜ poetry add torch
Using version ^1.12.1 for torch

Updating dependencies
Resolving dependencies... (0.2s)

Writing lock file

Package operations: 10 installs, 0 updates, 0 removals

  • Installing pyparsing (3.0.9)
  • Installing attrs (22.1.0)
  • Installing more-itertools (8.13.0)
  • Installing packaging (21.3)
  • Installing pluggy (0.13.1)
  • Installing py (1.11.0)
  • Installing typing-extensions (4.3.0)
  • Installing wcwidth (0.2.5)
  • Installing pytest (5.4.3)
  • Installing torch (1.12.1)

~/pytorture via 🐍 3.10.5 took 1m 3s

[tender current]

Yeah, now try to add torchvision 😦

[gentle solstice]

done

Installed v0.13.1

[tender current]

what

[gentle solstice]

Try updating python?

[tender current]

it's 3.10.6

[gentle solstice]

This isn't yours?

[tender current]

It is

[gentle solstice]

Says 3.8

[tender current]

at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/repositories/legacy_repository.py:393 in _get Oh yeah. Weird

I had ran poetry env use /path/to/python3.10

[gentle solstice]

delete the old env

[tender current]

and running python --version within the poetry shell gives me 3.10.6

[gentle solstice]

oh yeah, it's poetry that's running on 3.8

Remove the extra registry block in your pyproject.toml. You don't need it.

Unless you want the absolute latest

[tender current]

Ok, removed the env

How do I make sure I'll be running on the right python version this time?

(And I didn't have a registry block in the toml)

[gentle solstice]

It may have been this. ```toml
[[tool.poetry.source]]
name = "torch_rep"
url = "https://download.pytorch.org/whl/cu111/"
secondary = true

[tender current]

Oh, indeed, yeah I removed that

Ok it worked now, but I believe it's equivalent to running on Cuda 10.2:

[gentle solstice]

With that gone, you shouldn't get any "Authorization error" errors

[tender current]

Rather than on 11.6 (which I want)

Because the first one is just default installation, the 2nd one requires that wheel link

Yeah I was right

 python -c 'import torch; print(torch.version.cuda)'
10.2

Whereas I'm running on 11.7 (and torch latest is 11.6)

 head /usr/local/cuda-11/version.json
{
   "cuda" : {
      "name" : "CUDA SDK",
      "version" : "11.7.0"
   },

[tender current]

So...yes to this

[gentle solstice]

Ok, without the repository in your pyproject, run this? ```
poetry add https://download.pytorch.org/whl/cu116/torch-1.12.1%2Bcu116-cp310-cp310-linux_x86_64.whl

[tender current]

That seems to have worked:

 python -c 'import torch; print(torch.version.cuda)'
11.6

[tool.poetry.dependencies]
python = "^3.10.5"
torch = {url = "https://download.pytorch.org/whl/cu116/torch-1.12.1%2Bcu116-cp310-cp310-linux_x86_64.whl"}

It added the torch line. No torchvision nor torchaudio there, but I'll add my other deps first and try running my code, see if vision and audio are even needed

[tender current]

@gentle solstice Done with adding & installing deps through poetry. Imports still don't resolve. Do I still need to setup.py file & running pip install -e . for this? I'd understood that poetry replaces it, maybe I was wrong

[gentle solstice]

no.

just do poetry install

[tender current]

Already did

[gentle solstice]

it does that for you when you run poetry add

[tender current]

poetry install
Installing dependencies from lock file

No dependencies to install or update

[gentle solstice]

run via shell or poetry run python...

[tender current]

I am

[gentle solstice]

Do you still have your module not found issues?

[tender current]

├── LDS
│   ├── __init__.py
│   ├── book_loader.py
│   ├── summarization.egg-info
├── __init__.py
├── dev-requirements.txt
├── experiments
│   ├── __init__.py
│   └── ex_hf_summarizers.py
├── poetry.lock
├── pyproject.toml

from LDS.book_loader import BookLoader
ModuleNotFoundError: No module named 'LDS'

[gentle solstice]

Can you push to a git repo so I can take a proper look?

[tender current]

Alright, it'll take a little while to set that up

[pine fern]

@rapid sparrow is this true?

[rapid sparrow]

Partially, yeah. Serious companies would run their stuff only in Managed Kubernetes and AWS ECS systems or their analogs. I see kind of no point to try upgrading something in place, I replace stuff too.

But... sometimes we need a tool to make installation to something even if it is replacable cattle. Like... we don't get managed k8s to run CI runners or pet projects, because in both cases managed kubernetes a bit too expensive for this usage case. We install microk8s, k3s. And that's where we need ansible to adjust the server / making its installation. So ansible place still has its role, just more rare to use

[rapid sparrow]

It is possible to use Ansible in a cattle philosophy. All its scripts should run automatically always on server creation.

[tender current]

@gentle solstice Discord blocks my sending you the link privately without being friends. And I'd rather not share that repo here

[gentle solstice]

oh well

[tender current]

Would it be ok to add me temporarily?

[gentle solstice]

Sure.

[tender current]

Thank you

[rapid sparrow]

Plus you can consider that not each company is ready for k8s/AWS ECS xD That's where config management tools come into place

[pine fern]

i see, i see. thanks for the insight DW

[pine fern]

tru

[rapid sparrow]

Also, there is a point to consider using Salt in replacement of Ansible, i haven't tried it, but i heard it has state management in loop enforcing strategy

like Kubernetes it is checking in a loop server, and applies its state from time to time

making drift of configugration by manual hand pointless, sys admins will not be able to change something xD

Ansible has more popularity and ecosystem though. But I still can see that in some cases Salt stuff can be better

In any case this stuff we need only to deal with outdated companies, for CI runners / solutions and pet projects xD and for especially poor companies / client who are unable to afford proper cloud and skilled people to maintain EKS/ECS solutions

[pine fern]

yep

tbh i most likely wouldnt be touching too much of that stuff

since most of anything i deploy as an API would be on the aws side

[rapid sparrow]

It is nice to know that you can use it. Handy tool to keep in mind 🤷

[pine fern]

true

[rapid sparrow]

i made a software which offers installation of ansible script with VPN as service xD

[pine fern]

my boss made an in-house solution for automation of aws services

idk why but he showed it to me

[rapid sparrow]

what?

automation of AWS services?

like terraform/Pulumi/AWS CDK alternative?

[pine fern]

yeah

literally alternative

[rapid sparrow]

it sounds kind of stupid

[pine fern]

its obv not as extensive

the only thing it has going for it

is that it has easy-to-spin up UI/frontends

[rapid sparrow]

ah. that's not infra provisioning tool then