tools-and-devops

[rapid sparrow]

i worry for my pet projects only about a problem of that...

due to lack of statefulness in a regular helm and regular manifests

i could be having deployed smth i no longer wished to have

but i forgot to delete it with deleted code

[slim maple]

I didn't really encounter any of these problems because people develop and deploy applications they're responsible for, we didn't have a strictly devops team that would deploy apps for us

[rapid sparrow]

terraform(and pulumi) helps to ensure that is not a prroblem (when something is not removed from cluster, but the code was already deleted)
terraform and pulumi will delete all objects to which code is no longer present. Declarative code.

[slim maple]

Also dealing with standard k8s/helm manifests would get you farther than using terraform imo

I don't need another abstraction

[rapid sparrow]

😛 can we utilize full programming language at least? https://cdk8s.io/

it can form Helm packages

[slim maple]

No ...

Why would I need any of that 😅

[rapid sparrow]

for proper type safeness and power of a full programming language?

regular manifests lack... stuff

helm with its... templating language is kind of having limitations too
Full programming language has no limits 😊

[slim maple]

If you're trying to do anything complex in them that's just not the right tool or place to do that

[rapid sparrow]

exactly. with full programming language(or at least terraform or pulumi) we can build complex stuff with right tools

we can write reusable infra code

that we can save in shared infra library and reuse for different microservices

[slim maple]

Listen, stop 😅 I don't want to debate these tools really, I just don't need them

They seem overly complicated for what most people need

[rapid sparrow]

well, no need then no need 😅

[little yacht]

Quite a long discussion 😂
Thanks for helping me out

[mystic void]

thoughts on nats vs rabbitmq?

Am tempted to go with nats

[willow pagoda]

minor question about git/github, what happens if someone forks a repository and commits to the main branch, then tries to synchronize with the upstream repository? does a merge commit have to be made every time they need to pull and/or make a PR?

[mystic void]

that's up to the user. They may rebase or merge

[willow pagoda]

i see, so it's mostly an inconvenience/mess of commits if they forget to make a new branch

[mystic void]

That's one way to see it.
The world won't stop for the fork. There will be other forks, people will continue working on the main repo, etc.

I tend to always work from branches anyway, even when forking.
It enables me to keep master clean and closer to upstream as well as having multiple streams of work

[thorny shell]

if

• they commit to their main
• they submit those changes as a merge request
• you accept the merge request
• they do "git pull" from your repo
  they will find that they'll have to either merge from your repo, or rebase onto it, or just throw away their work on their main

I'd expect you could dry-run all of this with two separate repos that you own

[earnest forum]

Hi folks, what's your goto for integrating github actions with server deployment

[lunar drum]

Yes.

[lunar drum]

More specifically, what kind of server? What kind of application are you deploying to said server?

GitHub Actions is more than capable of deploying infra for just about anything. For basic deployments you likely don't even need Python.

[earnest forum]

Deploying to an ec2 instance with gitHub action

[thorny shell]

that seems pretty heavyweight for a github action

you want to deploy a new instance anytime someone merges to main?

[earnest forum]

Does anyone know why when running tests on my docker image one out of 10 the image builds and the tests run however for the rest of the time I get connection refused on port 5432.

[earnest forum]

No, I'm deploying to the instance

[thorny shell]

I wonder if you've got postgres in a second image, and it isn't ready by the time that the first image tries to run its tests

iirc postgres can take a little while to become ready

[earnest forum]

I have a wait for db command that pools the postgres container before the postgres container can actually run and it passes that test

[thorny shell]

that's weird. I assume that test specifically connects to port 5432

if your stuff is on github, gimme the url and I'll take a look

[earnest forum]

I have a wait_for_db command in django's management that pools postgres to see if its active before the tests start to run

[thorny shell]

*polls

iirc "docker compose" has something that will do that for you, but I've never tried it.

[earnest forum]

yep

[earnest forum]

i think its deprecated or something

[thorny shell]

aw 😦

[earnest forum]

this the script:

"""
Django command to wait for the database to be available.
"""
import time
from psycopg2 import OperationalError as Psycopg2OpError
from django.db.utils import OperationalError
from django.core.management.base import BaseCommand


class Command(BaseCommand):
    """Django command to wait for database."""

    def handle(self, *args, **options):
        """Entrypoint for command"""
        self.stdout.write("Waiting for database...")
        db_up = False
        while db_up is False:
            try:
                self.check(databases=["default"])
                db_up = True
            except (Psycopg2OpError, OperationalError):
                self.stdout.write("Database unavailable, waiting 1 second...")
                time.sleep(1)

        self.stdout.write(self.style.SUCCESS("Database available!"))

[thorny shell]

looks reasonable

[earnest forum]

It shows the Db is available but whenever, its time to run the test then i get this callback:

Waiting for database...
Database available!
Found 37 test(s).
Creating test database for alias 'default'...
/py/lib/python3.10/site-packages/django/db/backends/postgresql/base.py:405: RuntimeWarning: Normally Django will use a connection to the 'postgres' database to avoid running initialization queries against the production database when it's not needed (for example, when running tests). Django was unable to create a connection to the 'postgres' database and will use the first PostgreSQL database instead.
  warnings.warn(
Traceback (most recent call last):
  File "/py/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/py/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/py/lib/python3.10/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/py/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/py/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/py/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "db" (172.18.0.2), port 5432 failed: Connection refused
    Is the server running on that host and accepting TCP/IP connections?

[earnest forum]

@thorny shell do you have any idea what might be the issue

[thorny shell]

I wonder if the name postgres resolves to the correct IP address

[lunar drum]

If you're just deploying code changes to ec2, you can do something simple like this using ssh:
https://stackoverflow.com/questions/62087265/deploy-code-directly-to-aws-ec2-instance-using-github-actions

[earnest forum]

I later got it fixed based on what you said about postgres not waiting. For whatever reason the wait for db was passing but postgres wasn't ready. I had to use the command you talked about and the stack overflow post that said that was deprecated was misleading

[thorny shell]

oho!

I wonder why your test didn't work -- it sure sounded like it should have

[earnest forum]

It works on local builds though and I never had the issue on local builds

[earnest forum]

Thanks, I already tried some had issues with getting it to work which is why I had to ask on here

[lunar drum]

Yeah? What kinds of issues are you having with the SSH method?

[earnest forum]

Tried the easing-boy action, didn't work for me, got permission denied when the action tries to ssh into the instance

[earnest forum]

meant to write *easing-theme

[harsh panther]

How will I merge mine current commit and the new commit which happened on the main branch at once

Suppose something happened on main branch and i worked on some other branch and used git commit but now main branch is ahead of 2 commits and i also have some commit which isn't on main branch how will I combine both?

[brazen forge]

either rebase your branch onto the main branch or merge the main branch into your branch

[harsh panther]

Locally?

I want to open pull request on main branch

[brazen forge]

you could also open the PR and do these later

or (depending on repo settings and whether there are conflicts) not even have to do these

[harsh panther]

Ok so I just have to accept the incoming changes and all commit will be saved along with my local commits inline right?

[small jay]

Usually I do git pull --rebase in situations like this

[harsh panther]

Oh alright ty

[brazen forge]

if you have conflicts, check the changes you're accepting

[small jay]

If there are no conflicts it should be pretty smooth yeah

Otherwise you'll have to fix them

[harsh panther]

Got it ty very much : ) have a nice day/night

[glass plinth]

hello

if any one interested in Knox or Samsung I have a theory

[brazen forge]

We're not a general tech support server. Please use an off-topic channel.

[gilded cave]

Does anyone have a recommendation for a way to integrate docstring checking / linting into ci?

[rapid sparrow]

pytest can run doctest tests in docstrings

sphinx can build documentation with docstrings through autodoc 🤔 (not sure how much it is safe from submitting wrong data though)

otherwise.. pylint can check pressence of docstrings in general. linter.

[slim maple]

I would run ruff, it should include a lot of rules from pydocstyle, but you have to enable them

Also depends on which git hosting platform you use, I'd say CI in gitlab is easier to set up than GitHub actions

[gilded cave]

I already have ruff set up in gha so ill look into that, thank you

[alpine horizon]

how can I see if a string ever appeared in my source code with git? (and get the commit hash)

[earnest forum]

hi guys, i'm trying to add https to my docker container and was aiming to use certbot and integrate it with my nginx container but someeone suggested using traefik. Does anyone has experience working with the latter

[buoyant cairn]

hi

can some someone help me with git

how i discard these changes

[rare anvil]

i use this and its wonderful https://github.com/eugene-khyst/letsencrypt-docker-compose

[earnest forum]

thanks would check it out

[willow pagoda]

from the command-line? i would get the hash of the commit im going back to (git log --oneline helps), then use git reset <hash>
if you want a graphical UI from vscode, i suggest installing the GitLens extension so you can see your commits in the same side tab

[earnest forum]

i already have an nginx container setup using the nginx unprivileged user, i only need to be able to plugin certbot into that container

[rare anvil]

eh honestly it would be much easier to migrate to this, takes like 5 mins, I tried setting up certbot + nginx myself w docker by following articles online and couldnt get it to work 😛

[thorny shell]

wow, I've been looking for something like that @rare anvil ; thanks

[earnest forum]

what about using the nginx image as a reverse-proxy for other services

[rare anvil]

yeah it does it

[mystic void]

<@&831776746206265384> looks like someone trying to get voice access (spam across a bunch of channels)

[livid schooner]

i'm getting really frustrated trying to set up a virtual environment for my pygame project in vscode. i've tried just about every command i've found online, but i'm not able to do it

here's how my vscode looks rn

i think i managed to activate the environment, but i'm not sure

in any case i can't install from the requirements.txt, it says everything is already installed

[earnest forum]

I think this container fits the use case in terms of being able to manage multiple docker services from a single nginx proxy and also managing the static files easily, however, how would this fit into like a github-actions pipeline so everything would have to be automated and no manual interventions has to be made to allow the script run

[rocky granite]

it's not active, there also doesn't seem to be a venv directory in your project, in order to create one, enter
python -m venv venv
After which vscode should ask you if you want to use it automatically (i don't remember the exact content of the prompt), but agree to that, then after you restart vscode it should automatically enter venv every time. You'll know by
(venv)
prepending your current working directory (screenshot)

You can always activate it manally by doing this
.\venv\scripts\Activate.ps1
Since by default vscode uses powershell, that's the script to activate it, otherwise for default cli use
venv\scripts\activate.bat

After venv is active, install your requirements by
pip install -r ".\requirements.txt"

That should do it

[green rampart]

Basically im trying to get the electron sublevel configuration by getting a useful remainder

How do i do step 3 in a way such that i start subtracting X' left to right such that the code gives me the name of the term and the number of electrons in it

[earnest forum]

Hi guys, has anyoone used this image for nginx_proxying before: https://hub.docker.com/r/jwilder/nginx-proxy/

[toxic bough]

Hey @green rampart, your question is offtopic for this channel. Please see #❓｜how-to-get-help for instructions on how to claim your own help channel.

[green rampart]

I did but no one answered then the submission got closed

[earnest forum]

i need some docker help

[astral apex]

(You have to ask your question for us to be able to respond to it)

[slim hearth]

Guys

[earnest forum]

i already asked. Would ask again.
I'm using the jwilder's nginx-proxy image to create an nginx-proxy which is coupled with let's encrypt functionality to automatically add SSL encryption which is a service on its own.

I have another service which is the app itself which registers with the proxy to get requests. I expose port 443 and 80 to get access to traffic and then in the container i have uwsgi running on port 9090. I'm not able to get traffic to uwsgi mainly based on how the proxy is setup. Have gone through the docs but maybe i'm missing some. Here's how my docker file looks like

The proxy + lets encrypt docker-compose:

version: "3.7"

services:

  reverse-proxy:
    image: "jwilder/nginx-proxy:latest"
    container_name: "reverse-proxy"
    volumes:
      - "html:/usr/share/nginx/html"
      - "dhparam:/etc/nginx/dhparam"
      - "vhost:/etc/nginx/vhost.d"
      - "certs:/etc/nginx/certs"
      - "/run/docker.sock:/tmp/docker.sock:ro"
      - static: /var/www/html/static
    restart: "always"
    networks: 
      - "net"
    ports:
      - "80:80"
      - "443:443"

  letsencrypt:
    image: "jrcs/letsencrypt-nginx-proxy-companion:latest"
    container_name: "letsencrypt-helper"
    volumes:
      - "html:/usr/share/nginx/html"
      - "dhparam:/etc/nginx/dhparam"
      - "vhost:/etc/nginx/vhost.d"
      - "certs:/etc/nginx/certs"
      - "/run/docker.sock:/var/run/docker.sock:ro"
      - "acme:/etc/acme.sh"
    environment:
      NGINX_PROXY_CONTAINER: "reverse-proxy"
      DEFAULT_EMAIL: "[e-mail]"
    restart: "always"
    depends_on:
      - "reverse-proxy"
    networks: 
      - "net"

volumes:
  certs:
  html:
  vhost:
  dhparam:
  acme:
  static:

networks:
  net:
    external: true

#create the network: docker network create net

The apps dockerfile:

FROM python:3.10.13-alpine3.17
LABEL maintainer="Walexero"

ENV PYTHONUNBUFFERED 1

COPY ./requirements.txt /tmp/requirements.txt
COPY ./requirements.dev.txt /tmp/requirements.dev.txt
COPY ./scripts /scripts

ARG DEV=false
RUN \
 python -m venv /py && \
    /py/bin/python -m pip install --upgrade pip && \
    apk add --update --no-cache postgresql-client jpeg-dev python3-dev pcre-dev && \
    apk add --update --no-cache --virtual .tmp-build-deps \
        build-base postgresql-dev  musl-dev zlib zlib-dev linux-headers && \
    /py/bin/pip install -r /tmp/requirements.txt && \
    if [ $DEV = "true" ] ; \
        then /py/bin/pip install -r /tmp/requirements.dev.txt ; \
    fi && \
    rm -rf /tmp && \
    apk del .tmp-build-deps && \
    adduser \
        --disabled-password \
        --no-create-home \
        django-user && \
    mkdir -p /vol/web/media && \
    mkdir -p /vol/web/static 

COPY ./app /app

RUN chown -R django-user:django-user /vol && \
    chown -R django-user:django-user /app && \
    chmod -R 755 /vol && \
    chmod -R 755 /app && \
    chmod -R +x /scripts && \
    chown django-user:django-user /scripts/run.sh && \
    chmod +x /scripts/run.sh

WORKDIR /app
EXPOSE 80
EXPOSE 443

ENV PATH="/scripts:/py/bin:$PATH"

USER django-user

CMD ["run.sh"]

The docker-compose for the app service:

version: "3.9"

services:
  app:
    build:
      context: .
    restart: always
    volumes:
      - static:/var/www/html/static/app/static
    expose:
      - "9090"
    environment:
      - DB_HOST=db
      - DB_NAME=${DB_NAME}
      - DB_USER=${DB_USER}
      - DB_PASS=${DB_PASS}
      - SECRET_KEY=${SECRET_KEY}
      - ALLOWED_HOSTS=${ALLOWED_HOSTS}
      - DEV=false
      - VIRTUAL_HOST=myapps_domain_name
      - VIRTUAL_PORT=9090
      - VIRTUAL_PROTO=uwsgi

    networks:
      - net
      - backend

    depends_on:
      - db

  db:
    image: postgres:16.0-alpine3.18
    restart: always
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=${DB_NAME}
      - POSTGRES_USER=${DB_USER}
      - POSTGRES_PASSWORD=${DB_PASS}
      - PORT=${DB_PORT}
    networks:
      - backend

volumes:
  postgres-data:
  static:

networks:
  net:
    external: true
  backend:

And the run.sh script of the dockerfile that starts uwsgi:

python manage.py wait_for_db
python manage.py collectstatic --noinput
python manage.py makemigrations
python manage.py migrate
uwsgi --socket :9090 --workers 4 --master --enable-threads --module app.wsgi

[slim maple]

Don't run makemigrations during startup

It should only be ran by a developer during development

[earnest forum]

There was some issues with the migration files i had made and i was using github actions and had to add it for the db to start up, it was hindering the db. The more likely solution would be for me to clear up the migration files but that isn't the cause of the issue right now and i can remove it

[slim maple]

If you forget to create your migration it would silently create and run it

That's bad 🙂

If you want you can validate that migrations are up to date in CI

[earnest forum]

i'd do better

[slim maple]

But I'd say delete that from your startup script

[earnest forum]

yep i would

[rare anvil]

why wouldnt it able to be automated

[earnest forum]

i would have to input the letsencrypt info in the terminal which means i have to interact with it

[rare anvil]

you setup https once at the start then your project should run allg

yeah you set it up once with the ./cli.sh config stuff

then after youve setup the letsencrypt info you just launch your docker compose services

[earnest forum]

i'm hoping to make the deployment from github actions so i'm trying to prevent any manual interactions

[rare anvil]

oh ive never used github actions before sry

[earnest forum]

that was the reason i didn't use the image you provided because some manual interactions still has to be done with the setup

[rare anvil]

from what i just read about github actions i think u can just setup the config manually once for https, then have github actions launch based on docker compose up

[earnest forum]

i'm aiming to have multiple services run on a single server which is why i need the process automated. If that wasn't the case i would have gone with it. Plus the previous setup i was able to get working. Each service had its own nginx container and if i were to have 3 services with different nginx that would be a waste of server resources when i can have a single nginx that proxies requests to all services. And that way the letsencrypt certification would be able to work automatically for them as well which is what the jwilder container provides. Some said i should look into traefik but i think there's a learning curve to it

[elder solstice]

I've been trying to share configuration files between packages in a monorepo, which has the following layout:

$REPO_DIR/
  config/
    js/
      .eslintrc.json
      tsconfig.json
    python/
      .flake8
      pyrightconfig.json        
  packages/
    lib1/
      js/
        .eslintrc.json
        tsconfig.json
      python/
        .flake8
        pyrightconfig.json
    lib2/
      js/
        .eslintrc.json
        tsconfig.json
      python/
        .flake8
        pyrightconfig.json

I've managed to achieve this for the JavaScript devtools by providing the extends option in the JSON files.

However, this is not as straightforward for the Python devtools.

• For flake8, I added the following CLI options: ["--config $REPO_DIR/python/setup.cfg", "--append-config setup.cfg"], when running flake8 inside the libN/python directory. This appears to work.
• How can I achieve the same for Pyright? The documentation mentions multi-root workspaces but that requires placing the configuration file inside the root of the repository, which I'd like to avoid, as that would lead to clutter. I'm also unsure whether this would merge the configs together or only adopt the closest one.

[livid schooner]

i ended up asking a friend who showed me how to do it properly. turns out i'd forgotten to activate it in the screenshot i sent, but the issue turned out to be that i was installing from requirements.txt using my global pip install, and not the one in my virtual environment

thank you for your answer, though

[stoic tiger]

do you guys know of a small tool that takes a requirements.txt / setup.cfg and checks if there were news updates ? to monitor potential breaking changes in case version are not all pinned

[visual oxide]

I use pip-compile from pip-tools

Then dependabot on top of that

[stoic tiger]

@visual oxide superb

[slim maple]

pdm/poetry, if you don't mind switching to pyproject.toml

Both can export dependencies to requirements.txt format though

[tawdry needle]

i don't know if this is too ugly, but git does support cross-platform symlinks to some extent

[tender cobalt]

I´ve been trying to configurate my OpenCV using CMake so i can use it on my GPU it only says there is an error, and their may be a problem with the project files if anyone would like to take a look a the build feel free to dm me

[slim maple]

Maybe switch to ruff?
https://docs.astral.sh/ruff/settings/#extend

Or are you using plugins a lot? 🤔

[elder solstice]

I'm managed to side-step the problem by using the same config for all of my packages. In that case, I can just set the --project option in the command line without having to worry about inheritance. Unfortunately, there seems to be no way to specify a config file for VSCode's pylance server, so I have to repeat the options once.

[elder solstice]

That looks promising, I'll try it out as a replacement for flake8 and other code linters/formatters. Too bad it doesn't support Pylance.

[tawdry needle]

i switched from flake8 to flakeheaven to ruff, it's a big improvement

[elder solstice]

Just made the switch. Having everything in one place makes things so much easier to manage

[slim maple]

Now do select = ["ALL"] to select all rules 😅

[hoary shale]

oi any way to specify a dependency for windows only using pdm?

[willow pagoda]

normally you'd use an environment marker for that
https://pdm.fming.dev/latest/reference/pep621/#dependency-specification ```py

Requirement with environment marker

"pywin32; sys_platform == 'win32'"``` (see also: https://packaging.python.org/en/latest/specifications/dependency-specifiers/#environment-markers)

[hoary shale]

ooo neat, cheers!

[spice ginkgo]

I am going to send the key event to the wxpython dialog to input in the focued field.But can't
Please help me

[indigo zenith]

#❓｜how-to-get-help

[spice ginkgo]

Do you know how to add proxy to the wxpython dialog?

[spice ginkgo]

Do you know how to add proxy to the wxpython dialog?

[rapid sparrow]

Sphinx pure rst or sphinx markdown? Which u think is better for long term support? 😅

i am kind of liking that markdown will be read by devs from Github GUI without problems

Although github supports RST looks like too https://github.com/DevDungeon/reStructuredText-Documentation-Reference

not a fan of rst though... markdown is 20% cooler.

probably will roll with markdown

i think markdown is more readable from notepad (in case nothing is present at all)

[tawdry needle]

rst is probably better for authoring api docs because of the support for directives and roles

not sure how sphinx handles that with markdown

[rapid sparrow]

i am using with autodocs module to document library in python

[tawdry needle]

i really wish sphinx had an asciidoc frontend, because asciidoc maps much more closely to rst, but with the relatively good readability and similar ergonomics of markdown

i see, sphinx-markdown uses the myst extension, which does support rst-like roles and directives https://myst-parser.readthedocs.io/en/latest/syntax/roles-and-directives.html

that's strongly worth considering...

i might try it in my next project

i still prefer adoc but i'll settle for not-rst

[rapid sparrow]

got started with it.
it has embedding Rst feature inside

So i am able to utilize Sphinx-Rst features freely in my markdown when necessary

Excitement. Nice to use markdown sphinx autodocing documentation 😊

[tawdry needle]

that's useful

[brazen forge]

!rule 9 We're not a recruitment board

[rancid schooner]

Rules

9. Do not offer or ask for paid work of any kind.

[heavy knot]

I won’t pay

[brazen forge]

why not use #1035199133436354600 then

[neat flame]

any one know gmail api

[brazen forge]

!rule 6

[rancid schooner]

Rules

6. Do not post unapproved advertising.

[brazen forge]

!rule 6 This counts as an advertisement. Please read the #rules

[rancid schooner]

Rules

6. Do not post unapproved advertising.

[opaque egret]

Hello everyone!

[atomic dew]

Wondering if there are any good tools and approaches for deploying python portably in offline environments?

Basically, building flexible deployment methods in general where it's a little tougher to have package management etc set up.

[tawdry needle]

1. Use pip-compile (part of https://pypi.org/project/pip-tools) to generate a "locked" requirements file
2. Use pip download to download binary packages ("wheels") from the locked requirements file
3. Use python -m build (https://pypi.org/project/build/) to build a wheel of your own package
4. Put all the wheels onto the offline system however you normally would, along with the desired version of Python

the only tricky piece is that pip-compile does not have any support for "cross-compilling" the requirements file, so you might have to run it in a VM or container to get the precisely correct requirements (this can matter because some packages require certain deps based on operating system or python version)

pip download however does have support for fetching wheels based on platform, python version, etc https://pip.pypa.io/en/stable/cli/pip_download/

[atomic dew]

Thank you!

[full abyss]

Hi! I was messing up with moviepy and i noticed that the videos that i get when i concatenated them for example, have a lower size that the original ones.

Does moviepy compres them by default?

[cedar nacelle]

besides pdb which debugger is good to use for python ?

[rapid sparrow]

I like using vscode visual debug
And running not just as app
But installing Pytest explorer extension and running visual debug for specific tests

[cedar nacelle]

I never really got the hang of that but I'll try it again. Thanks

[placid hemlock]

I was planning to put my dotfiles on github, in which I include gitconfig, is it safe to include the line signingkey = in the gitconfig file?

[brazen forge]

yes, if you only put the fingerprint of the GPG key or a public part of the SSH key, if you use that

[placid hemlock]

how do you obtain fingerprint of the gpg key?

[brazen forge]

gpg --fingerprint

[rapid sparrow]

pytest visual debug works mostly out of the box

it is only missing ability to easy turn on "visual debug navigating through third party libs too"

i need to put json config when i need this feature. Otherwise it works out of the box

or may be i just missed to utilize GUI that should be already having easy turning it on 😅

[tawdry needle]

https://pypi.org/project/pdbp/

[cedar nacelle]

Cool. Will check that out Thanks

[soft hare]

Ok my question is only tangentially related to Python, but I couldn't find a better place to ask.
https://github.com/docker/setup-qemu-action
Can someone who understands docker/qemu explain what this does? All I want to do is run my cross-compiled binaries on the arm64 image 😭 They have no runtime dependencies, so I want to just copy the file over and run it 😦

[tawdry needle]

it looks like this just sets up QEMU in the GH runner environment, you still have to run it yourself. so you're looking for something like "how to run docker container with qemu"

(i don't know how to it, but i should)

[soft hare]

do you know anyone on this server who had a hand in the cibuildwheel python project?

because they know how to do all this, and idk how to understand what they've done 😦

[tawdry needle]

nope, sorry. maybe there's insights to be gained from their source code

[buoyant atlas]

Hi, does anyone tried to implement CICD to format code using black?

[rapid sparrow]

don't format code, just check if it needs to be formatted in CI!
black --dry .
and then ideally in PR person will need to fix it before merging 😅 otherwise even in master branch person can get a hint, when will start receiving spam of emails regarding failed CI

CI should not change Git repository code.

[buoyant atlas]

This is how I feel about this, yet I found few sources that were saying that in "this day" that kind of task should be done by CI not developer.

[rapid sparrow]

as a DevOps engineer (which has its a job to do it in a right way) i can assure you the correct way is going with --dry check and not changing Git repo code.

Changing git repo code is pretty much not allowed nonsense that is never justified

CI should be only running checks and building artifacts into external storages. CI is allowed to put results of its executions into Tag/Releases/S3 buckets/Docker registry/even remote hosts and etc, but not into git repo itself back

[buoyant atlas]

I also work and DevOps but i tend to question my judgment

anyway

Thanks!

[rapid sparrow]

u a welcome. Also, we have turned on via terraform Github setting for our repo that automatically forbids merging PR unless it passes CI checks, for unit tests and formatters and etc

that ensures that we never had not formatted code merged to master

as well as hopefully rarely having code that breaks CI in master branch 😅

obviously PR contributing strategy is needed to be used for that

[glacial ember]

Hey, switched back to Windows, having bought a Surface recently. Running into odd behavior on Powershell. I have ".py" in $env:pathext and I have ".../python312/scripts" in $env:path. If I run a Python script from the scripts directory, Powershell acts as if I'm opening an unknown file from the File Explorer as it shows the "Select an app to open" dialogue box. However, if I execute the script after cding into the ".../python/scripts" directory, it'll run normal in the terminal. Any Windows experts with a clue as to why this is happening?

[rapid sparrow]

back from what?

try running everything from git bash 😊 https://git-scm.com/downloads
it will bring more... normallity experience into windows

it emulates... having less windows inside.

paths will be back to normal like ../python/script too 😊

[glacial ember]

sweet, that sounds like a good tip. thank @rapid sparrow

[rapid sparrow]

👍 also, it gets back normal shelling like. mkdir, cat, echo $STUFF and etc.
I think even shell scripts are runnable from it.
Since it has name git bash. may be it has bash inside actually, if to be precise.

i remember i liked very much how i stopped having confusion that my servers and dev machine had different console navigating commands 😅 (reflexes are hard to fight)

[glacial ember]

i've been scratching my head for a couple days until now. thanks again! yea, powershell is not a strength of mine, but at least it's there nowadays making it somewhat closer to linux. that being said, i already miss zsh

oh my zsh*

[tawdry needle]

powerrshell is actually kind of c ool once you get used to it

[tawdry needle]

why not just run python .../python312/scripts/myscript.py?

windows doesn't have the same "shebang" system as in unix-like systems

for one thing, it probably isn't even finding the .py file, as you need to modify $env:PATHEXT https://superuser.com/a/437853

that might be all you need, but i'm not sure about that

actually on windows you'll also have the benefit of the py.exe launcher if you used the official python.org installer

[glacial ember]

Already did all that. Would be nice to avoid this absurd issue but GitBash seems to be working sufficient for the moment.

Actually, ran 'py python/scripts/suricata-updata' and it does work but I'm stumped how Powershell is shitting the bed right now when environment is setup. Tried to do the 'set-alias -name scupdate -value suricata-update' and that seems to malfunction too. Might be this new PS version. I'm retreating to the age old notion that Windows just sucks 🤷

For dev, I'm sticking to WSL and Vscode on remote. But I'd like the suricata tools on host env.

Correction* set-alias -name scupdate -value py python/scripts/suricata-update

[rapid sparrow]

or u could just install Linux as main machine if it is an option 😊 Then u can skip using linux through seconds hands of WSL and remote connections
I love defaults for everything to work from Kubuntu 22.04 LTS.

[tawdry needle]

it's very possible that powershell just doesn't know how to run a python script as an "executable". i'm a little confused here though, because it sounds like this is not a script in the casual sense, but actually a "console script" as installed by a python package. is that right?

[glacial ember]

Correct, suricata-update was installed by Pip

First line of the script points to python. Eg "#!C:/.../python.exe"

[elder folio]

I'm trying to get into linting (and formatting) with ruff. I've been avoiding it because I didn't want to manage another tool config but alas it seems it is the right thing to do...

So in terms of managing config I believe I need to set ruff to ignore something I find common in flask, importing items after some non import expression. Does anyone know the right term/setting that I am looking for to tell ruff to ignore this?

from flask import Blueprint

bp = Blueprint('main', __name__)

from app.main import routes

by default ruff just deletes the late import

found it, E402
https://docs.astral.sh/ruff/rules/module-import-not-at-top-of-file/

my bad for not realizing the output was telling me exactly what I needed 😉

[dawn ridge]

Hi I’ve been trying to make a San key diagram and have been having trouble getting it to display because there’s a lot of components. I posted in #1035199133436354600 but my case was closed. I simply want to display the relationships, I tried making them in to a webpage and that didn’t work either (there’s a lot of connections)

[tawdry needle]

is it just me or is it a lot of fuss to use pip-compile the "right" way? i develop on mac but deploy in linux docker images, so i need to run pip-compile inside a container in order to get a requirements file for use in the container. so now i need a second container image just so i can build my container image...

[thorny shell]

I am only vaguely familiar with docker, but I believe "using a container to build your container" is reasonably common

[tawdry needle]

sigh

i wrote what amounts to little more than a script to move data from system A to B with nearly 0 transformation along the way. it's something like 50 lines of functional code, 150 lines of tests, another 50 lines of boilerplate for input validation/logging/aws credentials, and 300 lines of scripts and dockerfiles and other mumbo jumbo

it makes me want to not do things the "right" way

so much fuss just to shuffle data around

i feel like the world needs a better way

[slim maple]

Could you run pip-compile during build?

[tawny temple]

Wouldn't that defeat the purpose of locking the dependencies? May as well directly install from the requirements files that would've been passed in to pip-compile.

[slim maple]

Ah, I'm just not familiar with the tool sorry

I used poetry but switched to pdm around a year ago, also pydantic uses it

[tawny temple]

Now that I am thinking about it - what is the benefit of locking dependencies? It seems to go against being full CI/CD. Instead of automatically getting updates to dependencies when deploying, updating dependencies has to be an explicit manual action. There are some tools designed to help with this like dependabot (especially for security updates), but it still requires some manual action I think or otherwise generates a lot of noise.

I suppose that prevents situations where you want to make some change and suddenly your build fails due to dependencies coincidentally being updated.

[slim maple]

It is so your docker build is reproducable

And installing that set of dependencies in general

[rapid sparrow]

🤔 somehow I am afraid in advance of packager that inspired from npm

Suspicious manager a bit

What is its point over poetry

[tawny temple]

Right, good point. Though I think that doesn't necessarily mean we have to check in our locked dependencies like is common practice. Would it be weird to instead lock dependencies at build time and export a requirements.txt as a build aretfact in case we need to reproduce the build later?

[rapid sparrow]

@tawny temple
Recommendation to use
pip freeze> constraints.txt
pip install -r requirements.txt -c constraints.txt

As minimal simple alternative at least

Constraints aren't installed, they just lock deps. Can be a mess machine generated.
Use written for humans only main deps in requirements.txt.
It will make possible to make sense which deps u a no longer needing and can delete.
With lack of main dep. All secondary deps will not get installed in this way

[slim maple]

Pip freeze sucks

[tawny temple]

My point is that if we lock stuff we don't get automatic updates when we deploy. If we use a constraints file, doesn't that also have the same limitation?

[slim maple]

Good luck figuring out what dependencies are transitive

[tawny temple]

I guess we don't need updates to be on the same cadence as deployments. There can be some automated job running at a regular interval to relock with latest updates and commit the lock file. That seems noisy though.

Just thinking out loud here, apologies for my rambling

[slim maple]

So, regarding your question about automatic updates - what if your dependency isn't pinned? E.g. it's pydantic and pydantic v2 releases

[rapid sparrow]

It will be annoying when your code will be breaking from any third party dep update.
Easier to figure them out in controlled updates with reading which stuff is breaking in changelogs of according libs

Otherwise u will have potentially disappointment from your code base instability very quickly

[rapid sparrow]

Also with frozen locks, u have environment reproducible between multiple developers

With lack of lock, u all can be having eventually different deps
And lack of understanding why u all code with different results

[tawny temple]

I'll have to come back to this discussion tomorrow.

[tawny temple]

Yeah brought this up as a downside in an earlier message

[tawny temple]

Locking means pinning to an exact version. We can still pin to e.g. A major version and get update for minor versions

But we're not in a perfect world and packages use inconsistent versioning schemes (or none at all) for breaking changes. And also what is a breaking change can be hard to practically define

So yeah, not an easy problem

[slim maple]

You mentioned updating your versions automatically, so, that would make a container with broken dependencies/codebase

[tawny temple]

Well hopefully you'd have tests catch it or the build would fail

[slim maple]

That depends on how you run the tests too

For example I don't run them in the same container at all

Honestly pdm kind of just works for me - you can update dependencies (including transitive dependencies)

This only updates a lockfile, and if I want to bump minimum version of a specific package I could pdm add <package> (the same as installing one) or I could update dependcies manually

Also it takes care of dev dependencies too

[rapid sparrow]

Can u update specific packages, without updating everything else? 😁

[slim maple]

Yep

pdm update <package>

Since I have tests for everything I just bump all the packages

[rapid sparrow]

Unbelievable. Same command in pipenv is still updating everything hehe

[slim maple]

Let me try

$ pdm update httpx --dev -G testing
Updating packages: httpx.
🔒 Lock successful
Synchronizing working set with resolved packages: 0 to add, 1 to update, 0 to remove

  ✔ Update httpx 0.25.0 -> 0.25.1 successful

🎉 All complete!

It only updated httpx in the lockfile

[rapid sparrow]

What if u just recently updated all already, no?

[slim maple]

I didn't commit anything to git

So I just rolled back to previous commit and did pdm sync --clean, --clean removes dependencies that aren't in your lockfile

Also pyproject.toml is just nicer to work with:

requires-python = ">=3.11"
dependencies = [
    "uvloop>=0.17.0; sys_platform != 'win32'",
    "uvicorn>=0.23.2",
    "fastapi>=0.104.0",
    "pydantic>=2.4.2",
    "pydantic-settings>=2.0.3",
    "asyncpg>=0.28.0",
    "sqlalchemy>=2.0.22",
    "alembic>=1.12.1",
    "python-dotenv>=1.0.0",
    "orjson>=3.9.10",
    "sentry-sdk>=1.32.0",
    "result>=0.13.1",
    "aioinject>=0.15.0",
    "aiosqlite>=0.19.0",
    "passlib>=1.7.4",
    "argon2-cffi>=23.1.0",
    "pyjwt>=2.8.0",
    "cryptography>=41.0.5",
]
[build-system]
requires = ["pdm-pep517>=1.0.0"]
build-backend = "pdm.pep517.api"


[tool]
[tool.pdm]
[tool.pdm.dev-dependencies]
dev = [
    "black>=23.10.1",
    "ruff>=0.1.3",
    "mypy>=1.6.1",
    "radon>=6.0.1",
    "deptry>=0.12.0",
]
testing = [
    "pytest>=7.4.3",
    "coverage>=7.3.2",
    "httpx>=0.25.0",
    "anyio>=3.7.1",
    "sqlalchemy-pytest>=0.3.0",
    "asgi-lifespan>=2.1.0",
    "pytest-archon>=0.0.5",
    "typeguard>=4.1.5",
    "freezegun>=1.2.2",
]
types = [
    "types-passlib>=1.7.7.13",
]

[rapid sparrow]

Useful

[slim maple]

And it manages your venv for you, I think you can also switch interpreters 😉

[rapid sparrow]

can I manage venv on my own with it? 😅
I need it in exact location

[slim maple]

You can manually create it and work with pdm inside of venv

And I think you can specify a location too

There's a pdm-python file, which is local to your pc

(in a project root, next to pyproject)

It basically points to a python interpreter

C:/path/to/your/project/.venv/Scripts/python.exe

@tawny temple What do you think?

[tawny temple]

Will catch up on this tomorrow, sorry!

[slim maple]

Haha, no worries

There's also a linter that could validate usage of your dependencies:
https://github.com/fpgmaas/deptry

For example so you don't accidentally use a dev dependency in your code

It should work with requirement.txt too 👀

[tawdry needle]

i've actually been thinking about this in response to my high annoyance level, and i think the main benefit is in fact preventing your build from unexpectedly breaking, which i have definitely seen a couple of times

whereas if dependencies are tightly locked, it's pretty easy to revert just to get your build working, then you can figure out the problem later

however i really should check out PDM, it seems to have become quite a mature project since i lasy looked at it

@slim maple can pdm lock deps for a different platform? or would i still need a container/vm for that?

[slim maple]

I think it's platform independent

You can export it to requirements.txt, then they'll be platform specific

[finite fulcrum]

I think there would be a bit more annoyance with builds breaking more in ci instead of locally and having to put in tighter constraints instead of just not re-locking if it's something new

but I do agree that bumping is a bit annoying because it's either done automatically like with poetry's default updating and things can rbeak when adding a new dep, or if it's not done automatiaclly people neglect it

[tawny temple]

If you export a lock file at build time then you can use that to revert to a working build. But yeah having pipelines be blocked unexpectedly is a trade off for having automatic updates

[thorny shell]

I think https://github.com/GoogleContainerTools/kaniko will build your docker image for you without requiring docker to be running

[alpine horizon]

how can I add optional dependency groups to poetry that can be used like this: pip install my_package[group1], pip install my_package[group2] ?

[willow pagoda]

those would be extras, and apparently poetry defines them this way
https://python-poetry.org/docs/pyproject#extras ```toml
[tool.poetry.dependencies]
mandatory = "^1.0"
psycopg2 = { version = "^2.9", optional = true }
mysqlclient = { version = "^1.3", optional = true }

[tool.poetry.extras]
mysql = ["mysqlclient"]
pgsql = ["psycopg2"]
databases = ["mysqlclient", "psycopg2"]``` multiple extras can be selected using pip install my_package[group1,group2] btw

[cold meadow]

.

[alpine horizon]

I will try this thank you!

[tawdry needle]

definitely not trying to add kubernetes to this project 😆

[tawdry needle]

true. absolute worst-case scenario is we could theoretically docker cp the exported requirements file from the container image. but yeah we really should save artifacts like that somewhere. currently we just throw out everything on the build agent after pushing to our container repository (i didn't set this up)

[thorny shell]

For sure. But that thing works without kubernetes too.

[hazy echo]

Have you looked at dagger?
https://docs.dagger.io/sdk/python
Some of the original Docker team started a new project, kind of picking up where Docker leaves off.

[gleaming gate]

is it intended behaviour that the mypy from pre-commit does not use the exclude settings comming from my pyproject.toml?

So i get different behaviour from running mypy by myself and by calling pre-commit

[rapid sparrow]

There is no magic 🪄 ✨
Mypy is same CLI tool in both cases.
If u have difference in its working, it means u run it with different CLI args 😉
(For example your IDE could be adding additional ARGS or smth)

[gleaming gate]

Ran both just by myself in the terminal

Ah maybe I've found the issue (it's always the user ;))

[buoyant atlas]

Does anyone worked with databricks? Can you suggest me good structure for databricks repositry?

[split crypt]

Hi, I have been building a simulation tool for underwater gliders to simulate sawtooth and spiral motions. Code: https://github.com/Bhaswanth-A/AUG-Simulator
I am now trying to implement a PID pitch controller for 2D sawtooth motions but I'm having trouble doing it. I am trying to use the simple PID library (https://simple-pid.readthedocs.io/en/latest/reference.html) but the motions just go haywire. I have a feeling I am not calling the function properly. Would really appreciate some help 🙂

This is how Im calling the controller in the Modeling2D/dynamics_2D.py file:

class Dynamics:
    def __init__(self, z):
        self.initialization()

        self.n1 = np.array([z[0:3]]).T
        self.Omega = np.array([z[3:6]]).T
        self.v = np.array([z[6:9]]).T
        self.rp = np.array([z[9:12]]).T
        self.rb = np.array([z[12:15]]).T
        self.Pp = np.array([z[15:18]]).T
        self.Pb = np.array([z[18:21]]).T
        self.mb = z[21]
        self.theta = z[25]
        
        print(math.degrees(self.theta))

        self.g, self.I3, self.Z3, self.i_hat, self.j_hat, self.k_hat = utils.constants()
        
        pid = PID(0.001, 0.001, 0.01, setpoint=self.theta_d, sample_time=None)
        
        self.theta = pid(self.theta)
.
.
.

[brave moat]

Hi guys, I have been working on developing my own Voice assistant I have added few things need some ideas for what I can add to it? Pls suggest some ideas (Either here or in DMs too)

[dense solstice]

Hi, I'm trying to make a pip package but I don't know for some reason a post0 tag is being add to my package name. can anyone help me ?

[slim maple]

To your package name or to it's version?

[astral apex]

How are you building your package?

[mental heron]

can someone help me with this

[heavy knot]

i'm having a issue with bitbucket, my repository uses a submodule, but the link is like:
https://bitbucket.org/COMPANY/REPO/src/

when i click from bitbucket web UI it's points to:
https://bitbucket.org/COMPANY/REPO/REPO/src/

anyone already got that?

[dense solstice]

@slim maple looks like this is the culprit

[astral apex]

Ah

[dense solstice]

I have a question regarding a python project.
I'm keep getting WARNING: swarm-cid-py 0.1.0 does not provide the extra 'lint'

here is the setup.py file https://github.com/Aviksaikat/swarm-cid-py/blob/main/setup.py

feel free to dm me

[willow pagoda]

pip prefers to use pyproject.toml for project metadata if it's present, and since you've declared poetry as the build system which doesn't use setup.py (exclusive to the setuptools backend), it won't run your setup.py at all

i suggest either sticking solely to poetry (i.e. removing setup.cfg and setup.py), or sticking solely to setuptools but using one configuration file (i.e. removing setup.cfg and setup.py)

[dense solstice]

My goal is to use poetry only for development

[willow pagoda]

whats wrong with using poetry for the build process/installation?

[dense solstice]

And for dependencies & all use setup.py is it possible??

[dense solstice]

Nothing I just don't know whether it's recommended or not😅

[willow pagoda]

setuptools and poetry are different build backends, they aren't supposed to intermingle with each other

[dense solstice]

So if I switch to poetry solely it won't be a problem?

[willow pagoda]

assuming you're able to migrate all your necessary metadata to poetry's format, sure

[willow pagoda]

see here for declaring extras using poetry

[dense solstice]

Thank you so much

[willow pagoda]

personally im a fan of setuptools since it added support for pyproject.toml following the PEP 621 standard (if you see a [project] table that looks similar to [tool.poetry], that's PEP 621)

if you're curious how it looks, here's some examples of mine: discord-ws, berconpy, discord.py-i18n-demo (also uses setup.py), thestarboard, discord-ext-pager

[dense solstice]

I see most project use setuptools

That's what I was doing. I used poetry just to maintain a separate virtual env

Can setuptools be used for virtual environment management?

[willow pagoda]

setuptools doesn't manage venvs, but in practice i don't mind it because i only need to run py -m venv .venv or virtualenv .venv once per project and let my IDE activate it for me (or if im on linux i'll use pyenv to handle it)

[dense solstice]

Ah ok

[willow pagoda]

afaik setuptools by itself only serves as a build backend and has to be supplemented with other packages like build and twine, where poetry's frontend tool is all-in-one

you might be interested in hatch or pdm which, alongside being PEP 621-compliant build systems, also have CLIs that can manage virtual environments

[gleaming gate]

okay this is what my problem was:
https://github.com/psf/black/issues/1985#issuecomment-778659648

and for isort its filter-files

and for mypy its currently not implemented: https://github.com/python/mypy/issues/13916

[young island]

I need image exploit

[deft grail]

!rule 5
We won't help with any sort of exploit

[rancid schooner]

Rules

5. Do not provide or request help on projects that may violate terms of service, or that may be deemed inappropriate, malicious, or illegal.

[dense solstice]

cool. what would be the best way to create a pip package & publish ?

[willow pagoda]

if you don't mind doing it by hand, you can create an API key on pypi for your account and then use it with poetry's CLI to publish your package

the same can also be done with github workflows, though i would recommend configuring the newer trusted publishing for it instead (see also pypa/gh-action-pypi-publish's readme and my publish workflow for berconpy)

[dense solstice]

If I use pyproject.toml file I have to define the dependencies in certain way.

Ex. If i want to

Pip install .[dev] will it work on projects without setup.py file ?

Or even if I do include the setup.py file will it work ?

Or I have to do poetry install -E dev

[willow pagoda]

all build backends should be able to declare core metadata (i.e. PEP 621), so having the setuptools-specific setup.py isn't necessary unless you're relying on setuptools and setup.py to define your package metadata

that is assuming your package is correctly configured to be built as a .tar.gz/.whl (source/built distribution)

(my thegamebot project for example doesn't declare any files to be included, so poetry has nothing to build)

[willow pagoda]

once you have your distributions built, pip should automatically install the poetry-core backend as declared in your [build-system] table, so users won't need the poetry tool except if they want to contribute poetry-related changes to your repo

also in case you're wondering, poetry.lock won't apply to your users if they install it via pip
https://python-poetry.org/docs/basic-usage#as-a-library-developer

[dense solstice]

ok so users can just normally do pip install the package ?

[willow pagoda]

if it's configured correctly then yes, but it's fairly easy to misconfigure it if you miss some piece of documentation or whatnot, so after you run poetry build i suggest opening the .tar.gz/.whl distributions with an archive viewer to make sure that it only has the files you intend to distribute

[tawdry needle]

Do you know an example of a trusted publishing service that is not github?

I want to use it but I don't want to use GH for CD

[dense solstice]

what If I use build ?

[willow pagoda]

unfortunately not, im not particularly involved in its integration but afaik pypi only lists github in their publishing tab

[willow pagoda]

build is another frontend like the poetry CLI that runs your build backend to generate the distributions, so it should basically be the same

correction: poetry build can only invoke the poetry-core backend, while build frontends like build and pip are able to invoke any backend defined in the [build-system] table, poetry-core included (see also this glossary)

[astral apex]

Why?

But to answer your question, I don’t think they’ve released support for any others yet

[warped latch]

I'm considering having my servers be in Germany even though I'm in the US. But I wonder if im gonna regret that if I need some kind of customer support. Ive been looking at nettcup for a hobby commercial project (if that makes sense). I know a lot of people recommend them, but I wonder, are there stories of people outside the EU struggling to have them as a provider?

[dense solstice]

sorry for being repetitive. what would be the best way a develop a python project & manage it's dependencies & use setup.py to specify the dependencies & all. I can see hatch uses pyproject.toml which again will cause the same issue as poetry does

same with pdm

I'm trying to use pyproject.toml along with setuptools

[tawdry needle]

hatch, poetry, and pdm all do similar things.

if you use setuptools you end up needing to manually set up venvs, and run build, pip-compile etc manually as needed. or use a tool like rye that does what poetry/hatch/pdm do, but not actually acting as a build backend

[dense solstice]

curretnly I use poetry to manage my env for dev. but as I mentioned it's creating a pyproject.toml file which will trigger pip to ignore setup tools

or should I bite the bullet & move complete to pyproject.toml ?

[willow pagoda]

If you define setuptools as the build backend in pyproject.toml (see their docs), it can parse both pyproject.toml and setup.py at the same time
This is what i did for discord.py-i18n-demo, but only because i wanted to automate compiling my gettext translations during the build process - for new projects, setuptools recommends you stick to declarative configuration files like pyproject.toml:
https://setuptools.pypa.io/en/latest/userguide/quickstart.html#transitioning-from-setup-py-to-setup-cfg

To avoid executing arbitrary scripts and boilerplate code, we are transitioning into a full-fledged setup.cfg to declare your package information instead of running setup(). [context: declarative configuration with setup.cfg was introduced in 2017 whereas PEP 621 for pyproject.toml was written in 2020]
...
New projects are advised to avoid setup.py configurations (beyond the minimal stub) when custom scripting during the build is not necessary. ... Note that you can still keep most of configuration declarative in setup.cfg or pyproject.toml and use setup.py only for the parts not supported in those files (e.g. C extensions).
looking at your setup.py there's nothing that can't be defined in pyproject.toml, so you can move all of it there while switching to setuptools in case you want to use setup.py later

as for managing my dependencies with setuptools, i use pyproject.toml to define them, pip install -U --upgrade-strategy eager -e . to install or upgrade them, and optionally pip-compile if i want to generate a lockfile

[wintry plinth]

@edgy basin

[edgy basin]

package building and publishing kinda go together

rye just seems to be doing everything, which... okay

pyenv is also environment management too

[wintry plinth]

yeah. If Rye becomes unmainttained, I'd be fucked. I'll just swap to pdm

[wintry plinth]

blame the fact that they're using an incorrect venn diagram XDD

[edgy basin]

xd

[young island]

Can you suggest me plz?

[brazen forge]

no

[deft grail]

I think you misheard what I said. We will not help with any sort of exploit.

[heavy knot]

Hey not sure where to start but what are the best tools for creating mobile apps? I'm trying to find the right community for this. Thanks for any info!

[dire citrus]

learn JavaScript

with VSC

[dire citrus]

@heavy knot

[heavy knot]

Thank you @dire citrus

[dire citrus]

you are welcome!

[dire citrus]

thats what i heard

but i would still think about it

[heavy knot]

Okay! I have seen none code stuff like flutter flow and adlo so I wasn't sure if there are better options out there 😅 thanks again @dire citrus

[dense solstice]

hi, in my pyproject.toml if I make build-system section something like this then pip will use setuptools instead of poetry right?

[build-system]
requires = ["setuptools>=51.1.1", "wheel", "setuptools_scm[toml]>=5.0,<8"]

[dire citrus]

ok! happy to help : )

[brazen forge]

you only need

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

see docs for details

[dense solstice]

by this I can make pip to use setup.cfg & setup.py ?

[brazen forge]

that'll make it PEP 517 compatible, yes
you don't need to do that, if you're using setup.py though, AFAIK

speaking of which, it's recommended to move away from setup.py

use pyproject.toml or if you really need to, setup.cfg

PyPA has a user guide on this, if you haven't stumbled upon it yet

[dense solstice]

I was thinking to move to pyproject.toml but I'm not sure how to define specific dependencies to install using pip.

like I want to define dependencies only for liniting & using setup.py I can define them in such a way that I can do pip install .[lint] & so on

poetry uses grouping system but I don't think pip can identify them, have to use poetry run

[brazen forge]

https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#optional-dependencies

[dense solstice]

thanks

[astral apex]

If you only have a setup.py, a PEP 517 builder will say it doesn't know what to do
If you create the minimal stub pyproject.toml, the builder will kick off setuptools, which has it's own parser that will see the setup.py and work with that

[brazen forge]

I see
I only said that because using pip install with the --use-pep517 option seems to make it generate a pyproject.toml file for projects that don't have it

[astral apex]

IIUC, at least some PEP 517 builders will just kick off setuptools anyways if they can't find anything, to see if they can find anything

[dense solstice]

what tools do you guys suggest for initialising a project template ? I use pyscaffold rn.

[brazen forge]

copier or cookiecutter

[astral apex]

I just copy/paste

But cookiecutter looks nice

[dense solstice]

hey if it's working for someone else why bother right ?🥴🥶

[astral apex]

It works perfectly
I have scores of packages! Scores!!

.wa s how many in a score

[signal cloak]

Wolfram Alpha

20

[astral apex]

okay maybe just 1 socre

[dense solstice]

what would be a great project which I can use as a dough ?

[brazen forge]

https://cookiecutter-hypermodern-python.readthedocs.io/en/latest/ is one I've used before
it has a lot of stuff, so you might wanna remove things you don't need

[dense solstice]

how can I avoid rewriting the the values again & again ?

[brazen forge]

Poetry doesn't quite support the [project] section last I checked

[astral apex]

You shouldn't have both [project] and [tool.poetry]
You should have one or the other

[dense solstice]

which one setuptools will parse ?

[astral apex]

!pep 621

[rancid schooner]

**PEP 621 - Storing project metadata in pyproject.toml**

Status

Final

Created

22-Jun-2020

Type

Standards Track

[astral apex]

Everything except Poetry will expect your pyproject.toml to match PEP 621's format

Poetry is the ONLY one that doesn't use [project]

[dense solstice]

if I just keep [tool.poetry] then setuptools will parse it right ?

[astral apex]

No

setuptools will completely ignore something that declares that it's for Poetry

[dense solstice]

https://tenor.com/view/steve-carell-the-office-yikes-this-is-the-worst-cringe-gif-21219465

[dense solstice]

then it's time to use rye I guess

[astral apex]

?

[dense solstice]

stash poetry & use other tool

as I'm getting rid of setup.py & relying completely on pyproject.toml

[astral apex]

You can do that and keep using setuptools without issue

[dense solstice]

where should I put the project info then ?

[astral apex]

!pep 621

[rancid schooner]

**PEP 621 - Storing project metadata in pyproject.toml**

Status

Final

Created

22-Jun-2020

Type

Standards Track

[dense solstice]

name, version etc.

[astral apex]

https://packaging.python.org/en/latest/tutorials/packaging-projects/

When you get to "Choosing a build backend" switch the tab to setuptools

[dense solstice]

yah I mean I was going to use poetry for env management & not as a build system

[dense solstice]

The lack of a [project] table implicitly means the build back-end will dynamically provide all fields. does this mean If i use poetry then poetry build will provide the data ?

[astral apex]

See the sentence before that
Only if you set your [build-system] to that system, and if that system has added supported for fetching data from other places
For example setuptools has support for going to a setup.py to find the project information

[dense solstice]

so if I get rid of setup.py my pyproject.toml should have a [project] field. else with my current setup I have to use poetry as build system. Have I got this right ?

[astral apex]

If you want to keep setuptools, then you fill out the [project] table, and then you specify the [build-system] like --

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

That's standards compliant. So, in the future, if you want to switch to PDM, for example, you update your [build-system] to declare PDM, and it will understand everything exactly the same

[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"

BUT, Poetry completely ignores all of that
SO, if you want to use Poetry, you take all of that, and you throw it in the trash, and you start over with a [tool.poetry] instead

[astral apex]

There is no reason that you would "have" to use Poetry
You can configure whatever you want, you just have to configure it

All I said was just "don't have multiple different configurations at the same time"

[dense solstice]

ok cool. thanks for explaining. and one last question. if I wish to use poetry which setuptools as build system then how should I configure the boring project parts?

[astral apex]

I don't understand how you plan on using them together

They're competing tools
Well, setuptools does a lot less
But they both manage your project's build

[dense solstice]

that's what I was asking. if I use poetry the have to use poetry as build system right ?

[astral apex]

Yes
... If I'm understanding the question correctly
I mean, you could try to keep doing what you're doing, but....
You'd have to duplicate everything
Most tools these days check for PEP 517 first, so if you do something in Poetry and forget to copy it, the thing would use setuptools and not see it

[dense solstice]

great thanks

[dense solstice]

pdm doesn't have the shell options but hatch does. also looks like pdm is taking more time installing dependencies.

[dense solstice]

ok I have swtiched to pdm now. will I be able to use pip install .[dev] now? I'm using setuptools for build system

[astral apex]

Uh.... Not with [tool.pdm.dev-dependencies]
That's specific to PDM, so you have to interact with it how PDM tells you to

The pip install .[dev] syntax comes from package "extras", which you setup with [project.optional-dependencies]

You should be able to just replace that one line and be good, and then you'll interact with it the way you want to

- [tool.pdm.dev-dependencies]
+ [project.optional-dependencies]

[dense solstice]

can I keep both ?

[astral apex]

If you duplicate the dependencies in both places

[dense solstice]

is there a way to use variables ?

[astral apex]

no

[dense solstice]

hard ctrl + c, ctrl + v ?

[astral apex]

Yeap

[dense solstice]

https://tenor.com/view/friends-ross-geller-david-schwimmer-huh-raised-eyebrows-gif-15947145

thanks

if I keep both there is warning I hope it'll not be a problem

[willow pagoda]

according to their docs, [tool.pdm.dev-dependencies] is only if you want to define extras that aren't part of the package metadata (i.e. regular pip users won't be able to select your dev dependencies)

therefore it doesn't really make sense to duplicate the same extras in both tables

[dense solstice]

what if someone wants to contribute without pdm or to run a workflow without pdm ?

[willow pagoda]

if it's in dev-dependencies, they would have to manually install them

usually its easier for contributors to install pdm/poetry/hatch/etc. if they need its features, and workflows can do the same thing (see this part of their docs for a github example)

well, easier if the alternative of moving away from tool-specific configuration isn't feasible (for example you might have a bunch of developers already working on your project which are used to how you do things, so changing it would cause unnecessary headache for them)

[dense solstice]

understood thanks

[alpine horizon]

hey guys, not sure where to ask, but I need to download a very big file from S3 almost every day, however it gets updated only like once a month, so how can I avoid redownloading it every time?

(maybe I could use the Etag? I'm not sure how it works)

[brazen forge]

you can make a HEAD request for the object to get the ETag, which is "an identifier for a specific version of a resource."

[alpine horizon]

and then I would need to compare it to my local Etag, how do I calculate it ?

I might as well compare the timestamp, no?

[brazen forge]

if you have a ctime/mtime that doesn't change with every download, then you can use that
the ETag is the HTTP way

with ETags, you'd store the initial ETag and compare against that for subsequent requests

[alpine horizon]

where would you store it?

[brazen forge]

anywhere you want
a file, DB, etc.

[alpine horizon]

I see

I thought you could do all of this in a single web-request, like I would pass the Etag to S3, and it will return the file-content only if the etag changed

[brazen forge]

you can use the ETag value in a POST/PATCH/DELETE request in the If-Match/If-None-Match header to update/delete a resource only if it matches the "version" you know

it's for avoiding conflicts

[brazen forge]

the HEAD request is way lighter than a GET request, so it doesn't matter, I think

[finite fulcrum]

streaming a get should also work

[alpine horizon]

thats true, I think I will use the HEAD request to compare the LastModified timestamp (to the local file) as calculating the Etag is more complex

@brazen forge thanks for the help

[brazen forge]

np

[heavy knot]

Hello
Any idea why a .app created with nuitka (for a PySide6 code) does not work when tested on another mac pc ?
it works well but on the PC it got generated on, while gives "the application can't be opened mac" when tested on any other PC

[static kraken]

I have a problem with tkinter. The import of module doesn't work :

[thorny shell]

what's the name of your file?

@static kraken ^^

[static kraken]

Thank you but I already solve the problem

[warped latch]

I have a question. So my server is configured to only allow logins via ssh key, and there is an ssh key on the computer I access it from. What would I do if like... a space hurricane came through and destroyed this entire building and left a massive flood and i had to migrate to the other side of the planet to survive. How would I get back into my server? I would have to backup the key, but how would you do that securely?

[warped latch]

another question... why do people build images locally and store them in container registries, when they could just pull down the repo on their server and build the images there

[rapid sparrow]

When u build image, it is buildable for interpreted languages only now.
We build once in order to freeze its state to unchangeable.

Then we use same image to test with unit tests, in staging and then launching in prod.

Image saved to docker registry can be pulled then across all servers the same one for months of time

[rapid sparrow]

When we later will try to deploy newer version, we can discover that newer image version is broken for some reason

Frozen previous version allows us to rollback back to previous one, even if its dependencies no longer downloadable or their dynamic installation changed in versions and would break its build today

[rapid sparrow]

I use two approaches

1. the lazy home one. I have git-crypt encrypted repository with my secrets and documents which I git push to GitHub and Gitlab, and even pull at my second PC-laptop just to be sure from time to time

2. the very lazy enterprise one:
   I create ssh keys with terraform. They are kept stored then in my terraform state in remote S3 bucket.
   They are used with same terraform to be applied to hosts.
   I can just query those keys to my local machine as long as I have sufficient IAM access/authorized to this AWS account

[warped latch]

i see ty

[rapid sparrow]

both approaches are bad for real enterprise btw

at aws a better approach is using SSM connections, since u don't have ssh keys then and can at any moment revoke access from some person

real security is when u can revoke access per user without replacing key at each infra object 😅

[warped latch]

interesting, i'll probably never be in a situation where there is anyone other than me to think about

[mossy topaz]

Hello there. Is there a Python package which allows get financial markets footprint data? In the photo I sent the footprint is the amount of volume traded "inside" every level of every candlestick, and normally it's given by a data feed. It's different from normal volume and to volume profile. All I find on the internet are things related to the things I just said, which is not what I am searching for. If anyone could response I would appreciate it a lot!

[bitter sedge]

I do offsite backups to AWS S3

[thorny shell]

I'd back up my key to some cloud-based password manager -- I use Bitwarden, e.g.

apple's keychain and google's password manager are equivalent. I assume Microsoft has some equivalent too

[warped latch]

that's not a bad idea i already have a bitwarden.

[pastel hare]

Hi guys.
I have a question for you.
which is the best and most affordable laptop for programming (less than 2000 dollars)?
Please, tag me

[thorny shell]

"best" is insanely subjective. I prefer macbooks.

[low inlet]

same, and they have lasted me forever

[uneven steppe]

Hi everyone, I'm very new to Python (my 3rd day), I am ready about setting up environments, I am working in this folder on Windows, C:\Learn\Python\TryItOut, I ran this command: python -m venv MyVirtualEnv C:\Learn\Python ... What it did is that it created few folders inside that folder AND also created a folder with the environment name inside C:\Learn\Python\TryItOut.. if open CMD and go to my folder and type "where python" it always shows me my user folder/appdata .. and doesn't show any indicator about my environment.

in VSCode though, it shows me the environment name, but if I browse to the root of the C: folder, it still shows that (MyVirtualEnv) between parentheses .. I'm just confused

[pastel hare]

Do you think M1 MacBook Air is good enough?

[thorny shell]

yep

it'll be great, but there are probably much cheaper options

they won't be a nice, but then ...

[pastel hare]

Thank you for your answers 🙂

[mossy topaz]

Is there someone who knows about the MetaTrader5 pip package? I am running through an error since some time which I can't resolve. MetaTrader5.order_send(request) returns None and should return a MqlTradeResult type

[hexed osprey]

I need help to setup the old github account with my current computer. I have an old github account that i used in the previous laptop now i want that account to be linked with the current computer. I have generated a ssh key and add it to the github account that i want to linked with. But when i do git push i get this message

fatal: Could not read from remote repository.```
so i have changed the git config like below

git config --global user.name "username:
git config --global user.email "myoldgithubemail"```
still get the same error. where to look for "anotherusername"? and how can i fix it and use the old github account in the current computer?

[rapid sparrow]

Probably u have too open ssh file permissions
Delete them
Open git bash. Generate again and don't create or move files on your own, do it all with ssh-keygen
Also may be .ssh folder permissions are broken too. So consider deleting and resetting it too

[tawdry needle]

follow the instructions here to test your ssh connection: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/testing-your-ssh-connection that way you can rule out a problem at the ssh level without worrying about git being involved. user.name and user.email configs shouldn't matter

[hexed osprey]

Hi oldusername! You've successfully authenticated, but GitHub does not provide shell access.

I get this.

[hexed osprey]

if i delete the ssh keys then will not give me problem to use the other account. i have generated two keys for two different github account

[hexed osprey]

thank you. it was me who forgot a step to complete when i generated another new key to link with the github account.

$ eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519```

after trying for three days finally figure out the step i forgot

now i can continue with my old github account in my current computer

[gentle solstice]

oh-my-zsh has a ssh-agent plugin you can use.

[radiant jasper]

Is this the right place to ask about the Google Drive API?

I am not retrieving all the subfolders in a folder, only two of them. I have granted the service account
https://www.googleapis.com/auth/drive scope. Still nothing.

[toxic bough]

Sorry, this is not the right channel for your question. Please claim a help channel. See #❓｜how-to-get-help.

[remote glen]

is there a formatter I can run with explicit detailed settings for all of it's rules, not just rule-sets? also, I'd like to learn by seeing one rule at a time get applied to my codebase, allowing me to keep or bypass them as desired.

[astral apex]

You can pass in single rules to Ruff, i.e. ruff check --select F401 src/

[remote glen]

ooo I didn't see that. I'll go look for the list "F401 vs F402 etc" now. thanks!

[astral apex]

The full list of rules is at https://docs.astral.sh/ruff/rules/

You can see https://docs.astral.sh/ruff/linter/#rule-selection , but it talks about setting up the rule selection in your pyproject.toml, not on the CLI, but it's effectively the same either way

[remote glen]

thanks so much!

[slim maple]

I don't think these are formatting rules though

[remote glen]

yeah I'm getting this problem... I have a ruff.toml like:

[format]
select = ["F401"]

and getting the error:

unknown field select

[willow pagoda]

^ the black code style is supposed to be opinionated right? i wouldn't expect either black or ruff to provide much customization for it

[slim maple]

Yeah, for me black style is the standard, you should have much better things to do than configuring your formatter

Especially reviewing it's changes too

[willow pagoda]

if you're using ruff/black you can read about the black code style here
https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html
but for incrementally introducing it to your codebase, i think you'd have to suffice with adding # fmt: off to files that you don't want it to format or some other similar approach

or yknow, just give in and let the python software foundation decide what's best for your code 🙇

[remote glen]

thanks for the advice!

[clever palm]

Any frequent users of devpy-server ?
Im getting a "no project named 'packagename' was ever registered"|

but no idea how to register

[daring escarp]

Can anyone tell me what's wrong with my Dockerfile? My container is crashing after installing dependencies.

# Dockerfile for Django backend with Pipenv
FROM python:3.11.5 AS builder

# Create a non-root user
RUN useradd -ms /bin/bash admin

# Set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# Create and set the working directory
WORKDIR /api

RUN chown -R admin:admin /api

# Switch to the non-root user
USER admin

# Install pipenv
RUN pip install pipenv==2023.10.24
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . /api/ai

WORKDIR /api/ai

# Collect static files
RUN python manage.py collectstatic --noinput

# Database migrations
RUN python manage.py migrate

# Expose the port your Django app runs on
EXPOSE 8000

# Start the application
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]

[brazen forge]

what's the error?

[daring escarp]

It just stops

[brazen forge]

the build fails?

[daring escarp]

ye

[brazen forge]

have you tried building with the --progress=plain option?

[daring escarp]

no, what does it do?

[brazen forge]

shows you all of the build output without hiding it as buildx does by default

[daring escarp]

I altered my requirements file and now it looks like this:

# Dockerfile for Django backend with Pipenv
FROM python:3.11.5 AS builder

# Create a non-root user
RUN useradd -ms /bin/bash admin

# Set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# Create and set the working directory
WORKDIR /api

RUN chown -R admin:admin /api

# Switch to the non-root user
USER admin

# Install pipenv
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . /api/ai

WORKDIR /api/ai

# Database migrations
RUN python manage.py migrate
RUN python manage.py makemigrations

# Expose the port your Django app runs on
EXPOSE 8000

# Start the application
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]

It says that build will be deprecated in favour of buildx.What's the difference

?

I managed to build my project in 15 minutes

[brazen forge]

buildx is the new (it's been here for a while, you should update Docker) backend for building container images

[brazen forge]

cool, no errors when running either?

[daring escarp]

The installation part gave me no errors, but I had an issue with the collectstatic command

[brazen forge]

I don't see a collectstatic command in the new build

[daring escarp]

exactly, it was in the old. I'm still trying out the new because it's building as we speak

I said it took 15 minutes before because the meat of the build is in the package installation part

Is there a way in which I can increase the RAM accessible to the container

?

I'm gonna be using this in a production environment and I once had errors with that

Any idea?

[brazen forge]

how are you going to be running the container?
raw Docker? Docker Compose? Kubernetes? anything else?

[daring escarp]

I would like to use CI/CD in this way: https://www.freecodecamp.org/news/how-to-set-up-continuous-deployment-in-your-home-project-the-easy-way-41b84a467eed/

So rn I'm trying to dockerise my django app

this will have ai as well

I'm still new to docker but I'm trying to get the hang of it

Also I'm not sure if it's a good idea to use poetry rather than raw pip with a requirements.txt file

I previously used pipenv but it's mighty slow and I can't afford to have that

I'm also getting a bunch of these if you can help:

  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

It installed all, It stopped, I started it again, it stopped again, now the container disappeared on docker desktop

I'm starting to believe that there's something wrong with my install

[hexed osprey]

i find out that this is not a permanent fix. i have to do it every time when i try to push after restart

[clever palm]

what do you guys use to host your own python packages?

[tawdry needle]

Just pypi. Source code on github and sourcehut

[tawdry needle]

most systems have a way to permanently add a key to the ssh agent

that or you have to do it in a script upon login

[clever palm]

Ok. I am using the same. Problem is that I just followed the https://hub.docker.com/r/muccg/devpi/ to get the whole thing running in a docker. I can see the webserver and devpi-web frontend. But when i try to upload my first package i get
"no project named 'packagename' was ever registered". But i fail to find how i should register projects. Any ideas ?

[topaz aspen]

Looking into the best/most common workflows to managing releases. Ie - release from tag branches like pandas, have some dev branch for testing then merge dev -> main or whatever. I'm wondering if anyone is aware of any nice write-ups around this workflow... A lot of packages don't have this information in the CONTRIBUTING.md file.

I might just be missing some keywords though, which would be nice.

[warped latch]

I'm having an issue with getting the real client ip from an nginx docker container. Here's an example of something you would see in my access.log

172.18.0.1 - - - - [24/Nov/2023:22:28:44 +0000] "HEAD /admin HTTP/1.1" 301 0 "-" "curl/7.81.0"

Every request, even requests I didn't make, are coming 172.18.0.1, which if im not mistake is an IP associated with docker. So the nginx container cannot tell where requests are coming from, it just thinks they are all Docker.

I know that you're supposed to set the header or something, so Ive tried that.

    log_format logformat '$remote_addr - $http_x_forwarded_for - $remote_user [$time_local] '
                     '"$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"';

    access_log /etc/nginx/access.log logformat;

    location @proxy {
        proxy_pass http://django;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
    }

    location /api {
        try_files $uri @proxy;
    }

so as far as I can tell, Ive done what every guide says to do and it still doesn't give me useful information.

Ive also confirmed that X-Forwarded-For is not in the headers usiing curl.

[tawdry needle]

you don't need devpi do publish packages to pypi. are you trying to host your own package repository? most people don't do that (and it won't work with most end users' workflows anyway, unless it's something internal for your company)

if you just want to publish packages that other people can use, see https://packaging.python.org/en/latest/tutorials/packaging-projects/

[brazen forge]

do you have IPv6 configured? https://github.com/nginxinc/docker-nginx/issues/18

[warped latch]

I have not specifically done anything with ipv6 but if that is a default its maybe worth looking at.

I keep returning to the fact that the X-Forwarded-For header is not being added at all where it should be. If it had the wrong value that would make sense but its not there at all.

i just had a bit of a break through, I changed it from proxy_set_header to add_header to this and headers came through

    location @proxy {
        proxy_pass http://django;
        add_header X-Real-IP $remote_addr;
        add_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
    }

the issue now is that it still has just the docker ip

$ curl -I https://0.0.0.0/api/pages/ -k
HTTP/1.1 200 OK
Server: nginx/1.25.3
Date: Sat, 25 Nov 2023 05:24:13 GMT
Content-Type: application/json
Content-Length: 5319
Connection: keep-alive
Allow: GET, HEAD, OPTIONS
X-Frame-Options: DENY
Vary: origin
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Cross-Origin-Opener-Policy: same-origin
X-Real-IP: 172.21.0.1
X-Forwarded-For: 172.21.0.1

[warped latch]

yeah i basically give up lol, if people are going to send maliciouis traffic from the same ip i guess theres nothing i can do about it

every answer on the internet has the same solution and it doesnt work

[sudden plaza]

could somebody help me with git? i accidentally put credentials in a file during commit and unable to remove it from the history

[rapid sparrow]

git rebase -i commitHash~1

replace pick with squash for commits between added creds and deletion

Beware, it will overwrite commits

Not recommended in branch usage of which is shared with other devs

[sudden plaza]

this throws an error " could not apply abc1234.. Add code to check if post is uploaded successfully"

[sudden plaza]

all the commits?

[rapid sparrow]

In range from current one to the one commit or one commit before commit for the selected CommitHash during command

[sudden plaza]

gotcha

[rapid sparrow]

What is your git status writing

[sudden plaza]

nothing to commit

[rapid sparrow]

Full msg?

[sudden plaza]

on branch refactor
nothing to commit, working tree clean

[rapid sparrow]

U can

git reset --soft CommitHashUAddedCreds~1

It will cancel all commits up to this troubling commit
But will keep fill changes and u can commit them as single commit after that

[sudden plaza]

so all the commits before this commit or the ones after this commit?

[rapid sparrow]

After commit and the commit in question included

[sudden plaza]

gotcha, tyvm!

[rapid sparrow]

Each section at this server includes some important subject though.

[thorny shell]

This discord is an anarcho-syndicalist commune. We take it in turns to act as sort of executive officer for the week, but all the decisions of that officer have to be ratified at a special bi-weekly meeting by a simple majority in the case of purely internal affairs

[thorny shell]

just because I've memorized all of Monty Python and the Holy Grail doesn't mean I'm a nerd!!

[tawdry needle]

"reset" in general moves the current branch pointer to the chosen commit. soft/mixed/hard determines how your files are modified as a result of moving it.

the ~1 notation (or just ~ would work) means "1 commit before"

so this moves your current branch to 1 commit before the problematic commit

[agile stag]

I wrote my docker run command inside of my crontab configuration file and the command is piped to var/log but what will the name of my file that will log the stdout of the Docker image application?

[agile stag]

Also how can I confirm that crontab will execute the command as scheduled?

[tawdry needle]

cron just runs the command as a shell command, it does whatever you ask it to do

[agile stag]

So essentially I need to create a sh file and write an executable command inside of it and crontab will run that script on a schedule?

[agile stag]

Someone shared a linux manual of crontab and I can’t find it

[brazen forge]

man 5 crontab?

[thorny shell]

https://linux.die.net/man/5/crontab ?

[agile stag]

Thanks

[agile stag]

0 6-23 * * 1-5 /root/run-app.sh >> /root/var/log/the-app.log

The syntax of this cron job is correct right? It should execute

[brazen forge]

At minute 0 past every hour from 6 through 23 on every day-of-week from Monday through Friday.

the command looks… okay

[agile stag]

I'm not seeing the output of my Docker image file being printed in the the-app.log file. I'm trying to troubleshoot this as we speak.

[brazen forge]

is that file mounted to a file on the host?

[agile stag]

Is this is what /var/log/syslog shows:

Nov 28 17:00:01 chili CRON[166949]: (root) CMD (/root/run-app.sh >> /root/var/log/run-app.log 2>&1)

[agile stag]

Which file?

This is what is written in my run-app.sh file

#!/bin/bash

/usr/bin/docker run --network host my-app/poolswim-api:1.1

[brazen forge]

the log file

oh, you're running the cron job on your host

do you not see anything in the log file then?

[agile stag]

No there's nothing written in there.

[brazen forge]

do you see any output when running the shell script manually?

[agile stag]

Yes it says

Unable to find image 'my-app/poolswim:1.1' locally
docker: Error response from daemon: manifest for my-app/poolswim:1.1 not found: manifest unknown: manifest unknown.

I typed my repository wrong. Oops.

Nice debugging tactic

But I would think that message should have been written in the var/log/the-app.log file

[brazen forge]

is that shell script executable?

[agile stag]

Yes. It's executable.

It's executing right now

[brazen forge]

the docker error was likely output to stderr

you're only redirecting stdout to that log file

[agile stag]

Doesn't 2>&1 redirect stderr messages to the stdout?

[brazen forge]

it does, but that's not in your cron tab

[agile stag]

0 6-23 * * 1-5 /root/run-app.sh >> /root/var/log/my-app.log 2>&1

I changed it. So now this cron job should be executable right?

[brazen forge]

should be logging to the file, yeah

[agile stag]

Hmm. The output of my application is not getting piped to the log file for some reason

[brazen forge]

could you share the current crontab and the shell script?

[tawdry needle]

as suggested above, just run it in the shell without crontab first

[agile stag]

I did that already

It’s executable

[tawdry needle]

it sounds like you figured it out, but what i meant by comment about shell commands a few days ago was that the part after the schedule specification is just a shell script. so in your example cron does sh -c '/root/run-app.sh >> /root/var/log/my-app.log 2>&1'

(but it can't hurt to put all that in a wrapper script so you have more freedom to configure the output handling etc)

[agile stag]

Yes

One moment

[tawdry needle]

@agile stag so when you run sudo sh -c '/root/run-app.sh >> /root/var/log/my-app.log 2>&1' it works as expected?

[agile stag]

No I just did bash run-app.sh

run-app.sh

# Add the following line for debugging
exec >> /root/var/log/my-app-script.log 2>&1

# Update your script
# Add the following line before the docker run command
echo "Executing docker run" >> /root/debug_log.txt

/usr/bin/docker run --network host my-app/poolswim-api:1.1

crontab file

# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command

0 6-23 * * 1-5 /root/run-app.sh >> /root/var/log/my-app.log 2>&1

[agile stag]

I will try the command you said though

[tawdry needle]

well then you're not really testing the script that you have in crontab

(ideally you wouldn't run this script as root unless necessary anyway, but that's another store)

e.g. you could su to a less-privileged user and run your service that way

[agile stag]

it says

sh: 1: cannot create /root/var/log/my-app.log: Directory nonexistent

It's not suppose to be a directory though. it's a log file?

[brazen forge]

does /root/var/log exist?

[agile stag]

Well I'm in my log directory and I typed

pwd

and this was the output

/var/log

I'm logged in as root so I assumed that all of my directories are the children of a directory called root because that's what I am logged in as.

[brazen forge]

/var/log is at… /var/log

that's not the same as /root/var/log

[agile stag]

All right. So I'll take /root/ off then

I actually don't know exactly where my crontab file is placed in on this virtual machine so I assumed /root/ was the way to access the /var/log directory

Appparently it's in

/var/spool/cron/crontabs/

Does this mean I need to write my cron job as

0 6-23 * * 1-5 ./run-app.sh >> ./var/log/my-app.log 2>&1

?

[agile stag]

I typed in sudo sh -c 'run-app.sh >> /var/log/my-app.log 2>&1' but I don't see any output on my terminal.

[eager rose]

you redirected all the output

[agile stag]

I typed in cat

root@mymachine:~# cat /var/log/my-app.log
sh: 1: run-app.sh: not found

but I type in pwd and it prints:

/root

Confusing.

Well the

sudo sh -c '~/run-app.sh >> /var/log/my-app.log 2>&1'

command is executable

[tawdry needle]

/root is usually the root user's home directory, it's not the same as /

it's confusing because / is the "root" of the filesystem tree, which is different from the super-user conventionally called root

[agile stag]

Why wouldn’t I?

Is it bad to run scripts as root?

[eager rose]

it's bad practice

[agile stag]

How so?

[eager rose]

root can literally do anything. non-root users have limited permissions

[compact violet]

Does anyone have any recommendations for android keyboard for development? I found a recommendation for a hacker keyboard app but it seems abandoned

Although, not sure if right place to ask

[agile stag]

This automation program I am working on: How the process works is that it saves the date of the last time the program ran and saves it into a .pkl file. When it runs again it looks into the .pkl file and grabs the date of the last time the program ran and it recurs ask the program runs.

Once the program gets built into a Docker image file, runs from a virtual machine, the program isn’t saving the date of the last time the program ran into the .pkl file. This is a problem that I don’t know how to solve.

[thorny shell]

maybe it is saving it, but your docker container has its own file system

[rapid sparrow]

Two options.
Use docker volume for persistence of data
Or use external data storage

[thorny shell]

so when the container goes away, so does the file

also, what Darkwind said

[agile stag]

Check this out:

cat -n /var/log/my-app.log | grep "LAST UPDATED TIME:"
    21  LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00
   366  LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00
  2213  LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00
  4072  LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00
  4146  LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00
  6017  LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00
  7888  LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00
  9759  LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00
 11630  LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00
 13501  LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00
 15372  LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00
 17243  LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00
 19114  LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00

Same date and time on every line

It shouldn't be doing that.

It doesn't do that on my local machine

Only on the virtual machine

when the docker image file runs

[agile stag]

Ohhhhh shit. That explains it

[agile stag]

Oh I know what you mean

[agile stag]

This is written in my run-app.sh file

echo "=== $(date) ==="
/usr/bin/docker run --network host -v shark_volume:/path/in/container my-app/poolswim-api:1.1

This is correct syntax right?

[thorny shell]

beats me

[agile stag]

What does that mean?

[thorny shell]

it means "I don't know"

[agile stag]

I can tell you that "in" is highlighted whenever I use vim to edit the file where that line of code is written for some reason

[tawdry needle]

that's correct. but note that this doesn't delete the container, so you'll get accumulating containers over time that will need to be cleaned. another option would be to use docker create to create a container with a given name, and just docker start to run it

are you using the volume to share some data between runs?

[agile stag]

Yes. I needed something for data persistence.

[agile stag]

So docker start instead of docker run?

[tawdry needle]

you're jumping too far ahead. i made a suggestion about something specific. read it, don't just scan messages for commands and guess that you need to use a different command

[agile stag]

So I could have ran the same container instead of using docker volume?

[twilit inlet]

I am using sphinx, I haven't separated source and build folders, meaning the source files are scrambled at level docs/. I have added images to docs/_static. When I add images using the image directive in my rest files, after I generate the docs, I noticed that inside _build/html, the images have been copied in both _images and _static. Is that normal?

[tawdry needle]

yes, or you can do both. containers are meant to be somewhat disposable, whereas volumes are designed to persist long-term. the main benefit of reusing the same container is just avoiding having a lot of single-use containers piling up that you need to clear out. or you can just use docker run --rm to dispose of the container after execution. that might be simpler

[agile stag]

Ok great. Thanks for the help

[agile stag]

I’m not sure how this docker volume is supposed to work because my data isn’t being persistent every time the application runs

Is the docker volume suppose to created on my virtual machine? Or it suppose to be inside my docker image file?

[rapid sparrow]

use relative/absolute paths for volumes
-v /path_to_host:/past_in_container

it will work literally just forwarding this folder between host and container

u need to activate it during container Run command or inside docker-compose file

[agile stag]

/usr/bin/docker run -it --network host -v chili_volume:/jb_cw/core/res mydockeraccount/myapp-api:1.2

How's this?

[agile stag]

Oh really? I wrote these instructions:

# Use the official Python base image
FROM python:3.9-slim

# Set the working directory inside the container
WORKDIR /app

# Copy your application files into the container
COPY . /app

# Copy the requirements.txt into the container
COPY requirements.txt /app

# Install project dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of your project files into the container
COPY . .

# Specify the volume mount point
VOLUME /jb_cw/core/res

# Set the command to run your Python application
CMD ["python", "main.py"]

inside my dockerfile. I thought this would work. I added the

# Specify the volume mount point
VOLUME /jb_cw/core/res

part

[rapid sparrow]

i could recommend reading this awesome book for getting docker

[agile stag]

Am I not mounting correctly though?

[rapid sparrow]

There is Dockerfile syntax for VOLUME https://docs.docker.com/engine/reference/builder/#volume
But i never used it, it is some kind of more recent less obvious addition (because volumes aren't intended to work without having specified both Host and Container path for their functionality)

I prefer
docker run -it -v $(pwd)/data:/jb_cw/core/res mydockeraccount/myapp-api:1.2
running and attaching my volumes like that
docker run -it -v /host_absolute_path/data:/jb_cw/core/res mydockeraccount/myapp-api:1.2
or like that

there is instruction for specifying volumes by name
but i don't like it too... because it is less obvious and less controllable 😅

[agile stag]

All right I’ll try $(pwd)

[agile stag]

This is quite a challenge

root@chili:~# cat -n /var/log/my-app.log | grep "LAST UPDATED TIME:"
    21  LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00
  1888  LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00
  4125  LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00
root@chili:~#

my command

docker run -it --network host -v $(pwd)/data:/jb_cw/core/res mydockerusername/poolswim-api:1.1 >> /var/log/my-app.log 2>&1

These are the functions in the program that saves and gets the date in the .pkl file btw

def save_engine_update_time(obj):
    with open('jb_cw/core/res/system_engine_update_time.pkl', 'wb') as outp:
        pickle.dump(obj, outp, pickle.HIGHEST_PROTOCOL)


def get_engine_last_updated_time():
    with open('jb_cw/core/res/system_engine_update_time.pkl', "rb") as f:
        return pickle.load(f)

version: '3'

services:
  myapp:
    image: your_image_name
    volumes:
      - my_volume:/jb_cw/core/res
    networks:
      - my_network

volumes:
  my_volume:

networks:
  my_network:

Is this what I need to put in my program? Will the data persistence then work?

[agile stag]

I have done a deep investigation and apparently the user inside my containers do not have writing permissions to write the pkl file

It looks like I need to write USER 1000:1000 in my Dockerfile?

[thorny shell]

I wouldn't be surprised.

I haven't been following closely

[rapid sparrow]

How do u generate such answers?

And why do u ignore previous answers :/

[agile stag]

What did I ignore? I was trying to follow along. I even used (pwd) in my code

[agile stag]

Which answers?

[rapid sparrow]

@agile stag i provided solution how to mount Volume

u could provide some errors u receive from this

docker run -it --network host -v $(pwd)/data:/jb_cw/core/res mydockerusername/poolswim-api:1.1 >> /var/log/my-app.log 2>&1

also don't run like this

it is incorrect

docker run -name service_name -d -v $(pwd)/data:/jb_cw/core/res mydockerusername/poolswim-api:1.1 will run in background pretty much find, because u just used -d flag

docker logs -tail 100 -f service_name can be used to access logs in a comfortable way (tail requests last 100 lines, -f request for live following). U can even request docker logs service_name_OR_container_id, but if logs amount is too big, not the best way to get it

or if service is not having assigned name, u can just insert container ID which u found in docker ps, docker ps -a command

Otherwise, u really need to read the book I recommended ^_^

[eager rose]

important to note docker logs reads the stdout. if you're logging to a file, that won't work

[agile stag]

To clarify service_name is a placeholder for a name of a service? So I would need to execute docker service before I execute the command you just told me to execute right?

[eager rose]

isn't docker service a swarm thing? --name service_name is just giving a name to the container

[agile stag]

Okay

Thanks

I’ll take another jab at this tomorrow morning; This is some advanced DevOPs engineering to me.

I was at this for hours today

[rapid sparrow]

As snow main said.

--name thing is just asigning alias/name to container, for easiest quering commands to it later (instead of needing to use container Id, if no name is assigned)

[eager rose]

you'll get a randomly generated name otherwise

[tawdry needle]

this shouldn't be so hard. i think there might be some simple mistake we've overlooked

[agile stag]

And this is the crux of the challenge in data persistence with Docker. Unfortunately for me, containers are meant to be disposable. After I run the image, the container closes so I’m not sure how my virtual machine will be able to retrieve that pkl file in the container after the program writes the date in that file after it finishes running.

Another problem is when the Docker image gets containerized, the root user doesn’t even have the write permission activated 🤷

[eager rose]

volumes are persisted. they stay on the host

as for the second one, you're probably mistaken

[agile stag]

Does mounting the docker image start at the beginning of it being run or at the end?

[eager rose]

are you talking about mounting the volume? that's done as part of the container startup

[agile stag]

Yeah I am talking about mounting the volume.

[tawdry needle]

can we backtrack a bit? you just want to run some script in a docker image, which saves or loads some data, and you want to persist that data between runs?

what you had before should work

#!/bin/sh

if ! docker volume inspect my-app-vol >/dev/null 2>&1; then
  docker volume create my-app-vol
fi

docker run --rm --network host --volume my-app-vol:/jb_cw/core/res my-app-img \
   >>/var/log/my-app.log 2>&1

if you put this in /root/run-my-app and run it in crontab with /root/run-my-app does that not work?

(i'd also recommend storing the timestamp as an integer or float timestamp as with time.time() and not a pickled object, but that's a separate conern)

[agile stag]

Thanks for the post. I’ll get back to you on this tomorrow morning

To answer your question, yes. Exactly.

[agile stag]

Timestamp is saved in the program

[grand mountain]

Hi, i'm trying to construct a simple live data view on a monitor (such as the one you see on queue monitor or livestream), where you can feed data into the machine and would show it into the screen. I have been searching for a particular keyword for this, is it possible to make in python?

[agile stag]

It runs on schedule with crontab. The data (saving the new timestamp of the app finish running) doesn’t persist.

I called in sick today so I will take another jab at this tomorrow

[tawdry needle]

maybe a "dashboard"? it's definitely possible to make that in python

[violet nebula]

how do i use linux

[tawny temple]

To do what?

[flat path]

to become the best developer out there obviously

[frozen oasis]

Hi, do you have any idea why the deployment script in this guide was working fine, but then overnight it stopped working? https://www.pythondiscord.com/pages/guides/python-guides/docker-hosting-guide/ Look at the status of my workflow: https://github.com/ElieTaillard/PSTHC-Alert-Bot/actions

[rapid sparrow]

u need to improve logging accessability for deubging this issue.
Add next code lines into your Dockerfile

ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1

it will make python loggin operating better for docker usage

And/or, just access logs directly at the server

docker ps -a for checking present containers including dead ones

docker logs --tail 100 containerIDOrContainerName for checking its logs before it died
optionally docker-compose logs can be executed without knowing exact name of the container

[frozen oasis]

@rapid sparrow I've checked the list of containers and only one is running. I've accessed the logs of this one and nothing abnormal.

Ok I was able to fix the problem by updating my deployment script. I added this step:

- name: Stop and Remove Existing Container
  run: |
    if docker inspect psthcbot &> /dev/null; then
      docker stop psthcbot
      docker rm psthcbot
    fi

Maybe I'm the only one to have had this problem, but if other people have the same problem as me, it would be interesting to update the guide: https://www.pythondiscord.com/pages/guides/python-guides/docker-hosting-guide/

[willow pagoda]

(notice: skip to bottom for possible answer)
Out of curiosity, does anyone know why my github workflow doesn't appear to cache all dependencies? Here is the workflow file, and the last and second last runs from it. The Run pip install .[docs] step shows that some of the packages were cached, but others had to be downloaded despite having not changed my pyproject.toml in either run. I'm not too concerned about it since I suspect majority of the execution time is spent resolving and installing dependencies rather than downloading them, but regardless it seems weird for the cache to omit. I've also referenced the actions/setup-python documentation to double check that my cache: and cache-dependency-path: properties were used correctly.

Actually as I was writing the above question, I realized that on the third last run where I changed my pyproject.toml, the Post Run actions/setup-python@v4 step showed: Failed to save: Unable to reserve cache with key setup-python-Linux-22.04-Ubuntu-python-3.12.0-pip-e6773eedc98d3ffc3bb5fa1e7dc01de601fcd096a97f624f23e806dfc108a904, another job may be creating this cache.

My lint run was the one that managed to save the cache, and the workflow for that runs pip install .[tests] instead of .[docs]. Therefore, I suspect that my docs workflow is being forced to use the lint's cache rather than its own with the docs extra. I guess using multiple requirements.txt files would be slightly more efficient for each worker than one pyproject.toml with extras?

[tawny temple]

Yes, I think this is what's going on. I believe the cache key is just a hash of the file plus some other identifiers. The extras used are not part of the cache key, so whichever workflow runs first when there is a cache miss will be the one that gets cached with its extras.

The cache action does allow customising the key (so you could include the extras names and effectively have separate caches), but unfortunately it does not seem like the setup-python action exposes this configuration. Though in your case it'd be pretty simple to manage the cache yourself with the cache action instead.

Or yeah, you could have separate requirements files for different workflows. I've done that with pip-tools.

[flat path]

@warm blade sorry for taking so long. What's your question? (ping me when you reply pls)

[warm blade]

@flat path i had created a PR, then the maintainer had made some other changes so he had told me to rebase and what not, somewhere i messed up and hard reset or something of the sort, so the PR got closed and now it says

KevinRodriguez777 wants to merge 0 commits into Nuitka:develop from KevinRodriguez777:develop
now my question is, can i reopen that PR, or do i have to create a new PR

[flat path]

You should just open a new one IMO

[flat path]

You may be able to get it to reopen by pushing new commits, but the PR timeline sounds messy already.

You still have the original commits, right?

[warm blade]

i managed to restore my fork to the commit that it was at when i did the PR

i would've thought that'd be enough to allow the PR to reopen but that didn't work

[flat path]

On the PR's original branch right?

But huh, I would've expected that to work

[warm blade]

yes

i only have a single branch forked anyways

[agile stag]

/usr/bin/docker run --rm --network host -v app-vol:/jb_cw/core/res myusername/poolswim-api:1.1 >> /var/log/poolswim-app.log

I am trying to persist data of my application inside he /jb_cw/core/res. I know my docker command is doing something incorrectly because my data is not being persistent.

This is what I typed

-app.log | grep "LAST UPDATED TIME"
LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00
LAST UPDATED TIME: 2023-11-24 16:09:08.538670+00:00

Same date is showing after I ran the program twice 😦

[agile stag]

Oh boy. I am seriously not understanding how this mounting is suppose to work.

This is my error in my program

Traceback (most recent call last):
  File "/app/main.py", line 50, in <module>
    greg_v_0_0_0.update()
  File "/app/jb_cw/core/system_engine/update.py", line 36, in update
    automation_start = dr.get_engine_last_updated_time()
  File "/app/jb_cw/core/system_engine/directives.py", line 185, in get_engine_last_updated_time
    with open('jb_cw/core/res/system_engine_update_time.pkl', "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'jb_cw/core/res/system_engine_update_time.pkl'

I deleted system_engine_update_time.pkl because I thought that mounting it on my host would persist the data inside of that file and then the Docker container would have access to that file

This is my script to run the Docker image

usr/bin/docker run --rm --network host -v "$(pwd)"/data/system_engine_update_time.pkl:/jb_cw/core/res mydockerusername/poolswim-api:1.0 >> /var/log/poolswim-app.log 2>&1

[agile stag]

I found the solution

[astral apex]

https://xkcd.com/979/

xkcd: Wisdom of the Ancients

All long help threads should have a sticky globally-editable post at the top saying 'DEAR PEOPLE FROM THE FUTURE: Here's what we've figured out so far ...'

[agile stag]

I didn’t need to mount anything

I can just start the same container

lol

[rapid sparrow]

containers are short living cattle. u will not be able doing that for long.

their life is temporal and not promised to be long

[agile stag]

Omg

How will this work then?

I tried the mounting and it’s not working unless if I change the function that will save the pkl into a mounted directory somehow

[rapid sparrow]

well, that is the exactly idea

u need to save you pkl into mounted directory 😅

[agile stag]

https://tenor.com/view/the-universe-tim-and-eric-mind-blown-mind-blown-meme-mind-explosion-mind-explosion-meme-gif-18002878

[agile stag]

docker: Error response from daemon: invalid volume specification: '/root/data:app/jb_cw/core/res': invalid mount config for type "bind": invalid mount path: 'app/jb_cw/core/res' mount path must be absolute.
See 'docker run --help'.

I don't know how to fix this.

My updated functions to save and get pkl files:

# This is for bind mounting the Ubuntu filesystem in our Linode server.
bind_mount = "root/data"
filename = 'jb_cw/core/res/system_engine_update_time.pkl'
final_path = (
    os.path.join(bind_mount, filename)
    if os.path.exists(bind_mount)
    else filename
)


def save_engine_update_time(obj):
    with open(final_path, 'wb') as outp:
        pickle.dump(obj, outp, pickle.HIGHEST_PROTOCOL)


def get_engine_last_updated_time():
    with open(final_path, "rb") as f:
        return pickle.load(f)

My run-app.sh script

#Running main script
echo "=== $(date) ==="
/usr/bin/docker run --network host -v "$(pwd)"/data:app/jb_cw/core/res/ mydockerusename/poolswim-api:1.1 >> /var/log/poolswim-app.log 2>&1

unsure what is going on.

[agile stag]

Bind mounting complete 🙂

[ionic ridge]

!warn 1162361466016510052 We are not an advertising board. Please don't post this sort of thing again.

[rancid schooner]

:incoming_envelope: :ok_hand: applied warning to @visual remnant.

[rapid sparrow]

#web-development message u may wish to clean same msgs in other channels if u delete in this one then

[short path]

I also posted this in help/other, but then realized this exists... I'm trying to build an aws lambda function with python with native dependencies following https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-native-libraries. One of the third party dependencies that don't have any native code stopped distributing wheels at some point, and now the pip install fails because it can't find the whl for the package. I've hacked it by removing the dependency from requirements.txt and running a source pip install after the --platform one... Is there a cleaner way to tell pip to use source dependencies where wheels are not available?
pip install \ --platform manylinux2014_x86_64 \ --target=package \ --implementation cp \ --python-version 3.11 \ --only-binary=:all: --upgrade \ -r requirements.txt

[tawdry needle]

you're using --only-binary=:all: which specifically tells pip never to use source distributions

by default, pip will always prefer a wheel over a source dist when available, but will use the source dist if needed

[short path]

i get this error when i try to do anything else: ERROR: When restricting platform and interpreter constraints using --python-version, --platform, --abi, or --implementation, either --no-deps must be set, or --only-binary=:all: must be set and --no-binary must not be set (or must be set to :none:).

[willow pagoda]

seems a bit odd that those arguments are available for pip install but not pip wheel, but i guess they expect you to also use -t/--target <dir> for ease of bundling ready-to-use dependencies alongside your scripts

regardless, that error message is there because you can't build wheels from source distributions on a different platform than your system, so you'll need to either use an older version of the library that still has prebuilt wheels, or use a virtual machine/docker with the matching platform to compile those dependencies

[short path]

what i was able to do... and was wondering how hacky it was... i removed the dependency from requirements.txt and run "pip install --target=package <package with no wheel>". Since the package with no wheel doesn't have anything native, it seems to work. But I'm not sure how bad this is and whether there is a better way. Since this is going to run as an AWS lambda, i'd have to spin up an ec2 instance to do a native build. Which I would rather avoid

[willow pagoda]

ah, so the package is pure-python but the maintainer stopped providing wheels? hmm, there is the risk that the package could at some point become platform-dependent, in which case pip install would unintentionally create binaries not supported by linux... pip install --implementation py <package> would prevent that, but that still requires combining --no-deps or --only-binary=:all: which defeats the point...

i guess as long as you can verify afterwards that the package is pure-python, for example by reading the .dist-info/WHEEL metadata: py Wheel-Version: 1.0 Generator: bdist_wheel (0.41.2) Root-Is-Purelib: true Tag: py3-none-any then i guess running a separate pip install command is a sufficient workaround

[short path]

yea, the package flatdict hasn't been updated in 4 years, but at some point they went from travis ci to github actions and stopped doing binary distributions. okta sdk depends still depends on it

[tawdry needle]

do they allow you to upload packages somehow? then you could do pip download to get the wheels and sdists that you need, then upload those to s3 or whatever

at work currently i just use ECS so it's all docker images, makes things easier

[short path]

you CAN deploy a container image as a lambda, but i was trying to avoid that as well because then you're responsible for the environment. This way AWS manages the runtime and you just provide the python code (plus deps...). I don't think there is anyway to download with a lambda without getting fancy with extensions since the first thing that happens that you're "aware" of is the handle method get's invoked. I could also maybe run a conditional download from the handle code, but that's seems ugly as well. With the pip download, isn't that basically the same thing as what i'm doing with the extra pip install?

[tawny temple]

I think that is similar if not the same as what I'd suggest: install the dependency separately, and then install the rest of your packages. This should be fine as long as you constrain the versions (pip install -c) when you do the first install.

[heavy knot]

can anyone help me i made a web scrapper of coinmarketcap.com

[peak wraith]

(posting so other folks don't respond): This has been answered in pydis / help thread. This is not something we can help with. Please don't cross-post.

[heavy knot]

kk

[tawdry needle]

yeah, if you're writing a custom script to download packages, then i think the best thing to do here is just do it in two stages. you can still use a "constraints" file to ensure that you are downloading the exact desired versions

[rugged flame]

@eager sapphire Sorry, we don't allow paid offers of work in this server.

[heavy knot]

Where can I learn to create tools

[vague edge]

ayo does anyone use vim here

bc i have a question about the ending newline

vsc and any other self-respecting ide would show this

but vim doesn't, it just stops at line 12

[vague silo]

is there actually an empty line at the end of the file?

if not, why would vim display it?

just Go and you have it

[brazen forge]

Vim doesn't show a newline at EOF

[flat path]

It will show in the status bar whether the file terminates with a newline or not.

It'll say noeof if there's no terminating newline AFAIK.

[vague edge]

god that's so cursed

[vague silo]

yeah it is really cursed to show things in files that are not there :D

[vague edge]

consider this: vsc, notepad, intellij, every other editor i use shows the final newline

is it really incorrect if so many other editors do it?

[eager rose]

it's just different

[vague edge]

fair enough

[knotty sequoia]

qq about specifying package names: if i put [project] name = "foo-bar" pyproject/pip seems to convert this into import foo_bar after i pip install -e . - does anyone know if this is officially support behavior or if i really need to clean this up and explicitly specify somehow?

[flat path]

it depends on the build backend you're using, but yes, this kind of normalization is usually handled gracefully

dashes cannot be included in a Python import name

[flat path]

Presumably you have a foo_bar package in your project tree that is copied during the install? That's what determines the import name.

[knotty sequoia]

im reading this - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#auto-discovery

seems to be it will find my dunder init file?
yeah, so i do have a foo_bar folder with an init file

[alpine horizon]

anybody knows how to use poetry pubblish with 2FA enabled on PyPi?

like, how do I pass the token

[flat path]

Yeah, it will automatically find your package code and include it into the distribution

I actually am not quite sure how the name normalization works, but the dash/underscore distinction doesn't matter AFAIK either way

[astral apex]

https://python-poetry.org/docs/repositories/#configuring-credentials

[alpine horizon]

I tried smth earlier, and I only had success with generating the multiuse oauth token

but I dont think you can actually use the one-time 6 digit password from the 2FA app

[astral apex]

the.... what

[astral apex]

not that token

[alpine horizon]

yeah

[astral apex]

https://pypi.org/help/#apitoken

[alpine horizon]

yeah thats what I did

I gotta read the docs you sent and see if I can create a config file for poetry so I dont need to remember the token each time

[tepid stump]

Someone has quake 3 rcon?

[elder knot]

!pypi rcon

[rancid schooner]

rcon v2.4.4

An RCON client library.

[placid nacelle]

!pypi kanban-python

[rancid schooner]

kanban-python v0.3.7

Terminal Kanban App written in Python

[placid nacelle]

Nice bot 😂

[jovial whale]

Apologies for the re-submission :

Hello,

I am trying to understand the difference between using Nexus and Twine ? It is a bit unclear as to what extent they overlap in functionalities.

https://twine.readthedocs.io/en/stable/

https://help.sonatype.com/repomanager3/nexus-repository-administration/formats/pypi-repositories

[hazy echo]

Nexus is the (proprietary) server that hosts the package repo and Twine is the tool that puts together the package and pushes it to Nexus (or PyPi)

[bright shuttle]

hey, how can I merget "HEAD" into main branch?

i guess I can create a branch from here

and maybe merge that branch with main?

[hazy echo]

you will want to make a commit (HEAD doesn't really have a commit id, it's like saying $PWD)
then you can branch if you like.

[astral apex]

No one can tell you without the code
Also, <@&831776746206265384> -- attempting to steal Spotify accounts

[tawdry needle]

yes, that's what you need to do. try not to make commits with a detached head...

[languid blaze]

That’s actually really cool, I’m going to use that on my dev vm at work.

[placid nacelle]

Its my first package, Glad you like it :)
If you find bugs or just have feedback, feel free to reach out or create an issue. :)

[languid blaze]

I'm a sucker for time management tools, and before now I was manually grabbing TODOs etc and putting into the kanban plugin in Obsidian

[heavy knot]

Where do I learn to create things in Python I'm kinda new

[agile stag]

Is there a way to send my built Docker image from my local to my Linux virtual machine without pushing it to my Docker repository?

[eager rose]

technically, yes. but why?

[agile stag]

It’s better

[eager rose]

why do you think so?

[tawdry needle]

https://stackoverflow.com/a/23938978

[agile stag]

Incase my Docker repository gets hacked, it’s a one way direction to my VM, instead of two steps (push to repository then pull from VM).

Maybe pushing it to repository is better. To treat it like a Docker bank.

[small jay]

The chances of that happening are very slim

you generally want to upload it to docker hub or some other container repository

very similar to storing your code on GitHub, really. Push code, pull when deploying. Push containers, pull when deploying

And GitHub has a container repository imaginatively named GitHub Container Repository (GHCR), so you can have built images and code on the same platform

[tawdry needle]

i don't see any problem with copying images and not using a repo

well, i see some problems. but you don't strictly need a repo either

[eager rose]

using a container repository is a lot easier with CI/CD. but if you're just building locally then i guess you can cook up a script to copy images directly to the vps

[heavy knot]

does anyone know a tool or way I can make shortcut code in the cmd or vs code integrated terminal?
for example

code: echo hello world
shortcut: say hello
cmd: hello world

[thorny shell]

https://learn.microsoft.com/en-us/powershell/scripting/learn/ps101/09-functions?view=powershell-7.4#a-simple-function

[crimson spruce]

Struggling a bit to get a Poetry-based package to install a .pth file. From the last time I did this I remembered I have to manually add the .pth file to the tarball, but that doesn't make the .pth file appear in the site-packages of the venv I'm trying to install the package to. I'm struggling to find any info online about using Poetry with custom .pth files.

[tawdry needle]

is it even valid to include a .pth file in a sdist or wheel?

[crimson spruce]

If I manually add the .pth file to the wheel, and then pip-install the wheel directly, it works.

[tawdry needle]

yeah, not surprised. but i'm questioning if that's "allowed" or not

maybe poetry just isn't set up to recognize .pth files as "python" files, so you might have to add it as if it were a non-python data file

i don't know what mechanism poetry uses for that, i know setuptools still uses MANIFEST.in

[crimson spruce]

Well, letting the problem rest for a few hours seemed to have helped: I can just list it in include and it will work. Thanks for being my rubber duck

[unkempt wyvern]

Hi, is there anyone who ever worked with Pillow library?

[astral apex]

Lots of them
If you head over to #1035199133436354600 and post your code and your error, they'll be able to help you
See #❓｜how-to-get-help

[tidal marsh]

hey guys

how do i make a website for hypixel skyblock stats

i want to make a website where you can check every players stats

[willow pagoda]

#python-discussion message @shut wagon not sure if there's a github action specifically for this purpose, but you could write up a workflow in the shared repo to checkout the other repos with a PAT, bump their submodules, run tests, commit your submodule changes, and push it

alternatively you might prefer making a python script to semi-automate the process on your own system, perhaps using subprocess.check_call()

[rapid sparrow]

i have a github organization with three repositories frontend,backend,server, and then one repository shared that contains a bunch of shared types and library stuff which I git submodule into the other three repositories.

when I do dev work on the submodule and commit an update, I now have to go back to all three of the main repositories and manually update their version of the shared submodule to be the latest version.

Is there an easier way to do this? I would like frontend,backend,server to automatically update their shared submodule reference to the latest commit, anytime I commit to shared. Can I setup a Github Actions for this or something?
@shut wagon the answer depends on if you are developing alone or having mutiple devs and size of the code.
If it your single project and u are the only dev... consider just merging everything into single repository. Monorepo simplicity wins for a single dev usage when all those repositories are the same project

I am currently developing full stack application nicely, where front/backend/workers/taskscheduler/infra are just sub modules of this app and quite happy about it 😅

really to move quickly forward when all your code between all sub services is directly connected by static types

[shut wagon]

submodules as in, you are doing a multi repo appraoch?

or submodules as in, they are just sub folders within 1 single git repo

[rapid sparrow]

sub folders of 1 single git repo.
sub modules i mean at the level of programming language, making them recognized as programming language libraries/modules/packages
from there i just structure the code accordingly with building stuff

[shut wagon]

gotcha

[shut wagon]

currently i have all my static types in a different git repo, that I have submoduled into all other repos

it is not as simple but I think this will scale better

it is for a game engine

where there are many moving parts

[rapid sparrow]

now that is promising beginning. finally static typed dev in dynamic typed server 😅

[shut wagon]

refuse to do python without it 😅

[rapid sparrow]

so, in which languages your different services? u named existence of frontend/backend/server?

[rapid sparrow]

shared types in git submodule
in which languages u have those shared types? is it only python or smth else?

[rapid sparrow]

i would imagine with git submodules approach as one of approaches u could be actually just utilizing in addition... volume binding mounts ? 🤔

they allow hooking in real time some folders

so changes will be automatically propagated

now we need to ensure only that we run CI for latest too? I would imagine u can add into your CI step fetching latest commits in git submodule