tools-and-devops

[rapid sparrow]

i feel like i will be able easily persuading my team adding task

looks very simple in working and readability... and super fitting CI and local env

we are still not having anything except invoking docker-compose commands in Github Actions after all

and just documenting docker-compose commands to README.md

[rapid sparrow]

https://youtu.be/vH3v4miy80U https://taskfile.dev/ is a true level.

https://www.youtube.com/watch?v=KLppseOoAIA

[tranquil slate]

Hey, I just have a question related to how AI models are generally deployed. How I've been approaching it so far is:

• Make model via jupyter notebook
• load trained model into django app
• contanerize django application
• deploy django container to the web

Would this generally be the correct approach? From what i've been seeing online, this is usually what's done (may be missunderstanding it). In this case, would I need to even deploy the SQLite database (which is used to store training data for the model) with my app to the web? Im also looking to have some sort of dynamic training (ie, users could input new data via a web interface to train the model again), which makes me then think that a database with training data should be on a cloud as well (and hence I wouldnt use SQLite as was discussed earlier in the chat).

I think I am just generally confused on how an AI application overall would be deployed, ie do I train the model, load it into django, deploy that (no db needed), but then how would you manage retraining the model dynamically? Also in this case of dynamic retraining, if we have replica pods containing containers with the django app, would these pods sync if one of them has the model retrained (dynamically)? I would really appreciate any help/resources that are related to the same technologies

[rapid sparrow]

i can't find information about it, but i remember seeing that jypiter models are the first prototype, that can be rewritten to vectors/C++ to speed up their performance greatly. That is supposed to be their more mature deployment strategy.

Also i remember it is possible to tune trained model to fit better some specific date set. (and it is supposed to take much smaller time to retrain model then)

• https://stats.stackexchange.com/questions/331369/what-is-meant-by-fine-tuning-of-neural-network#:~:text=Finetuning means taking weights of,speed up the training
• https://flyyufelix.github.io/2016/10/03/fine-tuning-in-keras-part1.html
• https://stats.stackexchange.com/questions/255364/fine-tuning-vs-joint-training-vs-feature-extraction
• https://stats.stackexchange.com/questions/278977/cnn-retraining-and-fine-tuning

I am not an AI guy, i just did something for it during university
Also in https://www.dsicommunity.org/ u could try asking Hubt there, he is professional in this regard (Combining machine learning and DevOps) (just don't mention i send u to him 😆 )
or asking question regarding this stuff somewhere in #data-science-and-ml or in AI/machine learning communities https://github.com/mhxion/awesome-discord-communities#linux

[thorn portal]

i think the answer really depends on your project's scale and complexity.

low traffic, low volume of data (even GB scale is small imo)? this is perfectly fine.

running A/B test? need close-to-real-time model training? having more and more training input? then this setup will likely fall over + just have too much overhead on the developer/data scientist daily workflow.

the keyword to google here is MLOps. unless you have specific pain point that you really want to get rid of, this might be more just for general knowledge instead of something practical though. it's way too easy to over-engineer these things imo. (i am a proud victim of this)

[thorn portal]

another convert to the church of task! one of us one of us!
glad you are liking it 😄

[tranquil slate]

I think the most complex thing in the project will be evaluating and retraining the model. Ie: a user can evaluate the model, and input new training data to re-train the model from the web interface. I assume this itself may require a different approach mainly with the database? I may just be understanding it wrong, but if its dynamically retraining, wouldnt this retraining occur on the cloud?

[tranquil slate]

Thanks! Will look into those

[tranquil slate]

Just out of curiosity, would combining persistent volumes (https://kubernetes.io/docs/concepts/storage/persistent-volumes/) and stateful sets (https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) potentially enable SQlite to work for deployment? (I understand though that for larger scale applications this would probably not be feasible, but in terms of small scale). If I am understanding the docs correctly, wouldnt these ensure that the SQLite database within the django app container would be synced between pods - in turn allowing for sqlite to exist in the different replica pods and avoiding the issues detailed here: https://sqlite.org/useovernet.html ?

EDIT: Looking more into it, I may be wrong with allowing sqlite to exist in the different replica pods - in that case, wouldnt it be fine to just use a single persistent volume that has sqlite, and then let my django app pods know to use that for data read/write? (Another issue with this approach (along with the sqlite docs issue) may be that SQlite is an embedded database?)

[thorn portal]

I may just be understanding it wrong, but if its dynamically retraining, wouldnt this retraining occur on the cloud?
that's correct, of course i am assuming you are hosting containerised django on the cloud already

in fact once your training process reach a certainly amount of complexity, you will inevitably opt for training in the cloud (unless you are happy with waiting for days, then no.) -- however it's worth noting people usually separate the process of training to another system (i.e. not your containerised django)
just remember training a model is not necessarily a one off process of training a singular model, it's highly likely that you will be training many models while searching for proper hyperparameters to use for your model, and this is not a short process.

---

so the question now boils down to, how fast do you train your model currently? is this going to balloon anytime soon?

if you can train it in seconds then i wouldn't bother making any drastic changes, assuming the user is happy to wait for seconds while django is potentially frozen

[tranquil slate]

Yes it trains in seconds

its very very small scale atm

But in that case, I have a question in terms of after retraining. So once retraining is done, how would this new model be applied to all of the different replica pods?

From my understanding, the replica pods dont sync

[thorn portal]

you need a model registry and a way to signal a new model is to be used then

(or you can skip model registry and just retrain across all pod)

how come you are using k8s?

[tranquil slate]

Im taking a project course atm related to AI. There arent many fixed requirements, but one is to use K8s. Apart from that, we have the freedom in terms of architecture, structure etc for the project

[tranquil slate]

Alright thank you very much! Im going to take a look into both of these approaches

[thorny shell]

I doubt that sharing persistent volumes would work, since iiuc that's effectively the same as sharing across the network.

[rapid sparrow]

StatefulSets for separate volumes per Pod.
Even if u will find a way that shares between multiple volumes... Having multiple writers to Sqlite is a sure way to corrupt database

Better accessing from multiple pods over network available sqlite at least. Having SQLite in one pod only i think

[tranquil slate]

Ahhh so basically a single instance/pod of sqlite would run? Something like: https://kubernetes.io/docs/tasks/run-application/run-single-instance-stateful-application/ but with SQLite instead of what they use In the guide?

[rapid sparrow]

👍

[rapid sparrow]

Or Deployment with single replica pod

In recreate Deployment strategy (previous pod in this strategy deleted first, before new one launched)

Technically, same result should be.

[tranquil slate]

Ahh I see, with this way, would I just make my deployment with the Django app a single replica? Hence my SQLite can stay in the same container with my Django app?

[rapid sparrow]

Make Django as separated Deployment of pods with any amount of replicas

Keep sqlite separated from them and accessable only over network to Django

I am trying to make u having stateless scalable main backend app

With aim for painless replacing Sqlite3 with something else later without making any architectural changes 😐

[tranquil slate]

RIGHT ok yes, it’s very clear now!

This is really great! Thank you all so much for your help! I really appreciate it!

It’s made these concepts, and how to approach deployment with K8s so much clearer for me!

[tranquil slate]

I have a question related to this, I may be miss understanding, but wouldnt there need to be some server for this to work?

[rapid sparrow]

write a bit more clear. in comparison to what? u wished to deploy Django without servers, only locally?

[tranquil slate]

My bad, I would be deploying django to a cluster on GKE

[rapid sparrow]

so... deploy Sqlite as to some simple computing instance with backups or something.
or deploy to kubernetes if u wish.
your choice to deploy it anywhere, as long as your django applications in GKE will be able to access it over network

[rapid sparrow]

May be it even makes sense deploying just straight along side with Django? if u don't need content of this Sqlite3 being persistent, and fine with each Django having copy of Sqlite3. May be u have your Sqlite3 changable only with deployment release versions

[tranquil slate]

Ok I see, thank u! Yes then i think maybe a persistent volume could be useful for this

[strong spire]

Hello, everyone! Does anyone here could help me with PLY in Python? I'm building a compiler, but I'm having trouble with if_statement there

[rapid sparrow]

[BaseGood]
base = br01_05_base
MarketGood = dsy_spatial_package, 1, -1, 1, 1, 0, 1, 1
MarketGood = dsy_firefly_package, 1, -1, 1, 1, 0, 1, 1
MarketGood = dsy_large_train_package, 1, -1, 1, 1, 0, 1, 1

[BaseGood]
base = br01_06_base
MarketGood = dsy_somtaaw_lf_package, 1, -1, 1, 1, 0, 1, 1
MarketGood = dsy_somtaaw_hf_package, 1, -1, 1, 1, 0, 1, 1
MarketGood = dsy_somtaaw_fr_package, 1, -1, 1, 1, 0, 1, 1

i have complex data to parse... is there any tool that can do it in a nice way?

[sharp elm]

Is there any way to get setuptools to deal with and install entry points? Like every other build system does?
pip install . with poetry, pdm or flit produces entry points nice and easy. But vanilla setuptools + wheel does not.

[project.scripts]
mybin = "project.cli:cli"

[build-system] 
requires = [ "setuptools>=41", "wheel" ] 
build-backend = "setuptools.build_meta" 

I'd expect ~/.local/bin/mybin to get installed. Any ideas?

[languid blaze]

Does anyone know if there is a good interface and/or api for working with jenkinsfiles in Python? Groovy is giving me issues and I ain't going within 20 smoots of XML if I can help it.

[rapid sparrow]

the best decision in this regard is not using Jenkins and migrating to Github Actions/Gitlab CI or any other modern CI tool
you can mitigate damage from using your CI tool by writing local workflows (chained tasks) through https://taskfile.dev/. Recommending to research it and learning how to chain tasks + using defer stuff in it / supplying variables into tasks, making them reusable. And then invoking in your CI tool only end commands. Thus getting easily debugged locally workflows

[languid blaze]

Okay, I'll bring it up with the MBAs in the cost center.

In case you don't pick up the sarcasm, I'm in an enterprise environment. I use Jenkins not because I want to but because I am required to.

[rapid sparrow]

it is very easy to insert Taskfile into any other CI tool xD it is lightweight one binary. As i said, u can mitigate damage from Jenkins usage through using Taskfiles xD
functionality at the level mimicking Ansible to some degree / CI worklow syntax

The best that it is language agnostic yaml. So... good for any project in any language.

[rapid sparrow]

just invoke Taskfile from Jenkins basically, if u have to.

[languid blaze]

I'll have a look, forwarding the link to the SRE chat on teams.

[rapid sparrow]

https://github.com/darklab8/darklab_article_docker_python in this repository i have examplw btw how to use almost zero of CI cloud tools with taskfile

variables:
  IMAGE_PIPELINE_RUNNER: docker:22.06.0-beta.0-cli
  # dind
  DOCKER_HOST: tcp://dind-service:2375

stages:
  - test

.setup: &setup
  before_script:
    - apk update && apk add curl
    - sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d -b /usr/local/bin

build-test-django:
  image: ${IMAGE_PIPELINE_RUNNER}
  stage: test
  <<: *setup
  variables:
    COMPOSE_ID: ${CI_PIPELINE_ID}_django
  script:
    - cd example_frm_django
    - task dev:test

build-test-fastapi:
  image: ${IMAGE_PIPELINE_RUNNER}
  stage: test
  variables:
    COMPOSE_ID: ${CI_PIPELINE_ID}_fastapi
  <<: *setup
  script:
    - cd example_frm_fastapi
    - task dev:test

before_script inserts into gitlab CI taskfile, sends variable i need through environment variable to taskfile, and from there its pipeline goes

equally same with github actions

name: Build

on:
  pull_request: {}
  push:
    branches: ["master"]
    
jobs:
  build-test-django:
    name: Build and test Django
    runs-on: self-hosted
    steps:
      - name: Check out source repository
        uses: actions/checkout@v2
      - uses: ./.github/actions/cleanup-docker
      - name: Install Task
        uses: arduino/setup-task@v1
      - name: Test
        run: task dev:ci_test
        env:
          COMPOSE_ID: ${{ github.run_id }}_django
        working-directory: example_frm_django

  build-test-fastapi:
    name: Build and test FastAPI
    runs-on: self-hosted
    steps:
      - name: Check out source repository
        uses: actions/checkout@v2
      - uses: ./.github/actions/cleanup-docker
      - name: Install Task
        uses: arduino/setup-task@v1
      - name: Test
        run: task dev:ci_test
        env:
          COMPOSE_ID: ${{ github.run_id }}_fastapi
        working-directory: example_frm_fastapi

[rapid sparrow]

under the hood both invoke

  ci_test:
    cmds:
      - task: compose:ci_run
        vars:
          CMD: web pytest
          PULL_CONTAINERS: db redis

  ci_run:
    internal: true
    cmds:
      - defer: docker-compose -f {{.COMPOSE_FILE}} -p {{.COMPOSE_ID}} down --volumes --rmi local
      - task: build
      - docker-compose -p {{.COMPOSE_ID}} -f {{.COMPOSE_FILE}} pull {{.PULL_CONTAINERS}}
      - docker-compose -p {{.COMPOSE_ID}} -f {{.COMPOSE_FILE}} up -d {{.PULL_CONTAINERS}}
      - sleep 3
      - docker-compose -f {{.COMPOSE_FILE}} -p {{.COMPOSE_ID}} run {{.CMD}}
    dir: ./{{.DIR}}

my ci run of tests through docker compose

x-services:
  app_poetry: &app_poetry
    build:
      context: .
      dockerfile: Dockerfile
      target: dev
    environment:
      DJANGO_DEBUG: "true"

version: '3.8'
services:
  db:
    image: postgres:15 # 5432
    environment:
      POSTGRES_HOST_AUTH_METHOD: trust
      POSTGRES_DB: postgres
  redis:
    image: redis # 6379
  job_migrate:
    <<: *app_poetry
    command: python manage.py migrate
  web:
    <<: *app_poetry
    command: python manage.py runserver 0.0.0.0:8000
    ports:
      - "8000:8000"
    depends_on:
      - db
      - redis
      - job_migrate

in any result, taskfile makes sure to destroy containers and volumes and images through defer

build container, pull services up with databases, run tests, destroy

[languid blaze]

I can guarantee I'll be the only person on the team doing the destroy step, based on the weekly emails we get from the cybersecurity team

[rapid sparrow]

anyway, it will work the best, if u will be able to get them also installed Docker with Dind in their jenkins. Surely that would be possible feat, right?
Without it, it will work too, just at lesser degree.

[languid blaze]

God I hope so.

Given I'm dealing with people who thought it was impossible to automatically add the build date to the image tag (so we can track the every-45-days-please-rebuild-the-image rule in datadog)...

[dusky cloak]

Hi there, anyone has version of Python-Rust or Rust-Python Docker image which he could share?

[subtle quarry]

Looking for "standard" refactoring tool. Like refurb, bowler, sourcery. Standard as for example black for formatting.

[rapid sparrow]

checked sourcery front page https://media.tenor.com/Hav5Q8L6w_oAAAAC/tbbt-the-big-bang-theory.gif

it is disaster tool.

decreasing readability through making comprehensions at every step. What a fine idea

[rapid sparrow]

Refactoring can't be done automatically beyond usage of classic linters (pylint for example)
you can heavily improve your code quality with usage of static typing like mypy though. Nothing about it is automated though

[rapid sparrow]

if u wish really improving code quality, learn unit testing and code complete book as starting books in the journey of refactoring and code quality

[subtle quarry]

Thanks will check these

[loud bear]

How can I create a different 32 and 64 bit wheel just for windows, but for all python versions it should be the same?

[tawny temple]

I believe a wheel is always tied to both the architecture and the Python version, unless it's a -any wheel

Sorry, I'm wrong. The spec does apparently allow for py3 as the Python tag or even py2.py3 for both. https://peps.python.org/pep-0425/#python-tag

It would be helpful if you described what your app is. Is it a C extension for Python, for example? How are you building it currently?

[real hedge]

Hello, Anyone know if VSCode have a reliable a static type checker plugin for Python?

[real hedge]

Nevermind, seems like I did turn on Pylance 😆 ( it ain't plugged In - BillyBob voice )

[weak heath]

I created my ecommerce website using django.now I am trying to horizontal scale by creating some containers of it.Should I use a different container for sqlite3 and media files,static file or should I create a volume ?

[short peak]

do you use sqlite3 for production?

[weak heath]

It was a side project

*is

[short peak]

for db, I'd suggest to use something else (like PostgreSQL) if you want to scale. I don't like the idea of using multiple instances to access a single file. For static files, I'd suggest to use web server or services only for that purpose, like another container with another NGINX, S3 like service, etc

[mossy sage]

Are there anyone who can help me integrating AWS SageMaker?

[weak heath]

I don't like the idea of using multiple instances to access a single file didn't understand?
*will update to postgresSQL
*The idea is to use NGINX as Reverse Proxy ,Load Balancer for multiple container running the application with a common DB for all containers

[short peak]

having multiple containers mean that you will access to a single sqlite3 shared through a volume, right?

[weak heath]

yes.I think having a common db is ok,because if each container will have its own db then ,consistency is lost

[short peak]

correct. For one container, it's ok. For multiple, I don't think so

[civic grove]

how does platorm engineering differ from devops?

[heavy knot]

yo

[loud bear]

Its pure python but uses a 32 and 64 bit DLL

I can save space by making a 32 bit wheel and 64 bit wheel

Also I really want to place it at the root dir in a vendor/ directory but the package-data doesn't seem to include the DLLs into bdist or sdist

[tawny temple]

I wonder if all you need to do is change the name of the wheel. As far as including DLLs, I had to do this for a project once and I believe I worked it out. I'll have a look tomorrow when I'm on my pc.

But be careful about licensing if the DLL is not yours. I got hung up on how to distribute LGPL DLLs in my wheels

[rapid sparrow]

😐 spammer. ask only in one channel your question. / or as help request post in help channel

[heavy knot]

What is the best way to learn GitHub

I know the commands, but it seems like I am missing best practices (when to commit changes, and when to branch off)

Like one of the questions I had was, is it acceptable to commit changes that are non functional?

[daring skiff]

Hello guys, quick question, can you install pip packages on a poetry environment, poetry add seems to work just fine, but are pip packages also poetry compatible?
also, how do you install a package from git with poetry?

[rapid sparrow]

#software-architecture message

[rapid sparrow]

poetry IS installing pip packages xD

it is just... wrapper around for more comfortable usage

automatically creating venv, installing packages to them

possible separation of installed packages into groups

pinning top level dependencies / and secondary deps

[heavy knot]

Is this not the right channel for version control? This is not a software design issue

[rapid sparrow]

the current channel is the right channel, the first person asked me in the wrong channel xD

[heavy knot]

You mentioned me lol

ah alr

[rapid sparrow]

1. Best practices regarding GIT (yes, just learn 5 xD) https://deepsource.io/blog/git-best-practices/
2. Learn possible workflows, and generic essentials quickly here https://www.atlassian.com/git/tutorials/comparing-workflows
3. Learn Git to very good CLI level here in interactive tutorial https://learngitbranching.js.org/
4. Learn git to advanced level with going through this book https://git-scm.com/book/en/v2
5. also get familiar eventually with conventional commits https://www.conventionalcommits.org/en/v1.0.0/
   can be useful for more order in naming, which are more meaningful. Especially considering that they are easily enforceable by git hooks or CI checks
   Copying to here for clarity then

https://i.gifer.com/70l3.gif

[rapid sparrow]

;Bases must follow the template and be sorted alphabetically. Ships native to the faction-owner, if not generic (civilian, border or edge worlds tech) can have rep requirement (0.5), generic ships should have negative rep requirement, which will allow anyone who can dock the base be able to buy the ship (-0.5).

rlds tech) can have rep requirem

rlds tech) can have rep requirem

how to write regex query so it would match only text of first line after ;? without having ; in a match

(?<=;)(.*)
found xD

https://stackoverflow.com/questions/5006716/getting-the-text-that-follows-after-the-regex-match

[mystic void]

https://regex101.com/r/6f6wQS/1 ?

[rapid sparrow]

yeah i know this way. but it matches only in first group of a expression
; gets caught into match too

i try to catch as match only due to limitations of a tool i have

[mystic void]

; wouldn't be in the group

[rapid sparrow]

it will be in a match though

[mystic void]

yeah

[rapid sparrow]

well, probably i just should use function in tool at my disposal: Get me all subgroups and getting from it

otherwise i will be breaking mind over even more complex regex

[mystic void]

you could also do something janky like [^;]*$

[rapid sparrow]

nah, that matches all other phrases too

too junky xD better using groups i guess

Found in my tool how to access groups in a nice way. I am good now, thanks

[barren epoch]

Hello, when i have file which is untracked and i want to update this file by pull, then my untracked code will lost? i know that i can do git stash but without stash, does it remove?

[thorny shell]

pretty sure git will say "that file would be overwritten", and will refuse to pull

👿1 [2022-12-02T04:38:14-0800][x] % git merge main
Updating c4e528f..14a2deb
error: The following untracked working tree files would be overwritten by merge:
    a_new_name
Please move or remove them before you merge.
Aborting

[barren epoch]

so and if i have untracked file and by pull only will change other file, these untracked files won't remove?

[barren epoch]

so what i meant, i added these two file in local but before push i want to pull new code, so when i pull will anything happen to these files?

[thorny shell]

why not back up your folder, and then just try it?

[barren epoch]

what do you mean in back up?

[thorny shell]

make a copy

[barren epoch]

yes its simple but i think very bad practice, if you have 100+ file you should copy these file? i think there would be better option

[thorny shell]

the better option is to learn git

[barren epoch]

yes 😄

[rapid sparrow]

if u committed those files, it will be safe to make next operation then.
u can check in a careful mode, with git fetch stuff, and there should be command checking if u will have merge conflict in some dry-run way
but that is not really needed, since u can just go git pull and jump into solving merging problems i guess (i think this operation can be canceled too? at least it is identified as rebase, which we can always rollback by git rebase --abort

[barren epoch]

but why will be merging problem? these file isn't on remote

it's my new local files

[rapid sparrow]

if nothing of your saved files were commited by any new author with versions newer than u have locally

then u will have no merging problems

merge conflicts arrise, only if someone committed work to same branch for files which u changed locally too. Only in this intersection merge problems appear (when u changed same files at the same time and commited results)

[barren epoch]

okay thank u very much!

[rapid sparrow]

solving merging problems is pretty common operaton though, nothing really should be feared about it

[barren epoch]

i have seen in videos but they have only one line but when u have this problem in big code i think then it will be difficult

[dusky osprey]

Hi
I'm looking for a way to **continuously **trigger a python code every time when an SQL query result is not blank.
I tried using Airflow's SqlSensor but it seems like the sensor's work is done once the condition is satisfied. I'd need to schedule more sensors over time and I can't even tell it not to schedule new sensors if one is already waiting. On the other hand I wouldn't want to schedule a dag every 10 seconds.
I know I can do a while True: time.sleep(10) if db_change_detected(): do_work() but isn't there a more elegant solution ?

[thorny shell]

I can't imagine any clean way to do this, unless the database can "publish" information about changes that it's making.

otherwise you've just got to "poll" (i.e., query it every 10 seconds)

I'd see if your database has something like this https://www.postgresqltutorial.com/postgresql-triggers/introduction-postgresql-trigger/

I've never used that feature, but it's conceivable that you could create a "trigger" that, in effect, "publishes" information about the table that the database is changing.

you'd have to

• figure out if a DELETE removed enough rows so that your query would return a blank result
• figure out how to get the trigger to publish that information
• figure out how to get your python program to consume that information

[dusky osprey]

my query is already designed in a way to only capture what I explicitly need (it calculates a hash and if the hash has changed, then I need to perform some actions). Long story short, I do not concern myself with "what exact change happened".
Triggers are an okay method however I only have read access to the database, so can't really create objects. I mean I could ask for one, but it's not really worth it.

thank you

[heavy knot]

so annoying that i cant speak..

[silver finch]

What do you mean by "every time"? You want to run a query, do things, and then run that query again more or less instantly?
There are a lot of important little details there. How long between runs can it go? Do you need to wait for the first one to finish before starting the next? What happens if you exceed your SLA? What happens if it breaks?

Also, are you aware of CDC, and does that solve your problem?

[dusky osprey]

It would but it's also an overkill of my case.

[dusky osprey]

I want every 10 seconds to run a query and check if it has results. If yes, I do_work() which lasts less than 10 seconds (let's say a second), then after 9 seconds I run the query again. No need to wait unti the previous one is finished because it will not last long. If it doesn't have results, I run it again after 10 seconds and so on indefinitely. If there are no results for a week, well then there are no results for a week.
If it breaks, I want it to restart itself,but I know how to cover that.

[thorny shell]

I wonder if all you need is ```py
import datetime
import random
import time

def run_the_query():
print("The time is now", datetime.datetime.now().isoformat())
if random.choice(range(2)):
time.sleep(random.randint(1, 3))
print(" Query is done", datetime.datetime.now().isoformat())

while True:
start_time = time.time()
run_the_query()
wait_until = start_time + 10
time.sleep(wait_until - time.time())

python3 arik.py

The time is now 2022-12-03T05:43:14.127803

Query is done 2022-12-03T05:43:14.127829

The time is now 2022-12-03T05:43:24.132650

Query is done 2022-12-03T05:43:24.132935

The time is now 2022-12-03T05:43:34.133123

Query is done 2022-12-03T05:43:37.138309

The time is now 2022-12-03T05:43:44.137469

Query is done 2022-12-03T05:43:47.142514

The time is now 2022-12-03T05:43:54.142446

Query is done 2022-12-03T05:43:55.147601

[topaz aspen]

is there a way to search open source python project code ? Sometimes I feel it would be nice if i could search the top 1000 python projects or whatever for a particular method usage or something

[thorny shell]

I wouldn't be surprised if github has a way to do that. Of course it'll only search projects that are hosted there, but many of them will be

[topaz aspen]

hm yea, maybe there's a way to search for a line of code across all projects

[dusky osprey]

That's the most basic level of how I see this. An ideal solution would have a comprehensive user friendly UI for turning it on and off, observing the logs for errors and status updates etc. The script does that but in a less user friendly way.
I've got an idea to still use airflow's SqlSensor but have the DAG on @once` schedule and after each successful (or any) trigger (and execution of tasks) I'll start another instance of the dag. This way I'll have more user friendly control over the process anddo less code since Airflow handles auth errors, timeouts, unknownhost and etc. accordingly out of the box.

Thank you for your help.

[heavy knot]

anyone knows about apache airflow

[dusky osprey]

I know some, yes.
Better to post the question right away.

[heavy knot]

my manager asked me to explore regarding that i have knowledge in python how to kickstart learning it can u help me out

[dusky osprey]

Plenty of youtube/udemy courses out there. Manning (and/or paktpub) should also have books available.
Try to read more on v2 but it's mostly the same, they just optimized how a few core functionalities operate.

[heavy knot]

Can we speak regarding this in pvt??

[dusky osprey]

Sure

[tawny temple]

Sorry for getting back to you late on this. Here is the code I used to copy DLLs. This was for a Python extension created with CFFI. You may be able to simplify it for your needs. I honestly can't totally remember how this all works. https://paste.pythondiscord.com/opumerarib

Hmm maybe it won't work for you since you don't have an extension, but this command is the build_ext command.

[loud bear]

No pure pyproject.toml alternative?

This is exactly where python packaging really sucks.. bad path handling

[tawny temple]

Maybe but I don't recall seeing anything that worked for my use case.

There's a bunch of stuff about including files here https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html

[loud bear]

Yes I tried this

It just doesn't get included unless its in a subdirectory with the code

In my repo, I want to place my code in say project/ and DLLs in vendor/ and in packaging I want to place DLLs beside code, any good way to do that?

[tawny temple]

Does manifest.in help or would it still be limited to data files within the package dir?

[rapid sparrow]

https://www.npmjs.com/package/git-conventional-commits
Who knows how to update current version correctly? xD

which u get in git-conventional-commits version

nvm, it looks like working on my tags

[tawny nacelle]

what is devops please?

[thorny shell]

https://en.wikipedia.org/wiki/DevOps

[rapid sparrow]

This book explains it

[floral smelt]

Using poetry, is there a way to not copy dev-related stuff in the containers for production? In example, I have a subfolder for tests separated from the main package but I don't want it. Thing is, my pyproject.toml is at the root of the project.

The structure looks like the following:

.
├── Dockerfile
├── README.md
├── docker-compose.yml
├── fastapi_crud_async
│   ├── __init__.py
│   └── main.py
├── poetry.lock
├── pyproject.toml
└── tests
    ├── __init__.py
    └── test_main.py

[rapid sparrow]

That is not job of poetry.
It is job of correctly writing Dockerfile and optionally .dockerignore

[rapid sparrow]

In an ideal world u should write dockerfile without COPY . . Command.
Copy only files and folders u really need.

And if u have some folders/files located nested in the stuff u copy... Add them to .dockerignore like .gitignore syntax

[floral smelt]

Like tests/*? The inconvenient with poetry are the pyproject.toml and poetry.lock files located at the root. Or else a Dockerfile inside the main project would've been enough.

[rapid sparrow]

I don't really see 🙈 why having poetry files at root is problem.

Copy in separate command from root

[floral smelt]

Through the same Dockerfile?

[rapid sparrow]

yes

[heavy knot]

multi tool

[proper edge]

Hey guys. I'm trying to get good at git and have run into a little problem. Using terminal, I created a new branch, added, committed, then pushed a file. Then I made a PR. I had to do another update that was not related to the current branch so I did git checkout again for a new branch. Added, commited, then pushed another file. Then made a PR for this update. For some reason my new PR says that I've done 2 commits. Each commit from a different branch. I made sure to git add only the file I wanted so why is another branches commit showing up in my new PR?

[rapid sparrow]

When u create branch, it creates by default copy of your current active branch

So from master u made A branch
Then based on A u made B branch
Thus commits from A became basis of B because u created B when u already added new stuff to A

U need creating new branches from master. For that u need to switch to it first.
Or u need to find how to add argument to branch creation from explicitly told branch u want, instead of using current active

[proper edge]

ohhh I see. that makes sense. So before I make a new branch I should always do git checkout -b main or whatever my master branch is called?

[rapid sparrow]

No no

-b it is argument to create branch xD

So do
git checkout master/main
git checkout -b new_branch

[proper edge]

ahh ok. that would make the basis main

ok that makes a lot of sense. Thank you so much

[rapid sparrow]

I would also recommend actually doing
git checkout master/main
git pull
git checkout -b new_branch

So u would get created new branch from latest stuff in master

[proper edge]

git pull wouldn't overwrite anything i've edited locally right?

or stuff from other branches that ive pushed already

[rapid sparrow]

if those local files aren't in git, then no.
If u committed all stuff on other branches then no, because u a currently in another branch master

In general(99.99999999% cases) master branch pulling never breaks anything in industrial environment because of branch workflow principles

U can get git merging conflicts only when needing to merge latest updates from master to your still developed feature branch

At least if u follow feature branch workflow principles. (U a forbidden directly committing to master, only pull requests can)

[proper edge]

I see ok. That's really helpful to know. Thanks once again. I appreciate you taking the time to help me understand this

[rapid sparrow]

U would not have asked it if went thrrough this interactive tutorial
https://learngitbranching.js.org/

[proper edge]

:o

Ill check that out

[silver finch]

Fyi GitHub's search does this without any tricks, but I think you need to be logged-in for them to show you the option; otherwise you only have search for users/repos/etc

[wispy garnet]

Hi! Is there a way to make an image for a prediction algorithm, but have the model file outside of the image itself. I want to update the model.joblib file and then not have to push a new version of the image.

in Docker, I forgot to add

[rapid sparrow]

Yes. Use Docker volumes if u use pure docker or docker compose
Or AWS ECS, or Kubernetes.
This is supported everywhere

[pine fern]

just heard a podcast with kelsey hightower. cool devops guy. core contributer to OG k8s project

[thorny shell]

bleah

k8s makes my life worse, not better

[pine fern]

you can always use a managed k8s service

[thorny shell]

we do

[rapid sparrow]

K8s wun love 😘 uwu.
Better than using AWD ECS

[pine fern]

thats what we use

[thorny shell]

it's probably our own fault: we're using k8s for something where we're not getting much benefit from it, but are paying all the "costs"

[pine fern]

oof. not the right use case huh?

i heard debugging pods is not fun.

[thorny shell]

indeed it is not

also finding your damned output

also trying to remember the eight thousand "kubectl" subcommands ("lens" helps a lot here)

[wispy garnet]

okay, can I have the volume mount in the docker compose? or do I have to mount it every time I update the file? or do I only need to add the mount command in the image creation command?

[rapid sparrow]

Just add volume syntax into docker Volume yaml, and make sure u launched it with already added code xD

[wispy garnet]

docker volume yaml?

[molten furnace]

You need to add the volume in the syntax of docker- compose something like this

services:
  app:
    image: nginx:alpine
    ports:
      - 80:80
    volumes:
      - /var/opt/my_website/dist:/usr/share/nginx/html:ro```

guys, I have a question, have you ever tried migrating requirements.txt to poetry? Or another package management

[thorny shell]

I think I did that once with pipenv

[molten furnace]

Did you migrate pipenv to poetry?

[thorny shell]

no

probably should have, though; pipenv is a pain in the ***

[rich heart]

Hi everyone,

I am sorry if this is the wrong thread but after 1 day python bot hasn't been that useful on this matter.
I developed a library that acts as mapper between python and GraphQL languages for clients (it transforms python classes in GraphQL objects and operations and viceversa).
Please let me know what you think about it:

repo: https://github.com/dapalex/py-graphql-mapper/

[wispy garnet]

version: 'v1.7'
services:
  titanic_mit_flask_und_docker:
    build: .
    image: titanic_mit_flask_und_docker
    ports:
      - "5000:5000"
    volumes:
      - /home/evo/Documents/MachineLearning/Docker_Mit_Flask_Und_Titanic/model.joblib:./app/:ro
      

Is this correct?

my WORKDIR is /app so I put that

idk if its correct

[wispy garnet]

I just want the model.joblib file to get used, I dont want any persistent data. Can't figure out how to do this without knowing full path of the WORKDIR

[rapid sparrow]

since u are clearly using your project folder, easier will be doing

version: '3.8'
services:
  app:
    build: .
    ports:
      - "5000:5000"
    volumes:
      - ./model.joblib:/app/model.joblib:ro

[wispy garnet]

Okay I will try this

[wispy garnet]

Thank you! That seems to be working now!

[wispy garnet]

I was having some problems when I started the container, but turns out sklearn was being a meanie and was not cooperating with a joblib file which was made using sklearn 1.1.2 and being used with 1.1.3 in the image..

[rapid sparrow]

if u will learn this book, u will be able to handle this stuff on your own

very good dev friend for documented dev env setup through docker

[wispy garnet]

Thank you!

[fleet halo]

tox 4.0.0rc3 is now out, this will soon become the stable release unless someone reports any more release blockers, help us test it with your project if you can! Thanks! https://tox.wiki/en/latest/changelog.html#v4-0-0rc3-2022-12-05 (tox 4 should be mostly backwards compatible with v3 minus some features long deprecated in v3 and now using isolated builds by default; as shiny new features I think the biggest are built-in wheel support instead of sdist for packaging, editable wheel support and fancy colors for reporting!)

[mortal zodiac]

hi

I use Github Desktop and want to know what the best way to pull down a PR locally to test it?

I don't want to fetch it into my main repository, so is there anyway to create a folder and fetch the contents there?

[feral badge]

guys i need help with github, i have a repository that doesnt have the right language. i wrote it in tyepscript and then compiled it to javascript, but the repository's language is 100% javascript while it should be 100% typescript

[still sentinel]

I am keep facing this problem from yesterday. Does anyone have idea to overcome this. I have tried all but also it's not working on my macbook pro.

[thorny shell]

i imagine you need to install some "portaudio" pacakges. Surely the pyaudio web page has instructions for this

[copper plover]

You might need to install a few extra libraries

[wooden flower]

Hi guys I wanna be able to dynamic pass the and docker compose up the container at the same time that I wanna bash and execute airflow so on inside of the container what is the right syntax plz ??

Btw I’m using GitHub actions

[heavy knot]

im on a branch and forget to switch back to main and make a commit on the branch that i then push... okay i realize i messed up so i cherry-pick the commit into main and try to push that

hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes

okay so I git pull but it only works if it's configed to rebase. I dont understand why that is. why can git rebase it but not fast forward it

[thorny shell]

because you have a commit locally that isn't present remotely.

The commit that you cherry-picked is not present remotely on that branch (although a very similar one, the one from which you cherry-picked, is.)

[stray tree]

i would say that you have to install wheel files for pyaudio and then install the wheel file using pip

https://www.lfd.uci.edu/~gohlke/pythonlibs/#pyaudio

Choose the lib which satisfies your system requirements.

and install it.

[still sentinel]

I tried that too, it shows an error.

[heavy knot]

I have a tool

but its in pyarmor

is there a way for me to unpack it?

[pine fern]

@rapid sparrow theres a conference im considering going to. one of the talks there:

stateful workloads

[rapid sparrow]

In the honour of k8s name origins with a hint on resistance being futile and u will be assimilated into k8s

[pine fern]

join the k8s

[heavy knot]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> lxml

how to fix this? python 3.11.1

[thorny shell]

first, get a help channel #❓｜how-to-get-help
Then, post all the output, not just a few lines. Include what you typed that provoked it.

@heavy knot

[misty magnet]

Hello everyone!
I have a question regarding pyproject.toml , setup tools and editable install.
I'm trying to understand if it is possible to define extra requirements for the editable installation.
The reason why I want to do this is because I want to have specific requirements just for development, given that editable installs are usually used specifically for development.

Now, at the moment, I'm getting away by creating an extra-options simply called dev to which I attach an extra set of requirements (pylint, pytest, and so on).

Is it possible to have these requirements installed only when the package is installed as editable?

[somber falcon]

is there any documentation anywhere about what exactly happens when you run pip install . while at the root of a python package that uses poetry (pyproject.toml + poetry.lock; build-backend = "poetry.core.masonry.api" under build-system etc.)? does it pip install directly from poetry.lock somehow?

[rapid sparrow]

poetry is augmentation on top of pip.
if u install via pip, you get installed pip package according to vanilla pip system. It got installed into pip, but not added to poetry
if u install package via poetry, it is added/or already tracked by poetry, and gets installed into pip venv system (there is flag allowing to skip venv, and installing straight to system pip)

[harsh linden]

Is the scripts section in pyproject.toml of a project managed by Poetry used for e.g. scripts that deal with putting things in PATH etc?

[mellow sapphire]

Hello guys! how can iaccess argument values like this args: a=1, b=2 ?

[thorny shell]

!d sys.argv

[rancid schooner]

sys.argv

sys.argv```
The list of command line arguments passed to a Python script. `argv[0]` is the script name (it is operating system dependent whether this is a full pathname or not). If the command was executed using the [`-c`](https://docs.python.org/3/using/cmdline.html#cmdoption-c) command line option to the interpreter, `argv[0]` is set to the string `'-c'`. If no script name was passed to the Python interpreter, `argv[0]` is the empty string.

To loop over the standard input, or the list of files given on the command line, see the [`fileinput`](https://docs.python.org/3/library/fileinput.html#module-fileinput "fileinput: Loop over standard input or a list of files.") module.

See also [`sys.orig_argv`](https://docs.python.org/3/library/sys.html#sys.orig_argv "sys.orig_argv").

Note

On Unix, command line arguments are passed by bytes from OS. Python decodes them with filesystem encoding and “surrogateescape” error handler. When you need original bytes, you can get it by `[os.fsencode(arg) for arg in sys.argv]`.

[thorny shell]

@mellow sapphire

[tawny temple]

Not to my knowledge. Extras will be available regardless of whether the install is editable. It's not unreasonable or uncommon to see projects suggest something like pip install -e .[dev]. Some projects opt to use tox or a makefile instead

I'm not a fan of using extras for dev dependencies cause I feel like it pollutes the extras with stuff end users don't care about. This is especially true when your project gas other extras that are intended for end users

[misty magnet]

I followed with the pip install -e .[dev]

I feel like it pollutes the extras with stuff end users don't care about

That is a fair assessment.

[graceful light]

I've been researching what tool I should use for listing / tracking projects I have, as well as tracking personal stuff like my PC & related inventory of stuff and other general things
I was using GitHub's project boards for each project I was working on, and presenting it in Notion, but the GitHub integration is extremely basic and is basically just a link to a repository (doesn't work when linking to a project)

I was looking at Notion alternatives and came across Joplin, which looks to be an open-source complete alternative to Notion, and Obsidian which tries to map out relationships between different pages you have
Do you guys use anything like Notion, Joplin, or Obsidian for such things? Do you have any other recommendations?
I'm looking for something that I can host locally that I can also access on my phone or tablet remotely

[tawny nacelle]

@graceful light I hate to counter your question with a question, but will you teach me the ways of the force so I can become a Jedi like my father?

[graceful light]

I know where the line is from but I'm not getting the joke here

[tawny temple]

What about good old pencil and paper? Or does that not fit the bill for remote access?

I feel like that often gets overlooked. But I don't know to what extent you need to track things so it may not work well.

[graceful light]

Between my atrocious handwriting, me being a much better typist than hand writer, plus with all the notes that I take, I don't think it'd work out well

[rapid sparrow]

Mind maps as a text
https://mermaid-js.github.io/mermaid/#/mindmap

Or
Fully fledged diagram drawer https://app.diagrams.net/

And both u can render from
https://www.mkdocs.org/

I just have repository with my docs, where I keep track thrrough..
https://sqlitebrowser.org/

Documentation in markdown of Typora is also great. Good for storage into git.
Mkdocs can combine all of it into one though, autobuilding menu / rendering mind maps / serving markdown

[graceful light]

I was thinking more along the lines of this:

1. Task boards: either integrated/pulled from GitHub projects / repo PR's / repo issues directly, or let me make the boards with custom stages, tags, priorities, etc. (Something like Trello, Asana, FocalBoard, etc.)
2. Create basic doc pages (like Notion, Google Docs, Markdown pages, etc.)
3. Create databases of pages with column property types like text boxes, formulas, single-select tags, multi-select tags, hyperlinks, checkboxes, etc.
4. Lets you create relations / references to other pages
5. Locally hosted with data syncing, preferably open-source
6. Works on Windows, Mac, Android, and iOS/iPad OS

Notion has most if not all of that (lacks proper GH integration or makes you pay for having more than one table get pulled from GH)
There's a few promising ones I could find on GitHub like AppFlowy and AFFiNE but they're both in extremely early alpha with very little in the way of functionality

[blazing flax]

Hey all, What's the correct syntax for specifying an exact version number within a setup.cfg? This is what I currently have but I'm guessing this is going to allow updated versions to be installed if available and that's not what I'm wanting. I've tried replacing the >= with both = and == but it doesn't seem to want to play ball, Anybody have any ideas??

install_requires = boto3>=1.26.24 pandas>=1.5.2 python-dotenv>=0.20.0 tornado>=6.2 websockets>=10.4 python_requires = >=3.11 package_dir =

[hollow roost]

in gitlab pipeline it says syntax is wrong

[graceful light]

Not sure if you're aware but cat prints out the contents of a file
If you're just trying to print that private key (for some reason) then you'd want to do echo $DEPLOY_PRIVATE_KEY
And the same for the next line, you don't need cat there

You might also want to wrap that variable in quotes so it doesn't accidentally break on escape characters

[hollow roost]

ok

tahmks

[misty magnet]

Printing the private key doesn't look safe :/
Out of curiosity, why would this be necessary?

[graceful light]

I mean I agree that printing the private key is most definitely not safe and unnecessary, but that's what they're doing their script
Seems they're saving the private key to a file to use for SSH stuff later and possibly printing out the key before hand to make sure it's correct as they're testing the code?

[misty magnet]

Yeah, I was thinking that. Debugging purposes.

[rancid schooner]

@rustic pawn Per Rule 6, your invite link has been removed. If you believe this was a mistake, please let staff know!

Our server rules can be found here: https://pythondiscord.com/pages/rules

[loud bear]

Is 2.0.0a7.dev13 greater than just 2.0.0a7?

[rapid sparrow]

The opposite

[loud bear]

ok alright

then its what i want

i just started using setuptools_scm

and it bumped the current 2.0.0a6 to 7

[ripe cave]

Hello, everyone, which server to choose to place the script there?

[zinc oak]

Hi everyone, I had Anaconda in my laptop and uninstall it (Anaconda creates its own virtual environmet). Now, I am facing issue in running python modules. I activated virtual environment in VScode and run a simple file there like import numpy as np. While numpy has been installed in VScode, after running the simple code (in the screenshot) again it does not recognize numpy. Any solution have in mind? Thank you

[fallow smelt]

My step #1 is to reinstall your OS from scratch.
So maybe wait for the next bit of advice.
But in that exact situation in the past, that has been my exact solution shrug

[midnight roost]

Hey everyone, 
I made my own bot! It was a cool project, so make sure to read the story: https://medium.com/@rickarentsen/i-made-a-bot-which-can-tell-you-anything-about-computer-science-14da74d57cf5?source=friends_link&sk=bd362e5f691d08c6904c36eb474b4dc6 feedback is appreciated

[compact cosmos]

https://github.com/python-poetry/poetry/releases/tag/1.3.1

New poetry release which includes the new poetry lock format v2

[lapis cloud]

Hi guys, i have a certain python file that imports the module "jinja2", i keep getting the error ModuleNotFoundError even though "sys.path" includes the directory that contains the module.
any ideas?

[compact cosmos]

if you run command pip freeze, do you see jinja2 in the output?

[lapis cloud]

Hey thanks for the followup, i actually fixed it!

[midnight roost]

Hey everyone,
I made my own stock market with python! Make sure to read the story: https://medium.com/@rickarentsen/welcome-to-my-own-made-stock-market-with-python-cb52f268944a feedback is appreciated

[harsh linden]

I need to develop some programs/script for for my company which I need to distribute some remote computers with very scuffed connections. Some of them run Windows, some run Linux. The Python versions also differ. Since setting up virtual environments and catering to each computer's needs is too much, I decided to just package my scripts with PyInstaller which sounds promising. I guess I need to to use WSL when I want to bundle for Linux from Windows.

Furthermore, my company does not really use any tools for properly distributing things. So far, I was just SSHing into the computers and manually uploading my scripts. Then I adjusted the configs for that particular setup, created systemd services/timers or task schedules depending on Linux/Windows.

Are there some tools for me to automate this process, starting from the actual shipping of the files and creating the services/timers and task schedules?

[harsh linden]

Not sure if I understand correctly: Poetry and PyInstaller do completely differrent things, right? Poetry cannot bundle applications so they just run, right?

[graceful light]

Pyinstaller creates an executable for a given system (IE it makes an EXE of your app on Windows)

Poetry is a package management tool that handles creating virtual environments for your app, installing the necessary dependencies into that virtualenv (thus keeping your global environment clean), and it handles dependency conflicts (like when one package requires a specific version of another package, but the latter is not compatible with the version of Python you are using, among other conflicts)
Poetry also lets you package apps into wheels and publish them on pypi

[harsh linden]

So when I just want to make my scripts self-contained so that I don't have to bother with the in-and-outs of the target system, PyInstaller is the tool to use

[graceful light]

Correct - Pyinstaller will package the entire app into either an EXE with some misc files OR it can package everything into a single EXE
This also packages the related Python runtime needed to run your application so it is 100% portable

[harsh linden]

Is running it through WSL enough to build for Linux or do I need a proper installation of some Linux distribution to be able to actually build for Linux?

[graceful light]

It should be fine to run it through WSL

If your company uses GitHub then you can use GitHub Actions to deploy to the related machines
You can do a similar thing with GitLab's equivalent deployment scripts
There's also Ansible which functions similarly

[harsh linden]

If only it would 😔

[graceful light]

Yeah trust me I now the feeling, previous job used old versions of Jenkins and Ansible and both were a pain to work with

[harsh linden]

We don't even have an old version of that. We don't even have any version control at all. I just build my own local repositories whenever I write something.

[rapid sparrow]

Ansible can install stuff over SSH remotely in a modern way. it configures chain yaml idempotent instructions for it, and obviously works for multiple server targets at the same time.
And yeah, pyinstaller is one of choices, but mind u that essentially u will compile only for Windows Amd64 then (your current OS+Arch), and Linux Amd64 with WSL then?

It is important matching compiled OS + CPU Architecture.
That is if u have 32 bit/i386 Windows, this is another env
Windows Amd64 another env
Linux amd 64 another env
Linux arm64 is another env

python in general sucks for cross compiling regrefully. So Windows + WSL is the easiest choice to get two OSes for amd64 (which should be enough as all modern PC in 98% capacity usually amd64. Arm64 is macOS quirk, +Arduino-embedded stuff + some cloud specialized stuff)

i just developed with golang when needed cross compiling

it is natively capable cross compiling for... a very wide range of systems (as long as CGO is not required 😆 but even in this case it is solvable with reasonable difficulties)

[harsh linden]

Yeah, the more I read about packaging and bundling Python, the more I see that it just sucks at that

[rapid sparrow]

https://freshman.tech/snippets/go/cross-compile-go-programs/ check how easy it is for golang. Just a matter of changing ENV vars during build (when u don't use CGO. If u use CGO it is solvable nicely with 'zig cc' stuff)
total supported different cross compiling combinations out of the box https://go.dev/doc/install/source#environment

$GOOS    $GOARCH
aix    ppc64
android    386
android    amd64
android    arm
android    arm64
darwin    amd64
darwin    arm64
dragonfly    amd64
freebsd    386
freebsd    amd64
freebsd    arm
illumos    amd64
ios    arm64
js    wasm
linux    386
linux    amd64
linux    arm
linux    arm64
linux    loong64
linux    mips
linux    mipsle
linux    mips64
linux    mips64le
linux    ppc64
linux    ppc64le
linux    riscv64
linux    s390x
netbsd    386
netbsd    amd64
netbsd    arm
openbsd    386
openbsd    amd64
openbsd    arm
openbsd    arm64
plan9    386
plan9    amd64
plan9    arm
solaris    amd64
windows    386
windows    amd64
windows    arm
windows    arm64

[rigid tundra]

I hope this is the right place. Is there a way to build a 32bit exe while on a 64bit computer without going through something like vmware?

[rapid sparrow]

U a in luck. Pretty much answered it right above u
#tools-and-devops message
i386 Windows is easily compiled... by Golang or other alternative modern solutions.

[rigid tundra]

Is there a python only way? Am I being too narrow for only wanting to use python?

What are the odds tho loll

[rapid sparrow]

Essentially no

python in general sucks for cross compiling regrefully.

[rigid tundra]

Oh right. Well I’ll read into it LOL

Thanks!

Is golang generally easy to pick up for this specific purpose?

[rapid sparrow]

in general yes, it is very easy to enter language. It has some unusual difficulties, but they get away once u get used to it
Recommending this book to get into it

[rapid sparrow]

also funny note: i have no idea what is your specific purpose

[rigid tundra]

err building a 32bit compatible exe on a 64bit system

[rapid sparrow]

if u would have only tried to read comments i linked above already, how easy it is
#tools-and-devops message
Repeating again that 32bit Windows is 'Windows i386'

[rigid tundra]

Yessir yessir mb was in a docs appointment and the doc finally stepped in

[heavy knot]

Does anyone here use docker?

I want to know what should I name my Docker Account name

And whether it would be good to put something silly or my actual name

And will my account name be shared with other people

[pine fern]

@rapid sparrow

[pine fern]

usually not unless you are planning on creating and releasing a lot of public Docker images for others. but someone who uses it more than me feel free to weigh in.

[heavy knot]

I'm using poetry and I found isort, how do people set it up? Do people just install the vscode extension and it will use the settings in pyproject.toml?
What would be the point of installing isort through pip / etc then. Since seems that to use black formatting we do need to install it through poetry but doesn't seem to be the case for isort?

[graceful light]

isort is specifically for sorting your library imports that happen at the top of the *.py files
black is for following PEP formatting guidelines
They're related but do totally different things

Installing black and isort through poetry just means that they will exist in your poetr-created virtualenv. Inside that env you can call isort followed by black to pretty much do everything you'd need
As an aside, a lot of people also follow that up with flake8 to make sure there aren't any lingering issues in your code

I personally just use the terminal to run those commands but if the VS Code extension is easier for you then by all means use that instead

If you get to a point where you're developing DevOps automation, running isort then black then flake8 in that automation would be pretty standard

[outer garnet]

Hey guys, I was deploying my service on k8s, when I run the task locally (through a local celery worker) it works fine, but when I run the task in the deployed celery worker I get this error.

I have the same, with the same versions on python and dependencies

[topaz aspen]

I want to create something in bitbucket CI which will enable me to

• setup a database
• apply migrations to this database

for this i think i need two docker images, but I need those images to be able to communicate - I need the apply migrations part to be able to see the database image from the first part in order to make sure they work.

Any suggestions on how to go about this ? I'm not doing very well at googling - maybe because I don't know what the terms are

[short peak]

if you need them to communicate via network, you have two options:

• popular option: docker-compose
• my preference: just docker but setting a network

You might want to ensure database is already running before attempting connections, some engines like PostgreSQL provide a handy healthcheck you can easily use

[topaz aspen]

i assume i need to use a network, but it would all be within bitbucket CI ? Basically i need to spin up a database image, then using a python image i want to apply some database migrations to make sure that they work - if that makes sense ?

(it's a postgres db fwiw - it's a timescale image)

[short peak]

it makes sense, I guess you are checking that migrations will apply without a problem

[topaz aspen]

yea that's all - i don't need to network with an external db or anything for this tho

[short peak]

also consider that if you are not using "real data", data migration is not asure

[topaz aspen]

what do you mean ?

[short peak]

if the db you just setup in CI doesn't have the same data as the target database, you are just ensuring python scripts and schema changes. Sometimes data migrations also produce problems and without this "real data" you won't know ahead of time

[topaz aspen]

ah i see yea - i might add in some data at some point

another outcome will be to have these migrations applied then test out some API package on a database that has the migrations applied to it

but i can't apply migrations in ci atm ha 😬

[topaz aspen]

i thought that perhaps i would have to

• create some ubtuntu image
• within the ubuntu image create images for the database and the python processes
  is that 'docker in docker' ? not sure if that's a complete pain or not

[short peak]

docker in docker is not recommended unless specific scenarios like docker development itself. Another options for these cases is blue-green deployments but definitely requiere a bit more of work to have it fully automated

[topaz aspen]

im not sure what you meant by network

[short peak]

docker network create my-awesome-django-app

[topaz aspen]

if i create different steps then they can't talk to each other - there's a way of creating a network within a ci process or something so they can all talk to each other ?

[topaz aspen]

but if i do this within a CI step - it'll be gone when the next step starts ?

[short peak]

sorry, I don't know that detail, I don't have access to internet and my data is limited to 2021

kidding 😄

I don't know that 😄

[topaz aspen]

fair - I'm not sure what your network is meant to be solving then i guess 🤔

[short peak]

yeah, that's for the app to communicate to the db. When you create containers using the same network and a specific name, then you and access then via their names. e.g.

docker run --name my-db --network my-net db-image
docker run --name my-app --network my-net app-image
# wait until db is up
docker exec  my-app pgshell -h my-db```

testing CI/CD is painful though

that means the you need to dynamically change the db host during the CI/CD

I rather go directly to blue-green

[topaz aspen]

ah ok - yeah this might be alright locally but in CI i think (?) i can only have one image for a particular stage

what's blue-green ?

[short peak]

https://en.wikipedia.org/wiki/Blue-green_deployment

[topaz aspen]

unless this could all run within an ubuntu image :S but idk if that's some docker in docker weirdness at that point lol

[heavy knot]

Yeah but it's more that for some reason VSCode uses the pip installed one for black, but then doesn't seem to do the same with isort. How do people set up this automation without VSCode? I'm curious

[graceful light]

Aside from macro keys I don't really see how people set up automation for that within VS Code itself
Most people use the automation tools I mentioned previously

[heavy knot]

I see, thanks for reponse! What's the most used combination of tools atm? type checking, formatting (i've seen black is the most widely used), linting, etc

[graceful light]

In order of what to run first to last:
isort for sorting your top-level imports
black is for automagically fixing your formatting
flake8 and/or mypy and/or pytype for finding any errors and telling you about any remaining linting/formatting stuff that black didn't handle
typeguard for run-time type checking

Then on top of all that you van add unit tests that you can look more into with the pinned comments in #unit-testingbut you'll most likely be looking at pytest or the builtin unittest

[heavy knot]

Oh thanks I'll check them! I have a confusion, are stuff like flake, mypy, pytype or pyright, etc? just type checkers? or linters too? I'm struggling a bit to understand the difference between them both and/or the tools

[graceful light]

Those 4 tools are very similar but may report things differently, or some tools report things that others do not
It'd be better to look at their docs or related pypi pages

!pypi flake8

[rancid schooner]

flake8 v6.0.0

the modular source code checker: pep8 pyflakes and co

[graceful light]

!pypi mypy

[rancid schooner]

mypy v0.991

Optional static typing for Python

[graceful light]

!pypi pytype

[rancid schooner]

pytype v2022.12.15

Python type inferencer

[graceful light]

!pypi pyright

[rancid schooner]

pyright v1.1.284

Command line wrapper for pyright

[pine fern]

reduced a cold start issue from 30s to 5s.

wanted to get subsecond times but deploying ML models in lambda custom containers is kinda hard.

i hope i dont get asked to do this again. i kinda suck at it.

would rather just put up a sagemaker endpoint or something

[pine fern]

fyi the original, original time was like 70s lmao

this looks promising though https://aws.amazon.com/blogs/aws/new-accelerate-your-lambda-functions-with-lambda-snapstart/

[rigid tundra]

Does anyone have any experience connecting to mqtt from outside of lan?

[mystic void]

yes

[rigid tundra]

How do you connect to it? I set my user and pass but it’ll say that the target machine rejects it

[devout quail]

Hello, Should I install Jenkins LTS or normal one ?

i wanna get started some simple stuff

[lusty sky]

hi I want to update version on each push -f master or on merge request but not on PR. https://github.com/gogo2464/cryptatools-rs/blob/master/.github/workflows/deploy-pypi-on-windows.yml#L40 This workflow creates an infinite loop https://github.com/gogo2464/cryptatools-rs/blob/master/.github/workflows/deploy-pypi-on-windows.yml#L43 how can I uodate the version in a way that does not loop infinitely please?

[rancid schooner]

.github/workflows/deploy-pypi-on-windows.yml line 40

bump2version patch .\Cargo.toml .\setup.py```
`.github/workflows/deploy-pypi-on-windows.yml` line 43
```yml
#- uses: stefanzweifel/git-auto-commit-action@v4```

[tawny temple]

Are you trying to automatically bump the version number?

Also why are you using a force push? You really shouldn't be for the main branch

[rapid sparrow]

I made actually a tool to calculate automatically next semantic version of a product

• Generating changelog, what new features and bug fixes were added
  https://github.com/darklab8/darklab_autogit/releases
  U can see in releases of the product, example of its own work

Today going to finish to state going more public.. although 0.3.0 version is already usable too, first development is almost finished

Main goal was to create CI friendly tool in addition

It assumes u save versions as git tags

It parses previous git tags and commit messages to calculate next version and changelog

It has part, helping u to enforce u write correctly git messages too

Within how u use, better to make github workflow with checkout of .git folder
And saving through git tag version of release in workflow too
And since u use fully automated CI way to release

I would recommend also using flag --build to which u add GitHub actions current commit hash or something additional stamp just to be sure for uniqueness. Not necessary but could be useful

P.S. force push should not be ever used to deploy releases

P.P.S. tool can calculate next alpha/beta/rc releases too

P.p.p.s. good idea to write automated GH workflow as example how it works during deployment I LL do it tomorrow
I remembered to write only GH workflow for its validator run

P.p.p.p.s. for main functionality it does not need having installed git to work. It has inside inbuilt git (no need to have installed git in pipeline). I LL eliminate last non essential remains of external git usage tomorrow

[lusty sky]

I have already bump2version. Could you just give me the bash to modify the version from github action yaml files plrease?

[lusty sky]

it is all what I need

if possible with bump2version

[rapid sparrow]

i will write workflow how to do it with my program autogit right now

[rapid sparrow]

interesting. Code is certainly way below my solution, i'll research if they made any interesting ideas i should implement.

my solution was specifically made to be CI friendly

their is not, already checked code

[rapid sparrow]

https://github.com/darklab8/darklab_autogit/blob/master/.github/workflows/autotag.yml
Auto tag workflow

[lusty sky]

could you check it out? https://github.com/stefanzweifel/git-auto-commit-action/discussions/271#discussioncomment-4442301

[rapid sparrow]

A very horrible and dangerous github action action that should be never used in repo?

What is there to check, except to unsee it

[lusty sky]

why?

[lusty sky]

how can I fix it please?

[rapid sparrow]

it is bad idea to have tools autocommiting into your git repository :/

builds should be located somewhere outside of repo

main branch of commits should be used by developers only

it will not prevent dev workflow then

Anyway, here goes release of my product
https://github.com/darklab8/darklab_autogit/blob/master/README.md
CI friendly tool for automatic following of Git Conventional commit rules (standard for more descriptive commits)
Based on it, the tool generates automatically changelogs of new features and bug fixes for your product releases
and also calculates next semantic version your next product release should be having (you can choose to have alpha/beta/rc releases as option)

[lusty sky]

I am getting a look. I am afraid because you have 0 star

[rapid sparrow]

😆 it was just released.

[lusty sky]

yes exaclty

[rapid sparrow]

Fair point
UPD: Yay 😁, i got whole one star!

[lusty sky]

@rapid sparrow https://github.com/gogo2464/cryptatools-rs/blob/master/.github/workflows/deploy-pypi-on-windows.yml#L43 I modified my code. WHat should I do with your project?

[rancid schooner]

.github/workflows/deploy-pypi-on-windows.yml line 43

#- uses: actions/checkout@master```

[rapid sparrow]

example how to use it in CI is provided. https://github.com/darklab8/darklab_autogit/blob/master/.github/workflows/autotag.yml
Since u use Windows runner, u could be wishing to use windows amd64 highly likely binary though (instead of linux one in example) (they are all provided at release page)
(or better yet, switching to using linux)

[lusty sky]

I must use Windows in order to publish to pypi

in myhumble opinion you must package this: https://github.com/darklab8/darklab_autogit/blob/master/.github/workflows/autotag.yml#L22

[rancid schooner]

.github/workflows/autotag.yml line 22

wget https://github.com/darklab8/darklab_autogit/releases/download/v0.4.0-a.2/autogit-linux-amd64```

[rapid sparrow]

Package? To where? and why?

It is binary file, why would it need packaging

It is self containing whole program without any dependencies

[lusty sky]

to install it more easily

you probably do not understand my need

bump2version works for me

[rapid sparrow]

:/ curl is enough. Wget for easier way.

[lusty sky]

I just want to run a command to update code on each commit

from a workflow

[rapid sparrow]

Why 🤯 would u wish to do that

[lusty sky]

so that when a PR is merged a version will increse

[rapid sparrow]

Code should not be self committing itself

[lusty sky]

ah is it a bad idea?

[lusty sky]

ahhhhhhhhhhh ok!!!

[rapid sparrow]

Yes, it is bad idea

[loud bear]

i recently shifted to setuptools_scm

what files should i exclude in sdist>

i am excluding all git / github related files

[crude furnace]

Hi! I'm having a problem with the push command. Whenever I try to push changes from my local rep to a remote one, it gives an error saying that I have no upstream breach, but when I try adding it, it gives another error: " warning: could not find UI helper 'GitHub.UI' "

Idk if I did something wrong, but idk how to solve it

I tried to serach for an answer and I found that the problem maybe be in within the latest version of git and I should try installing the previous version

But i'm not sure if this is the answer so i'm asking here

[meager osprey]

Hi, I've recently seen myself needing to have multiple versions of Python for the work I'm doing, which has lead me to installing pyenv-win.

I've run into a small issue though.

I use VSCode almost exclusively, so I was trying to verify pyenv working from within VSCode.

However, when I try to run pyenv --version in the integrated terminal, I get the CommandNotFound exception.

Even weirder, if I open a Powershell terminal seperately and run the same command, I can get the pyenv version installed on my PC.

Is there any fix for the integrated terminal?

[desert canyon]

where does poetry put installed dependancies after i run poetry install in a directory with a poetry project in it?

this is the first time i'm ever running that command, and poetry seems to be doing some work downloading all the dependencies for my project, but at no point does it tell what it's doing with them nor am i able to run any scripts my project exports after running it

i've just been using poetry2nix so far, but i'm trying to set up a CI/CD on a windows machine and need to figure out how to do things with just poetry

i could probably export a requirements.txt with poetry and checkout that to the vcs, but i'd like to stick with as little intermediaries along the way as possible

and i'm losing my mind, cause searches like "poetry install destination" bring up questions about where does the poetry binary itself get installed to, not where the dependencies fetched from poetry install do, and the documentation doesn't mention that one either

[desert canyon]

oh, it just installs it to a virtualenv path

i thought it would be trying to put it somewhere like site_packages

but i suppose that would make less sense

[rough hearth]

@rapid sparrow sorry to bother you, i got into web scraping lately and used the book you recommended too, i can see now why u mean it's a powerful tool

[unique violet]

I am trying to do a multi-stage build with Docker so as to inject dependencies into an image, as follows:

FROM python:3.8-slim-buster AS builder
RUN apt update -y && pip3 install pipenv

WORKDIR /opt/build
COPY requirements.txt .
RUN bash -c 'PIPENV_VENV_IN_PROJECT=1 pipenv install'

FROM ...
WORKDIR /opt/app
COPY --from=builder /opt/build .
COPY ./my_project ./my_project
ENTRYPOINT . .venv/bin/activate && python3 -m my_project

For some reason, when I run with docker run -it ... I am getting an error saying that the libraries listed in my requirements.txt could not be found.

Any hint?

[short peak]

try building the builder image only first, create a container and inspect it so you can ensure the builder is correct. Something like docker build -t builder:local . && docker run -it --rm builder:local bash

[unique violet]

Hi, did that before posting here ofc, and the build is fine, I can find dependencies from under site_packages

[rancid schooner]

:incoming_envelope: :ok_hand: applied mute to @heavy knot until <t:1671961287:f> (10 minutes) (reason: duplicates rule: sent 4 duplicated messages in 10s).

The <@&831776746206265384> have been alerted for review.

[broken crest]

hello. I have a few questions for anyone familiar with Docker. I just need a sanity check on a process I'm thinking of using.

I have a Python package (a web app using Flask) and I want to include some built artifacts from Vite (JS/CSS/SVG/PNG, etc) in the distribution. I have a linear pipeline that is straightforward for the "package and ship" approach. It builds the frontend assets and dumps out a directory. I copy that directory into the package namespace at build time and it's included as package data. It's a single artifact to ship and I can access the assets from Python (with importlib.resources) or host them with Nginx just by extracting them from the distribution archive.

But I'm trying to set up a development environment that can autoreload off my local edits rather than running this whole pipeline. The concept I've worked out seems like it will be a horror show and I'm wondering if it's even worth pursuing. If someone's around to talk it through, I can explain a bit more. Thanks.

[tawny temple]

I've used the "watch" build feature of Webpack to automatically rebuild files that get changed. Vite seems to support something similar.

If you want to run the server in Docker during local development then I suppose you can if you just make a bind mount

[broken crest]

Thanks. I've considered a bind mount for the source directory, but want to avoid the containers modifying the contents of the mount (I could get into that, but for brevity I'll just say I've got justifiable reasons). Mounting the source directory read-only mode could work if I configure the Python and Vite builds running in containers to spit their artifacts out to another volume/mount. Unfortunately some tools like npm really want to put node_modules in-situ and wrangling it to avoid putting stuff where the source code lives is doable, but a PITA (--prefix flag could work).

Another option I've been thinking of is to set up a (Docker) service that watches the source directory and uses rsync or similar (when a change is triggered) to copy the source tree to a writable mount where tools can do whatever they want, but then I run into issues like rsync deleting generated things like node_modules

So my effort to simplify the process is complicating things greatly.

I'm leaning to just abandoning the entire idea of using vite build --watch and the flask dev server entirely, and using some Docker-oriented feature to run the conventional builds triggering off changes to the source directory, but I'm not sure what that would be.

I'm really trying to avoid inotify pong

So I have a number of paths that could work, but are varying degrees of hairbrained lunacy.

Looking for a pattern that that lacks that particular fault.

[tawny temple]

Sounds like you've thought through this quite a bit already. Those do seem like valid issues.

I think the constraint of not modifying the bind mount is very much at odds with using Docker for development.

[tawny temple]

Only thing that immediately comes to mind is something like this https://code.visualstudio.com/docs/devcontainers/containers I'm not sure if that still violates your constraint, and it's also ide specific

[tawny temple]

Even if this does exist or you write a tool to do this, it sounds like it would be significantly slower

[broken crest]

yeah, the read-only constraint is a major stumbling block and maybe I'm fighting it unreasonably.

I'll read up on that. I already use VSCode, so that's pretty sensible.

As to why I don't want it modifying my local dev code:

• I switch between Windows and macOS and the code lives in Dropbox. That's why I don't want node_modules anywhere near those directories.

[tawny temple]

Code living in cloud storage instead of version control is a major red flag to me

[broken crest]

yes, there is a git repo and I'll commit code before swapping environments. I just don't want build artifacts being thrown in and the easiest way to stop that is read-only mount or using Docker's COPY behavior.

[tawny temple]

If you have a git repo then it's trivial to ignore build artefacts. And what would drop box be for then?

[broken crest]

Avoiding authenticating to a remote SCM in containers obviates the need to pass credentials into docker.

But you have a point. If a cloud storage system is creating a constraint, I could just not do that.

Mostly I'm prodding at this setup to see what I can give up that costs the least, heh.

I'm used to a bit more freedom than this.

I do use a git remote for the repo, and pulling from it on the workstation instead of using cloud storage is an option. I could also give up on the constraint to avoid secrets in docker and just use a proper secrets service (which Docker has).

[tawny temple]

Why would docker need to contain git credentials? The host can auth with git and clone the repo. Then the docker image build can copy from the host (I don't think this is even relevant for local dev since those containers will be getting source code from a bind mount anyway).

You could have the image build run the git clone, and that would require passing it credentials, but only during the build. They don't need to be saved in the image.

I've never tried it that way for a repo that needs auth though

[broken crest]

One value to is that you can automate releases and such with Docker. That's part of the project - trying to build my own pipeline (alternative to Jenkins or Github actions).

There is a need for secrets though. If I'm hosting Nginx, I'll need an SSL cert and such. So some way of getting secrets in is important.

Flask secret key, etc.

I was considering cloning my local dev repo into a volume as a bare repo which could serve as a common local filesystem remote for code in containers and my local workstation. It could even do things like stage up tags and release branches, closing them off entirely before pushing to the real remote SCM.

but that's more hairbrained lunacy.

Yes. So I'm struggling NOT to invent Frankensten.

[tawny temple]

I see, so I suppose a Docker container is the "host" and with that you'd run steps to build and deploy your application?

Building a pipeline service sounds like a difficult engineering problem, not one I have experience with..I would look at existing open source CI tools for guidance.

[broken crest]

The idea is that could build and host the entire thing in a single docker service - on my laptop, for example. Then ship the images/containers to run... wherever. If ever.

The pipeline isn't the challenging part - setting up Jenkins would be. And using a third party service for this (like Github actions) is a last resort, ideally.

essentially I want the following in docker:

• The dev environment. I don't want to run code on whatever workstation I'm using (and don't want those containers touching that workstation any more than necessary). Hence the desire to run Flask and Vite in docker (and other tools that contribute assets like data files).
• The "prod" environment. Where I can get the entire site rigged up and tested before hosting it somewhere (or only locally, if I prefer).
• The pipeline - Takes a release branch, builds the artifact(s), installs them, runs tests, merges, cuts the tag, pushes it with the artifact. I'm comfortable with this just being a pile of bash scripts I trigger.

And oddly enough, the only thing that's presenting the hard problems is the dev environment. If I just fall back on making dev exactly like prod, abandon Vite and Flask dev server, this all goes away and I'm good to go.

But I feel like the second I introduce another dev, they're going to look at me and go "You ditched all the dev tools?"

Avoiding third party services (like Github actions) is mostly because I don't think I need them for what I'm trying to do and don't want to grow dependent on them. Maybe that's misguided. And I'm avoiding Jenkins because it's a behemoth and way overkill for what I need to do. And Docker would be an entirely adequate substitute anyway.

I realize I've built my own minefield here. But I have numerous paths out. I'm just looking for the path of least regret and clean ways others have solved a similar problem. I'm sure bundling built assets in a Python release is an age-old problem, and Docker being the best practice for this sort of thing, this ordeal must be a solved problem.

Thank you for taking the time to help talk it out. I'm wandering away for the moment. Will check back later.

[light cedar]

So I am currently trying to dockerize development environments of angular and spring boot applications. The main reason for this is that our clients do not, yeah I can repeat, do not want their applications to be upgraded to newer versions of java/js so we are facing developers devices with tonnnnnsss of security issues regarding old and not maintained anymore more development software. I successfully created dev environments with docker containers, the images that those containers run have the development software required for the development process so the devices do not have it installed. But the real issue with this is that the software is still required by our IDEs (vscode/eclipse (I fcking hate eclipse)) so they devs has lints/autocompletion/etc.
I found that Microsoft is testing a vscode extension to run Visual Studio Code servers on containers and allowing the IDE to connect to them (very similar on what they are doing with WSL) but Im stuck on thinking about a solution regarding Eclipse, any ideas on how to solve this?

[storm phoenix]

So i deployed a fastapi docker image on aws ecs succesfully but for some reason i cant access the public ip address, it saysthis site can't be reached.
I have tried disabling firewall, have added all traffic in security groups inbound and outbound

[tawny temple]

I've used IDEA for Java and it was possible to select the language version to target, which could be older than the sdk version installed. The ide would lint appropriately if something too new was used.

[shy gazelle]

hello

here would be the right place to ask for help in a bug that I have find it on PyInstaller?

I need to compile a script with all its dependency's in just one file to run it in zabbix

that which I'm trying to do was already done in the past, I can see a script that was compile that way and actually works.

When I run the command pyinstaller --onefile file.py.... I get the compile script but when I do run the compile script he shows me that some modules were not imported

[inner matrix]

Hello All,

Cross-posting from "programming-pedagogy. I'm trying to teach myself github best practices. I've confirmed the following as necessary topics to master:

1. Enable MFA for the account
2. Use branch protection rules
3. Use a security scanner on your code.

I'm trying to learn more about #2. Specifically what I've done is I've created a github repository and pushed a project from my WSL machine. I then setup a branch protection rule with a name pattern that exactly matches "main" which is the name of my branch. I also selected "Require approvals" as set it to "1" and I've also set "Require review from Code Owners". After making this change I'm still able to push from my WSL machine as an anonymous "root" user. Therefore my questions are:

What mistake did I make?
What resources is anyone else aware of that I can use to learn github best practices?

[rapid sparrow]

as a suggestion, I can offer using docker volumes, and having through them node modules available locally(Even if they are installed in image). Then your eclipse should work.
At least assuming there are no absolute path troubles

A bit troublesome option since missmatch between libraries and node version can be at Dev machine, but i think it is acceptable solution

[cedar heron]

Im a beginner in Visual Studio and a beginner in Django.
I'm following a Microsoft tutorial and I'm stuck!
Haaalp! 😭😭💔💔

[rapid sparrow]

I know right. AWS networking is a pain. At least there is book to learn it

[cedar heron]

Hope this helps.

[rapid sparrow]

#❓｜how-to-get-help read the guide, how to ask good questions and channel info in how to get help please.
And never post code as screenshots

https://pythondiscord.com/pages/resources/guides/asking-good-questions/

[rapid sparrow]

Appropriate place would be opening git cloud issue (GitHub issue?) on their main repository 😅

[clever lagoon]

Have you made sure that the security group is attached to the ECS task/service and not just the EC2 instance? If you are running the ec2 instance in your VPC ensure that the subnet is also allowed to reach out to the internet (this is not a default setting for some AWS regions)

[vale wedge]

Anyone aware of a minimal ci server that plays well with gitea?

[clever lagoon]

Did you try GitLab Runner?

[vale wedge]

I fail to see how that plays well with gitea

[clever lagoon]

If you want to run your CI pipeline from a Gitea server, you can use GitLab's GitLab Runner software to host your runner. For testing, I use GitLab Runner for Linux, and it works well with my Gitea server.
It's the same product as GitLab's SaaS CI system, but self-hosted. They have a docker image for the Linux version on the GitLab GitLab Runner GitHub page. GitLab Runner is very easy to integrate with Gitea and GitLab's container registry. You just mount the gitea volume and you're done. Haven't done it in a while though.

Why do you fail to see how it plays well with a Gitea? Did you face issues with it?

[vale wedge]

I hadn't seen any docs about it that made it work for a non-gitlab forge, im very explicitly avoiding gitlab as to the poor experiences i had to make with it before (and the the gitlab ci was a big part of the pain)

[clever lagoon]

I'm sorry then, wait for someone else 😄.

[jagged badger]

Hey guys, What's up? These days I fronted up a problem I couldn't get out with a good answer. I've searched a lot of StackOverflow questions, sites, and everything, but it all seems to give a glue answer. How can I manage/handle multiple editions (Community, Enterprise, Pro) of software in a Git Repository?
The answers I got said I should manage its editions on separate branches and keep merging back and forward between them. This answer had a lot of criticism of this method because Git was not design to have long-lived branches. Another one is to keep editions on separete directories. It's the same problem of copying code back and forward, but without the merge conflicts. I was looking for how Gitlab mantain its editions, but I didn't get any indication. It seems they have Enterprise edition as a subdirectory of Community, but I didn't understand how they build each one. Could you share your thoughts about this? Is there a right way to do this management? Thanks.

[willow pagoda]

i dont have experience on managing different editions, but i do know that long-lived branches are common (e.g. your main and dev branches) so i dont see how Git could not be designed for them
as for a real example, CPython's repository uses multiple branches for the different versions they're maintaining (3.7 through 3.11), alongside a main branch where (i believe that) PRs are merged to first and backported afterwards
https://github.com/python/cpython/pulls?q=is%3Apr+is%3Aclosed

[jagged badger]

For this case Git works pretty well. It's the same code with different versions. What I mean is when I have a base code and need to change some piece of the code and add new features in the others editions. What will happen is a bunch of merge conflicts between branches of different editions. Maybe long-lived branch isn't the correct term here. I've spent a lot of time trying to figured out how to do this in a clear way. Nowadays I'm using multiple branches and it sucks. There must be a better way I haven't found it yet.

[willow pagoda]

i guess features are a bit more significant than patches, but with a reasonably organized codebase, maybe cherrypicking is sufficient for this? the two options im thinking of are adding all the features/patches into the base branch and cherrypicking them into their respective branches, or directly committing new edition-specific features in each branch and cherrypicking common changes from the base branch

and if common merge conflicts still happen, ive heard git-rerere can automatically resolve them for you after the first time

[jagged badger]

The big problem is reasonably organized codebase haha. I never heard of git cherry-pick, seems interesting. I'll take a look. I was thinking that using rebase instead of merge can improve conflicts. I appreciate .

I'am really curious of how Gitlab is doing. It seems that CI/CD is doing the trick. Some kind of script running in a pipeline but I could find any script. I don't know if a bash script with grep, sed and etc don't solve hehe

[somber lion]

I think for having multiple versions of the same application, it'd be better to try and keep them in the same repository, but then pick which edition you want when building. In like C++ you might use a pre-processor to strip the pro code when building the free version, but in Python you could potentially do the same sort of thing by ensuring most of the code is present in separate modules. Building the free version would deliberately exclude those modules, so they'd raise ImportError which you'd catch and then disable the button or whatever.

[jagged badger]

This approach really seems to be the best but I don't know (Don't have the knowledge yet) how to do it hehe. I couldn't find any example on how this is done at the build stage. Do you know some source: article, book, video, anything. Or maybe keywords that I could use to search?

What tools is used in the building stage of CI/CD??

[spiral bridge]

I've got a code that needs to be able to download a group of files from some cloud storage location. Currently I'm already using Google sheets api for this program, is Google drive api the ?most suitable option or is there another better option I should use for this?

Also sorry if this isn't the best channel to ask, there isn't a dedicated api channel as far as I can tell

[rough hearth]

Terraform, Maven, Docker, Jenkins, and Kubernetes can work on windows and mac? or just linux

[jagged badger]

For Docker there is Docker Desktop that can run on Windows, Mac and Linux.

[rapid sparrow]

Easier to say just Linux.
Use WSL in Linux mode with ensured full Linux virtualization in order to avoid any OS incompatiblities, if u that want to have it at Windows

Docker is not OS cross platform technology yet. Aiming to be with WASM though

The rest of tech stack is even less cross platform in general

[hidden narwhal]

What are valid methods for implementing performance benchmarks/profiling as part of a CICD pipeline?

Rust has this

If I wanted to add performance checks to my CICD pipeline and monitor performance improvements/degradations for each commit, what would I do? I'm aware of profilers like cProfile and such, but I'm not sure how to get around the variance that would come with different build hardware etc. Any suggestions?

[flat path]

Unless you have a dedicated server that you can tweak the hell out of (for stability), the best you can usually do is to comparative benchmarking (I just made this term up). So when CI triggers, you test the performance of some baseline revision (probably main or the latest release) and the target revision (the PR or a commit to main) and compare the two. You won't be able to measure trends across builds, but this will at least let you know whether a patch causes a performance improvement or regression.

[lunar drum]

I'm not sure I understand the scenario. If you are running performance checks in a CICD pipeline, you elect where to run the build, the performance tests, etc. Why would there be different build hardware each time?

[hidden narwhal]

My understanding is informed from this article https://pythonspeed.com/articles/consistent-benchmarking-in-ci/

Which says that things like " GitHub Actions might hand you a completely different virtual machine for each workflow it runs."

[lunar drum]

Oh! That's true. In my experience, many production applications are hosted on docker or auto-scaling virtual machine clusters. The point being you wouldn't have that level of control in practice, so why worry about it in theory?

If you're in a situation like the SQLite team, where the .5% improvement/degradation in performance is critical, then yeah, I could see that now being a huge problem.

If that's the case, I think @flat path is on the right track. You could try a self hosted runner for the "benchmarking" step and run your profiler tests on that target environment.
https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners

[hidden narwhal]

Use case I'm thinking of is python code written to be called in aws lambda, so we're fighting the 15 minute lambda limit

[opaque gorge]

Gotta love windows github workflows. Spent 2 hours debugging this...

[opaque gorge]

worth it though, finally got this out: https://github.com/ItsDrike/setup-poetry

[digital lotus]

I have initalized a repository on my local mahine, how do I add it to my github?

[thorny shell]

I think you have to use the github UI to create a repo, then copy the URL from that new repo into your local repo by doing something like "git remote add origin [e-mail]/you/that-new-repo"

https://github.com/new

see where it says "skip this step" at the bottom? You gotta skip that step 🙂

🙂 [2022-12-31T07:33:06-0800][/tmp] % mkdir tess-ting
🙂 [2022-12-31T07:33:10-0800][/tmp] % cd tess-ting
🙂 [2022-12-31T07:33:11-0800][tess-ting] % git init
Initialized empty Git repository in /private/tmp/tess-ting/.git/
🙂 [2022-12-31T07:33:13-0800][tess-ting] % echo "Allo allo control tower? Lisbon plane taking off in 20 minutes" > the-file
🙂 [2022-12-31T07:33:43-0800][tess-ting] % git add the-file
🙂 [2022-12-31T07:33:46-0800][tess-ting] % git commit -m "My first file"
[main (root-commit) c4d639f] My first file
 1 file changed, 1 insertion(+)
 create mode 100644 the-file
🙂 [2022-12-31T07:33:52-0800][tess-ting] % git remote add origin [e-mail]:offby1/tess-ting.git
🙂 [2022-12-31T07:34:26-0800][tess-ting] % git push --set-upstream origin main
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Delta compression using up to 8 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 280 bytes | 280.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
To github.com:offby1/tess-ting.git
 * [new branch]      main -> main
branch 'main' set up to track 'origin/main'.
🙂 [2022-12-31T07:34:34-0800][tess-ting] % 

et voila

[soft axle]

Git question: I've managed to scuff up my history somehow when using PyCharm to edit a commit. How would I clean this mess up, and combine the two commits with the same commit message?

(this is on the main branch if that's relevant by the way, and yes I do know that committing to main is bad practice)

[vale wedge]

At this point not without force pushing to main, which is typically strongly recommended against doing

[vale wedge]

Oh, wait,if its still local and not puahed, just do a interactive rebase and drop the commits you don't want

[soft axle]

If this could be fixed by a force push to main, what would be the cost?

[vale wedge]

Any fork using the prior history would have to be updated

[soft axle]

And if I have no forks of the repository?

[vale wedge]

Then it's still "safe"

[soft axle]

Alright, thanks. In thaht case, do you know how I would fix that mess, allowing for force pushes?

[vale wedge]

AFAIK branch protection has to be turned off (please make backups, it's a footgun)

[soft axle]

Okay, let's assume I've turned off all protections on the remote repository as necessary.

How would I actually squash the duplicated commits and the merges into just one commit?

[vale wedge]

That's part of git history editing tutorials and /or part of the ux of your ide

[rapid sparrow]

Interactive rebase to "squash"
That is the best. Don't listen to suggestions about dropping commits xD

Squash joines/combines

Drop deletes

[rapid sparrow]

👆 #tools-and-devops message

[eternal gate]

Is there any linter that will catch and warn/error bare exception raising? i.e. raise Exception("some error")

[rapid sparrow]

only possible in your code parts, i mean if your code wrote it to have explicitely

technically nothing prevents catching in libs too... but what would be the point

[rapid sparrow]

anyway, just some linux command can catch for you problem like this

some kind of recursive "find blablabla" | exit 1"

[eternal gate]

flake8 will nicely report caught bare exceptions but they should never be raised from our code.

[eternal gate]

Yeah, it should be quite trivial to grep for but would prefer it as part of some style guide enforcement tool

[rapid sparrow]

Pylint caught except Exception: I remember

[narrow timber]

Hey there, I hope this question is in the right place here … There is this package: https://github.com/kbr/fritzconnection One can install it via pip install fritzconnection and then run some cli commands provided by it. But, is it somehow possible to run it without installing it systemwide, just from within the project directory?

[eternal gate]

Look into using virtual envs https://pypi.org/project/virtualenv/

[heavy knot]

Does somebody maybe have an Idea to the following issue/discussion?

https://stackoverflow.com/questions/74984613/how-to-create-a-color-profile-of-a-video-using-python

[dreamy kernel]

is there a difference between using the sync fork vs. deleting and reforking after a PR?

[rancid raven]

Hi guys I'm having an issue with docker, when we do multi stage builds, if i have a Dockerfile like this:

FROM base AS build
...

FROM base AS deploy
...```

Specifying the target as `deploy` does not execute `build`. I would like to execute these sequentially since build makes some files that deploy needs.

[subtle quarry]

i am looking for inspiration on how to install dataiku dss on gcp using terraform 🙂 thanks in advance 🙂

[brazen forge]

the usual way is to copy the files you need from the build stage

COPY --from build /path/to/some_folder /path/to/dst

[winter birch]

Are there any BPMN tools in Python that can support business automation models for containers

[barren matrix]

Hey all

Ahm, where can I find the help section in here?

[tiny jungle]

#❓｜how-to-get-help

[old thistle]

How do you run a Dockerfile ? I tried googling it but it came up with docker load with a tar file? I copied a project with a Dockerfile and I assume the way to run the project is by somehow running the Dockerfile, which i don't know how to do

[indigo zenith]

https://docs.docker.com/engine/reference/commandline/build/

If you're in a folder with a Dockerfile and eveything else it needs docker build . should do it. I usually add -t some-relevant-image-name so I'll know what the image is

that's just to build the image, then you'll need something like docker run --name some-relevant-container-name -d -it some-relevant-image-name to actually bring a container up

[old thistle]

Cool, thanks!

[soft hound]

(AWS) I have a process where every second Tuesday of the month I:

1. create an EC2 from old AMI I created the month before
2. run SSM to patch the EC2
3. run CloudFormation that builds environment with new AMI
4. other stuff

best way to automate steps 1,2,and 3?
in fact if I could automate step 2 to run like...at 5am or some shit, that would be dope

[indigo zenith]

I've not done anything like that myself but if I were going to, I would look into boto3 (https://blog.knoldus.com/how-to-create-ec2-instance-using-python3-with-boto3/) and/or Terraform (https://www.techtarget.com/searchcloudcomputing/tip/How-to-launch-an-EC2-instance-using-Terraform)

[rapid sparrow]

Could u clarify how many times it is supposed to run per unit of time? Your writing was a bit unclear

Every second Tuesday of a month?
I wonder if we can express it as Cron time

Clarify what u do under 4

Clarify if u expect automatically self destruct this raised instance after job is over

[soft hound]

Yeah that's probably how i will time the lambda.

Just needs to run every second Tuesday of the month as that is patch Tuesday

[devout lodge]

Hi 👋
Is there a way to make this command python setup.py bdist_wheel --universal upload -r server-1 upload to multiple local pypi indices ?

[devout lodge]

Or what's the best way to mirror pypi index, so that uploads to an index gets mirrored to other indices ?

[lusty forge]

I've created a dir full of kubectl manifests, currently 4 in kube/postgres for my db, and one in kube for my Discord bot. Running sudo kubectl apply -Rf gives me: dan@ubuntu-2gb-ash-1:~/Overseer-Beta$ sudo kubectl apply -Rf kube deployment.apps/bot unchanged configmap/postgres unchanged deployment.apps/db unchanged persistentvolumeclaim/postgres-pv-claim unchanged persistentvolume/postgres-volume createdBut oddly enough this gives me two bot instances, and everything ends up stuck on Pending: dan@ubuntu-2gb-ash-1:~/Overseer-Beta$ sudo kubectl get pods NAME READY STATUS RESTARTS AGE bot-5d7d84f75d-dfxfm 0/1 Pending 0 60s db-b97b9b945-wzhrs 0/1 Pending 0 60s bot-54b8f65b47-xpbkz 0/1 Pending 0 60s

[rapid sparrow]

Show us your code?

Solid option would be using Helm btw.
Helm automatically removes previous applied group of manifests

Helm is like Python package uniting multiple Kubernetes manifests

Together with Skaffold to simplify its applying, it is very enjoyful to use

[bold elbow]

I'm trying to package a project using hatch and I'm trying to include some json files in my package, I'm not exactly sure how to do this, I see the shared files option of the wheel builder but that sets up the assets globally for the environment, here's my project structure

pyproject.toml
src/
  gpt_assistant/
    __init__.py
    __about__.py
    main.py
    model.py
    api.py
    plugin_manager.py
assets/
  message_schema.jsonc
  response_schema.jsonc
  sample_response.json
  sample_schema.json

[lusty forge]

Ah, gotcha, though I'd like to experience the manual stuff first before abstracting away the tedious stuff. Before I gather up my code, could it be pending because the available ram is ~1gb?

[faint umbra]

👋 has anyone worked with the PyGithub library that has some insights into how you can get a Github App working and connect to an installation of the app without having to tell it a specific repo?

ex:

integration = GithubIntegration(integration_id=GH_APPID, private_key=open(PEM_FILE).read())
installation = integration.get_installation(GH_OWNER, GH_REPO)
access = integration.get_access_token(installation.id)
g = Github(access.token)
org = g.get_organization(GH_ORG)
...

i can't seem to get to a point where i can get an object of my organization without having to specify a random repo, which seems odd. like i could just give it a random repo, but it should be installed as "any" repo on the org, so why would i have to supply it with a repo?

[rapid sparrow]

Sure. 1 GB is really small amount for fully fledged kubernetes

fully fledged one should start from 2 GB probably

1 GB can be for smaller kube versions, microk8s/k3s

since they work on Sqlite3, they take less ram

[lusty forge]

Aye, I'm using k3s. I seem to have found the issue, mismatched volume claim names and adding mountOptions when not supported.

The replication is a huge pain in the ass though, it's impossible to remove existing pods when the one you delete is instantly inserted in its place. Can this be disabled?

[rapid sparrow]

When u create/deploy Deployment, it has self healing property(Replica set) to pods.
Delete Deployment (object) to delete it with pods

kubectl get deployments, or smth like this to see them

Existing deployment can be scaled to zero without deleting if desired

[lusty forge]

Ah I see, seems like kubectl delete -n namespace deployment deployment works to delete the pod.

[lusty forge]

Does K3s support local images? I ran docker image build . -t overseer-bot on Windows, and then provided this image with a pullPolicy of Never yet it says there was an error pulling it

[rapid sparrow]

Kubernetes supports any images it can pull somewhere over network.

Use docker hub or gitlab docker registry for free.
Mikrok8s has option for creating local docker registry. Perhaps k3s has it too.

If u will find u can use your client Docker in a network exposed server way, u will be able to apply it too directly

[heavy knot]

IS this about hacking?

[desert viper]

'Error: Invalid value: File does not exist: ./app/app.py >>>> #!/bin/bash
streamlit run ./app/app.py

app.py is inside rootfoler inside app folder

[lusty forge]

Gotcha, will give ghcr a shot

[rancid raft]

https://github.com/serpapi/serapis-ai-image-classifier

[lusty forge]

I've got a kubernetes dashboard running in kube-system (namespace), though signing into this with the admin user gives me: https://cdn.overseer.tech/file/the-void/screenshots/firefox_LGgkcC4H3B.png
My config is pretty much identical to https://github.com/poketwo/kube-manifests/tree/main/kube-system/kubernetes-dashboard, and I run this by sh cd /home/dan/Overseer-Beta/kube/kube-system/dashboard sudo k3s kubectl kustomize --enable-helm | sudo k3s kubectl apply -f -

[sly sleet]

that's not the same path listed in your run.sh

[soft axle]

I've got a strange issue with Git/GitHub - how do I pass a required status check for a commit which hasn't been pushed yet?

I was under the impression that this would be for pull requests only

oh never mind, I can't read :p

in which case, I'll correct my question - is there a way to only enforce this rule for pull requests?

[winter birch]

Are there any python 🐍 based reporting or analytics tools and business intelligence software that are free and open source

[rapid sparrow]

what kind of info is expected to be received and from what

[winter birch]

For now I’m not sure let me know if there are any that are not open source too

Any software framework that is available on GitHub and can be used to develop analytics and reporting tools using python 🐍 as the backend

[rapid sparrow]

:/ what kind of analytics

It is too broad word

[winter birch]

Yes I get that

[rapid sparrow]

Provide any example or smth

[winter birch]

Maybe 🤔 I will ask again after a day or two

For now I have no clue either

Even non open source should be enough to understand what kind of tools are there

[rapid sparrow]

All right. Let's simplify it. What kind of data u have, from which u want to extract analytics

[winter birch]

I found something called Domo and Dundas

It’s a ERP and workflow kind of data that requires some dashboard, analytics and reports to be made

[summer maple]

Can someone help me make my selenium bot faster, I think it works just not fast enough when it sees something it needs to click.

[lethal mantle]

I hate git

[rapid sparrow]

I love git 💖

You probably just don't know how to cook it

[bleak iron]

git is good

[rapid sparrow]

git good 🤣

[bleak iron]

ya

@rapid sparrow btw do you have any config recommendations for vscode to make it more useful

i am used to pycharm and it feels like what i have set up right now doesn't give me many insights on what's wrong with my code before i run it

[rapid sparrow]

Head first recently made book about git btw

[rapid sparrow]

Perhaps u wish gradual static typing with mypy/pyright and vscode extensions for automatic error showing with them in order to see errors before code running

[bleak iron]

is there any extension that lets me know if i what i typed for a function / class use from another library matches the expected type signature?

[rapid sparrow]

Extension mypy 🙂

[bleak iron]

that's basically what i want, for my editor to yell at me before i run

oh cool!

thank you 🙏 you've been very helpful

do you have any thoughts of vscode + neovim extension vs straight vscode for productivity?

[rapid sparrow]

Not sure if understood correctly your saying though 🤔

Mypy catches typing missmatches between different functions input and output types at least
And variable missmatches when it is possible clearly getting their types

Also heavily recommending pydantic models. It gives run time ensured typing checks. Perfect data carrier

[bleak iron]

our project uses pydantic, seems pretty good

[rapid sparrow]

I am not pervert enough to use neovim yet
(Hard learning curve for dubious results with less feature richness than vscode)

[bleak iron]

there's some performance issues with pydantic for the IoT work we're doing, but for validating our pipeline in development and in prod we still use it initial data type validation for sources

[rapid sparrow]

Python in general is having performance issues (let's not recall Multithreading pitfalls of GILL)
And its typing is limited

I switched with high enjoyment to golang

[bleak iron]

this code was initially written by a bright data scientist i am working with, but we're slowly optimizing it

I want to change most of the code to golang, as i don't think most of it belongs in python

[rapid sparrow]

When using static typed languages like Golang, finally code architecture, Refactoring, TDD. Everything works correctly and easy in good typed language

And parallelism is so easy to use

[bleak iron]

So one particular thing I am curious about is I want to leverage some sort of structured streaming approach like in spark

because we have these various data sets that have different precisions and semi overlapping records that come at different times

that way we can emit records out as we patch the merged data set from multiple sources

Does this functionality exist within golang?

https://d2slcw3kip6qmk.cloudfront.net/Lucid4Edu/2020PreviewImages/Venn_Diagram.jpg

I am not sure if you know if golang has this functionality to merge streaming sets and emit a new record in an efficient way

The idea is basically this, you have source blue and source pink, which overlap at purple

You want to output a record whenever blue or pink change of the combined blue and pink data set

(ID, attr1, attr2), (ID, attr2, attr3) -> (ID, attr1, attr2, attr3) as an example

with each parts changing at different times

[rapid sparrow]

Sounds some kind of Kafka stuff

Not sure I understood correctly, but as fully fledged language u can implement anything in Golang or reuse library/database streaming system that for most part already implemented it

Sets do not exist in Golang though. Only maps(dictionary).
U would have to implement sets based on maps or finding set library

[bleak iron]

we use kafka as our message broker so you're not far off

[rapid sparrow]

If your recieved blue. Where did you expect to query pink in order to emit merged purple:)

[bleak iron]

at time t1 Blue provides the record {"id": 1, "attr1": "dolphin"} no combined record is created
at time t2 Pink produces the record {"id": 1, "attr1": "monkey", "attr2": "silly"} the combined record becomes {"id": 1, "attr1": "monkey", "attr2": "silly"} with time of t2
at time t3 Blue provides the record {"id": 1, "attr1": "dolphin"} the combined record becomes {"id": 1, "attr1": "dolphin", "attr2": "silly"} with time of t3

i guess my hesitation to move to golang is what support does it have for stream processing

[rapid sparrow]

Strange rules. I don't see a pattern.
T1 blue did not create
T2 pink created record with all it had
T3 blue overlapped previous record and made combined, but why T1 record did not change anything 🤔

[bleak iron]

As there was no previous history with Source pink emiting a record would result in unknown data for attr2

[rapid sparrow]

Ah. Attr2 is required.

[bleak iron]

Just as an example

It's a common pattern for our data

A lot of this is easy to handle with historical records, you just iterate through, but doing it in real-time without a library seems like we'd be giving up some guarantees

so TLDR, what do people use for stream processing in golang and what does it provide

I barely understand the golang ecosystem so it's hard for me to know if I should jump to it

[rapid sparrow]

Streaming and time series databases are language agnostic, and other dbs

Just a matter of checking if library was written for language X. Usually golang is among supported for any kind of infra stuff.

Be it time series databases timescaledb, Kafka or anything else. 🤔

With stuff like that I did not work though.
Depending on amount of workload, i suspect even stuff like AWS DynamoDB can be used in your case 😐

Surely full support to all stuff like this should be present in Java though. It is static typed too xD and way older and library rich

[rapid sparrow]

One of main benefits of Golang in regarding to your complex stuff like this, that golang is super friendly to serverless architecture of AWS lambda stuff.
It is fine to run with all installed any libraries inside it

[bleak iron]

Scala would support what we need, I am just not sure about introducing a FP language into a team with no FP experience

[rapid sparrow]

That is main reason I managed to persuade using it. We are deep in AWS Lambda and step functions

[bleak iron]

We're going to be migrating to Azure, though I'd prefer AWS, and golang isn't well-supported

Neither is Scala haha

[rapid sparrow]

Ergh. I am fan of object oriented programming and golang despite its weird syntax imitates through compositions all OOP stuff more or less

[bleak iron]

I think FP gets a bad rep because a lot of FP devs like to write cryptic stuff

good news is scala supports FP and OOP, but again i'd be the only one with FP experience

and picking a language for spark support seems yuck imo

[rapid sparrow]

Scala i haven't seen among adopted languages for backend at all. (No infra integrations)

I think it is choice only between Java, C# .net Core and Golang today
Among backend static typed languages with ecosystem supports to databases/monitoring/logging systems

[bleak iron]

Right

The ideal would be Golang + AWS + some golang streaming processing library

[rapid sparrow]

https://github.com/avelino/awesome-go
Check if what u desire exists

[bleak iron]

this is a good list

thank you

[rapid sparrow]

Stream processing section is present there 😁

U a welcome

[south forum]

Hey guys,
Has anyone of you ever worked with Dryscrape?
(I get an error after installing and trying to run the "tutorial" scraper.
"in init self.driver = driver or DefaultDriver()"
"FileNotFoundError: [WinError 2]")

Or else what would be the fastest way to scrape a js website. ( selenium is way to slow)
Thanks

[loud bear]

Are single file script projects deprecated?

Seems unnecessary to me to make a folder and put the same code in __init__.py instrad

[dusty swallow]

deprecated by who

who is the authority in project setups?

[loud bear]

I vaguely remember reading something like that in setuptools docs

[dusty swallow]

oh you mean if you want to publish a package?

[loud bear]

Yes

If its just a single file can I just avoid the src or the code folder?

And put the script at project root?

[dusty swallow]

I haven't ever tried to create a single file package so I wouldn't know sorry

I don't see why you couldn't though

[hearty torrent]

hey, which category in this help area does selenium go into in this topical chat/help area

[rapid sparrow]

!rule 5 partially kidding, but mostly not

[rancid schooner]

Rules

5. Do not provide or request help on projects that may break laws, breach terms of services, or are malicious or inappropriate.

[rapid sparrow]

Ensure u use it for appropriate.. things

[hearty torrent]

oh no i am, i am trying to use it tu automate logging into some websites when i wake up

[south forum]

hey guys

what could be the reason for not finding an element By Xpath.
I block till parent element found and have copied the xpath from website.
Still I get
"Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id='root']/div[6]/div[1]/div/div[4]/div[1]/div[5]/div[3]"}"

[rapid sparrow]

!rule 5

[rancid schooner]

Rules

5. Do not provide or request help on projects that may break laws, breach terms of services, or are malicious or inappropriate.

[rapid sparrow]

Web scraping is often against TOS, unless proven otherwise

[south forum]

ok

[alpine horizon]

The currently activated Python version 3.10.7 is not supported by the project (>=3.70). huh?

3.10 is bigger than 3.7,so it should be supported

[rapid sparrow]

3.10 is bigger than 3.7
but 3.10 is less than 3.70.
you have missspelling in version by requesting non existent 3.70 python version. Zero matters, you can have infinity Minor versions, even 3.111 version

[alpine horizon]

oh fucking hell

[stone kestrel]

the most satisfying feeling is when your code works on the first try

[subtle quarry]

It's nice for sure. But most satisfying is when you struggled and then made it to work 😜

[stone kestrel]

that too yep

[unique sage]

anyone use any of the sugarcrm packages, i can get none to work

[midnight ember]

Hey not sure if DevPi falls under development tools but it is my best guess: I am having some issues with configuration(I think) on the server everything works great with the global index set to:
http://localhost:3141/root/zeuspi/+simple
but on other machines I can't get pip to work. I have tried:

http://172.31.94.115:3141/root/zeuspi/+simple```
but both fail with the error
    ```ERROR: Could not find a version that satisfies the requirement six
    ERROR: No matching distribution found for six```
The package install works fine on the server so I know the upload works fine.

For authentication it is default username of root with no password since the network is isolated. Not sure if this is something wrong I have on my devpi server or the client machine any help would be great if someone has familiarity. I have checked the network path between the machines on port 3141 all looks good there.

This is an isolated private repo so no mirroring of pypi it is clean with only selected packages approved with no backup.

[rapid sparrow]

sounds like working VPN or weird networking firewall potentially 🙂

or also... missing list of available dependencies unable to download

missmatching some global dep also can be

[midnight ember]

Testing is currently in an AWS VPC, traceroute between them is direct with no hops.

I would think if it was missing deps the install on the server would fail but it is possible. What is a good really base level package with no deps if you know of one by chance.

[rapid sparrow]

AWS has inbulit messy firewall, that can prevent incoming traffic too 🙂

[midnight ember]

I will check flow logs but would assume the firewall would not come into play since it is all inter-VPC traffic with no outbound

[rapid sparrow]

six is external stored third party lib

[midnight ember]

I downloaded six and used devpi upload to add it to the devpi index. I had pypi enabled as a backup to save trouble of getting all the deps for it but your comment on deps makes me think that may be the root cause. I am building a empty python package with no deps to see if that works

[modest storm]

hey i need help on this code i have been working on, im trying to make the output text that displays on screen a variable that is saved to a list and then is showed as a history in the output any help would be nice

pls dm me if needed fuller description or code to help you understand or debugg

[rapid sparrow]

see #❓｜how-to-get-help and https://pythondiscord.com/pages/resources/guides/asking-good-questions/ plus you can read dontasktoask.com
provide detailed information about your requests, otherwise nobody will help you.
write: What you think you were trying to achieve, then write reason why you was trying to achieve, code examples what you already tried to do (text only, screenshots are really bad). And everything should be in public channel or post in #1035199133436354600

[marble ledge]

Anyone finding Pytype super slow in CI builds and have a good approach to speeding it up outside of their suggestions? https://google.github.io/pytype/faq.html#why-is-pytype-taking-so-long
Trying to see if it's cache-able between builds (using CircleCI): https://circleci.com/docs/caching/#saving-cache
but haven't gotten this to work

[soft anchor]

git question: is there a way to sort of rebase a branch but also keep where it used to be?

[soft anchor]

[solved] git cherry-pick main..feat-a

* d9fb18a (HEAD -> feat-b) second commit on feat-b
* 88cef9b first commit on feat-b
| * 0c2fdfe (feat-a) second commit on feat-a
| * e4bd4e1 first commit on feat-a
|/  
* c830d66 (main) second commit on main
* 299d773 first commit on main

* 06fb063 (HEAD -> feat-b) second commit on feat-a
* 7818dd5 first commit on feat-a
* d9fb18a second commit on feat-b
* 88cef9b first commit on feat-b
| * 0c2fdfe (feat-a) second commit on feat-a
| * e4bd4e1 first commit on feat-a
|/  
* c830d66 (main) second commit on main
* 299d773 first commit on main

[hardy jay]

Can I use Selenium to automate filling out a form on a webpage?

[light seal]

yes

[lean bison]

selenium can literally do anything

[novel sinew]

That being said, if there's no Javascript involved but it's a simple form, Selenium is probably an overkill

[hardy jay]

What is a simpler way to do it?

@light seal @lean bison @novel sinew Is it possible to have it run automatically? Like on a virtual machine or something? It basically just needs to go to a website, and enter in information to download a report

[real hedge]

Does python have a bulitin that will report method usages across python application?

I'm curious how often devs are using the std lib over external libs

[tawny temple]

There is sys.settrace but it's fairly low-level

[empty rover]

The profile and cProfile modules will do this for you. Profiling comes with caveats, but if you have a specific enough question, it might help you answer it.

[rapid sparrow]

Python unittest mock library provides it as method to use. Pretty much useful at least during unit testing sometimes to ensure correct amount of calling

[rough nova]

Any one can support in python asn1tools, we used it for decoding binary streams but it takes too much time for decoding which affect on performance

[steel vapor]

Hi all - got a Python script that is using multiprocessing and taking forever on my laptop as I need more cores. Anyone know the cheapest service that I can run Python scripts online? Preferably one that runs based on CPU usage rather than hours if that's a thing?

[short peak]

major infrastructure providers offer CPU optimized instances (I only know DigitalOcean and AWS)

[steel vapor]

Thanks, checking out DO now.

[rapid sparrow]

CPU usage instead of hours is a thing. Biggest providers have like AWS (and GCP/Azure), it is called AWS Lambda in AWS
but AWS is expensive.

Cheapest amount of Core per price is given at Hetzner (but its hourly)

[iron falcon]

Raw cores / CPUs are not always the problem... it could be other things. Did you profile your program to see where all time is spent?

[short peak]

DigitalOcean has its own version for serverless

though I'm not sure if serverless approach can provide flexibility to set the amount of cores

and pricing here - https://docs.digitalocean.com/products/app-platform/details/pricing/

I found AWS documentation better than DO but I found DO pricing more predictable than AWS

[steel vapor]

Thank you

[steel vapor]

Yes, it's I/O. Just tried moving my sqlite3 db from disk to /dev/shm and it's absolutely flying!

I've gone from 400 items per second to 12,000 😮

[blazing gyro]

Hey guys

Anyone is open to work on a custom module for ansible?

[rapid sparrow]

And the music plays in my head with your words. https://youtu.be/kBqGhUMovtY

[novel sinew]

https://stackoverflow.com/questions/42884087/how-and-when-to-use-dev-shm-for-efficiency

[blazing gyro]

lol

[shadow crow]

What's a good way of making a pros-and-cons list in MkDocs Material? Is there something built-in?

[heavy knot]

Can someone help me with my requests?

import requests

url = "https://panel.socialheaven.xyz/api/v1/"
key = {'key':'7HUq1G5E6tVoIE5ikevj'}
action = {'action':'add'}


x = requests.post(url, json = key, add, )

print(x.text)

[mystic void]

I hope for you it's not your real key

[astral apex]

🚨 You posted your token in your message. This means that your token has been compromised. Please change your token immediately

[short peak]

hey. I replied this question in telegram 😄

[finite fulcrum]

For what reason would docker compose build an image after it successfully pulled it? I defined build in the config for the service to get rid of having a separate script for it, but now in CI where it has a separate build step that publishes it with a custom tag and the run/up has that tag in the config override, compose both fully pulls it and then builds

[tawny temple]

Do you have a pull_policy set? https://github.com/compose-spec/compose-spec/blob/master/spec.md#pull_policy

If pull_policy and build both presents, Compose implementations SHOULD build the image by default. Compose implementations MAY override this behavior in the toolchain.
What happens when you use the --no-build flag in the Docker Compose command?

[glossy valley]

Anyone here have a recommendation for semi-budget keyboards for development and gaming?

[slim maple]

Depends on the budget 😅

and also your preferences

[opaque thicket]

what is the best way to extract a list of URLs on a website, specifically on a blog or personal site?

for example:

• takes in a list of blog top-level-directory URLs, ie: ["http://paulgraham.com/index.html", "https://www.vitalik.ca/", "https://medium.com/@FEhrsam", "https://openai.com/blog/"]

• extracts a list of URLs for each blog/article

[indigo zenith]

https://github.com/openfaas/faas

[indigo zenith]

There are all kinds of existing tools that might fit your requirements, here's a really simple one: https://hackertarget.com/extract-links/

If you want something simple but customizable, a Python script with requests and re might be sufficient

[finite fulcrum]

The config doesn't have pull_policy set, and unfortunately it doesn't look like there's an option for no-build in docker compose run. I trieddoing exactly waht CI is doing with the remote repository but it worked fine locally and immediately started after the pull instead of also building

[tawny temple]

I don't have any other ideas unfortunately

[flint ruin]

Hello guys, i hope this is the right channel

I would like to connect to my jupyter lab remotely from a server like ovh or something else

how to do it ?

[gentle solstice]

You should use urllib.parse.urljoin

[thorn portal]

google for jupyter notebook/lab port fowarding, it's one of the ways you can access jupyter notebook/lab remotely.
e.g. https://thedatafrog.com/en/articles/remote-jupyter-notebooks/

let me know if you are still stuck if the above didn't help.

[devout lodge]

Anyone aware of any PyPi Cluster setups ?
We are looking to setup a HA Local PyPi Index
or a solution to upload package to multiple servers is fine

[fluid token]

Should I let my Docker related files (Dockerfile, .dockerignore) to be committed to my private GitHub repository? I think I need them since I have a GitHub "Action" set up, that builds a Docker image and uploads the image to ghcr.io on push. But I am not sure if this is the ideal way of doing it.

[rapid sparrow]

it is surely ideal to commit those files. they are meant for this.

you are not supposed to commit only resulting docker image 😆

(they can be saved locally as docker save image)

docker images are supposed to be saved only to docker registries (docker hub, or AWS docker registry and etc)

[fluid token]

Ahh I see lol, thanks so much!

One more question, I have been getting error at the "Build and Push Docker Image" step of my workflow file

It looks like it's failing to build the docker image, I'm not sure why that could be.
All of the steps above it seem to be functional.

name: Build and Push Docker Image

on:
  push:
    branches:
      - master

jobs:
  build-and-push:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Code
        uses: actions/checkout@v2

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Set Owner Name to Lower Case
        run: |
          echo "OWNER_LC=${OWNER,,}" >> ${GITHUB_ENV}
        env:
            OWNER: ${{ github.repository_owner }}
      
      - name: Build and Push Docker Image
        uses: docker/build-push-action@v3
        with:
          context: .
          push: true
          tags: ghcr.io/${OWNER_LC}:latest

This is what my workflow file looks like ^^^

[rapid sparrow]

error clearly states, invalid tag. ${OWNER_LC} is not replaced to any meaningful tag

github has different syntax for how variables are written

{{ I.dont.remember.what }}

if you would have used run: to write command, u could have used shell way to insert variable, due to it being shell

but u are at the moment in pure Github Actions syntax, which has different mechanism

ghcr.io/${{ github.repository_owner }}:latest will work

[fluid token]

Ow that makes sense. I'll try ${{ OWNER_LC }} and see if it makes a difference, ty!
I didn't research how actions work enough and here's the effects of it heh, sorry for the trouble.

[rapid sparrow]

nah, it will not

variables in jobs are available only as
predefined variables like example above
or it has certain mechanism like ${{ steps.url.outputs.url }} to input variables. i did not get too much how they are created in previous steps of jobs
well, or passing secrets like ${{ secrets.bla}} is possible too

[fluid token]

I've tried that actually, it failed because (apparently) I can't have uppercase characters in the tag, and my username happens to be MasterPrasanjit 😓

I've verified the OWNER_LC variable returns the lowercase version of it though

[rapid sparrow]

well, read the official docs of GH about variables 🙂 github has its quirks

[fluid token]

Ah, it really does, ty for the help! ^^

I should've learned how the thing works at all before diving into it :hidethepain:

[gleaming forge]

So I may or may not have accidentally disable SSH on a digitalocean droplet I'm hosting a website on. How on earth do I get back in to enable SSH again 👀

[gleaming forge]

Nevermind all good now

There's recovery options provided by digitalocean

[slim maple]

devoops moment

[gleaming forge]

Question for those familiar with uvicorn, how do I specify a file in a different directory when doing something like bash uvicorn main:app --reload What if I wanted something more like ```bash
uvicorn "directory/subdir/main":app --reload

Just because I want my working directory in a specific place

[rapid sparrow]

linux compatible command. Can be probably executed in git bash on Windows

sh -c 'cd directory/subdir && uvicorn main:app --reload'

[rapid sparrow]

uvicorn directory.subsir.main:app --reload is also an option, which will work only if you have this specific... python packaging situation, where directory and subsir are packages too

[gleaming forge]

ah true didn't consider that, yeah that should work just fine

cheers

[solemn osprey]

How do you go about health checking scratch-based docker containers?

[rapid sparrow]

technically should not be a problem.
Add second executable as health checker

[solemn osprey]

Will probably do that, thanks

[gleaming forge]

Quick question. So I've got a digitalocean droplet running a webserver, having a little trouble getting my domain name to correctly serve the webpage though.
So I've got a basic config file for nginx that looks like this server{ server_name <server_ip_here>; location / { include proxy_params; proxy_pass http://127.0.0.1:8000; } }And I'm just running a little fastapi app using gunicorn and uvicorn like so```bash
gunicorn -w 4 -k uvicorn.workers.UvicornWorker main:app

On the digitalocean side of things, I've got my domain setup and it seems to be fine as seen in the image. However, when I try to access the site via the domain name instead of just the IP I get the default "Welcome to nginx!" page instead of the webpage I expect, using the IP directly works fine.
I assume this is just some nginx configuration thing I've skipped over since I'm reaching the default nginx page

[rapid sparrow]

server_name <server_ip_here>; server name is an instruction limiting from which adresses to accept requests
and it must contain same address as user inputs in a browser url. If user inserts ip address -> then ip address
if user inserts domain path -> them domain path

if u will exclude this server_name instruction, nginx will accept for this web server requests from any browser url address.

[gleaming forge]

Ah I see, so I can just get rid of that altogether and it'll accept basically any option I'd throw at it?

[rapid sparrow]

yes. it is useful instruction for using one nginx to serve multiple web sites from one server

u can make it listening to different domains at the same time, and redirecting to different server configs

[gleaming forge]

Yep gotcha

Thank you

Hmmm, when I remove the server_name instruction it's just serving the default "Welcome to nginx!" page. For both just manually entering the IP and when trying to use the domain

[gleaming forge]

Yeah weirdly even if I try add my domain as well as the IP in server_name it then doesn't work with either, whereas just the IP would at least work when entering the IP in a browser

[quartz grove]

apologies if this is the wrong channel but, how can i prevent autodoc/docutils from adding the <em> in sphinx

it does not do so if i manually use Optional[:class:`int`] but that is unnecessary when it can read from the signature

[gleaming forge]

Oop fixed it, turns out the default config was shadowing my one

[last hearth]

is it possible to disable a plugged i usb storage device with python?

[wraith sparrow]

I am using PyCharm as it seemed to have the most features, is that accurate or are there another IDE with more features?

I also liked it as it works with github copilot

[slim maple]

Depends on what features you need really

If you just need an editor vscode would do

[wraith sparrow]

I am new so just want the most features to see which features I like, what does editor mean compared to IDE?

or are they the same thing

[short peak]

https://realpython.com/lessons/ides-vs-code-editors/

[true vapor]

There is realistically very little difference between an editor and an IDE at this point - VSCode has nearly as many features as Pycharm, Vim with Treesitter and LSP goes almost as far

[short peak]

true but I think IDEs setup is usually "click, click, click...". Text Editor setup is mostly text
Also IDEs have magic built-in, text editors usually lean on extensions/plugins

[true vapor]

VSCode works far better out of the box than PyCharm - I just think it's an almost entirely unhelpful distinction at this point

[short peak]

depends on needs. If you need debugging on remote containers you need extensions. And don't get me wrong, I prefer text editors but I tried PyCharm for 2 years a while ago

[true vapor]

depends on needs. If you need debugging on remote containers you need extensions

Both of those things are easier in VSCode than Pycharm

devcontainers, and debugging that is

[short peak]

I didn't say harder or easier. I said click based vs. text based as a difference

[alpine horizon]

whats the best AWS product for launching a python script every once in a while?
basically smth like crontab, and its gonna run maybe once a day for a few minutes

and there is no need to store any state permamently (other than the script it self...)

[short peak]

you could try AWS Lambda + Eventbridge

[rapid sparrow]

AWS Lambda function
Launched by AWS cloudwath event rule tied to AWS cloudwath event target
(Those are names of Terraform resources participating, first run can be done by AWS lambda invokation)

[alpine horizon]

so the Lambda is like a "one time use" EC2?

[rapid sparrow]

Lambda is reusable serverless scalable function.
It can be triggered in different ways

AWS Lambda function invokation is one time run

Event bridge targeting lambda is Cron job

[alpine horizon]

oh I see, thanks for exaplaining that

do you have any idea how much does it cost if im using say 30 minutes of CPU time a month for a Python script?

[rapid sparrow]

AWS resource calculator can be opened. Official thing at their site for calculations like this

[alpine horizon]

lemme try that, thanks bro

[stable sinew]

.

@heavy knot what do you think?What might be the problem with that stuff.

[heavy knot]

I have no clue. I'd repost your question here and maybe open a post in the help channel.

also have you tried to look up the warning you are getting?

[stable sinew]

yup it is saying that i have no github

[heavy knot]

but you just made a new repo

[stable sinew]

which isnt true because i set the mail id and also the repository from where i cloned the stuff.

[heavy knot]

hmm have you set the uername and email config in git?

[stable sinew]

yup i already configured it

[heavy knot]

Well repost and hope for the best. I might also consider trying a linux / ubuntu server.

[rugged hare]

can you show the problem again?

[rugged hare]

hmm, sorry, i don't know what a UI helper is for.

[stable sinew]

hmm.

this shitty stuff

[heavy knot]

no one can do anything w it

iused a temp token on my API

[heavy knot]

someone know what i must delete to log out from steam?

[knotty osprey]

I've recently added a dependency from a private git repo and I'm not sure how to get it working with docker.

Initially, I just used the ssh URL of the repo locally in my pyproject.tom since I have a key, but I'm running into an issue where Docker doesn't have a key and can't access the repo.

What's the most secure way to clone the dependency with either the SSH or HTTPS clone link? The most obvious solution to me is to just put an access token in a secret environment variable and pull that in with the requirements file we export and use for docker deploys, but we manage dependencies with Poetry and a pyproject.toml file and I'm not sure how to use tokens from environment variables there. Poetry's docs do mention using environment variables for private PyPi repo authentication, but we don't have a repo, we're just pulling directly from the git repo.

[last lion]

i got a question, i have a github repository, created it using 'share project on github in pycharm, that repository is long gone and i dont have access to it anymore how can i commit to that repository again, because i used to commit using commit button in pycharm

[rapid sparrow]

Get access if u don't have access
Or commit to your own forked repo and offer PRs from it

[charred sigil]

Is there a way to populate the input field of a terminal with some python library? Maybe subproces? By input field I mean the where you type your commands. I want to execute a python script and have it populate the terminal with a command, BUT does not run it

[rapid sparrow]

https://pyautogui.readthedocs.io/en/latest/

[rapid sparrow]

pyautogui as direct answer
but https://pypi.org/project/pyexpect/ is used to automate CLI tools with input where values are requested

[alpine horizon]

after I do poetry add smth is the package ready to use? do I need to do update or build poetry again?

[rapid sparrow]

i as far as i remember it is immediately gets installed.

[ancient vapor]

Hi guys

assume you subscribed for a product you can use daily for productivity purposes, do you think it's better if you receive a tip email once per day for a month or get all tips in a single email instead ?

[tribal path]

how would you commit a file without saving its contents to git? I have a repo with a few files that have private info to upload to github, but they need to be there to run the program

[rapid sparrow]

Github Secrets as a choice (most simple solution compatible with CI-CD deployment workflows)
or using AWS Secret manager and supplying to application only secret to secret manager (compatible with bigger range of fantasy regarding infrastructure deployments)

for home projects we can use more silly solutions like saving to encrypted git in public import parts 😆 git-crypt can do it. (less bothersome solution having everything in git xD not very secure though kind off though)

as advanced infra solution people also go to stuff like Vault (at least i hear it is kubernetes choice). Some kind of programmatic access secret storage... AWS Secret manager technically should be doing the same. Vault is just open source may be.

[lethal mantle]

vault is also not tied to aws

[alpine horizon]

sounds reasonable, I think the issue is that Pycharm isnt recognizing the poetry file and therefore its using the global enviroment

[flat lotus]

I've seen many times that PDM is faster than poetry, according to benchmarks. Why? In my experience, this is usually related to the other tool just having less features.

But if someone here worked on either poetry or PDM: do you maybe know specifically why PDM is faster?

Note that I am heavily biased towards poetry because I've used it everywhere for the past 2-3 years so there is no need to explain to me that speed is not always the most important thing :)

[alpine horizon]

guys how can I set a windows enviroment variables from CMD ? (globally)

[alpine horizon]

I rather just use a good old requirements.txt but thats what my team wants me to use 🤷‍♂️ lol

[rapid sparrow]

%VAR%

[rapid sparrow]

U could find tempting to just work in Git Bash CLI. It will be same Linux experience. Less confusion in environments between desktop and servers

echo $VAR then

[short peak]

has anyone replaced Docker with podman? can you share your experience?

[alpine horizon]

thanks darkwind

[golden kestrel]

Guys how i can host my website in django on localhodst allowed only access on local?

I can use xamapp for to do this?

[rapid sparrow]

U a welcome. Consider also using https://taskfile.dev/

very cool makefiles in yaml. It works on inbuilt bash, so it will be same bash for you in windows too

[alpine horizon]

interesting, you should make a list with all these tools and utilities for devops

[vapid fox]

I mean, if they supply you with a VM, then everything's fine. What about free-tier friendliness?
@mystic void I'm mainly charmed by Azure's 200$ credit

[mystic void]

The main differences are in the philosophies of the services. As a rule of thumb, Azure is more about solutions, GCP about developers, and AWS as building your own legos

Beyond that, a VM is a VM

if you got the student credits, then explore and play with it. It won't hurt 🙂