tools-and-devops

[rapid sparrow]

Docker Desktop creates a second VM even if it is running inside Linux. Don't use Docker Desktop, use Docker CLI, it access current Linux machine directly

you have Ubuntu VM 1 running in Gnome Boxes

and Ubuntu VM 2 running for Docker Desktop inside Ubuntu VM 1 then

if u use Docker CLI client, docker ps, on Ubuntu VM1, it will access Docker Daemon in Ubuntu VM1 (or may be because of installed Docker Desktop, it can be accessing Ubuntu VM2)

[carmine scroll]

There is only 1 Ubuntu VM

[rapid sparrow]

if u use Docker Desktop on Ubuntu VM1, it access docker daemon in Ubuntu VM2

[rapid sparrow]

Docker Desktop automatically creates another Ubuntu/VM for its working

[carmine scroll]

Ohk understood

[rapid sparrow]

simple solution would be reinstalling VM, with not installing Docker Desktop

install CLI Docker only version for linux ^_^ https://docs.docker.com/engine/install/ubuntu/

then your python script and docker CLi will access same docker daemons

[rapid sparrow]

👆

[carmine scroll]

Yes, works now. Deleted Docker Desktop and restarted VM. Python SDK and terminal are consistent now. Thanks!

[steel sundial]

This is a shot in the dark, but

Anyone can point me to some more indepth Sphinx/rst info, in regards to extensions and programmatically handling nodes and so?

Or

somewhere sphinx question can be asked and discussed?

(Please do not point me to the shinx or rst docs, I have already almost memorized them and sadly they leave a lot out)

[vague silo]

this is probably a good channel to ask questions about sphinx

for extensions I think the best starting point would be looking at the builtin ones

[steel sundial]

My current problem is mostly, that sphinx seems to do its magic in 2-4 places, depending on the node or directive/role. I have no idea where to find out which node gets modified, by the builder, which by the translator, which by the transformer and which by all 3 or none.

I tried to create nodes and children inside a directive, but it seems most nodes do not actually have their extra_attributes tuple filled out it seems. (See image node for example) and I cant find a description of what attributes each special node needs to have set.

So generally i was hoping that someone either already has this mapped out or there is some special docs I have missed

Or would be the best way to create nodes to instantiate the connected role/directive?

[gentle solstice]

I managed to extend dokku-scheduler-kubernetes to add support for secret, configMap, and hostPath volumes.

https://paste.pythondiscord.com/inakofariq

dokku k8s-ext:mount hostPath /host/path /container/path

[heavy knot]

https://pastebin.com/HARDfNbN

recreation of auto-py-to-exe, il add more later, im bored asf

[willow pagoda]

mostly curious, is it possible to upgrade a pyenv virtual environment inplace? i have some expensive compiled dependencies and didnt notice any obvious way to bump my patch version, but for now ill use my lock file to reinstall cached deps

[exotic crown]

That moment when github codespaces is hilariously overpriced so you just make your own web based vscode on-demand service

[short peak]

and desktop version has profile syncing. I also don't get it 🤷‍♂️

[exotic crown]

get what?

[night narwhal]

ayo is there a good and WORKING alternative to pyinstaller

i am furious because pyinstaller seems to hate importing requests lib

no matter what i do it won't work with it

[cunning niche]

https://www.saashub.com/pyinstaller-alternatives maybe has one you can use?

[rapid sparrow]

Drastic measures: writing in easy compilable language

[night narwhal]

i hate c family ok ;(

[rapid sparrow]

There are alternatives of course though. Cython is compilable too

[rapid sparrow]

Golang = not C, easy to learn, easy to setup, easy to compile, cross platform compilable from any OS to any OS and CPU architecture

[vague silo]

so, just like C?

[brazen forge]

Si

[eager rose]

arguably C is all those things, except "not C"

[peak wagon]

Just stuff it into a flatpak

You can also give briefcase a try

https://briefcase.readthedocs.io/en/stable/

[rapid sparrow]

you understimate the power of how easy to compile Golang in crossplatform way

      - GOOS=linux GOARCH=amd64 go build -v -o dist/autogit-linux-amd64 main.go
      - GOOS=linux GOARCH=arm64 go build -v -o dist/autogit-linux-arm64 main.go
      - GOOS=linux GOARCH=386 go build -v -o dist/autogit-linux-386 main.go
      - GOOS=linux GOARCH=arm go build -v -o dist/autogit-linux-arm main.go

      - GOOS=windows GOARCH=amd64 go build -v -o dist/autogit-windows-amd64.exe main.go
      - GOOS=windows GOARCH=386 go build -v -o dist/autogit-windows-386.exe main.go
      - GOOS=windows GOARCH=arm64 go build -v -o dist/autogit-windows-arm64.exe main.go
      - GOOS=windows GOARCH=arm go build -v -o dist/autogit-windows-arm.exe main.go
      
      - GOOS=darwin GOARCH=amd64 go build -v -o dist/autogit-macos-amd64 main.go
      - GOOS=darwin GOARCH=arm64 go build -v -o dist/autogit-macos-arm64 main.go

compiling for all OSes (macOS included) and CPU architecture from Linux, without needing to install any dependencies

https://media.tenor.com/rDu4ehHMp5sAAAAC/you-underestimate-my-power-aniken-skywaker.gif

[subtle holly]

Hi everyone,
I’m looking for a Python package for maps that can detect the coordinates of the location where the user clicks. I’ve tried Folium, but I think I need something else. What packages do you recommend? Thanks!

[rapid sparrow]

pyautogui

[twin hull]

Hello, I'm trying to swap the docker SDK with aiodocker for async support and have been encountering issues,
old code

manim_stderr = dockerclient.containers.run(
                image="manimcommunity/manim:stable",
                volumes={tmpdirname: {"bind": "/manim/", "mode": "rw"}},
                command=f"timeout 120 manim -qm --disable_caching --progress_bar=none -o scriptoutput {cli_flags} /manim/script.py",
                user=os.getuid(),
                stderr=True,
                stdout=False,
                remove=True,
            )

new code:

container = await dockerclient.containers.run(
                config={
                'Image': "manimcommunity/manim:stable",
                'Cmd': ['timeout', '120', "manim", "-ql", "--disable_caching", "--progress_bar=none", "-o", "scriptoutput", f"{cli_flags}", "/manim/script.py"],
                "User": "root",
                "AttachStderr": True,
                "AttachStdout": True,
                "HostConfig": {
                "Binds": [f"{tmpdirname}:/manim/:rw"],
                #"AutoRemove": True,
                }})
            manim_stderr = await container.log(stderr=True)
            print("manim_stderr: ", manim_stderr)
            print("stdout: ", await container.log(stdout=True))

aside from the -qm -> -ql change in the docker invocation and print for debug purpose this should be identical, however the later fails, due to

2023-04-09 17:55:16 Traceback (most recent call last):
2023-04-09 17:55:16   File "/usr/local/bin/manim", line 5, in <module>
2023-04-09 17:55:16     from manim.__main__ import main
2023-04-09 17:55:16   File "/usr/local/lib/python3.8/site-packages/manim/__init__.py", line 106, in <module>
2023-04-09 17:55:16     from .utils.ipython_magic import ManimMagic
2023-04-09 17:55:16   File "/usr/local/lib/python3.8/site-packages/manim/utils/ipython_magic.py", line 13, in <module>
2023-04-09 17:55:16     from manim.__main__ import main
2023-04-09 17:55:16   File "/usr/local/lib/python3.8/site-packages/manim/__main__.py", line 9, in <module>
2023-04-09 17:55:16     from .cli.cfg.group import cfg
2023-04-09 17:55:16   File "/usr/local/lib/python3.8/site-packages/manim/cli/cfg/group.py", line 261, in <module>
2023-04-09 17:55:16     @click.option("-d", "--directory", default=Path.cwd())
2023-04-09 17:55:16   File "/usr/local/lib/python3.8/pathlib.py", line 1096, in cwd
2023-04-09 17:55:16     return cls(os.getcwd())
2023-04-09 17:55:16 FileNotFoundError: [Errno 2] No such file or directory

which seem to indicate the binding is failing.
This traceback comes from the docker App, weirdly the debug stdout/stderr in the code return as empty, which is one more indication that something fischy is going on

[ionic fox]

hey!1

[tame bolt]

For an on prem team that absolutely refuses cloud services, how can I let non technical users run my code or jupyter notebook? I’d like to avoid the exe route.

I have my local windows machine, our org has a GitHub intranet, and I can spin up a windows server via unity VM.

If the answer is docker, does anyone have a good resource that is specific to windows? 😅 I feel like I’ve tried to understand docker 100 times but get overwhelmed by the quantity of use cases

[thorny shell]

I'd think you could host it on some server your team controls

[rapid sparrow]

could you clarify, non technical users use which OS?

and I can spin up a windows server via unity VM.
you wish to provide network access to your jypiter for non tech users?

[tame bolt]

They use windows

[rapid sparrow]

This resource teaches it in structured and friendly way

except it is not having Windows part though. Ugh, can u just share it on Linux server ^_^ it will be way easier than dealing with Windows servers

and locally you can check you configured it correctly in linux VM / or WSL2

[tame bolt]

I was just listing some of the tools I do have, as it’s not many—

Really, I just want to understand how to be less reliant on my local machine and allow users to run a script that generates xyz

[rapid sparrow]

do u intend it for demonstrational purposes to incompany people, and wishing end users just accessing your jypiter page?

[tame bolt]

We use webex spaces which has webhook and websocket

I was thinking about some tool with persistent access so a user can run a slash command and it executes my script.

It’s data science team, so they would request a report > script executes > csv results exported

[rapid sparrow]

naming possible option
A:

1. raised LInux VM
2. where you installed your python script
3. people connect over SSH, execute, grab result if they are able / using SCP to download result
   P.S. can be augmented for easier deployment to use Docker, this is not changing anything except your experience to deploy it.
   P.P.S. you can provide users command that does over ssh script execution and download after that in single command

modifying situation
B:

1. raised Linux VM
2. Script can be launched by primitive simple web GUI
3. CSV result is autodownloaded them after that

modifying situation:
C:
No servers
you build docker image and upload to some docker registry
users run docker image to execute script and receive csv results locally
(probably too high level choice for users, since they probably will not be able to install docker locally)

D:
option four, actually compiling your python script for windows

Option A is what u seek, but it requires you only at least to know Linux

probably same stuff can be made with Windows Server though

[tame bolt]

I’ll see if these unity boxes will let me spin up a Linux server, that seems to be the biggest limitation

Thanks for the info! I’m going to look into it (and finally learn docker lol)

[rapid sparrow]

learning docker is not necessary for any of those cases. it will make just easier for you room to make deployments of script

(in fact, it will be not needed deployed at all, u will just save resulting docker image to docker registry (and install docker to server for users to use. Users automatically will download necessary application version during usage of your script automatically (if they have available docker installed))

[tame bolt]

Re: B — could i make a flask app or use fastapi and host it on a windows vm?

Windows server*

[rapid sparrow]

Sure. you can. I'll be honest though, that using Windows for deployment is very ancient archaic way, that is used by less than 1% of web developers today.
I am honestly not even knowing how to have it done (and if it is compatible with docker at all, i thought docker in windows for local dev env only)

All instructions and services aim to provide web dev deployment to linux

[tame bolt]

😅 yeah, I get that. If the org won’t let me get ec2 or even s3, the least they can do is add Linux to the virtualization

Okay one last scenario for you — can github act as a host for a script where a mechanism can be created for a non technical user to trigger the code from a repo?

[rapid sparrow]

Yes. it is already having this functionality, allowing to run script on demand. It can even yield artifact results (like CSV file). It is not meant to yield huge files though (you would wish to yield big files to some other storage ^_^)

button "Run workflow" on the right is how to launch your workflows

name: build

on:
  workflow_dispatch:

manually triggerable scripts have workflow_dispatch button

[tame bolt]

ah, cool I’ll read more about this too. I imagine I’d just have to figure out authentication for the mssql odbc conn

[rapid sparrow]

pretty simple, just inject Github Actions secret with credentials, while having exposed DB for connection
or you can even just run self hosted runner for this, which has DB accessed at private network locally available to runner
first option is simpler

[tame bolt]

that makes sense, thank you I really appreciate the details!

[rapid sparrow]

this is in fact, using Cloud. But perhaps they will not see it as a cloud ^_^

[tame bolt]

Yeah, i agree, I’m going to see what needs to be done to allow me to spin up linux

[rapid sparrow]

Github Actions aren't probably having option to inject parameters though for script run if it is necessary for you. It can be run just as it is programmed
if it is fine to you

[tame bolt]

They just don’t want to pay for aws or similar 🫠

[rapid sparrow]

Well, github actions is free (to some certain limit) (Github actions self hosted runner has no limitations, and there is no limit for public repostiries, but using public runners for private repositories has certain montly quota^_^

[tame bolt]

Free enough hah

[cerulean hollow]

Is there a book like this for Ansible and/or Terraform?

[exotic crown]

terraform is love and life

[naive pecan]

Hey all! Will schedule.every().monday.do(job) run every Monday at 00:00? The example at https://schedule.readthedocs.io/en/stable/examples.html doesn't have an explanation.

[pastel meadow]

i was guided here to ask

https://dev.harfang3d.com

what are the pros/cons of this .... im in search of a game engine to make fast hi-res graphics for a MCU scope

[rough hearth]

anyone know why my azure cluster is frozen?

[thorny shell]

IME the Azure UI is very flaky. Try reloading the page

[rough hearth]

yeah it started very slowly

i get this error now

[minor current]

Haven't found any good tool to limit myself from pushing huge diffs, so created this: https://github.com/skonik/git-limiter

Can be used with pre-commit hook(or even CI/CD, but I didn't test it in pipelines) .
Not sure that anybody interested, but just wanted to share with people and get some feedback

[rapid sparrow]

can be nice to add releases built with pyinstaller ^_^ built at least to amd64 linux
without being compiled to binary we can't say about CI friendliness

[rapid sparrow]

https://github.com/darklab8/darklab_autogit i made tool from similar category btw
Except my tool enforces good commit messages + generating changelogs (from parsed commit history) + telling next semantic version

[minor current]

Thanks. Really helpful information. Going to look into it

[minor current]

Looks interesting. Automatic generation of changelog might save a lot of time. One more star

[rapid sparrow]

also as an interesting note:
I have my tool made compiled to all different OSes and CPU architectures due to easy cross platform compiling for golang https://github.com/darklab8/darklab_autogit/releases/tag/v1.0.0
and even installed git is not requried for its functionality, because golang has go-git library https://git-scm.com/book/en/v2/Appendix-B%3A-Embedding-Git-in-your-Applications-go-git that embedded git into my tool ^_^
so it has zero dependencies on anything in OS

went with dependency free friendliness ^_^

kind of overkill, but ensures there will be no trouble with not discovered git in some OSes. git has different messed up forms of installations in windows at least, and can be seen in some cases only in git bash

[minor current]

Wow. This is awesome. Zero dependencies is right thing to do

[toxic silo]

Does anyone have a good bash one liner for determining the latest stable release of python?

I'm looking to integrate that into a script that downloads and compiles the latest stable release.

(does not really need to be a one liner)

[thorny shell]

maybe install pyenv, and ask it for the latest

🙂 [2023-04-11T07:05:13-0700][discord-recipes] % pyenv install -l | grep --extended '^ *[0-9.]+$' | sort --version-sort | tail
  3.10.4
  3.10.5
  3.10.6
  3.10.7
  3.10.8
  3.10.9
  3.10.10
  3.11.0
  3.11.1
  3.11.2

[toxic silo]

that is an interesting route, thanks I will look into that. I am not very familiar with pyenv yet, but hopefully I can script its installation reasonably well.

[thorny shell]

it's a big nasty shell script that downloads, unpacks, configures, builds, and installs whatever version of python you want.

it does a pretty good job.

[toxic silo]

haha yeah that makes sense, seems pretty cool, I don't have to hop versions often but it would be nice if it is good at prepping a new system with the latest, and it seems like it would be easy to keep up to date with new releases

[thorny shell]

well to state the obvious: your distro's package manager will probably be easier.

if you're on some old version of Debian, it could be tricky

but there are apt sources that hold newer versions; you could consume those

[toxic silo]

yeah the issue with the package manager is that they don't typically get updates with latest releases often.
If I could add an apt source that does that would be really great

[thorny shell]

sure, on Ubuntu there's a whole apt repo called "deadsnakes" iirc

[toxic silo]

#!/bin/bash

# Install Pyenv
curl https://pyenv.run | bash

# Add Pyenv to the shell environment
echo 'export PATH="$HOME/.pyenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init --path)"' >> ~/.bashrc
echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc

# Load the updated shell environment
source ~/.bashrc

# Determine the latest stable Python release using Pyenv
PYTHON_VER=$(pyenv install --list | grep -E '^\s*[0-9]+\.[0-9]+\.[0-9]+$' | grep -v '[a-z]' | tail -n1 | tr -d ' ')

# Print the result
echo "The latest stable version of Python is $PYTHON_VER"

This seems to work pretty well 😄
I guess I don't know how quickly pyenv will see new stable releases but it seems likely quickly enough

[thorny shell]

are you sure that gets the latest version? I'd expect you'd need to sort the output of pyenv install --list

huh I guess it doesn't need sorting after all. Dunno why I thought it did.

[stuck cloak]

Hi guys i have a question could the use of coding and programming be of use when trading stocks or cryptocurrencies like create an AI that buy when its at the lowest and sell when its it the highest
Its an idea i got but im new so i can just be all BS

[thorny shell]

nobody can predict when a stock or cryptocurrency is at its lowest.

or, rather: anyone who can is either lucky or doing insider trading

[rapid sparrow]

automated trading bots can be easily created btw, we created them as little projects during university and checked results on history

but you know... most of people just lost demo money anyway ^_^

market is pretty volatile stuff.

idea to use AI is nevertheless an interesting one. The only issue regarding that, that highly likely competirors use AIs as well

So it would be competion of AIs with each other. And Question araises, can u write AI better than your competitors. (And your competitors like Banks with hired teams of professionals ^_^)

[stuck cloak]

Yeah no no way i'll as good as a whole team💀

Thx for the response btw

[heavy knot]

Hi guys, topical question related to namespace packages. Does anyone know a good example of an open source repo where I would be able to look for how to implement namespace packages in a monorepo... Tagging mechanism , how to do versions for example...
Having a look how other people do it is always insightful but it's hard to find

[astral wharf]

Anyone implemented tyk for rate limiting?
I have a key which has unlimited quota limit, but when i am running more than 1500 requests per second, i am getting 403
time="2023-04-10T14:56:35Z" level=error msg="Request failed with status 403. Response body: {\n "error": "Access to this API has been disallowed"\n}" source=console

[astral apex]

I don't necessarily agree with everything they do, but the only monorepo of namespace packages that I know of is the Azure Python SDK
https://github.com/Azure/azure-sdk-for-python/

[cedar nacelle]

Anyone know a good batch job execution library does conditional running? Like you would have pre main post executions and custom conditions that could apply to each of them. say retry or alternate job

[lethal mantle]

what sort of conditions?

if all you need is the successful completion of another job, you could look at systemd

if you need arbitrary conditions, then the only real options are 1) very heavyweight "enterprise" job management systems or 2) a "controller" script that handles it

[mystic void]

like airflow or argo?

[cedar nacelle]

They a frameworks though? Was thinking more of a module that straps onto a script you might have.

It's hard to tell which ones are services and which are standalone. Luigi might be stand alone

[mystic void]

nope

[rapid sparrow]

Sounds like Celery as a possible fit

[cedar nacelle]

Celery is tasks in db?

[rapid sparrow]

horizontallly scalable Message Queue.
As a broker to queue tasks by default it offers RabbitMQ and Redis to use. AWS SQS is an option too. I saw some support to abuse Django ORM at least for results too

Monitoring system named Flower working for RabbitMQ and Redis can be checked to see status of tasks visually

it has certainly features of sending tasks executed in background (by worker) with retries

[lethal mantle]

it sounds more like he wants cron + logic to me

[rapid sparrow]

has Cron/periodic task feature too

[cedar nacelle]

I'm just trying to find a serverless library that can Handel conditional failures.

[rapid sparrow]

AWS SQS is serverless mod to run Celery

[lethal mantle]

i.e. do job A, if failed, retry up to 3 times. if successful, do job B; do job C; no matter what, end with job D.

that sort of shit

[cedar nacelle]

So sone class that can house a bunch of jobs in a task which realizes itself by some run logic

[lethal mantle]

systemd timers are sort of like cron but can have simple conditions

well, dependencies. but if a task fails, the dependency fails

so it's sorta like conditions

[rapid sparrow]

there are complex solutions like Apache Airflow to chain tasks, but it should be certainly overkill hard to learn

[lethal mantle]

the enterprise job management tools are overkill 99% of the time, IMO. typically, they are built for orgs that have hundreds/thousands of tasks over multiple machines.

[rapid sparrow]

Serverless, it is AWS lambdas + AWS Step functions

[cedar nacelle]

Isn't aws a server ?

[rapid sparrow]

AWS is a cloud provider.

[lethal mantle]

AWS is a full data center that does everything you can think of and more. for lots of $$$.

[rapid sparrow]

(we can say it is bunch of servers in the end hehe)

[cedar nacelle]

Server less is me aim to encapsulate a complex task

[rapid sparrow]

Same can be said about anything serverless

[somber glen]

🤔

[lethal mantle]

your cat is scary

[cedar nacelle]

Yeh def

There was a lib called tornado but that resolved job state

But that was kinda difficult to use

Looking for a lot cake I guess 🙂

[lethal mantle]

I really should write a cron replacement that also does simple conditions

and retries

and dependencies

or maybe something that wraps systemd

and provides a simplified/sane UI

I think a lot of smaller orgs would like it

maybe throw in some AI bullshit for marketing

[cedar nacelle]

Yeh because I do basically a parallel job run to avoid cicd frameworks insane DSL picks like groovy and kotlin. Why pick those as a DSL instead of python.

[lethal mantle]

could call it "pron", i.e. python+cron?

[cedar nacelle]

Well my stuff is a builder but it runs outside CI because devs want to build the stack without the builder

[mystic void]

the ones I cited earlier do use python or yaml

[mystic void]

why not? It's made for it

[lethal mantle]

I recall looking at airflow. it seemed overly complex for simple: do A, do B, if success do C else do D type situations

[cedar nacelle]

Too big 900 projects

[mystic void]

that's breakfast for jenkins or other CI/CD platforms

[cedar nacelle]

Depends dumped jenkins because of insanity like Groovy

And all the security patches that broke everything

[mystic void]

so trading one set of hell for your own home made one?

[lethal mantle]

it's not a real patch if something doesn't break 😋

[cedar nacelle]

OK not getting into that one. 🙂but my stuff outperformed jenkins

It's not hard

[mystic void]

jenkins is actually pretty good, albeit a bit old now

[cedar nacelle]

Again if you have done a big project you probably wouldn't recomend jenkins

[mystic void]

But anyway, in 2023, of all things tools and devops one could spend time on, rebuilding my own ci/cd would be the very last thing

[mystic void]

I actually did and would do it again

[cedar nacelle]

Hehe better than me then

[mystic void]

I don't think it's a competition or one has to be better than the other

[cedar nacelle]

I would not tuch that mess again

Anyway devs love my solution because they can run it on any bos no setup needed. I'm just looking for something that can resolve the random stuff they send.

I mean it could just be a json file that is specific to one job I guess and as @lethal mantle said the ci frameworks are overrated.

[tepid light]

Not sure if this is what you look for, but I’m the maintainer of a tool for the Polylith architecture that is for monorepos and heavily use namespace packages: https://github.com/DavidVujic/python-polylith

[cedar nacelle]

It was the very last thing but it actually worked out to be less hassle then setting up an new instance of jenkins with all the plugin madness and i really didnt like the groovy pipeline stuff. I went with team city which is ok but didnt want to build a script in kotlin either. I really liked buildbot because it's python but the ui was not great.

Luckily teamcity allows you to issue state on the stdout and is pretty flexable

And you can publish a html page to it with project state which was perfect. Something jenkins can't do

[mystic void]

nowadays, I would probably stick with gha, gitlab ci or drone ci

[cedar nacelle]

Can they be run on premises?

[mystic void]

not sure what you are referring to, but we were doing something like that 10 years ago

[mystic void]

indeed, they can

[cedar nacelle]

Maybe they are broken now as I said a lot of plugins are broken from security upgrades

At least 3 years ago they were

[mystic void]

no idea, I haven't used it in that long at least. So I will trust you on that

[cedar nacelle]

I'll have to take a look at these other ones you mentioned. Although I'm kinda done switching.

[cedar nacelle]

So let me ask this. The challange with project I delt with was that we had over 700G of linkable binaries that had to be staticly linked each runcycle and that was bigger on release builds. So we couldn't distribute the build over nodes to speed it up. It slowed it down. I didn't see anything to figure out dependencies. I actually made that too which I could then minimize build time.

So has anything you saw able to split up builds automatically by known its dependencies?

[rapid sparrow]

Hmm. At this point makes sense writing cli small script that utilizes multithreading of a programming language for its building
Like some golang script with go routines, or similar stuff in python and etc

And since programming language allows limiting to threading pools with fixed amounts, it leaves room to not over load machine completely

At least if issue is in lack of parallel

[cedar nacelle]

Ya. I did do that. I used threaded pool to match loading in parallel to cores. But the script can't split the build into smaller chunks. So can any Ci that you know do that effectivly

Or perhaps a system to only distribute binaries that changed

[rapid sparrow]

Btw, check https://taskfile.dev . Yaml makefile to chain tasks

It has parallelization too for dependencies

Tool literally made for your usage case @cedar nacelle 🙂

[cedar nacelle]

Haha

Nice I'll check it out👍

[mystic void]

looks like something like https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html?tab=Multi-project+pipeline

[cedar nacelle]

It's going to take a while to understand task it's interesting but I need anoter day. The git lab one what is interesting is they say auto generated configs but it's your script is responsible for auto generating it. Problem with this workflow is that it still does not understand the dependency unless you do. You can run make -n on each project to figure out what .o files are used and then build a tree that way but splitting that tree up is not trivial as all deps have to exist on each node. something still needs to put that together. I didnt see gitlab doing that from what I read so far. What's worse is the build is hard-coded into many small scripts that are not in the same workflow as a dev's makefile and would have to mirror a dev including a new dependency. This is where writing pipelines is not really speeding up the build.

[mystic void]

yeah, they cannot magically guess which target you want to call.
From a build and release perspective, it depends on how is your stuff structured in terms of repositories, dependencies and build. Ideally, you should have your flow encoded in your build and having it once so it gets exercises (ex: not having dev and build use a different flow and target). The CI/CD pipelines should be thin wrappers around build scripts as it reduces the maintenance burden and make it easier to debug and troubleshoot

[torn blade]

how do i profile pyqt applications? cprofile does not produce callback to the qt c++ codes

[sturdy tide]

Any recommended benchmark and profiling tools?

[mystic void]

<@&831776746206265384> looks like an ad?

[frozen zephyr]

yes it does...

!rule ad

[rancid schooner]

Rules

6. Do not post unapproved advertising.

[frozen zephyr]

we don't allow advertising here, @halcyon dune

[mystic void]

https://github.com/psf/pyperf looks interesting for microbenchmarks.
For general load testing, I tend to use gatling but heard good things about locust

[frozen zephyr]

for profiling CPU bound stuff, py-spy

for profiling memory usage, Memray (disclosure: I'm a Memray maintainer)

[thorny shell]

Dr Brown's Mem-Ray Tonic

[cedar nacelle]

Perhaps magic is a bad word to use here. Because i was able to infer by parsing all Makefiles, what the build dependency was (quicker than make -n on 900 projects) . Just sorting that build into building the least dependent projects build 1st, and then most dependant build last. You can do this by doing a topological sort to order the build so the most dependent is last to run. Then you can let loose as many co routines to build reach proj and pause any project that is waiting an its dependent project to be done. That's what i baked into that one script that i use to build at maximum speed today. So what are these CI's doing for us that is totally proven to be possible?

i disagree that we need a skeleton on top of some existing build system like Cmake + Make to have a valid build workflow. if Make is the build system, the builder should be able to infer from that system, what to do. Don't you think?

Well lets put it this way. I had no choice. the devs did not want to add more on-top of what they had.

So am I doing something that a CI framework finds impossible ?

[gleaming wadi]

Not entirely sure I picked the right channel, feel free to direct me elsewhere.

I am currently struggling with setuptools using pyproject.toml at repo top-level and installing it all with pip install in a venv. I cannot use a standard src layout as it's a big repo with many things, so I would like only actual Python modules to be picked up under the same namespace. I want to avoid explicitly listing submodules. One possible layout would be:

pyproject.toml
not-python
|- test.py
actually-python
|- src
   |- actually-python (intentionally serving as an implicit namespace)
      |- module_a

However, any attempt to utilize a [tool.setuptools.packages.find] section leads to either nothing being picked up or every single Python script in the repository being installed inside an implicit namespace package, i.e., I end up with a not-python module in my virtual environment. More specifically, adding the section but leaving it empty usually yields the same results as putting any settings in there - unless I only include a module that definitely does not exist; then it just installs nothing. Specifying the one namespace module I want (actually-python) to get picked up however leaves me with everything being added again. If somebody with some deeper understanding of the setuptools magic could help me out, that would be great.

[hollow wagon]

I made a thing. It makes Python CLI tools start much faster.

I now use it to make my pre-commit hooks blazing fast. It's nice for git commit to be near-instant again, even with pre-commit checks!

It's called JumpTheGun, because it makes Python tools win the race by cheating 😉
https://github.com/taleinat/jumpthegun/

So I'm looking for early adopters. If you like, give it a whirl and let me know how it goes.

[gleaming wadi]

Seems very niche to me.

[astral apex]

https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html
See the very last point in the tip. Try declaring the packages specifically, so you’re not not doing any finding.

As a worst case last ditch effort you could also try excluding the unwanted ones from your find.

[gleaming wadi]

Thanks for the reply! I was hoping to be able to still rely on finding, just steer it in the right direction. Excluding things might be worse than explicitly listing all modules as technically, there are no modules to exclude; just setuptools believes they are modules and gives them silly names like test or scripts because that's the name of the directories where those scripts happen to be. The docs say that I should rely on include/exclude/where to avoid this, but I just cannot fathom why specifying include = ["mypackage*"] does not work. Even when I put a sub-directory in where, it still picks up modules outside of that directory (i.e., in directories next to that one). Why on earth would it ignore its own settings?

[astral apex]

Yeah that doesn’t make too much sense.

Try your find again with the namespaces=false

Just to see what happens

[gleaming wadi]

It seems that regardless of what I do with the other options, I end up with no packages whatsoever.

I guess even more curious than it blatantly ignoring settings is that it will honor include = ["thismoduledoesnotexist"] by not adding any module, but if that include is set to the namespace package I actually want to include, it includes everything again, even the things that cannot have that name.

[astral apex]

Do they have that name in a sub folder?

Have you tried setting where?

[gleaming wadi]

Nope.

[gleaming wadi]

Yes. and when set, it will straight up ignore it and also add scripts as part of implicit namespace packages outside of that folder. In the structure given above, setting where to ./actually-python or ./actually-python/src will still give me a not-python module.

[astral apex]

[gleaming wadi]

Or perhaps I am haunted by some pip feature that I am not aware of? This is how I check my changes to pyproject.toml:

rm -rf .venv && virtualenv .venv && . .venv/bin/activate && pip install ."[dev]" && ll .venv/lib/python3.10/site-packages

[astral apex]

… no deactivate first?

I would try updating setuptools before installing the package

[gleaming wadi]

Requirement already satisfied: setuptools in ./.venv/lib/python3.10/site-packages (67.6.1)

Seems to be most recent one already.

Good catch with the deactivate, but it does not change anything.

At this point I either believe setuptools is quite broken or I did something very stupid elsewhere.

[astral apex]

Uh…

[astral apex]

- ."[dev]"
+ ".[dev]"

Try that real quick

[gleaming wadi]

No difference. Even just using . does not change a thing.

[astral apex]

I would python -m build and double check all the output

But I got nothing

[gleaming wadi]

That actually helped a little bit, but it doesn't make it easier. It seems that a "test" directory gets collected regardless as its own module; but in there, only files prefixed with test_ (they are pytests and conftest.py is not picked up). What is really new however is that another pythons script is not picked up at all. So the scope got smaller somewhat, but part of the problem persists.

I mean, setuptools toml support is labelled as beta... Perhaps I should do what all the cool kids do and just switch to poetry, hatch, or whatever alternative there might be.

[brazen forge]

AFAICT, this runs a daemon that executes the actual tool, right?

[hollow wagon]

Indeed, that's part of how JumpTheGun works.

[sweet warren]

Would this be the place to ask about the aws Toolkit for Pycharm?

[mystic void]

I wish.
There is no convention for Makefiles. Every single project will define their own targets and flow to call. Some projects will ask you to do make or make build or make release or make tarball or some other variation.
That's why it's very common to have the build as used by the developers, and to wrap that up for the CI/CD.

That would be different for better project management tools like Maven where there is a convention codified and you would know right off the bat how to build it without having to read any doc

[sweet warren]

well ill ask either way :x

For Pycharms AWS Toolkit plugin, I'm able to use it to make a connection using my regular account. However I'm not succeeding at using a specific role that has perms I need. Everytime I run the MFA to use the role credentials it tells me it connected just fine but unfortunately, running the script tells me i don't have the right perms.

Any help? Also is it possible to run my script locally using the exact same perms that a lambda has?

I've also tested using the aws console that I do get the exact same error message as my script when using my regular account, but once I assume the correct role, I can access the secret manager resource

[gleaming wadi]

No idea about AWS, but JetBrains have a track record of not implementing support for things properly. But if you don't succeed without using PyCharm either, then it sounds like an AWS config problem?

[sweet warren]

Ill look into trying the aws cli locally

[cosmic sparrow]

hello, when I try to make a pr in github it gives me a merge conflict. I make sure that my local branches are up to date and then I run git merge master from the relevant pr's branch:

Merge made by the 'ort' strategy.
 src/tests/PosicionGeograficaTest.java | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

it "successfully" completes the merge without asking me to do anything... but I'd like an easy way to see what those 16 insertions and 2 deletions were. How could I do that?

[cosmic sparrow]

in case anyone was wondering I just canceled the commit message and it allowed me to see what changes were staged

[gleaming wadi]

git show for the most recent commit. But merge might also have verbose output. Never really use merge.

[thorny shell]

git diff HEAD^

[dark latch]

how can i remove these 73 commits from contribution activity?

lemme give you context, all these commits are made by codespace, it basically made commit on every keystroke and i was new back then

and now it is making my contribution activity uglier

[thorny shell]

you could squash them all into one commit, then "force push" the result

dunno if that'd fix the statistics, but it might. If you're concerned about hosing your repo -- and you should be -- you might create a new repo (like a fork of this one) to test it out on

[dark latch]

can you tell me more specifically about git push part

[thorny shell]

git push --force-with-lease

what more do you want to know?

[dark latch]

like.. these re the commits that I've made 4 month ago

[thorny shell]

... and?

how is that relevant?

[dark latch]

now i want to remove the commit but i dont want to change the content

[thorny shell]

yes, I understand that.

[dark latch]

can i do git reset --soft <id>

ofc i have to do it will al commits

[thorny shell]

you can do it but I don't think that'll help you solve your problem

"squashing" is the obvious way. Are you not familiar with rebasing?

[dark latch]

no

[thorny shell]

ah that's the problem, then.

you should read about it; it's too complex to fully explain here.

but basically: "squashing" means taking the changes from a bunch of commits, combining them, and making a new commit from all those changes. it's exactly what you want.

https://git-scm.com/book/en/v2/Git-Branching-Rebasing

[dark latch]

ok, I'll read about it

[thorny shell]

https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History

[dark latch]

is it right?

it squashed the commits but it didn't changed the graph

[dark latch]

@thorny shellit squshed the commits but it changed 73 to 94 💀

[thorny shell]

dunno what you mean

and I've never used gitlens, so I can't be sure; but that screenshot looks reasonable

if that's a public repo, I could do it, and then tell you how I did it

[dark latch]

its not
and its not my repo

[thorny shell]

"Not my circus; not my monkeys"

[dark latch]

do you know cs50?

[thorny shell]

nope

[dark latch]

its a cs course
they made their repo for us

codespace automatically commited changes on every keystroke

[thorny shell]

I could try creating a similar repo, just to figure out how to use gitlens

[deft grail]

That suggests it is public - it would need to be at least accessible for all people on the course...

(Isn't CS50 free access??)

[dark latch]

the bot make repos for every individual

[brazen forge]

that's a weird choice

[dark latch]

and... i git push --force-with-lease after rebase
i think that commited more changes

can i completely remove this repo from my profile
from all places including graph

[bitter cedar]

so you mean is all the thing you commit is correct and the code is working

then it should be fine

like you commit everything that you change right?

[dark latch]

I just found out that others can’t see the graph anyway because it was a private repo

[bitter cedar]

yueys you can

[bitter cedar]

you can make the repo public

[dark latch]

read upper chats

other side

my side

lmao

[bitter cedar]

so is the code you commit and push available on the other side?

[bitter cedar]

so you want other to see the graph like your side

I am not sure about that

because private push right

[bitter cedar]

you can also remove repo from profile

delete I think

[vestal grove]

Iirc theres a way to show private repos commits in the green graph too

Some setting

[wary oasis]

When the repo looks like this 🧑‍🍳 💋

Like the New York subway

[thorny shell]

that's wrong; there should be a free transfer between main and dev-beta2

at least, during rush hour

[fresh timber]

Made a pretty simple 2D rendering app, but tried to support as much as possible, opinions? https://paste.pythondiscord.com/tocijobedi

[pliant drift]

Good afternoon everyone, does anyone have aws course recommendation, focusing on s3?

[spice wasp]

i tried to deploy my mern application using elasticbeanstalk and s3 and it always ends up giving this error

can anyone help me

[dark drift]

Having some Git issues, if anyone minds pointing me in the right direction.

User 1 owns a repo.
User 2 forked the repo.

User 2 commits to master x times and makes PR to User 1's repo.
User 1 squashes x commits and merges PR.

User 2's master now shows x commits ahead of User 1's master even when User 1's repo is correct

How does User 2 solve this problem and fix their repo/master?

[thorn portal]

what's user 2's git remote -v output?

assuming upstream is there already and pointing to user 1's repo

git fetch upstream
git checkout main
git merge upstream/main # OR rebase

[thorny shell]

has User2 fetched?

[rapid sparrow]

if User 2 has nothing to loose. Just make git reset --hard to original repo master, after git fetch

or just, make new branch based on original repo master. does not matter i guess.
(git checkout original repo master, git checkout -b new_branch_name)

[gray valley]

When i use top command there is a CPU%, is this for the whole system or per one cpu?

Another question i have 10 cpu on my machine , so the max load should be 1000% right? I mean I should see 10 jobs each took 100% so thats mean the max load

From the top command cpu percentage

[hollow wagon]

I'm writing a shell script that will run my Python code via exec, but first I need it to run a background process. Considering that this going to be run in a container as the main process, would it enough to run my background process as python my_background_task.py </dev/null >/dev/null 2>&1 &? Should I be using something like nohup, or something else?

[brazen forge]

so, the main PID 1 is something else and this is a related background program, right?

I'd say also disown it

and nohup it

nohup ... </dev/null &>/dev/null &
disown

[hollow wagon]

Thanks @brazen forge !

[thin crescent]

hello, does anyone have experience with this k8s repo? https://github.com/tomplus/kubernetes_asyncio

the whole thing has a weird implementation in my eyes

was wondering if other people felt similarly

[thorn portal]

apologies for digging out an old thread, i just thought this might be of interests to you re. this tool

i came across usage of importlib to query entrypoints when inspecting how pluggy works under the hood
after some toying around i discovered the following works perfectly for fetching the entrypoint funcs for all tools

import sys
if sys.version_info >= (3, 8):
    from importlib import metadata as importlib_metadata
else:
    import importlib_metadata
importlib_metadata.entry_points(group='console_scripts')

might be a future proof way to fetching the entrypoint func? just in case there is another change in the template used to generate the entrypoint scripts?

[hollow wagon]

That's great, thanks!!

[fallen flame]

hey what's the best library through which i can automate my xl work?

[mystic void]

can you define xl work?

[fallen flame]

like microsoft excel basic cell coloring and other stuff

formulas and all

[mystic void]

oh got it. Thanks!

[fallen flame]

This is my code currently and I want to color the cell according to the formula output
Do you guys have any idea how it can be done?

import openpyxl

# For Willow Weekly
workbook = openpyxl.load_workbook('Willow.xlsx')
print('File Loaded')

sheet = workbook.create_sheet('Output')
print("Sheet Created")

sheet1 = workbook['Input']
sheet2 = workbook['Output']

cell_values = []
for row in sheet1.iter_rows(min_row=2, max_row=770, min_col=1, max_col=1):
    for cell in row:
        cell_values.append(cell.value)

for i, value in enumerate(cell_values):
    sheet2.cell(row=i+2, column=1, value=value)

for col in range(2, 131):
    value = sheet1.cell(row=1, column=col).value.split()[-1].strip('[]')
    sheet2.cell(row=1, column=col, value=value)

for row in range(2, sheet2.max_row + 1):
    for col in range(2, sheet2.max_column + 1):
        cell = sheet2.cell(row=row, column=col)
        cell.value = f'=Input!{cell.coordinate}/(6*20*0.275)'
        cell.number_format = '0%'


workbook.save('Willow.xlsx')
print("File saved")```

[heavy knot]

what does allow password auth over ssh mean?

[thorny shell]

it means: you can type "ssh some-host", and the host will respond with "Password:" and then you can type your password.

[grim rock]

https://www.reddit.com/r/Python/comments/12ta44k/i_made_a_path_cheat_sheet_it_could_be_better_it/?utm_source=share&utm_medium=ios_app&utm_name=ioscss&utm_content=1&utm_term=1

[hearty copper]

anyone know a good lib to async download files from FTP?

aioftp is giving me issues

[pastel slate]

hey .. so im pretty new to using python.. mainly the tooling .. I got past the whole venv thing. all set there. I am trying to just skip all the flake8/black stuff and just use ruff since its seems like its pretty nice and fast supports all the rules from previous linters. However my issue is im trying to enfore single quotes so I have this in my pyproject.toml but it doesn't seem to do anything

[tool.ruff.flake8-quotes]
inline-quotes = 'single'

[thorny shell]

I've never heard of ruff ... but I'd advise you to use double-quotes, not single quotes, since "black" (mostly) enforces double-quotes, and lots of people use it.

If you insist on single quotes, you'll be a special snowflake.

[pastel slate]

im coming from javascript

thats fair though, either way the rule is not being enforced

i figured it out i have to enable the rules to be enforced in the first place

anyways ruff: https://github.com/charliermarsh/ruff#configuration

rust based linter

[edgy basin]

thats pretty ruff

[rough marlin]

I actually have a question about it as well

likely around the same issue you did: rule enabling

Yesterday, I merged a PR that seemed to enable linting

it worked locally

today, after pulling the remote changes, my local ruff doesn't work at all

[pastel slate]

are you using intellij?

[rough marlin]

PyCharm, but yes

[pastel slate]

so there was a breaking change to the plugin after ruff 260

[rough marlin]

So far, i've tried:

1. making a new venv and installing latest ruff
2. ruff clean before a run
3. moving things in my pyproject.toml [tool.ruff] around

[pastel slate]

it no longer shows inspections

[rough marlin]

I don't know if I have a ruff plugin installed

I'm running ruff via CLI

[pastel slate]

oo ok this is in clu

[rough marlin]

is this something shipped by default?

though, you made me think

[pastel slate]

heh i mean i barely know anything at all

[rough marlin]

that's actually a really important detail, ty

I didn't use ruff at all until the past two days either, so we're both learning

I just used flake8

[pastel slate]

all i know is that in my toml file i had to add to this select = ["E", "F", "Q"]


it selects the flake8 rules to enforce

like i wanted to enforce quotes so i had to add Q

[rough marlin]

Yeah, I added "W" to enable whitespace rules

https://github.com/pushfoo/arcade/blob/51adf7943884d34e434d525d29cba2362cf019c8/pyproject.toml#L86-L102

that was my PR the other day

[pastel slate]

im not the person to ask unfortunately

im like 24 hours into python linting

[rough marlin]

this seems to be the best channel for it on this server

btw, "W" will enable whitespace rules, but not all whitespace rules are implented

there's an issue with a check list tracking what's been done on their github page

[mossy gate]

could anyone recommend me some Python books?

[next gazelle]

what part of python are you looking to learn?

[nocturne palm]

ModuleNotFoundError: No module named 'tiermaker'

Using poetry, built a new version still getting this error. I have removed this line from the code and moved the function in the main file. This line itself no longer exists yet when I install with poetry it does show me.

How do I update the package using poetry while building from new lines of code?
I did the following

poetry lock
poetry build
poetry install
poetry publish

[heavy knot]

hi guys,
I wrote a port scanning tool with python, which has many parameters such as service,
version, threads, protocol, ports. Don't forget to check it out on my github.
https://github.com/HalilDeniz/Port-scanner

[vague silo]

looks interesting

what made you write this over using nmap?

[earnest forum]

Hi, is there anyone who has gone through the Docker Deep Dive book by Nigel Poulton? Please if you have, i hope you can ping on reply
I have a couple questions i need to ask to make clarifications

[rapid sparrow]

https://dsicommunity.org/nohello
https://dsicommunity.org/dontask

[earnest forum]

I'm currently going through the Section 5 of the Docker Deep Dive book which is about The Docker Engine and the section under this specifically i need clarification on is Securing client and daemon communication. The section in the book talks about configuring CAs and certificates and then nodes and also implementing the tutorial through a lab. I'm quite confused if i can actually try the implementation out from my local pc but he doesn't reference a specific lab setup in the book. I don't know if anyone can help with this and clarify this for me.

[rapid sparrow]

https://docs.docker.com/engine/security/certificates/ there is always an option to check official documentation for more details
here is a very detailed instruction how to do it: https://docs.docker.com/engine/security/protect-access/ from official docs

i can bet it is all automatable though in an easier way

[rapid sparrow]

i remember Docker in Docker image already offers it way easier https://hub.docker.com/_/docker

TLS
Starting in 18.09+, the dind variants of this image will automatically generate TLS certificates in the directory specified by the DOCKER_TLS_CERTDIR environment variable.

So just going through documentation, and finding how to use Dind, u will get easily setup secure dind with almost no effort. just a matter to read those dind docs

[earnest forum]

Okay, thanks for this. I would go through the links and read through the resource you outlined. I'm still new to docker though so was just following the instructions from the book

[earnest forum]

I just skimmed through the links you provided, it seems way to advanced for me at the moment as i can't fit it into the context of which the book presented the topic. But thanks for the links

Also this is the section of the book i'm talking about:

[tepid light]

I have just released a new version of the Python tools for the Polylith Architecture, containing a brand new command - poetry poly sync - that will keep the projects up to date with the bricks they are actually using, in case you would forget to add dependencies to your project file(s). I think this will be most useful during development, for a nice flow when adding the LEGO-style bricks to a Polylith Workspace, and during the exploration and trying out new features. You’ll find the sync command in v1.6.0 of this tool (yes, it is a Poetry plugin).

This was fun to develop - and an example of practice-what-you-preach about Polylith - the command is reusing already developed AST-parsing bricks + new command-specific functionality. Bricks are small reusable building blocks that you can combine to build features. You'll find the docs here: https://davidvujic.github.io/python-polylith-docs/

[gray knot]

Are there anyone fimiliar with ansible and vagrant here? Im going to setup a ansible project and configure Redis using vagrant VM. Just wondered if someone was able to help a little. Bit stuck with the configuring part of redis

[slow willow]

hiya! :) i'm attempting to download a docker image that has no tags but i keep bumping into:

Error response from daemon: manifest for cool_dev/censored-docker-image-name:latest not found: manifest unknown: manifest unknown

i tried adding the -a flag to no avail, anyone have any suggestions?

[rapid sparrow]

It does not exist to download. Incorrect address/name

[slow willow]

The "docker name" shown in the error is swapped out, the actual image exists on the docker hub but with no tags

Might just build it myself from github

[rapid sparrow]

tags are always present for image

if tags aren' tdefined, latest tag becomes default

[lean pewter]

Random question. Whats a good way to survey new Python users for their initial thoughts and experience using pip? Specifically looking for people with like < 30 days using pip

[arctic rose]

is it possible to update a GCE container using jenkins without having to manually restart the GCE instance after jenkins pushes new conatiner

[lean pewter]

https://github.com/mitsuhiko/rye/discussions/6

[warm bramble]

Hi! I'm not sure if this is the right place to ask my question, but here it is:

I'm trying to automate tasks on my Android phone from my Mac using adb (Android debug bridge). I'm pretty sure adb is used for testing, and that's why I'm in this channel. I've found 3 adb modules on GitHub, all of them are different. The first three have links to each other in their readmes.

• https://github.com/Swind/pure-python-adb
• https://github.com/google/python-adb
• https://github.com/JeffLIrion/adb_shell
• https://github.com/dtmilano/AndroidViewClient
  This left me very confused. Posts on stack overflow recommend different modules from each other, same with tutorials I found online. They don't explain the differences between the modules either.

My questions are:

• What are the differences between the four modules?
• Which module is the easiest to use, the most suitable for my use case, and has the most community support?

[heavy knot]

Hello everyone,
I wanted to share with you the url-scanner tool I recently wrote in python,
may seem like a simple tool, but, inside
(url, wordlist, cookie, http status, user-agent, etc.)
https://github.com/HalilDeniz/Url-Scanner

[tepid light]

Following up on this:
I have recorded a very much improvised video - with live coding - to show how Polylith changes the way you develop Python code, and how to use the tool that I've built. Phew, it is difficult to speak & code at the same time 😅

In the video, I’m taking the first steps to develop some kind of Dad Joke service using the Python tools for the Polylith Architecture 🥁 👴
https://youtu.be/oG4OFEer3Tk

[sacred violet]

Hi there, I have a conceptual question about jupyter notebooks and shared code. I have a project (it's a grpc server) that will be shared between several devs. The project runs in a docker container (technically in k8s). For documentation and a playgound, it also runs a jupyter notebook server. I have some shared code that should be used in each notebook (mostly basic python, but a few magic commands as well). I was originally trying to do this with profiles, but overriding profiles was pretty janky. Is there a better way for me to go about this? Edit: I have created a base notebook that all other notebooks should run first via %run base.ipynb. Any better ways?

[peak plume]

me when the stores out of whole wheat

[acoustic wyvern]

http://www.dabeaz.com/generators2/index.html

[crystal roost]

does anyone know of a hosting service that I can use to run my code every set time interval like 30 min, also will need some storage too.

I know that a lot of hosting services provide 500 hours free every month, but what I want is to run my code every 30 min and it takes only 1-3 seconds to complete.

It's a bot that checks for a new vid from a specific channel

[thorny shell]

AWS? Azure? Digital Ocean? Linode? Google Cloud Platform

Any of those would work

[crystal roost]

free?

[thorny shell]

of course not 🙂

I imagine each has something like "first 500 hours free" but in general, you get what you pay for

[crystal roost]

that's the problem, my code doesn't need 500 hours it's needs to be run every 30 min.

which would reduce the uptime by a lot.

[thorny shell]

if you don't mind

• spending a little money; and
• spending a lot of time struggling with Lambda
  you could use AWS lambda and cloudwatch events. Disclaimer: I've never done this, but I'm pretty sure it's not easy.

The other services will have something similar.

[rapid sparrow]

@crystal roost it is... easy.... if u know AWS (for example), and preferably Terraform, and preferably Golang (because using with interpreting language lambdas is its own challenge, compiled languages work way better for lambdas)

[brazen inlet]

Is there a better than android studio to develops mobile apps

Because my pc is not that good

And android studio makes it laging

[sacred violet]

any poetry folks around that can help me out for a sec, or is there a different room I should be in?

I asked chatgpt, and it hallucinated an entire long answer about poetry that is completely wrong 😆

[thorny shell]

I'll take a crack at it

get a #1035199133436354600 channel tho

[sacred violet]

@thorny shell thanks, I’ll have to post later, had to head out, but I think I figured out what to do. Was trying to recreate the idea of npm link for dev while using a git repo during deployment. Found out about environment markers 🙂

[thorny shell]

heh, I have no idea what those are; glad you figured it out

[wary oasis]

I was wondering if someone could help me out with this simple dockerfile setup. I am using compose to run all of the images btw. I have different servers with identical dockerfiles that simply run the Go binary (gobin). The problem is that when the dockerfile gets to CMD and tries running the binary, I am getting errors that the file/dir cannot be found. File tree: https://pastebin.com/qRKuJjuJ docker compose build/up output when run in top dir: https://pastebin.com/YP8VHhwR

All of the dockerfiles look exactly like this ```dockerfile

FROM alpine:latest

WORKDIR /go

COPY ./cmd/gobin .

EXPOSE 80

Start the API server binary when the container starts

CMD ["./gobin"]

and the compose file https://pastebin.com/7rE5r1dH

[neat tundra]

Hey everyone!

I wanted to introduce myself and share a new open-source project I've been working on called Apollo. It's an LLM aggregation library for Python that simplifies the process of working with APIs, AI models, and Django.

I noticed some of you have experience working with APIs, Django, and AI models, and I'm curious about your experience. Have you worked with AI models via API or used Django to manage multiple channels for AI models in your projects? I'm currently adding support for a full API to manage in-app integrations, user subscription management, and preferences, and would love to get feedback from those with experience in these areas.

By the way, if you're interested in contributing to the project, we're always looking for new contributors!

Thanks for your time, and have a great day!

[brazen forge]

do you have a .dockerignore in each of those service directories?

[wary oasis]

No

what should be in included in that?

[brazen forge]

if you don't have one, that's okay

[wary oasis]

I'm just trying to copy one file and run it

[brazen forge]

did you try debugging the image to see whether the image contains the gobin binary after the COPY command?

[wary oasis]

I ran ls in the CMD command of one of the dockerfiles and it shows gobin which is the one I want to run, however in the terminal it shows the error

[brazen forge]

did you check the permissions of the binary in the image?

[wary oasis]

I did not

i will try it

[rapid sparrow]

RUN chmod 777 gobin

[wary oasis]

quick explanation?

the chmod part

[rapid sparrow]

Allowing file executed by every possible user (+reading and writing)

[wary oasis]

sounds like a good chance, one minute

[wary oasis]

hmf same problem

FROM alpine:latest
WORKDIR /go
COPY ./cmd/gobin .
RUN chmod 777 ./gobin
EXPOSE 80
# Start the API server binary when the container starts
CMD ["./gobin"]

[rapid sparrow]

Also, change CMD ./gobin
When no brackets, it is shell (shell syntax is allowed)
When brackets, executing specific executor with arguments directly

[rapid sparrow]

Did chmod run successfully?

[wary oasis]

currently have this file/output ```dockerfile
FROM alpine:latest
WORKDIR /go
COPY ["./cmd/gobin", "."]
RUN chmod 777 ./gobin
EXPOSE 80

Start the API server binary when the container starts

CMD ["ls"]
txt
gosqueak-web-service-1 | exec ./gobin: no such file or directory
gosqueak-message-service-1 | exec ./gobin: no such file or directory
gosqueak-auth-service-1 | exec ./gobin: no such file or directory
gosqueak-web-service-1 exited with code 1
gosqueak-ecdh-service-1 | gobin
gosqueak-message-service-1 exited with code 1
gosqueak-auth-service-1 exited with code 1
gosqueak-ecdh-service-1 exited with code 0

"gosqueak-ecdh-service-1 | gobin"

[rapid sparrow]

Can we have more full output of docker build

[wary oasis]

https://pastebin.com/TirmA1j3

[brazen forge]

also, focus on only one service for now

[wary oasis]

ok

[brazen forge]

by full output, DW probably meant ls -l

[wary oasis]

Ill go for the html server

[brazen forge]

also also, build with --no-cache for now, so you can verify the build steps

[rapid sparrow]

Nah, I meant seeing docker build

[rapid sparrow]

Looks fine to me as it is

[brazen forge]

¯_(ツ)_/¯

[rapid sparrow]

Run
docker compose down ; docker compose up --build

Down stops and deletes previous containers

[wary oasis]

is it 404?

[rapid sparrow]

Yes

[wary oasis]

https://pastebin.com/Ym6rcEm1

[rapid sparrow]

CMD ./gobin
And running same down and up build?

[wary oasis]

I tried with that and got the same result, but slightly different ```txt
gosqueak-web-service-1 | /bin/sh: ./gobin: not found

https://pastebin.com/hJbHpuuP

[rapid sparrow]

docker compose run NAME_OF_Service_in_compose_file sh

This will get u inside container for debugging

Try quicker there ./gobin and just trying to find what is wrong

[wary oasis]

we're in

🤡

[rapid sparrow]

Wait. U did not execute with down

[wary oasis]

wait wait

what do you mean by that

[rapid sparrow]

I told u run like this

It is one full command

[wary oasis]

and then get into the shell?

[rapid sparrow]

No, just print log for starters

[wary oasis]

https://pastebin.com/CzYqK6ve

[rapid sparrow]

New error. That is a progress

[wary oasis]

the "orphan containers" are all of the ones im not currently running, I commented out those parts of the compose file

Should those be gone?

[rapid sparrow]

Out of curiosity try replacing your FROMv with
FROM debian:11.6-slim
And running down and up build again

[wary oasis]

ok

[rapid sparrow]

Should be, added flag about removal of orphans is nice, but not necessary for our current debugging step yet

[wary oasis]

https://pastebin.com/twCuqWCp 👀

it tries to run it now

[rapid sparrow]

Nice

[wary oasis]

maybe there is a way to install taht using RUN

[rapid sparrow]

That is cool

[brazen forge]

how was the gobin created?

[wary oasis]

go build -o gobin .../cmd

[rapid sparrow]

FROM golang:1.19.3-bullseye as build

RUN apt update
RUN apt install -y build-essential
RUN apt-get install ca-certificates -y
RUN gcc --version

WORKDIR /code

COPY go.mod go.sum ./
RUN go mod download -x

COPY main.go ./
RUN go build -v -o main main.go

FROM debian:11.6-slim as runner
WORKDIR /code
COPY --from=build /code/main main
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
CMD ./main

Adjust to yourself

Keep certs

[wary oasis]

I was hoping to avoid building during the container loading... I'm not sure how to manage the local dependencies I have, like I have modules for JWT and api tools that I'm not sure how to get included

Just gonnabe a headache

[rapid sparrow]

Building image fully is the right way

Container is not loading. It is building image

[wary oasis]

right it prob just needs to be built on the same "machine"

[wary oasis]

Youre right to be precise

[brazen forge]

FWIW, the issue is that you built the binary on a non-alpine OS (one that doesn't use musl as the libc) without telling go that you want a binary that works for alpine (one that uses musl)

[rapid sparrow]

Golang is built differently depending on used c libraries under the hood. It is not os cross platform fully :/

[wary oasis]

I CAN tell Go?

[brazen forge]

yes

with the GOOS and GOARCH env vars

[wary oasis]

So the default build is not built right

go os and go architecture?

[brazen forge]

DW's suggestion to build the binary during image build is a better idea though

[rapid sparrow]

How/? Isn't it having just generic GOOS=linux GOARCH=amd64

[brazen forge]

you can build for darwin from linux too

[rapid sparrow]

It is macos? Not alpine? Or they are related

[brazen forge]

well, yeah, mb, GOOS and GOARCH won't help with glibc vs musl

[wary oasis]

Is that a result of the libs on my actual machine?

[rapid sparrow]

Possible alternative using same Linux distro as your current os. What is your current os?

[wary oasis]

ubuntu

[rapid sparrow]

Okay. It was supposed to be compatible with debian, but lets try Ubuntu image

FROM ubuntu: your version

Indeed

[brazen forge]

||btw, if your code or any libs you use doesn't do any C FFI, you can also build the binary using CGO disabled
which will make a binary that does not link against any libc (glibc or musl or any other)||

[rapid sparrow]

We are just eliminating all possible reasons at this point

[wary oasis]

https://pastebin.com/2SEyTf7d

the container didnt crash

i think the server is running

[rapid sparrow]

Indeed

Hooray 🎉

[wary oasis]

hell yeah

thank you both*

[brazen forge]

tldr: build the binary during image build instead

[wary oasis]

this is the way

I will check that out tomorrow

Do you know any downsides to the ubuntu image though (besides size)

[rapid sparrow]

With proper build step, your docker will not depend on specific OS for building

[wary oasis]

so a downside is coupling to the OS

[rapid sparrow]

Not really. Size is pretty much the only thing. B there should be slim version may be

[wary oasis]

ubuntu server maybe

[rapid sparrow]

Yup. U kind of partially losing advantage of containers, since they make os independent stuff

[wary oasis]

updated 7yr ago ☠️ but probably OK https://hub.docker.com/r/circleci/ubuntu-server

[rapid sparrow]

U you aren't having as a code recorded building app

[wary oasis]

right, it's sort of cheating

[rapid sparrow]

Double hacking

[wary oasis]

just trying to get a prototype running to help debug the business logic stuff right now

[rapid sparrow]

Best not to use 7 years old images. Embrace big size of Ubuntu official for stability until u embraced proper docker image building

Can't have everything while breaking best practices

[wary oasis]

yeah, im not cramming on a aws server seomwhere

not right now

[brazen forge]

there is a way to cross compile a musl-dependent binary from a glibc-dependent OS btw

[rapid sparrow]

Better using best docker practices. Let's alternative paths not adviced and not mentioned xD

[wary oasis]

I like to believe almost anything is possible in CS 🙂

[brazen forge]

yeah, I would not advise that 😛

[wary oasis]

something I could install then run a command on an existing binary? That type of thing?

I could just use bash to do it all

[rapid sparrow]

🤫

Building in docker image is the way

[wary oasis]

lol

[wary oasis]

My one idea was to try and copy the entire project dir into the container then build the binaries, so it would be just like building the binaries outside the container. Well I guess it wasnt "just like it" enough....

Maybe I can get somewhere with that

I will implement the hack for now

[brazen forge]

yeah, the musl site provides a C compiler
you provide that compiler to go and build
again, don't do this over the image way

[wary oasis]

I can learn that, I could also learn how to do it the right way 🤷‍♂️

[wary oasis]

ot but s3 is so good, were on ep 3 or 4

I've got to not try to run all of the things at once again when bootstrapping things, good idea to cut it to one

[heavy knot]

Hello friends
I wrote a nice program called network scanner with python,
You can reach me at my github address
https://github.com/HalilDeniz/Network-Scanner

[rapid sparrow]

https://youtu.be/AX1gfJK96KY
Haven't seen yet

[wary oasis]

I thought it would be a very big program, very compact, nicely made

______The server still holding strong in that container ```go
func NewServer() *Server {
return &Server{}
}

func (s *Server) Run() {
http.Handle("/", http.FileServer(http.Dir("./static")))

if err := http.ListenAndServe(":80", nil); err != nil {
    panic(err)
}

}

not sure others will go quite as smoothly though (the others have more stuff to break)..

[rapid sparrow]

That can be replaced away by using nginx image

[wary oasis]

I was planning on going the easy route and using nginx reverse proxy because it has a nice GUI and easy SSL certs

I think last time I was able to get https in a few minutes

Ive already rolled my own* auth, I just want to get this thing goign right now

[wary oasis]

I could also just add that route to one of the other servers to eliminate a container

it just doesnt quite fit exactly anywhere, but it could be shoehorned / put wherever it works

[loud bear]

how can i conditionally add tool.setuptools.packages entry based on sys_platform?

[tool.setuptools]
packages = ["webview", "webview.js", "webview.lib;sys_platform=='win32'"]

[digital matrix]

how can you convert avif files back to jpeg

i only found code that does the opposite

[indigo zenith]

The same code doesn't work in reverse?

[umbral dove]

Might wanna try running it a as scratch or separately like Dow said, if not then might have to do with any dependencies and such also needing to be converted(?)

[south forum]

hey peeps,
Whats the easiest way to deploy my react, FastAPI, MongoDB application??
really prefering simplicity over everything rn 😄

[slim maple]

Depends on what you're more familiar with

For example I'd prefer to use docker so I don't have to manage any dependencies on host system

[south forum]

do you by any chance have a tutorial on how to dockerise and deploy my tech stack to DigitalOcean ?

[slim maple]

It's easy to find tutorials on how to use docker

[wary oasis]

I did get proper binary creation going by copying the project over, building, then deleting everything but the binary ```dockerfile
FROM golang:latest

temporarily copy project dir for building

WORKDIR /gosqueak
COPY . .

download deps & build

WORKDIR /gosqueak/src/webserver
RUN go mod download
RUN go build -o ../../../staticserver_bin ./cmd

del proj files

WORKDIR /
RUN rm -rf /gosqueak

EXPOSE 8080
CMD ["./staticserver_bin"]

still needs a few more lines to work but this is A solution for sure

Oh yeah I had to move the dockerfiles to the root dir 🗿

[rapid sparrow]

🥳

[thin jasper]

Docker problem:
I try to build an image, which includes a run step with pip install --extra-index-url https://${USERNAME}:${PASSWORD}@..... The password is handed in via --build-arg. Unfortunately, the password contains a $ which docker tries to interpret, effectivly cutting the password off. How can i prevent this?

[brazen forge]

it may not be Docker that is interpreting the substituted password, but the shell
try using '${PASSWORD}'

(single-quoting disables variable substitution)

[thin jasper]

It's not even a shell, but a Makefile. I use $(value PASSWORD) there, which helped to fix a docker login with the same credentials

Just built a minimal example to make sure its docker

Dockerfile

FROM alpine

ARG PASSWORD

RUN echo ${PASSWORD}

Makefile

build_minimal:
        echo $(value PASSWORD)
        docker build \
                . \
                --no-cache \
                --build-arg PASSWORD=$(value PASSWORD)

executing PASSWORD='SECRET$TERCES' make build_minimal

Output

echo SECRET$TERCES
SECRET
docker build \
    . \
    --no-cache \
    --build-arg PASSWORD=SECRET$TERCES
Sending build context to Docker daemon  3.072kB
Step 1/3 : FROM alpine
 ---> 9ed4aefc74f6
Step 2/3 : ARG PASSWORD
 ---> Running in b4ce644b8c5b
Removing intermediate container b4ce644b8c5b
 ---> c62b7ddb0903
Step 3/3 : RUN echo ${PASSWORD}
 ---> Running in e6056a25eb3d
SECRET
Removing intermediate container e6056a25eb3d
 ---> c5e91792a779
Successfully built c5e91792a779

So I guess its not make which cuts it off

[rapid sparrow]

if u would replace COPY . . to explicit folder/files copying, that will be even better

generic copying everything is bad ^_^

[rapid sparrow]

and in my example... i showed you multi layering building. you should copy paste resulting binary from building image to final one for running. it will reset present stuff / minimize the image to minimum

[rapid sparrow]

FROM debian:11.6-slim as runner
COPY --from=build /code/main main
u can see example of multi layering in my example

[thin jasper]

Just realised I didnt have a problem with $ in the password but with other special characters that need to be URI encoded

[brazen forge]

I meant, put the $PASSWORD in single quotes in the RUN

also, pretty sure the password's getting "chopped off" because of the way you're running docker build

run as docker build ... --build-arg PASSWORD='your password here'

and again, use single quotes in the RUN command as well

[wary oasis]

Thanks I will add this

I'm getting this race condition error when starting my containers (the auth server is not yet running, so my code panics because it can't download the public key) should this just be handled with retries in the code? ```txt
[+] Running 2/0
✔ Container gosqueak-auth-service-1 Created 0.0s
✔ Container gosqueak-ecdh-service-1 Created 0.0s
Attaching to gosqueak-auth-service-1, gosqueak-ecdh-service-1
gosqueak-ecdh-service-1 | panic: Get "http://gosqueak-auth-service-1:8080/jwtkeypub": dial tcp 172.20.0.2:8080: connect: connection refused
gosqueak-ecdh-service-1 |
gosqueak-ecdh-service-1 | goroutine 1 [running]:
gosqueak-ecdh-service-1 | github.com/rebeljah/gosqueak/src/jwt/rs256.FetchRsaPublicKey({0x8a7f63?, 0x0?})
gosqueak-ecdh-service-1 | /temp/gosqueak/src/jwt/rs256/rs256.go:19 +0xac
gosqueak-ecdh-service-1 | main.main()
gosqueak-ecdh-service-1 | /temp/gosqueak/src/services/ecdh/cmd/main.go:18 +0x3f
gosqueak-ecdh-service-1 exited with code 2

[austere summit]

Hiya... I'm having trouble with anaconda which I downloaded and installed last night on a new machine ... when I start up spyder (latest version) I got this error message. However, when I go to Anaconda.org and look for pylsp I only see version 0.6.6 ... I'm not sure what to make of it (new to Anaconda - 1st day using it). Can anyone help me understand how I should work through this? Thanks!

[rapid sparrow]

feel free to just put https://github.com/vishnubob/wait-for-it and having it launched as
wait-for-it.sh auth-server:port -t 30 -- command_to_launch_service

as a lazy solution

or handling in app code retries more explicitely

[wary oasis]

Damn is all bash this cryptic? I created a small generic function to do retries

[thorny shell]

I haven't been following the discussion, but I heard "retries", so I automatically suggest https://github.com/groodt/retrying

[rapid sparrow]

yes, this why i never write bash
unless it is my complex linux alias commands (makes me wonder if i can write them in python actually whatever, i use copy pasted solution anyway)

or it is single line pipeline chained commands

[wary oasis]

I'm using Go, but this looks useful

[rapid sparrow]

common @wary oasis , that's Python 2!

Python 2, Carl!

oh. they updated to python3. they forgot to update documentation

[wary oasis]

Ew don't touch that

func RetryThenPanic[T any](Tries int, interval time.Duration, fn func(...any) (T, error), args ...any) T {

    var v T
    var try int
    var err error

    for ; try < Tries; try++ {
        if try > 0 {
            time.Sleep(interval)
        }

        v, err = fn(args...)

        if err == nil {
            break
        }
    }

    if err != nil {
        panic(err)
    }

    return v
}
``` story of my life

[thorny shell]

seems straightforward. I'd be surprised if something like that isn't in the standard library though.

and if the function you're retrying is "fetch something from some server somewhere", it's common to use exponential backoff, instead of just steadily poking it every N seconds

[sweet warren]

So I'm trying to privately publish a python package on devops

And I got the pipeline working fine, if I download the .whl off devops and pip install locally it all works just fine

Issue is when I try to pip install directly from the artifact feed

I get an error message
ERROR: Could not find a version that satisfied the requirement packagename (from versions: none)
ERROR: No matching distribution found for packagename

[austere summit]

I'm hoping this is a simple thing and I'm just missing the bit of context I need to get past it. Anyone happen to know?

The Spyder version in 5.4.3

Or, if I'm in the wrong channel for this question, if anyone can kindly point me in the right direction I'd appreciate it. Thanks.

[sweet warren]

Got it working. Idk what exactly fixed it tbqh but it works.

gonna figure that out

[shadow gazelle]

bro can i ask something?

[mystic void]

sure, as long as it is related to this channel

[shadow gazelle]

i cannot find python for scientific computing channel

[mystic void]

you could try #python-discussion or to create a thread as described on #❓｜how-to-get-help

[austere summit]

are Anaconda questions a purview of this channel?

I'm hoping to get some help regarding the error message I'm getting in relation to dependencies for Spyder 5.4.3

[sweet warren]

ok update

so it turns out

pip install <PACKAGE_NAME> --index-url https://pkgs.dev.azure.com/<ORGANIZATION_NAME>/_packaging/<FEED_NAME>/pypi/simple

doesnt work unless I append my <FEED_NAME> with %40Local

[mystic void]

in doubt you can ask. Worst case, someone redirect you

[sweet warren]

which stands for @local and im so confused.

[austere summit]

As it turns out it was an issue with how the error message was phrased. It was not pylsp, but instead python-lsp-server 1.7.2 that was required. The error message was ... in error. In case that helps anyone else.

[wary oasis]

I need to manage these settings for both JS and Go and some values need to be changed depending on if it is dev or prod. Anyone solved something like this?

[brazen forge]

pick those from environment variables during build

[wary oasis]

hmm yeah I can just do environment vars

[frank flower]

which tool is best?

[brazen forge]

tool for what?

[loud bear]

why can't pip just add a version tag to a dependency instead of allowing only a single version of a module?

like package@2.3

[thorny shell]

Not sure what you mean. requirements.txt and pyproject.toml typically have those tags in the dependencies, and when you install, pip, or perhaps poetry or whatever, finds the best possible versions of each dependency that "satisfies" the tag.

[brazen forge]

what they mean may be the way NPM handles it

package a can depend on package c@v1 while package b can depend on package b@v2 without conflicts

[thorny shell]

ah, having two versions of a package in a single project at the same time? istr java does that too. My guess is: it's not that way because it'd be super-confusing, and only somewhat useful

[loud bear]

why would it be super confusing

[thorny shell]

can't explain why; I can just vouch from personal experience.

maybe because I'm dumb.

[willow pagoda]

whatever packages you install with pip get stored in a single site-packages/ directory, so two versions of the same package will usually conflict and overwrite each other, hence why pip uninstalls the previous version beforehand
you could try manually placing the package in a separate directory on sys.path, but now you're held back by the import mechanism which has no ability to distinguish which version of that package you want - whichever comes first in sys.path is what gets loaded

[rapid sparrow]

Or golang

Kind of simplifies things

One venv is enough for all package versions

[austere summit]

I'm having trouble getting pytorch in spyder (latest version) to recognize cuda. I have cuda toolkit just installed and is version: Cuda compilation tools, release 12.1, V12.1.105 ... however when I try to run torch.cuda.Is_available() I keep getting "false". I tried doing this install but so far no variation on this seems to work: conda install pytorch torchvision torchaudio cudatoolkit=12.1 -c pytorch ... every variation of version I try reports back the version was not available on current channels. I tried 12.1 I tried 12.1.105 and 12.1.10... At this point I'm cufused as to how to get this to align properly on versions. I'm new to the environment so I'm not sure how to address this kind of issue. Any ideas?

I installed the cudatoolkit directly from nvidia website installation.

from what I can tell maybe anaconda does not support this version yet?

if so, how do I address this?

thanks in advance

Anaconda shows my pytorch version as 1.12.1

how can I find out which version of cudatoolkit to use for it?

[subtle quarry]

From docs

# CUDA 10.2
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=10.2 -c pytorch
# CUDA 11.3
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch
# CUDA 11.6
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge
# CPU Only
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cpuonly -c pytorch

https://pytorch.org/get-started/previous-versions/

[austere summit]

Ah. Thank you. I didn't know of that site. Much appreciated. :)

So from this it seems that I have the option to use 10.2, 11.3, or 11.6 cudatoolkit with pytorch 1.12.1

[austere summit]

I am trying this one: conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge

However when I run this code after the installation is complete

import torch.nn as nn
import torch

print(torch.version)
print(torch.cuda.is_available())

1.12.1
False

so for some reason cuda is still not available

[reef wigeon]

What are some good Computer expo projects to make Using Python (preferably OpenCV Module) and including Physical Computing like Arduino,nodemcu

[austere summit]

I went to the pytorch.org site and selected the appropriate (I guess) options and got this install command: conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia ... I ran it and it ran successfully. However when I run the same code as above, once again, is_available() = false.

not sure what else to try. any ideas?

[rapid sparrow]

Unity / C# may be better helped there https://github.com/mhxion/awesome-discord-communities#c

[austere summit]

Ok, for anyone who may run into this as well I found the answer in this post: https://stackoverflow.com/questions/57238344/i-have-a-gpu-and-cuda-installed-in-windows-10-but-pytorchs-torch-cuda-is-availa ... as it turns out, conda install is, well, not 100% reliable, and pip uninstall and reinstall of torch fixed the issue.

[blissful pond]

hi ngl im running kali virtual machine but it requires domain name which my computer doesnt have what to do❓

[indigo zenith]

You do not need to have your own domain name to run Kali.

[blissful pond]

tysm

[blissful pond]

and ab ubuntu

its a vm so it wont affect my real files etc innit?

[indigo zenith]

If you are sure that screenshot is running on a VM with it's own virtual drive then yes, only the virtual drive should be "erased". The "Advanced Features" button maybe has more detail to help confirm this

[blissful pond]

yep it worked appreciate u

[rough marlin]

Is there a good unifying alternative to tox and Github Actions? I've seen these two plugins:
https://github.com/ymyzk/tox-gh-actions
https://github.com/tox-dev/tox-gh
The problem with them is they seem to force dependency on tox

I've also seen act, which is a general go-based tool not specific to Python https://github.com/nektos/act

I've also seen nox, but it doesn't seem to have as strong a community

I'm also confused as to how tox manages deps / python versions

Right now, I throw my 3.8 tests into a docker container for a local run and it works acceptably at the price of having to download the base images

[rough marlin]

update: act is very, very slow for Python

[vague silo]

yeah unfortunately a good, nicely CI-integrated tool that is equally easy to run locally still seems missing

[vague silo]

i think the best „solution“ is using a makefile with your test steps and using that as your source of truth for running tests

[rapid sparrow]

https://taskfile.dev

as a better yaml makefile alternative with CI friendliness

[vague silo]

i don‘t understand how this is better than make. the CI install instructions are literally „curl someweb.site | bash“. how is that friendly? make is preinstalled almost anywhere

[rapid sparrow]

more yaml developer friendly solution

makefile has its own alien syntax hard to read and weird rules

[vague silo]

https://yaml.org/spec/1.2.2/

so does yaml

[rapid sparrow]

we already too much familiar with yaml because of CI tools ^_^ (and docker-compose, kubernetes, ansible and etc)

whole ecosystem around same.

regular makefile is unique snowflake of stuff

[vague silo]

yeah, and each of these tools require you to learn the specific yaml data layout of them

so i don‘t see how that‘s a benefit over make

[rapid sparrow]

nevertheless, some rules are same.

[rapid sparrow]

makefile is weird and does not have any comprehensive easy to read documentation ^_^
taskfile.dev has

[vague silo]

https://www.gnu.org/software/make/manual/make.html
https://makefiletutorial.com/

https://earthly.dev/earthly-core

i think this could be a good middle ground

[brazen forge]

Dockerfile and Makefile had a baby

[rapid sparrow]

i tried it, i did not like it for its... too immature state. logging building information was crippled.
Last tried half of year ago though, may be it already changed

[gentle solstice]

my task file is a folder named scripts XD

[thorny shell]

do people still use Makefiles? 😕

[rough hearth]

Quick question, im used to pipelines written in yaml in azure, is the yaml structure different in google cloud?

[brazen forge]

yes

[thorny shell]

I used to think "Make" was cool, but eventually I realized that no human is capable of writing a correct Makefile for any reasonably-complex project 😕

[brazen forge]

exactly why automake and the like were made

[thorny shell]

shudders

also, very indirectly, why "golang" was made 🙂

[tepid light]

Yes!

[thorny shell]

in new code?

[tepid light]

According to the legend - no one but the very first one has created a Makefile from scratch. All others are copies originating from that one.

[thorny shell]

🤔

[tepid light]

I think it is a common thing to have in a lot of projects and build scripts/CI stuff out there: a make test and make run is a pretty common thing, and usually wraps linter calls, automated tests etc.

It’s a pretty convenient thing that works on all machines (if we forget about windows for a while) and, yes, the syntax is not easy to grasp sometimes. Just like any shell command.

[thorny shell]

TIL

[keen current]

hi i'm unfamiliar with a new repo i'm tasked to work on, but its in python/flask and i'm a js/react dev

[empty pasture]

you wanted to link the repo?

[rapid sparrow]

u a missing to ask some question ^_^

[keen current]

sorry im trying to find a service to post the readme onto since i realized the repo is private

and making sure its scrubbed

[rapid sparrow]

post README to public or to private?

[keen current]

is this normal?

[rapid sparrow]

terrible ^_^ screenshots are bad to show text, but do go on. We still haven't seen your question

[empty pasture]

using a venv is normal for Python, yes

those numbered steps are pretty typical for setting up a project

[keen current]

ok ok good just sanity check

pardon the dumb question

[rapid sparrow]

using requirements.txt, venv is normal. simpliest choice.
The code was tested against Mysql db v8? ergh, would be nice just seeing docker-compose setting it up in one command. (and preferably unit testing in CI code)
Config files are kind of complicated i mean, may be it is required there, but if it is web application (which it is), more common to use Environment variables for configuration (because they are compatible with any deployment system) (easy to fix though)

So... very simple setup u have there, not made friendly for infrastructure tools yet 😛 (gimme ready to use docker image)

[keen current]

ill be honest i was the insane one, have not been sleeping well and fully misread the readme

did not connect some dots

[keen current]

it never actually says how to instantiate/run the app though does it?

[rapid sparrow]

Yup. That is one of reasons docker images are preferred built for applications.
They are all launched in same way
Docker images for web app, as binary for desktop apps

[keen current]

so how do i launch this puppy

[rapid sparrow]

We don't see your code to tell you ^_^

[keen current]

with js its npm run dev

ok but where is the command usually created, in js its in package.json

[rapid sparrow]

There is no one recipe for flask applications

[keen current]

defined i mean

not created

run.py?

[rapid sparrow]

We don't see your code

Flask is micro framework, each person has its own way to configure it. Egh, there is need to see where app will be created in which file and untangle from it

[keen current]

@rapid sparrow

wouldn't the start up command be in one of these files?

[heavy knot]

I think it won't run python in phone like app

It can be run in internet

[keen current]

O_o

who said anything about a phone lol

thats just the screenshot i took...on my pc

would i use flask --app run run

?

[tepid light]

I would recommend you to browse the Python Flask docs, there’s a QuickStart in there that will guide you on the basics (and your suggestion on starting up looks correct, by the way).

[rough marlin]

It's a little more similar to the github CI format.

Ty for the alternatives though!

[rapid sparrow]

Yes it may be, but exact instructions will remain unknown without seeing its code xD

Not possible to guess how to launch application, without seeing the application

Only suggesting possible highly likely solutions which I refuse to do, since I don't like to guess

[rapid sparrow]

If u want to receive blind person guesses for sometime unknown, it is start for to use chatGPT xD live people like I am, not wishing to give not precise answers

And of course good advice regarding official documentation

If u are going to deploy to prod, make sure to run with gunicorn at least

[loud bear]

how can i clear conflicts introduced in main branch of the repo (i forked) from my separate branch, so that i could make my pr happen?

[obsidian imp]

Hello everyone, i wonder if you know any good devops/cloud bootcamp on Udemy (or other platforms) for Python web applications (Django, Flask, FastAPI).
Thanks

[obsidian imp]

you can use IDE like VSCode to show every conflict and fix them.
Or you can rebase your branch on the main branch and then execute the PR

[heavy knot]

Might be a really niche question but has anyone encountered issues with the Github Jupyter Notebook viewer only showing the light theme and is there a way I can change that and some colored text output looks better on black background than white

[thorn portal]

do you have a screenshot to show how it looks at the moment?
also what are your set theme in github? do you have some browser plugin to automagically create dark theme for all websites?

[heavy knot]

This is on mobile but it appears the same in any browser and I have the basic dark theme set on Windows and any browsers I use but it makes no difference whether it's set to light, dark, or anything else and I have dark theme set in Github

[thorn portal]

that's weird.. can i have the link if it's public? just want to see if it's reproducible on my end

[heavy knot]

I can get the link, but this isn't something that normally happens with Jupyter in Github? 🤔

[heavy knot]

Link: https://github.com/SpyrossS3/RPGUniverse/blob/main/dev_notebook.ipynb

[thorn portal]

how it looks on my end

[heavy knot]

Wha

I- 😭 I'm so confused then

[thorn portal]

reproduced on firefox

[heavy knot]

Ohh I'm using Opera GX

[thorn portal]

okay i know what's up

[heavy knot]

On firefox it did look like that too yea

[rapid sparrow]

different github (+potentially browser) themes ^_^

[thorn portal]

the dark theme on your github settings kicks in when you log in (obviously), and ultimately that determines the notebook background

the dark background everywhere could be configured by your system preference of dark theme even when you aren't logged in

[heavy knot]

Yeah I have it set on my system for dark mode

But since different ppl might see it different ways, my main issue is that, as you'll see on the page, there is some lime green text and that is meant to show on a dark background

Is there a way to have that specific part shown on a black background?

Oop nvm I did it

I solved it with my Python 😌

Thx for the help!

[cedar nacelle]

looking for a job chaining or pipeline API, that could be used to interface process steps like transforms on data. So a common task like [read csv] -> (optional user transform) -> [publish] could be made plugable. So im hoping to write a plugin type of interface for user transforms. Im considering this one

https://towardsdatascience.com/python-pandas-data-pipelines-515bcc678570

and wondered if there are other suggestions that people have tired and liked? Let me know if you can recommend one. Thanks !

[tepid light]

There’s a thing called Railway oriented programming - but that’s more of a programming pattern - that is useful for chaining and transforming data. There’s libraries for this, but also quite simple to implement yourself. Maybe not what you are looking for though.

[cedar nacelle]

Yeh done it myself too but I feel that the code seems messy and non-generic. Im probably overthinking it. Just thought that an interface that you can clearly set the boundary of input and output would be nice. ill take a look at this Railway thing on the way home.

[tepid light]

I experimented with it in Python a while ago - wrote a post, added code and recorded a short video about it: https://davidvujic.blogspot.com/2021/10/a-pythonic-railway.html

[cedar nacelle]

Very impressive! thank you.

[stuck badger]

https://github.com/YashIndane/platefetcher

[vague silo]

this isn't a project promotion channel

[heavy knot]

i have this errors

help plsss

[brazen forge]

!rule 5 this may be something you're learning as a cybersec student, but it's not something we will help with in this server

[rancid schooner]

Rules

5. Do not provide or request help on projects that may violate terms of service, or that may be deemed inappropriate, malicious, or illegal.

[indigo zenith]

What happens if you try pip3 instead of pip? The warning you have there from pip indicates Python 2.7 but hopefully whatever app you're using is more modern than that

[heavy knot]

Hello everyone.
3 days ago, you can pull certain information from the website with this python script. feel free to test the tool:
https://github.com/HalilDeniz/PathFinder

[brazen forge]

Hi, this (or any other) channel aren't for advertising per rule 6.
You might find forum platforms like Reddit more fitting for advertising your projects.
Would you please delete your message?

[tame iris]

does any know what this means, if it would be potentially bad?

WARNING: Ignoring invalid distribution -rotobuf (c:\users\hp\anaconda3\envs\spyder-env\lib\site-packages)

when installing a package in my conda env

[surreal ridge]

You can design a pipeline with pandas, but scheduling/orchestrating it with something like Airflow is also great practice

[willow pagoda]

ive seen this occasionally occur i think when cancelling a package upgrade from pip, and i can only guess that its a temporary folder that didnt get cleaned up properly; ive deleted them before to no apparent issue

[alpine horizon]

does anybody here know how AWS Batche queues work?

[tame iris]

how would I go about deleting it lol

[willow pagoda]

you open the site-packages directory shown in your warning and manually delete it

[tame iris]

I don't see any -rotobuf directory in the site-packages directory is that supposed to be the name, or is it something else?

[willow pagoda]

i cant remember what i saw exactly last time it happened to me, but whatever it was seemed to easy enough to notice

[tame iris]

chatgpt says protobuf?

[willow pagoda]

protobuf is the original package name

speaking of which apparently i have some artifacts in my site-packages too, but pip doesnt give me the same warning

yup this stuff is meant to be temporary directories
https://github.com/pypa/pip/blob/23.1.2/src/pip/_internal/utils/temp_dir.py#L202-L226

[tame iris]

so what do I do then? just ignore it?

[willow pagoda]

hm try looking for google/-rotobuf

judging from their source that's where the package would have unpacked to

[tame iris]

i dont have google folder in there

[willow pagoda]

oh

[tame iris]

maybe this has something to do with it

shows different versions?

anyways I dont think it matters I'll ignore it

[languid saffron]

Help me I made a Key System for My Telegram Bot and that only owners can create, add time and duration to the key and that key is generated automatically and that other users cannot do The above mentioned and I made a command called info that tells the information of the user who uses it and I also made a range system that every time you claim that key it gives you the Rank Called Premium and also give them their plan called vip And I don't know what error comes up I almost don't know about python

[vague silo]

dude, are you okay? you read like you had some seizure

anyways, this looks like you wrote something like

def myfunction():
if user_id in OWNERS:
# ...

you need to indent it properly like this

def myfunction():
    if user_id in OWNERS:
        # ...

[versed basin]

Been trying to make Docker work for the last 4 hours or so, but it just refuses to integrate with WSL

A timeout occured while waiting for a WSL integration agent to become ready.

I did look for some Github issues and forums on the topic, but to zero avail

I'm quite frustrated and nearly punching a hole through my screen

[tepid light]

You’ve might have seen this one, looks like there are a couple of solutions/workarounds suggested: https://github.com/docker/for-win/issues/13357

[surreal ridge]

Did you try running docker without docker desktop?

Directly in WSL/Ubuntu for example, you could install docker there

You'd lose the GUI from Windows but.. GUI are problems.

[versed basin]

Already migrated to Full Linux

[surreal ridge]

Ha! That works too 😄

[versed basin]

Guess that was just the push I needed

[surreal ridge]

Did you entirely switch OS or did you just deploy an actual linux box?

[versed basin]

Dualbooting with Windows

I'm not ready to lose my Windows privileges yet

[surreal ridge]

That's dedication 😄

Just so you know, I'm also not ready to give up on windows on my personal computer. Think about deploying a very small linux box for that kind of needs as well, it will also be an awesome oipprtunity to learn about servers, containers, devops, etc

I don't know a lot of companies using WSL or offering Linux laptops!

[versed basin]

Huh, that's an option worth considering

I still haven't done the same with my secondary machine; perhaps I can try on it

[surreal ridge]

Haha be ready to fall in a gigantic rabbit hole 😄 But you learn at lightspeed when you need to be your own dev, sysadmin, network admin, etc

I started by deploying docker on an 8yr old laptop at the time, so anything works really

Setting up a remote python dev environment is also a nice little project

@versed basin don't hesitate to ping me if you want to take that path, happy to discuss

[gentle solstice]

Well, I'm bored at work. I think I'll try to fix https://github.com/dokku/dokku-scheduler-kubernetes/issues/41

[gentle solstice]

Oh... ```diff
diff --git a/scheduler-deploy b/scheduler-deploy
index c565604..123de21 100755
--- a/scheduler-deploy
+++ b/scheduler-deploy
@@ -86,7 +86,7 @@ scheduler-kubernetes-scheduler-deploy() {
fi

 plugn trigger post-deploy-kubernetes-apply "$APP" "$PROC_TYPE" "$TMP_FILE" deployment

• done <"$DOKKU_SCALE_FILE"

• done < <(plugn trigger ps-current-scale "$APP")

  dokku_log_info2 "Deploy complete"

[surreal ridge]

Autoscaling?

[gentle solstice]

Yes.

[indigo zenith]

Heheh, that's one workaround. I think maybe I've only used Docker for Windows on Hyper-V instead of WSL (but mostly use Docker on Linux VMs)

[gentle solstice]

https://github.com/dokku/dokku-scheduler-kubernetes/pull/42

ey, merged 😄

[versed cedar]

hey , anybody want to talk about opentelemetry

[versed basin]

Currently taking that path; I'm just finish setting up my VSCode and then I'll get my hands on Docker

[surreal ridge]

Nice! Did you go for ubuntu + docker?

[indigo zenith]

Promoting random projects is off topic here

[granite cloud]

Thanks for letting me know, but what do you mean by "Random projects"?

[indigo zenith]

See the topic description. Your project might fit another channel

[granite cloud]

Thanks, it's because I am new here!

[indigo zenith]

No problem. Thanks for deleting

[golden barn]

One Question: (The answer must be subjective but I want to hear you opinion)
Should we bump the version after each (feat or fix) commit? or it's ok to bump that whenever we feel we had enough changes...
because for example let say the current version is 0.2.5, and then we make 3 commits:

1. a feat
2. a fix
3. a feat
   with automatic bumping and changelog tools (like commitizen in my case) if we bump on each the result will be:
   0.2.5 -> 0.3.0 -> 0.3.1 -> 0.4.0
   but if we do it once after all 3 the result will be:
   0.2.5 -> 0.3.0

(consider that all these creates unannotated tags.)

[willow pagoda]

in the context of a library with frequent commits, i would expect a major/minor increment to have more substance than one small commit, and version numbers would be massive if we had to bump them after every commit

unless that small commit is quite important like a security fix, then you probably want to release that sooner

[versed basin]

I went Arch, as it's the system I'm most familiar with

Managed to get it to work, but I'm not entirely sure how to proceed from here 🥴

I got a container going, but still trying to figure out what to do from now on

[glass shoal]

I need some help with my Python

[surreal ridge]

Remote dev container @versed basin? Are you already selfhosting your own git, maybe GitLab?

[versed basin]

Nope, still don't have anything important going on

I am quite clueless about where I should go from now on

This is supposed to be an isolated instance of vscode, right?

If so, I guess I could re-import my extensions back into this container

[surreal ridge]

Yeah, now your interpreter is running in a remote container. From here you can start creating custom images with your preferred stuff.

[versed basin]

It's gonna be a long road for me to figure out what exactly I'm doing

[versed basin]

In this case, I used "ENter a Python module instead..."

[viral marlin]

So let's do a basic chance to your dockerfile and change the CMD it's running

[versed basin]

As I didn't have any files

[viral marlin]

If the file your running isn't a module (aka no __init__.py) then you don't need the -m

[versed basin]

Removed it

[viral marlin]

What is the file called that you're trying to run?

[versed basin]

test.py

[viral marlin]

So the CMD should be:
CMD ["python", "test.py"]

[versed basin]

Done and executed the file, it worked

[viral marlin]

For understanding what each line does, Docker's documentation is actually pretty good for it's Python example: https://docs.docker.com/language/python/build-images/

[versed basin]

🙏

I'll take a look, thank you for your help

Really appreciate it

[viral marlin]

Docker (containerization more generally) is really useful, but can be finnicky

[versed basin]

I'm trying to get something going so I can get familiar with the tools

[deft dock]

So I downloaded a git and the developer changed something in the code and wants it too, I did too, but now I have the new code and the old code, what should I do?

[graceful rose]

That looks like you have the repository nested within itself. Which is the new code and which is the old code?

[deft dock]

the left ist the old and the right the new one

[graceful rose]

This appeared after you ran a git pull?

[deft dock]

yes

[graceful rose]

Well, that might be part of the changes the developer made. It's hard to know without a link to the GitHub repository itself. But for working with the new code, you want to go inside the new folder and work from there

[deft dock]

It's just stupid because I adapted the code myself

[versed basin]

@surreal ridge Heyy

So, I managed to get the container going with a folder

Soo yay, progress

I still need to figure out some stuff before I'm fully able to get anything running with it, but it's looking half good

[heavy knot]

yo guy anyone wanna do one little project whit me

is for my company i wanna do one app

to make work easier

if is that right chat room for that

[rapid sparrow]

!rule 9

[rancid schooner]

Rules

9. Do not offer or ask for paid work of any kind.

[rapid sparrow]

this server is not appropriate place for paid tasks to be offered

[heavy knot]

i will not pay

[rapid sparrow]

you are offering stuff of the level of a paid job at this point

even worse

[heavy knot]

oh

im just learing

python

need some help

[rapid sparrow]

helping person for entire project for some company for free, is not really appropriate you know

https://wiki.archlinux.org/title/General_guidelines
https://slash7.com/2006/12/22/vampires/

learn how to ask help in a good way, don't be a help vampire

[heavy knot]

oh didnt know it look like that

[rapid sparrow]

ask what you want in specific way, something like

I receive X (errors X, T, P). I work in OS T, i have frameworks of version H and etc

I want Y to do to fix it, because i think it is a solution

That is happening because my real goal is Z

I tried solutions A, B, C but they did not work. I was not able to Google my problem

Here are my code examples i think are relevant
(Possibly links to GitHub, repositories, code examples)

Help me please in some kind of wording xD

[heavy knot]

ph

ok

[rapid sparrow]

there is more detailed guide @heavy knot highly recommending to read ^_^