tools-and-devops

[tawdry needle]

you should not need the privileged or network options

this is why i use compose even for projects with 1 image/container/service. i prefer a declarative config instead of a big messy command line.

[red oxide]

Here's my settings:

Dockerfile

FROM python:3.10.4-alpine3.15
WORKDIR /app
COPY . .
RUN apk add build-base musl-dev mariadb-connector-c-dev  # You don't need this if you are not running DB
RUN pip3 install -r requirements.txt
CMD [ "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "1111" ]

docker-compose.yml

version: '3.1'

networks:
  oxy-node-net:
    name: oxy-node-net
    driver: bridge

services:
  oxy-node-server:
    build: .
    image: oxy-node-server:latest
    container_name: oxy-node-server
    restart: always
    networks:
      - oxy-node-net
    expose:
      - 1111
    ports:
      - 1111:1111

as @tawdry needle mentioned, use docker compose, makes things much easier

[tawdry needle]

doesn't compose implicitly create a "network" if you don't specify one? or do you just like having control over the name?

same goes for the image tag and container name

[red oxide]

I like having control over names cause now I can run another compose with same network work name if I would like to share it with some other app

very helpful under situations when, eg:
You run nginx-proxy with acme and stuff first under a network, then you can use the same network and start individual apps whenever you wish

[tawdry needle]

i like this, i'm going to start doing it. handy with multiple compose-based projects on the same machine

[red oxide]

yup, very much!

[tawdry needle]

i also just learned from the spec that you can set a top-level project name with name: in the compose file

i've been using the cli flag until now

[red oxide]

by default it uses the project folder name i guess?

[tawdry needle]

unsure!

[red oxide]

I saw this behaviour on docker desktop, let me try

indeed

version: '3.1'

services:
  my-ip:
    build: .
    image: my-ip:latest
    container_name: my-ip
    restart: always
    networks:
      - nginx-net
    expose:
      - 8000


networks:
  nginx-net:
    name: nginx-net
    driver: bridge

[red oxide]

Can pi-hole mess up the internet inside containers?
I'm running pi-hole on my pi4, I also wanna run speedtest exporter with prom and grafana but I notice I am not getting internet in any of my containers which just results into errors in speedtest exporter container. I checked my pi-hole query logs and dsn seems to be resolved there:

inside rpi speedtest exporter container

[heavy knot]

im getting this error how can i fix

[rapid sparrow]

Read some documentation to proper TOML format, and compare with what u wrote
Install TOML syntax linters to show u where are errors automatically (works great in vscode)

[tawdry needle]

it's interesting how many people come in here and ask questions because they simply did not read their own error message

[tropic olive]

I did that as well

[blissful ice]

Hi, I'm having trouble understanding how publishing packages to PyPI works. When I use pypa/gh-action-pypi-publish in GitHub Actions for uploading packages, how does PyPI determine the project to which the uploaded wheels belong to? There is no "create project" function in PyPI, and none of the GitHub/twine tutorials mention any kind of "target" to publish to, other than using the test-repository. And this one is giving me a

HTTPError: 403 Forbidden from https://test.pypi.org/legacy/            
         Invalid or non-existent authentication information.

[tawdry needle]

this error looks to be due to incorrect authentication setup, as per the error message

if the sdist or wheel is built correctly, the project name should be possible from the filename

[blissful ice]

Ah, so if the package-name determines the project under which the package lives within PyPI.. there are probably some restrictions for uploading packages with different users? I.e. PyPI probably won't let Alice upload a new package with a project-name that is already in use by Bob?

[flat path]

Yup, it's first come first serve

I'll note the distribution/project name is not always the same as the importable name

[crimson stump]

Currently struggling with a basic docker command but its not working, i have a wrapper script i've built which is in the dockerfile, called init.sh

I am trying to alias it but the alias isn't showing up in the file i am appending it too, am i doing something wrong?

RUN echo "alias init='/usr/bin/init.sh'" >> ~/.bashrc

When i check in my pipeline and read the output of bashrc i get

$ cat ~/.bashrc
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
# User specific aliases and functions

Figured this out

~ was causing the issue as its a unicode character

Now however i get the following 😭

$ init
/usr/bin/bash: line 182: init: command not found

[rapid sparrow]

mkdocs - love from first site
https://www.mkdocs.org/getting-started/
So simple.... yet powerful. Good thing when Sphinx is too much

[thorny shell]

if it's called init.sh, then you can't just type init; you have to spell out the complete name, including the .sh

[crimson stump]

i just read apparently it wont work at all because of how docker filelayers work 😦

so the alias wont register in the bash session for the pipeline

[crimson stump]

RUN echo "alias init='/usr/bin/init.sh'" >> ~/.bashrc

its called init.sh but the alias is just init?

[thorny shell]

aliases make my brain hurt; I never use them

also, they're meant for interactive use only, iirc; it's odd that you're creating aliases in a dockerfile -- most of the time, Docker is for (say) services, not interactive use.

[crimson stump]

im just really bad with bash but i really need a way to call a command in a pipeline which points to a script 😦

i thought alias would achieve that

Do you have any other ideas @thorny shell ?

I've just been struggling with this for days

omg you were right @thorny shell i dont think i even need the alias

i have init script stored in /usr/bin

so i can just call it

ffs

i just wasted 2 days

[thorny shell]

yeah I don't understand what you're trying to do, but ... aliases ain't the way 🙂

[crimson stump]

urghhh

thanks mate

[thorny shell]

heh

[crimson stump]

just need to set the permission for execution right

[crimson stump]

just in case you are wondering, preventing devs from modifying compliance testing within a pipeline, so wrapping up all the compliance testing in a script and executing it in a single command

[thorny shell]

you must work where I work 🙂 I spend 50% of my time with compliance

[crimson stump]

Haha

Lot of OPA yeah

OPA is like my strong suit

[thorny shell]

no idea what that is, but ... please don't explain

[crimson stump]

But i hate pipelines, in the wrong industry i think

hahaha

[thorny shell]

it'll just make me grumpy

[crimson stump]

Look into it, open policy agent, basically can traverse through any JSON or YAML and allows you to create custom rules

quite a popular devops tool these days

[thorny shell]

oh pipelines make sense to me. I don't particularly like dealing with them, but I will admit they solve a real problem, and I can't think of a better way to solve it

[crimson stump]

yeah, they do, but unless things are planned really well in advance the tech debt is astronomical over time

pipeline and solution planning within devops is like the one thing id advocate all companies spend more time than they think is necessary brainstorming with as many minds as possible

just seen so many botched platforms over the years ive worked

admitedly not that many yet 😄

[weak heath]

is this correct way run docker when starting file is in backend folder last line

[rapid sparrow]

Lock node to specific version
Don't copy . ., Copy specific stuff
Otherwise fine to me. Except expose does not make to much sense to have

[weak heath]

ok

[gentle solstice]

I usually have a PORT env var. Services like heroku tend to use those to tell your app what port it should use. And I don't think heroku does proper port forwarding via -p 1234:80

That's also assuming your app loads the port value from env like gunicorn does

[copper tundra]

Hi, really basic git question (figured I'd ask here as git is technically a tool)

Just as you can pin repos to your github profile, is there a way to pin a commit/PR to your profile?

Made a trivial PR to the cpython repo and after about 1.5 months it was accepted. Can I pin that PR or commit to my profile?

[rugged hare]

That's a GitHub question, not a git question. I don't think you can pin commits.

But your profile can have anything you want in it.

[copper tundra]

Sorry, you are correct I had a GitHub question

[rugged hare]

You can write your own readme.md to be your profile, and put what you want into it.

[copper tundra]

Ok thanks, didn't know if there was already a builtin way using the pin repo feature to pin a commit/PR

I'll look into the profile README

[rugged hare]

@copper tundra https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-github-profile/customizing-your-profile/managing-your-profile-readme

[loud bear]

You can pin an issue or PR in your repo, not your profile unless you link it manually

[copper tundra]

Thanks @rugged hare, will take a look

Unfortunately @loud bear it's the official cpython repo so can't pin the PR, but thanks for the info good to know

[loud bear]

Why would you want to pin a commit /PR in your profile tho

Ahh ok understood

[copper tundra]

i'm a fairly new open-source contributor and its the first "big" pr I've made to a notable organization

[crimson stump]

For anyone decent with Docker, why doesn't the bridge network, automatically at the IP hostname mapping to /etc/hosts on the Docker VM so that bridge network containers can communicate with each other via hostname, not just IP?

Also why does the bridge network not have its own DNS server included, it uses the Docker VM DNS server but additional networks created do?

That is also confusing

[thorny shell]

by "the docker VM" do you mean "the machine on which I run the docker daemon"?

The answer is probably: because it doesn't have root access

[crimson stump]

Yeah i assume so

Well I'm watching a tutorial

And its telling me the Docker containers sit on a VM, by VM i guess it just means the device you're using then

[thorny shell]

I guess

"sit on" isn't exactly a technical term, so who knows what they mean

[crimson stump]

Okay that makes sense, no root access. But why not come with its own DNS server to handle it?

[thorny shell]

er

[crimson stump]

And it necessary go through the bridge network and use the host DNS server

If new networks do have their own DNS

[thorny shell]

are you asking about name resolution on your laptop, or inside the container?

those are two different things

[crimson stump]

Sorry inside the container

So i have two containers

They can't communicate via hostname if they both sit within the bridge network

But they can if they are inside a custom network

Typo

[thorny shell]

huh, well I don't know enough about docker to understand how they can resolve each other's names

[crimson stump]

Because apparently custom networks have their own DNS resolver

[thorny shell]

I dimly recall there's a platform-specific trick whereby you can get some magic name to resolve, inside the container, to the IP address that corresponds to your laptop

dunno if that's helpful

[crimson stump]

It makes sense i was just curious, what is the limitation of the bridge network that means it cant have its own dns resolver, where custom networks can

but yeah thanks mate

I managed to fix that docker thing pretty quickly after your tip yesterday lol, just completely unblocked me

[thorny shell]

yay me

[dawn sedge]

Hello, somebody know another alternative of agora to play músic in another app?

[leaden tapir]

does anyone know why does buildozer failed to compiling a kivy project?

[dawn spade]

Hiii !!
I've to create an automated code.......to calculate the number of commits for each pull request which every user does.
So I've to fetch the data from the bitbucket repository and calculate the number of times the changes have been done to each pull request
Basically calculating the build statistics corresponding to each pull request and the total number of times build has triggered. What should be the approach ?

[loud bear]

How do I pin a dependency to use latest git commit in requirements.txt?

[brazen forge]

?ref=main
or whatever the main branch is named

although, it should just work

[loud bear]

It will just install latest published version if I don't use anything

Do I use the pip install git+ syntax?

[brazen forge]

yeah

[glacial sand]

I don't have version dependent code, should I run coverage on every tox test environment anyway ? Currently I only use coverage for the latest python version

[rapid sparrow]

why do u not have version dependent code, but having tox tests and therefore code?

something potentially wrong was made steps before that

[glacial sand]

It's to automate testing with gh actions and test with multiple interpreter versions. Also helps as sanity check to ensure it builds correctly everywhere (isolated build)

[rapid sparrow]

so u a having version controlled code then

u can't be in gh actions without git

i don't understand your question then

[glacial sand]

Oh yes, I meant that the code is not python version dependent (i.e. it doesn't do different things on different versions of python)

of course it is version controlled

[rapid sparrow]

well, run coverage at least once for one python version. All u really need having it outputed and easily being visible

[glacial sand]

yes that's what I'm currently doing, was just wondering if there would be a reason to run cov on every version

[rapid sparrow]

technically there is, in different version different code will be covered from having executed different amount of code, but i would say difference will be so minor and rare that i would have prefered to ignore it

unless u are building 100'000 code lines+ software in one repo, it should not be important

[glacial sand]

All for the better, fewer things to wait on 🙂

[urban needle]

Hello, in Docker I have a question:
FROM is for getting an image, and I've read that docker will automatically use the last FROM as the image, what if I want to get the python image into a ubuntu image? or a mono image to the ubuntu image?
like can I do
FROM ubuntu
FROM mono

• when I build a docker image downloading these packages it just takes way too long to build

[indigo zenith]

No, one image per container. If you need a custom image you can build it yourself. https://docs.docker.com/develop/develop-images/baseimages/

[urban needle]

so its about getting a base image or create from scratc h and downloading packages until the desired image satisfied?

[rapid sparrow]

essentially yes, but not always
your base image could be only BUILD stage, where you compiled result to binary
or made build of static frontend assets

Results in multi stage building is moved from base image into another layer after that
Example: First FROM contains node.js building frontend into static assets, second FROM contains Nginx to serve those static assets as web server.
Together they are still one multi stage dockerfile

[urban needle]

does the second track the first?

[rapid sparrow]

here example for you

FROM node:lts-alpine3.12 AS build

COPY package-lock.json package-lock.json
COPY package.json package.json

RUN npm install

COPY . .

RUN npm run build

FROM nginx:1.21.3-alpine-perl As runner
COPY --from=build dist/ /usr/share/nginx/html # we copied from Build stage dist/ folder into new FROM stage

[rapid sparrow]

After u apply FROM second (or more) time, final image will not have image layers from previous steps before that

this is used to minify size of final docker image

[urban needle]

wait like the first image is just used to download packages then, the second image just forgets about the first but still have the packages?

[rapid sparrow]

in my case, first image downloads packages and uses them to compile frontend content into static assets html/css/js of a small size

second image gets copied only html/css/js static assets and forgets about first image ( that it had packages to build them)

thus having very small size 🙂

[urban needle]

the build image owns its own directory right?

[rapid sparrow]

each FROM contains its own filesystem inside. So yes, it has entirely different OS inside

[urban needle]

ooh okay, so this filesystem is for the build image right?

--from= build

bcuz u specified AS Build

[rapid sparrow]

yeah, as u can see i made alias FROM node as build

[rapid sparrow]

i specify COPY from build stage in this case

[urban needle]

yes yes, okay then u copied that diretory to the current image were working on

[rapid sparrow]

yup. And current image is having entirely different filesystem

of Nginx image

i copied result to where Nginx expects files to serve as default web site location

[urban needle]

yes, I got that! Thanks, but I have another question.

[rapid sparrow]

sure

[urban needle]

why downloading packages just takes too much time for me. I dont understand if I am doing something wrong because its literally taking more than 2k+ seconds and I stopped it.
I tried to apt-get install packagees like tkinter

I use a ubuntu image

[rapid sparrow]

Umm. somewhere error i guess. Share code to see

[urban needle]

Okay just a min

FROM ubuntu
RUN apt-get update \
    && apt-get install --assume-yes \
    python3 \ 
    python3-pip \
    git \
    python3-tk

WORKDIR /app
COPY . /app

CMD python3 ./python-scripts/script.py

[rapid sparrow]

Also recommendation to deepen knowledge with this awesome book btw

[rapid sparrow]

print your log output

[urban needle]

its just keeps Building:

Building 1852.1s (5/9)
>
>
>
>
>

[rapid sparrow]

error is found. u would have found it if... u have read log

Setting up tzdata (2022c-0ubuntu0.22.04.0) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
Configuring tzdata
------------------

Please select the geographic area in which you live. Subsequent configuration
questions will narrow this down by presenting a list of cities, representing
the time zones in which they are located.

  1. Africa   3. Antarctica  5. Arctic  7. Atlantic  9. Indian    11. US
  2. America  4. Australia   6. Asia    8. Europe    10. Pacific  12. Etc
Geographic area: 

it encounters user interactive step

therefore unable to proceed next without user input

it is solved with ENV flag, a moment

[urban needle]

oooh okay, I have used assume-yes

[rapid sparrow]

not enough

RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get -y install tzdata

install like that before that

https://serverfault.com/questions/949991/how-to-install-tzdata-on-a-ubuntu-docker-image easy to google tzdata docker

[urban needle]

okay what about other packages how do I know if there is parameters?

[urban needle]

btw I put this before my main RUN right?

[rapid sparrow]

yes, like that

FROM ubuntu
RUN apt-get update
RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get -y install tzdata

RUN apt-get install --assume-yes \
    python3 \ 
    python3-pip \
    git \
    python3-tk

i would like also to note that u were... using

RUN apt-get update \
    && apt-get install --assume-yes \
    python3 \ 
    python3-pip \
    git \
    python3-tk
``` multi install in one command without knowing how it works in regards to docker

[urban needle]

Okay.... does that makes a problem?

[rapid sparrow]

a moment

[rapid sparrow]

okay, if u have code that it not trying to minimize docker image size, then it is more comfortable to have those steps on separate RUN commands (easier to debug usually)
But if u are trying to optimize Docker image size

https://www.metricfire.com/blog/how-to-build-optimal-docker-images/
Then it makes sense to have appropriate Apt cache deleting instruction at the end (rm -rf) (for ubuntu can be different)

FROM debian:stable

WORKDIR /var/www

RUN apt-get update && \
    apt-get -y --no-install-recommends install curl \
        ca-certificates && \
    curl https://raw.githubusercontent.com/gadiener/docker-images-size-benchmark/master/main.go -o main.go && \
    apt-get purge -y curl \
        ca-certificates && \
    apt-get autoremove -y && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

The point is Docker saves new Layer as RESULT of complete step completion. So the size increase of image only for files that changed/were added in step
That's why people compress into one step apt update, installations and deleting apt cache, so that it will not affect image size increase further

[urban needle]

so seperating RUN commands increase performance (w/cache) and decreases image size?
so, doing mine will naturally take that long?

bcuz its still taking way too long 500+ seconds until now

[rapid sparrow]

no no. Separating RUN commands is just easier to debug
But combining them into one allows to decrease image size if u used correctly to clean cache in it (intermediate files appearing in current step)

[urban needle]

oh okay yes I understood what u mean

[rapid sparrow]

500 seconds? that's weird. My PC builds your image in.. 60 seconds

may be not enough CPU u have

it can build faster though

[rapid sparrow]

reuse common existing official docker images

FROM python:3.10

RUN apt-get update \
    && apt-get install --assume-yes \
    git \
    python3-tk

WORKDIR /app
COPY . /app

CMD python3 ./python-scripts/script.py

10 seconds build now at my PC

google Docker hub python (or anything else) to find stuff

[urban needle]

Okay so it differs for CPU capiabilties

[rapid sparrow]

u install python in the way it required C compiling. It was very CPU heavy

[pine nacelle]

I need help I created an azure VM and I am not able access its server which i hosted using go

[urban needle]

is it possible to do that as an image alias so I can use python in my ubnutu image?

[rapid sparrow]

read docker hub explanations / dockerfile
python:3.10 is essentially debian image + installed python as far as i know

there is a choice to choose alpine based pythons too though

with different tags

debian is essentially same ubuntu, same OS family

[urban needle]

okay thats better so python image is enough

I also have another question, f its possible I think I took lot of time for that sry

I wanted to run a graphical program with mono, but it went wrong because there is no x server or similar. Also, is this for all graphical programs if I want to run with docker?

[rapid sparrow]

Graphical programs aren't meant to run in Docker. It is possible though at advanced level if u will pass necessary devices from host into containers (some perverts even serve Linux Desktop in container and allow remote connection into it)
Docker is meant to serve web (or CLI) applications, frontend, backend. If u want serving GUI with docker, make localhost served website that serves as a GUI to your service inside docker, that's how pgadmin does for example

[urban needle]

Okay, Thanks! very informative. Also, thanks for the book recomendation!

[tawdry needle]

what was your original objective? on linux for example, if you just want some amount of dependency isolation and maybe a little sandboxing, you can do it in flatpak or appimage

although i have gotten discord and spotify running in podman on fedora, under x11 (not wayland)

the latter was nontrivial, i'd have to dig up my files to show how it worked

[urban needle]

Theres a program that I want to make a container for it since it depends on other things to be run and installed, but its not that easy right?

[indigo zenith]

I doubt anyone can help without more information. How exactly are you trying to access this server. What does "hosted using go" mean exactly?

[ruby fossil]

I need some help with mypy

I tried making a custom log record factory and this happened...

[tawdry needle]

did you name your own module logging.py?

[ruby fossil]

Idk why this happens but it does

[ruby fossil]

I did but it is under src folder

Something like this

Should I change the name?

[tawdry needle]

yes. if logging is a python package in the python search path, then it will take precedence over the built-in one

[ruby fossil]

Alright

[tawdry needle]

you can put it inside another package, but it can't be a top-level name

so you can do this

src/
  myapp/
    __init__.py
    logging/
      __init__.py
      custom_record.py

because the module name will be myapp.logging.custom_record

but you can't do this

src/
  logging/
    __init__.py
    custom_record.py

[ruby fossil]

Ohhh

But why does that src folder not count

Like, my configs are in a folder called Applic and it has src as subfolder

something like this

Application root
 |- src
     |- logging
 |- config files

Another question but it's probably for #type-hinting

[fallow smelt]

Is there any way to get VSCode to show each parameter on a new line in the hover tooltip?

[rapid sparrow]

do u mean, showing parameter value during debugging session in visual interface?

[fallow smelt]

Would prefer if each property was on a new line since kinda complex signature

[rapid sparrow]

if u need reading complex signature, use Show definition functionality and traverse through the function code that made this tooltip

[rapid sparrow]

Document your function in docstrings

First string in the body of a function works as documentation added to tooltip too

It can be even used later in autodoc of Sphinx to autocompile static asset documentation after that

[loud bear]

It doesn't work

https://github.com/demberto/PyFLP/actions/runs/3322488584/jobs/5491578753

[brazen forge]

try

construct-typing @ git+https://github.com/timrid/construct-typing

[foggy snow]

Hey folks , can anyone help to get commit history between jenkins builds ?

[south forum]

Hey folks,
I run a py script with selenium to get some data.
I run this in a docker , and the official standalone-selenium-chrome docker with "docker compose".

Problem is: I get a lot of errors by selenium..
"ConnectionRefuseError: [Errno 111] Connection refused."
"urllib3.execptions.NewConnectionError: ... failed to establish a connection "
"urllib3.execptions.MaxRetryError: ..."

Anyone experience with setting up a selenium web scraper in docker? would appreciate some help. Thx

[loud bear]

Works. found it documented here https://pip.pypa.io/en/latest/topics/vcs-support/#git

[rugged flame]

Would there be any advantage to an organisation grouping together all of their packages into a single namespace package?

Also disadvantages.

[thorny shell]

if you mean "every python package that we produce in our org will provide modules named lx-org.something, that sounds reasonable

[rugged flame]

Yeah that's what I had in mind.

[thorny shell]

disadvantages: you already have tons of existing packages in random namespaces, and converting them will be a pain.

advantages: sanity, simplicity, stuff should Just Work® if you ever open-source it

[rapid sparrow]

There is recommendation to do it not through single folder defining all of them
but through defining shared_namespaces.your_package_name as a single name for folder
In this way u will keep ability to have their... locations independent

if u will ever do it

[silver cradle]

how do i upgrade pip in windows terminal?

[rapid sparrow]

it can self upgrade itself, python -m pip install --upgrade pip

[silver cradle]

my terminal keeps saying no module named pip (normal & admin priv) but my ide pycharm is using pip

https://pastebin.com/RriTaxjG

getting this too

i managed to update it using pycharm

im trying to install PIL now tho, and its saying this
https://pastebin.com/YubDi4h2

[civic grove]

try pillow instead of PIL

[leaden flame]

Hey guys, I'm working on a project that's actually not a programming project per-se but rather a guide for beginners on all the tools that will help them in their programming journey. Think like project templates, website bookmarks, blogs to follow and read, command line tools, desktop tools, things like that.

It's most similar to "tools and devops" subject-wise, but it's not just that

I haven't yet decided if it's going to be a book, or a course, or some other form yet, but I'm getting together all the information for it

And for now I'm primarily targeting windows and linux (I don't have a mac to try to find the mac alternatives yet)

So I was wondering if I could get some suggestions from you guys on what sorts of tools or things I might be missing

The idea is a beginner can pick this up and have the tools and know how to use them to be effective as they first learn programming, and then as they write programs and software as a developer.

What I have so far is...

First things First

• Chocolatey & Chocolatey GUI

System Tools

• Powershell 7 - Latest Powershell Update
• WSL 2 - Linux on windows
• Autohotkey - A programmable hotkey and automation tool
• Power Automate - Automate repeated tasks on your computer
• 7-Zip - Easily unzip anything from the context menu
• Winaero Tweaker - Make tweaks to the way Windows behaves
• Powertoys - A bunch of cool extra hidden features

Command Line Tools

• Posh-git - Shows your git status in your powershell prompt
• Win32 Peco - Pico editor for Windows
• Ripgrep - Grep for windows, better than GNU
• GNU Sed - Stream EDitor
• SnakeTail - Tail clone for windows
• Neovim - Terminal Power-Editor, Configured by lua scripts
• GNU Awk - Programmable Text processor
• Github CLI - Programmatic Access to Github from the command line
• Git - Distributed Version control system
• GNU WGet - Download files from the internet directly
• Curl - programmatically interact with web service
• Selenium - programmatically interact with websites

Helper Programs

• Ditto Clipboard Manager - Saves a history of clipboard clips
• XMing - Allows you to use GUI programs in WSL
• Keypirhanna - A highly configurable launcher
• Quicklook for Windows - Allows you to press spacebar on a file to preview it
• Wincompose - Allows you to compose special characters from the keyboard
• (Not Free) - Insync - Manage your Dropbox, Google Drive, and OneDrive
• Just Color Picker - detects the color value on screen and gives you various formats to replicate color

Primary Software

• VSCode - Super Powerful Code Editor with extensions and JSON config files
• Pencil Wireframes - Free tool to wireframe a program you're writing
• LibreOffice - You primarily want the drawing and math tools
• Kitty SSH - A Better Putty that will allow you to connect to remote servers
• WinSCP - Copy files using SSH
• Notepad++ - Good all around text editor
• Adobe Acrobat Reader - To read PDF files
• SQLite Studio - Open and mess around with SQLite data files
• HxD - Hex editor, very useful
• Ghidra - Reverse Engineering
• Gauge - Test Automation Tool
• NSIS - Installer Package Creator
• Obsidian - Markdown Based Notes in a hive
• XMind - Mind Mapping software

Online Accounts and Services

• Github Account
• Stack Overflow Account
• Gmail Account
• Discord Account
• Slack Account

[rancid schooner]

:incoming_envelope: :ok_hand: applied mute to @leaden flame until <t:1666830204:f> (10 minutes) (reason: links rule: sent 24 links in 10s).

The <@&831776746206265384> have been alerted for review.

[leaden flame]

I also have a set of suggested sites/pages to bookmark and how to organize them:

Links to Setup in Browser Toolbar

• Grimoire -> Links to articles and stack overflow posts
• Servers -> Links to manage servers and to connect to web interface for them
• Documentation -> Links to sites that host the documentation for a product
  • VSCode
  • Markdown
  • Markdown Guide
  • JSON
  • XML Official Docs
  • W3Schools
  • Chocolatey
  • MDN
  • Powershell
  • Power Automate
  • Powertoys
  • Winaero Tweaker
  • Windows Subsystem for Linux
  • AutoHotKey Documentation
  • Posh Git
  • Ripgrep
  • GNU Sed
  • GNU Awk
  • Neovim Editor
  • Github CLI Documentation
  • Git SCM
  • GNU Wget
  • cURL Documentation
  • Selenium
  • Ditto Clipboard Manager
  • XMing X Server
  • KeyPirhanna
  • Pencil Wireframes
  • LibreOffice
  • Kitty SSH
  • WinSCP
  • Notepad++
  • SQLite Studio
  • HxD Hex Editor
  • Ghidra
  • Gauge Test Automation
  • NSIS Installer Creator
  • Obsidian
  • XMind Mind Mapper
  • Docker
  • Microsoft API and Reference Catalog
  • IETF RFC Database
  • Regular Expressions
  • Geeks For Geeks
  • BEM

• Accounts -> Links to sites that you can log into
  • Github
  • Stack Overflow
  • Gmail
  • Discord
  • Slack
• Repositories -> Links to places to download software
  • Chocolately Community
  • Windows Store
  • VSCode Extension Marketplace
  • Ubuntu Snaps
  • Ubuntu Deb Packages
• Projects -> Project Hosting sites
  • Read The Docs
  • Documentation.Help
  • Github
• Tools -> Sites that work as tools for you
  • MXToolbox
  • DMarcian
• Stock Sources -> Sites that provide images, drawings, sound effects, music, fonts, etc
  • Unsplash
• Content Delivery Networks -> Delivery Networks to access libraries
  • Bootstrap CDN
  • jsDelivr
  • CDNjs

Extensions and Settings

• VSCode Extensions

• Windows Settings and little seen features

• Fonts
  • Fira Code Font
  • Places to get Fonts
    • Google Fonts

File Formats to Learn About

• Markdown
  • Markdown
  • Markdown Guide
• JSON
  • JSON
• XML
  • XML Official Docs

[rapid sparrow]

Idea is nice of course, but i am kind of strongly disliking how you threw into one pile ecosystem of Windows and Linux together

[leaden flame]

This is literally a brainstorm

Also WSL is a thing

[sly sleet]

w3schools

geeksforgeeks

[sly sleet]

aside from half of the cli tools, i dont think any of them are linux

and these are undercategorized

b/c you have web dev tools mixed with general software mixed with desktop app tools

[sly sleet]

...but you mention plenty of tools that are obsoleted by wsl's existence

[knotty kettle]

hey all, had a quick Jenkins question. has anyone run into an issue where Jenkins doesn't inherit system environment variables? I recently upgraded our Jenkins instance and now none of our jobs are seeing system variables (this used to work)

[tawdry needle]

i would distinguish "important utilities that you are likely to encounter everywhere" (git) from "nice fun cli tools" (posh git)

also posh git is just for powershell

ideally you would use tags instead of or in addition categories

e.g. posh git could be tagged with git and powershell, for example

[graceful light]

For having multiple Python versions installed on a Windows system, what's the best way to manage setting up / running different versions through something like poetry?
I've heard of pyenv but seems to not officially support Windows

[rapid sparrow]

Clarify usage case better
Different python for different projects
Or testing same project with different pythons

[graceful light]

I have different projects with different dependencies, where I want to upgrade to the newer versions of Python, and others I have to wait until the associated libraries release compatible versions

[rapid sparrow]

I solve it with docker and CI
For u I would recommend trying to solve with Poetry first.

Poetry is very nice package manager, best today

U can set Python version in its config

It will auto create venv with necessary python for project

[graceful light]

The issue is that installing multiple versions of Python on windows and adding them to the env path means that I HAVE to use poetry to make sure I'm using the correct version of python, otherwise I would be using the newest one by default in a non-virtualenv situation
My cheap workaround is to make a copy of each python executable and rename them to their version (IE Python 3.9 would go from python.exe to python39.exe

[rapid sparrow]

In Linux installed new pythons are actually by default gaining corresponding version namings xD
python3.9, python3.10 and etc

So u repeated Linux solution

I used at the first it with related venvs

[copper tundra]

Try asking in the Kivy discord, they have support channels for questions just like these :)

[tawdry needle]

just install them using the official installers, as long as they have different minor versions (3.9, 3.10, etc) they won't collide at all. use the py.exe launcher instead of invoking python.exe, py.exe is basically pyenv for windows.

there's also a windows port of pyenv

ideally you don't need to add any python.exe's to path, you just use py to create venvs, and invoke those directly

things get a bit trickier when using poetry i suppose

but if you install poetry into a venv to begin with, i think it will "do the right thing" and reuse that same venv. not sure if it requires the venv to be active though

[graceful light]

Oh my god, I didn't know py did that, I just thought it was a shorthand for existing python.exe, this fixes my issue completely, thank you!

[tawdry needle]

yeah, check py -0p to see a list of all the python installations that it can detect on your system

[graceful light]

My only remaining question is how to get VS Code to see that poetry has a virtual env to enter into - sometimes the prompt comes up when I create the env, but I don't know how to have VS Code enter it otherwise
And for context, I don't just mean having poetry's virtual env usable in the VS Code terminal, I mean VS Code seeing the imported libraries of the virtual env for autocompleting 3rd party library methods (IE: doing from tqdm import tqdm and then typing tqdm() would show you the available arguments).
Without the virtual env, the autocomplete stuff doesn't happen unless I install the library globally

[tawdry needle]

i believe you need to tell vs code to use the python.exe that's in the venv

[frigid plover]

Hello Everyone
anyone here who have taken kodekloud devops course ...

If yes, please let me know
I have some queries

[unreal fable]

anyone using airflow?
im a data engineer and running one on vineyard
trying to clear my xcoms like this

@provide_session
def cleanup_xcom(context, session=None):
    for task_instance in context['dag'].tasks:
        XCom.clear(
            task_id = task_instance.task_id, 
            run_id=task_instance.run_id, 
            dag_id = task_instance.dag_id, 
            execution_date = context['execution_date'])

but when a task finishes with failed state, the callback fails with

AttributeError: '_PythonDecoratedOperator' object has no attribute 'run_id'

so how shall one pass run id here ? im stuck

or if anyone has a debug environment set up, would help me too

[loud bear]

So TIL pypi doesn't allow publishing wheels having direct git dependencies

ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
         Invalid value for requires_dist. Error: Can't have direct dependency: 'construct-typing @
         git+https://github.com/timrid/construct-typing.git@ce871936d9e885c84f2f507c723fc5a7e99c9096'

[thorny shell]

not surprising; many git repos are non-public, pypi probably can't (or doesn't want to) figure out which are and which aren't; and they certainly don't want to distribute a package whose users will run into a 404 when trying to install it

[loud bear]

isn't it super simple to find that a repo is non-private?

what should I do now? i already told the library author about this

[thorny shell]

no idea, sorry

[thorny shell]

it'd take time and network bandwidth. Sure, not much of each; but they'd want to do this for every project, and they'd want to do it periodically, since a repo that's private today may become public tomorrow, and vice versa. I imagine that'd add up.

anyway I'm just guessing as to the reason; it might be for some other reason.

[loud bear]

but this arbitrary dependency can be installed via setup.py explicitly right?

imagine falling back from pyproject.toml to setup.py

[thorny shell]

dunno tbh

never run into this

also I've never published anything on pypi

I am but a consumer

[loud bear]

its hard to get good help about pip, pypi in general

some useful information hidden deep inside some github issue usually

[thorny shell]

well there is https://packaging.python.org/en/latest/

that site has a lot in it, and it's reasonably well written

[loud bear]

is that new?

[thorny shell]

I don't think it existed 5 years ago

[loud bear]

i didn't see this 2 years ago

oh

[thorny shell]

it feels new-ish to me 🙂

[loud bear]

I never ever remember seeing this
https://packaging.python.org/en/latest/overview/

[loud bear]

it didn't

https://readthedocs.org/projects/python-packaging-user-guide/builds/?page=47
Says 4 years 1 month ago

[thorny shell]

heh

gotta love open source

[loud bear]

!pep 582

[rancid schooner]

**PEP 582 - Python local packages directory**

Link

Status

Draft

Python-Version

3.12

Created

16-May-2018

Type

Standards Track

[loud bear]

seems like a node_modules copy surely

https://peps.python.org/pep-0582/#rejected-ideas python_modules 😆

[errant bison]

hey there :)
what are your experiences with pyinstaller, nuitka, etc.?
any good articles about speeding up python program execution and single executable distribution with these that compare them?

[tawdry needle]

that's exactly what it is. pdm i think is the only package manager that implements this

[tawdry needle]

i've only messed around with nuitka and mypyc a bit. but some applications (e.g. Black) are starting to use mypyc and finding substantial speedups.

i've used cython and numba a few times and seen significant improvements in hot codepaths

the major downside is having to now compile your code and worry about distribution and cross-compiling

pyinstaller i think occupies a different space, it's intended for distributing python applications as self-contained binaries

[leaden flame]

That's true, but even on linux I prefer ripgrep, and having these tools means not dropping into wsl for them. I only do that if I want to use a full linux environment. Those tools though are just nice to have on windows.

[leaden flame]

I have something similar to it set up on wsl and on my linux servers. I'll have to go back and see how I did that. I'm only just getting started.

[worthy dew]

I mean how to get into some Tools for example Hazard Nuker-a and then somehow get inside it and then remove the virus and work like before

[wet dune]

!rule 5

[rancid schooner]

Rules

5. Do not provide or request help on projects that may break laws, breach terms of services, or are malicious or inappropriate.

[wet dune]

Sounds like it to me 🫣

[burnt quiver]

!pip simpletwitter

[rancid schooner]

simpletwitter v1.2.8

Python package for making twitter bot with less code

Bad argument

Converting to "int" failed for parameter "pep_number".

[loud bear]

does vscode not understand pep582?

python extension scans 2500 files eveeytime i save one of my source files

and its extremely slow

ahh too bad vsc doesn't support it

[gentle solstice]

You configured it with the PYTHONPATH?

[tawdry needle]

most tools don't, unless you set pythonpath manually. pep 582 wasn't actually accepted afaik

!pep 582

[rancid schooner]

**PEP 582 - Python local packages directory**

Link

Status

Draft

Python-Version

3.12

Created

16-May-2018

Type

Standards Track

[loud bear]

No PDM has something

But the PDM suggestions break Python extension of VScode

[gentle solstice]

This? https://pdm.fming.dev/latest/usage/pep582/#vscode

[loud bear]

Yes

It makes VSCode scan tons of files

Everytime code is saved

Starts after about 10 seconds or so

[gentle solstice]

See the box that says "Using Pylance/Pyright?"

[loud bear]

Yes I added the config for pyright and mypy in pyproject.toml

Its not caused by type checkers

Its cause by the Python extension, not Pylance

[gentle solstice]

Did you read this? https://github.com/microsoft/pylance-release/issues/1150

https://github.com/microsoft/pylance-release/discussions/2581

[loud bear]

Its this config which causes that

{
  "python.autoComplete.extraPaths": ["__pypackages__/<major.minor>/lib"],
  "python.analysis.extraPaths": ["__pypackages__/<major.minor>/lib"]
}

[gentle solstice]

add it to python.analysis.exclude

[loud bear]

Add what?

pypackages?

[gentle solstice]

{
  "python.autoComplete.extraPaths": ["__pypackages__/<major.minor>/lib"],
  "python.analysis.extraPaths": ["__pypackages__/<major.minor>/lib"],
  "python.analysis.exclude": [
    "__pypackages__",
  ]
}

or it might be ignore instead of exclude

[loud bear]

Don't 2nd and 3rd line cancel out each other?

2nd line adds

3rd line excludes

Total: nothing

[gentle solstice]

then use ignore instead

[loud bear]

That would probably work

[muted cobalt]

Uhm so, I have this workflow that has worked forever, but when running it again now I get this weird error...

[gentle solstice]

Got a link to the file or is it a private repo?

[loud bear]

GA has become increasingly complaining, it once complained about " instead of '

Who even cares about ' or "

[gentle solstice]

Maybe the yaml parser they use was updated or changed

[heavy knot]

I have some github actions that are running when I click the 'Actions' tab but when I click the checkmark/x only a subset of my workflows are showing up. I know they updated the UI recently has anyone else seen this?

[heavy knot]

nvm figured it out

[heavy knot]

Hello guys how can i add proxy for tool ?

[rapid sparrow]

explain your wish better, it sounds quite vague at the moment. Proxies can be working in entirely opposite ways

[zealous timber]

anyone that can help me distribute an app with nuitka?

i get these errors after i try running it, compiled it with --standalone

[tawdry needle]

show the full error?

[zealous timber]

i fixed it, just used pyinstaller and worked first time

[fervent fjord]

is there a possibility to convert a mkv to mp3 with python?

[indigo zenith]

I googled it for you: https://www.codegrepper.com/search.php?answer_removed=1&q=python convert mkv to mp3

[fervent fjord]

I can add all of a folder keeping the same name

[graceful light]

Can someone with more knowledge of pyeprf help me understand the arguments that it takes? The documentation states the following: “runs”: Number of spawned processes, -p/--processes command line option “values”: Number of value per run, -n/--values command line option “warmups”: Number of warmup per run used to warmup the benchmark, -w/--warmups command line option “loops”: Number of outer-loop iterations per value, -l/--loops command line option “inner_loops”: Number of inner-loop iterations per value, hardcoded in benchmark.
But there's no other explanation of what these arguments are
What I can infer from dumping a benchmark into a JSON file is this:

1. runs is the number of individual processes that are spun up to run the benchmark, to prevent the Python interpreter from caching the function calls
2. values is the actual number of times the function is called within a single process
3. warmups are how many times the process will run the function to prevent outlier results to happen when calling a function for the first few times
   But as for loops and innter_loops, I'm not able to find anything mentioning what they actually do. Does loops handle how many times a process runs the function for a given value? IE 10 processes with 15 values and 5 loops would be 10x15x5 total function calls?

[vague star]

Anyone here I can DM that has experience using AWS Code Repos?

[old thistle]

How can I make my local directory in vscode match the last commit in my git history? I have tried git checkout -- and git reset --hard but it doesn't seem to actually change anything in my local directory

[rapid sparrow]

try git checkout branch_name

master / main branch if u wish default one

[rancid schooner]

:incoming_envelope: :ok_hand: applied mute to @iron lark until <t:1667476814:f> (10 minutes) (reason: duplicates rule: sent 4 duplicated messages in 10s).

The <@&831776746206265384> have been alerted for review.

[harsh linden]

Are there some tools which can help me build a proper requirements.txt, minimal required versions of packages and Python itself etc?

I developed my current project in a virtual environment via conda but a simple conda list -e > requirements.txt not only contains the packages required to actually run my program but also things related to development like black. Apparently, it also can't filter packages which have been installed as dependencies and were not explicitly installed.

[thorny shell]

probably -- look at https://packaging.python.org/en/latest/key_projects/#poetry

Warning -- sorta overwhelming 😦

[hazy echo]

Hi @harsh linden I have the same question, but not much urgency yet for the answer. I suspect that the matter of min req version will be a topic for the testing channel.

[floral smelt]

I need to sharpen my skills with data streaming and Kubernetes. So far I'm running kinda dry on mini-project ideas.

[wet valve]

poetry can be used to generate a version-locked requirements.txt based on your pyproject.toml

[loud bear]

pdm >> poetry :))

Just disable pep582 that's all

[crude hedge]

do you know offhand if that can include transient dependencies, or only whatever's defined in your pyproject.toml?

[tawny temple]

Yes, Poetry includes transitive dependencies in the lock file.

If you really meant "transient" rather than "transitive" then you'll have to elaborate on what you mean by that.

[burnt quiver]

!pip simpletwitter

[rancid schooner]

simpletwitter v1.2.8

Python package for making twitter bot with less code

[tardy isle]

Hey guys was referred here. I'm having an issue with my import location being cached. When I import models.blip from BLIP it looks like the import location gets cached and passed on to another import AdaBins that uses models inside one of its classes. I'm stumped!

Any help is much appreciated

[tawny temple]

I think whoever directed you here was mistaken; a help channel would be more appropriate. To briefly answer: I'm guessing these are two different modules both names models. You should use absolute imports to import them to avoid the naming conflict.

[cloud axle]

what is the best way to distribute a project which includes files that arent python like database files or templates for flask and some files which have to be generated on setup like certificates?

[rapid sparrow]

!rule 6 delete. I would not have asked it if u had a real quality stuff, but I have read your study plan.. and I think it is totally wrong one for 80% of it at least

Plus general scamness vibe from the offer is not helping too

[rancid schooner]

Rules

6. Do not post unapproved advertising.

[rapid sparrow]

<@&831776746206265384> rule 6

[tardy isle]

I kept getting redirected, and thank you for the response. This defenitely gets me on the right track. Thank you!

[floral smelt]

Can we use a Docker image exclusively used to pass and validate tests within a pipeline? As in, the container is launched, the tests are executed and when everything passes, the container is destroyed.

[rapid sparrow]

yes, docker run --rm image_name makes sure automatically destroying container once its main process is finished

image still remains though and u will need to clean it after that in addition

[floral smelt]

Ok that's good to know. 👍

[tawdry needle]

isn't this how most CI environments work?

[floral smelt]

Yeah I was wondering how Github CI worked behind the scenes

[tawdry needle]

it's the beauty of containers

[mighty cape]

anyone here to help pewease

[tawdry needle]

see #❓｜how-to-get-help

[heavy knot]

Hey lads, would you mind helping me figure out why my GNU gettext translation tool isn't working?

My folder structure is the following:

/
|- src
|    |- __init__.py
|    |- main.py
|- locales
     |- ru
          |- LC_MESSAGES
               |- base.mo
               |- base.po

the top of my main.py reads like this:

import argparge
import gettext

argparser = argparse.ArgumentParser()
argparser.add_argument("--language", required=False, default="en")
cmd_args = argparser.parse_args()
match cmd_args.language:
    case "english":
        cmd_args.language = "en"
    case "russian":
        cmd_args.language = "ru"
translation = gettext.translation(
    "base", localedir="locales", languages=[cmd_args.language]
)
translation.install()
_ = gettext.gettext

And yet, when I try to run the script in Russian, I still get an error:

FileNotFoundError: [Errno 2] No translation file found for domain: 'base'

What am I doing wrong?

[south forum]

Hey I get the following error on my ubuntu server wen trying to install deadsnake ppa
"Unable to locate package python-openssl"

Wen trying to "sudo apt install python-openssl" I get the same error. Also tried to update and upgrade.

Ubuntu version is 22.10 and kinetic
Thanks for help

[gentle solstice]

Isn't it python3-openssl?

did you sudo apt update first?

you also may need to specify the python version. python27-openssl if using python 2.7

You also may need to use pip if those libraries aren't provided

I personally would suggest using pyenv to install old pythons

[thorny shell]

I found that building with pyenv works but was surprisingly fiddly

I had to install a ton of dependencies before pyenv would build a "proper" python -- one with ssl support, sqlite support, lzma, &c

here's what I had to do for 22.04 "jammy"

sudo apt build-dep -y python3-pip python3-openssl python3-cffi python3-tk python3
sudo apt install -y libssl-dev libbz2-dev libncurses-dev libreadline-dev liblzma-dev libsqlite3-dev

[peak iris]

conda activate should work the same as source ${env}/bin/activate for venv, right?

i see the (conda env) indicator in my shell but it doesn't find any packages when i run it

[jaunty cargo]

does anyone know how to make/work discord tools?

[gentle solstice]

like a bot?

[frozen oxide]

I'm trying to wrap my head around pdm's pep582 support, I THINK I have read that when I enable pep582 support when I do pdm add <package> it should be added to __pypackages__ in my project directory. As that didn't happen I feel I'm missing part of the enabling pdm pep582 support.

I ran pdm --pep582 and added the output to the bottom of my .zshrc file, Is there more I need to do?

[frozen oxide]

aha, got it sorted. the docs page does not include this tidbit that the github page does. pdm config python.use_venv false

[gentle solstice]

pdm uses pep582 if the __pypackages__ folder exists

[native shadow]

an app that can read the content of a notification and display a custom icon or message to a 10x10 pixel display

[hallow cypress]

How do I know if hive metastore is active or not ? Using cli

[subtle quarry]

a question on github cli tool, i want to use gh cli tool on one machine with two different profiles. each will have access to different repos. i searched on gh cli docs but cant find a way to do it. is it possible?

[rapid sparrow]

What is github CLI tool

do u mean Git CLI tool, https://git-scm.com/downloads ?

if yes, then it is resolved with...
having ~/.ssh/config file

Host github.com-petprojects
  HostName github.com
  User darkwind-petprojects
  IdentityFile /home/naa/.ssh/id_rsa.petprojects
  IdentitiesOnly yes

Host github.com-work
  HostName github.com
  User darkwind-work
  IdentityFile /home/naa/.ssh/id_rsa.work
  IdentitiesOnly yes

Then all is needed to fix git clone command [e-mail]-petprojects:my_project_group/my_pet_project_infrastructure.git
or correcting for existing local repo through git remote set-url origin new_corrected_url

this helps to set different SSH keys without conflicts to same Git remote resource

[brazen forge]

you can have multiple hosts, but not so much different profiles for the same hosts with the gh CLI

[brazen forge]

https://github.com/cli/cli/

[rapid sparrow]

interesting. And equal gitlab CLI tool is present too
https://glab.readthedocs.io/en/latest/

[brazen forge]

yep, use it often
it's nice to review and merge PRs from the CLI

fun fact: it was developed by a volunteer before GitLab decided to take it under their umbrella

[subtle quarry]

How to set up different hosts? I have two personal access tokens. Would that work in this case?

[subtle quarry]

Yeah it's GitHub cli. We use pat tokens but will see if I can set up ssh keys

[brazen forge]

different hosts as in, github.com and some github enterprise domain

either way, I'd recommend using SSH

[subtle quarry]

That may work. I indeed need GitHub and GitHub Enterprise access

[brazen forge]

in that case, gh creates a hosts.yml in its config directory

there might be some docs about it, lemme check

apparently, you can just pass --hostname to gh auth login and it'll take care of it

https://cli.github.com/manual/gh_auth_login

[subtle quarry]

Thank you. Perfect. I missed that in docs. I mean i saw this flag but didn't think it is used for that.

[young marlin]

hello py maximalists.
our team is packaging python into a single binary (hooray).
went down the route of pyinstaller,
but we'd rather run cython pre-processor/compiler,
then bundle that into a binary w/ single pyinstaller file (main.py).
only insufficient tutorials on the internet.
anyone have a good example project on github to acheive this?

[tawdry needle]

i'm not sure how pyinstaller works specifically, but you'll probably have to build a binary "wheel" package for your application that includes the compiled cython extensions, and then try using pyinstaller with that

https://pypi.org/project/python-build/

[crude hedge]

oof yeah, i did mean transitive. that was silly. thanks.

[tawdry needle]

follow the cython docs instructions for using cython with setuptools. you can use python setup.py bdist_wheel but that's deprecated and python-build is considered the "correct" choice for new projects

you can also use pyproject.toml to specify everything and just use setup.py for the cython parts

[young marlin]

thanks @tawdry needle! cython gets us into a build state of .c files/.so libraries (quite easy, 4 lines of setup.py), but not to the executable. hence the additional step of pyinstaller into a single executable. we're having quite some trouble bundling the two (given the # of files, all examples online are trivially 1/2 py modules)

[tawdry needle]

what happens if you do that first, install it into your project environment, and then use pyinstaller?

[young marlin]

output into a /build folder .so files - that pyinstaller can/would bundle into a single primary file (main.py)
the challenge this last step! how does one bundle those 54 .so? looking for a good repo w/ example of this 🙂

[tawdry needle]

tell setuptools to include them in your package and/or use the setuptools cython extension to have them included in your package for you

maybe if you add build/ to PYTHONPATH it could work too

note that i haven't used pyinstaller, but it seems like it will pull in compiled modules as long as they're in a place where it knows to look

and it seems like that would be "anywhere in the standard module search path"

[young marlin]

....certainly does not auto index them from trial/error
....otherwise our approach to glob the folder would suffice!

[tawdry needle]

how do you currently run this? without pyinstaller

if you dump the .so's in to a build/ folder, python won't find those naturally

so you must be doing something special here

[young marlin]

nothing special. the .so's automatically generate into the build/ folder.

# app/compile.py
from setuptools import setup
from setuptools.extension import Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext
setup(
  name="mods",
  ext_modules=cythonize(
    [ Extension('mod.*', ['mod/*.py']) ],
    build_dir="build",
    compiler_directives={
      'language_level' : "3",
      'always_allow_keywords': True,
    },
  ),
  cmdclass=dict(
    build_ext=build_ext
  ),
  packages=['mod']
)

your typical cython compilation 🙂

[wise valve]

should i use Pycharm or VS code or Atom or Sublime?

[tawdry needle]

what happens if you take out build_dir? try removing that option and using python-build to build a wheel

that build_dir is outside any package that python knows about and it's not in the python search path. ergo python (and therefore pyinstaller) can't find them, unless you're adding that directory to pythonpath/sys.path in your application

although you don't even need to install from the wheel, you should be able to just pip install . to install your own package into your own environment

[pseudo kiln]

Hi does any know how to make a saltstack script that adds a task to the task scheduler on windows (the criteria for the task is to execute a python script every 1 min? If so could you maybe send an example? plz tag/reply or dm me.

[young marlin]

Thanks @tawdry needle - very kind. none of this working, we’ll ask else where 🙂

[short peak]

hello. I'm using a private repository with poetry and I want to install black (which is not coming from the private repo. I have this setup in my pyproject.toml

name = "my-private-repo"
url = "https://my-private-repo.hostname"```
it usually works without problems but I'm now getting this
```  403 Client Error: Forbidden for url: https://my-private-repo.hostname/black/```

I also execute this

```poetry config repositories.my-private-repo https://my-private-repo.hostname/```

it looks like it is trying to pull `black` from the private repo. Any idea why?

[brazen forge]

If package sources are defined for a project, these will take precedence over PyPI. If you do not want this to be the case, you should declare all package sources to be secondary.
https://python-poetry.org/docs/repositories/#project-configuration

[subtle quarry]

How can I find out hostname for GitHub Enterprise? 😅 tried some hostnames but didn't work.

[harsh linden]

Am I understanding Poetry https://python-poetry.org/ correctly: it is a more sophisticated way of using virtual environments like venv or conda environments? My main interest in it is that I can export a clean requirements.txt. Clean here means that it is not cluttered with development-related packages like black or packages which were only installed as a dependency, i.e. I only want to export the packages required to actually run the program or script.

[rapid sparrow]

i think, in first order it is a better requirements.txt alterantive

it controls venv too in addition just for nice dev env

[rapid sparrow]

your main root dependencies will be cleanly... shown in forgot the name of file.toml, with clear separated section for dev dependencies like black

secondary dependencies will be locked in poetry.lock or smth like that

and for docker image purposes u will be always able to export stuff to requirements.txt, which will be always messy but u will not care because your forgot the name.toml will be source of truth`

[keen junco]

Not sure if this is the correct channel for this; but does anyone have experience implementing a python script into a Rhino3D file using Grasshopper?

Strange question, I know. But I would like to run a python script that does some calculations based on the geometry in the part file...

If there's a better channel for a question like this, let me know.

[lament dirge]

Can find some here:

https://www.cohesive.so/blog/top-50-open-source-tools-2022?utm_source=cohesivedevs&utm_medium=discord&utm_term=discord

[subtle quarry]

in github actions i can create repo secrets and variables. to access secrets i can use ${{ secrets.SECRET_NAME_HERE }} how do i access repository variables?

[tawdry needle]

poetry does 3 separate things: 1) help you manage dependencies, 2) help you manage venvs, 3) act a as a "build backend" for building python packages for distribution on pypi

[opaque cliff]

Flutter or kivy is better

[civic grove]

Flutter

[vagrant heart]

Hi, I'm looking for some feedback on managing locked and unlocked dependencies for internal and externally used packages.
I want the option for end uses to easily install an application with the exact dependencies I used for testing.

This seems to be not solved by seperating requirements.txt and setup.py (or similar alternatives using pyproject.toml etc).

I've a process that seems to meet my needs but seems unique and I'd like some review/feedback - I've not seen it in my explorations around the web.
See #help-chocolate

[red locust]

how to check memory that i am using on remote ssh?

free -g shows combined for all users and not me specifically

[subtle quarry]

found it. it's secrets.REPO_VARIABLE_NAME

[pine fern]

@rapid sparrow have you seen this? https://www.docker.com/blog/docker-wasm-technical-preview/

[rapid sparrow]

Sure. nice to know it will be one day. But technically not really important technology too much in terms of docker

we need only Linux platform, the rest of platforms are optional ;b

i solved multi architecture problems also with buildx already

plus it is really immature at the moment. So one day later to include more non-linux developers, which aren't that needed in web dev in the first place.

[pine fern]

yeah just seems like web assembly cropping up everywhere

even pyscript was due to it

[rapid sparrow]

Although u know...

i wonder if this tech will be able to compile once and making cross CPU architecture image. One image for amd64/arm64 and etc

@pine fern in this case it could be really useful

[pine fern]

they were right when they said web assembly would unlock many other tech it seems like

[pine fern]

oh yeah

and faster runtimes

[rapid sparrow]

because apperently Arm64 tech is several times cheaper tham amd64

[pine fern]

[rapid sparrow]

it would be awesome. Having completely hardware abstracted programs

not caring what is under the hood except of raw CPU/RAM/SSD power xD

to hell with hardware details

[pine fern]

haha abstract it all away

would make dev even easier

[hazy echo]

@tawdry needle: suggested I come here from #unit-testing because we're talking about automating Sphinx and tweaking it a bit to consume Markdown.
(Disclosure: I'm not a fan of Markdown, but I can understand how .rST is not for everyone)
So, there's a byproduct of the Jupyter community which produces:
https://myst-parser.readthedocs.io/en/v0.15.1/sphinx/intro.html#enable-myst-in-sphinx
and markdown folks love it to pieces.

[tawdry needle]

to be clear: i'm not actually interested in markdown right now

people can write rst in docstrings

but i'm using apidoc because i don't want them to manually create .rst docs files

heck, i don't want to do that

[heavy knot]

am making a discord bot and i wanna make it when it runs a cmd procces i dont want it to pause the discord bot session i want it to make it so its like opens a new cmd tab and does the procces in it and if another and another and another user does it keeps on opening more cmd procces so it still doesnt pause the discord bot so i need someone to make this code "os.system(f"py read.py {args1} 200")" turned into a one thaat does the same function but it does it in a new cmd tab

[tawdry needle]

i'd rather have auto-generated docs than try to setup a ci check that enforces that modules are documented

[heavy knot]

:(

[hazy echo]

I see, in that case, have you considered sphinx-autoapi
https://github.com/readthedocs/sphinx-autoapi#installation
(I know, yet another confusing name)

[tawdry needle]

i considered it, but didn't want to mess with learning another tool

i just want to move my conf.py 🙂

[hazy echo]

I feel your pain. And we can do that.
For spectators:
Our colleague runs a Makefile with two targets, the first runs sphinx-apidoc to prepare a skeleton directory tree of rST files.
The next runs sphinx-build using source/conf.py

which means that make is running two python processes, each with it's own system.path

[tawdry needle]

that's a good summary. and i just want to move ./source/conf.py to ./conf.py and leave everything else the same

[hazy echo]

so you could probably get away with it by modifying the system.path in ./conf.py

[tawdry needle]

so you suggested setting source dir to . and then adding source/ to the system path inside ./conf.py, but that doesn't help sphinx find the rst sources in source

or am i misunderstanding?

[heavy knot]

can you drag and drop issues on the jira mobile app from one column to another? i swear i did this once but it's not working now

ok idk it works now weird

[heavy knot]

nvm it only works sometimes and i don't know why

[hazy echo]

The way I would think about it is: conf.py tells Sphinx where to find whatever files it needs.

• In the first stage, sphinx runs apidoc and uses conf.py to find the python files and create .rst files.
• In the second stage, sphinx runs build and uses conf.py to find the .rst files AND the jinja, CSS, etc. to create HTML. So conf.py will have the paths for finding HTML themes, Static images, CSS stylesheets, Jinja templates, and more.

@tawdry needle Where is gets tricky is that you're launching the python modules from make and passing along some references, which might not be what you intend.

[twin hull]

Is there a way to limit codecov access to my github? I only want to use it for public repo, so it doesn't need read access to private ones, nor write at all, and neither it needs my email address

[tawdry needle]

i actually think there's just a bug in sphinx-build with the -M option

i filed a bug report https://github.com/sphinx-doc/sphinx/issues/10968

it's certainly not behaving as described in the docs

[hazy echo]

Ok. I'm around today if you want to jump on a session together

[hollow aspen]

which one do you prefer, virtual env or conda env?

[rapid sparrow]

poetry. best virt env manager (+and package manager for that matter)

[tawdry needle]

venv, unless you specifically need conda

there's also virtualenv and virtualenvwrapper, both of which have extra features for creating venvs

and yes, a lot of other tools use venvs internally: poetry, hatch, pipenv, pipx

they're well supported and easy to work with

[hollow aspen]

thank you, it seems Ive used conda for too long.

[tawdry needle]

i like conda a lot, but for general software dev it's not the best choice

if you're doing data science, it's a better choice (although nowadays you can get by with a venv too, because so many libraries publish binary wheels)

[subtle quarry]

Why you say for too long? Just curious. Conda user myself:)

[hollow aspen]

oh I just thought if I aim at software development, I shouldnt use conda that much.
I even install my python via conda.

[rapid sparrow]

software development is quite broad thing to say.
we could clarify, do we use conda for regular backend development? then answer will be no.

[tawdry needle]

yeah, you can use conda for general-purpose development, but it's not done very often, so the ecosystem support won't be very good

[quaint fox]

how can i get data from a website / api but only spit out a certain part

[tawdry needle]

it depends on the format of the data. e.g. if it's json, json data types map cleanly to python data types (list, dict, int, etc), so you can parse them as python data types and then work with those.

html requires an html parser, xml requires an xml parser, etc.

[quaint fox]

no no just json im using a neat thing called "bored api" it sends you a random thing to do when your bored but i dont want to print "{activity: blah blah blah}

i just want what comes after activity

[tawdry needle]

{activity: blah blah blah} this looks like json to me

rather, {"activity": "blah blah blah"} would be valid json

so you need to make your http request to the api using whatever library you are already using (requests? httpx? aiohttp?), parse the body of the response as json, and proceed from there using the python objects that you get as a result

[dark grove]

We prefer you didn't - even if spoilered

@quaint fox

[quaint fox]

yes sir sorry didnt know i respect it!

[dark grove]

Can you edit your message to remove it.

[tawdry needle]

can someone with AWS experience help me to understand what Elastic Beanstalk does that ECS doesn't do? my naïve impression was that EB was a layer on top of ECS, but it looks like you can't use EB with Fargate, but you can use "plain" ECS with Fargate. i am so confused 😵‍💫

[heavy knot]

whats the difference between epics and tasks with subtasks in jira? I'm wondering why use epics

[rapid sparrow]

Out of curiosity scanned documentation of ElasticBeanstalk and can say...
difference between ECS and Beanstalk like difference between Kubernetes and Heroku.

in ECS u a essentially having Kubernetes cluster imitation, you create cluster, attach to its ec2 nodes, and u launch docker containers into it.
possible to launch container into fargate stuff for serverless option. You build docker image in order to deploy into ECS. ECS = Container orchestration service essentially.

Heroku ElasticBeanstalk strictly offers only already built images for your server base, for python they suggest
https://docs.aws.amazon.com/elasticbeanstalk/latest/platforms/platforms-supported.html#platforms-supported.python
Python 3.8 AL2 version 3.4.1 Python 3.8.5 pipenv 2021.11.9
Python 3.7 AL2 version 3.4.1 Python 3.7.10 pipenv 2021.11.9
Beanstalk expects you to have your dependencies only as requirements.txt saved for python, which it will install on its own. U do not have control what will be installed in image.

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create-deploy-python-django.html
you are only expected to write .ebextensions in root folder

option_settings:
  aws:elasticbeanstalk:container:python:
    WSGIPath: ebdjango.wsgi:application

which in their proprietary settings will tell how to launch wsgi of an application

if u need any customization... it is again their very custom proprietary settings. like those ones to make migrations before starts.

container_commands:
  01_migrate:
    command: "source /var/app/venv/*/bin/activate && python3 manage.py migrate"
    leader_only: true
option_settings:
  aws:elasticbeanstalk:application:environment:
    DJANGO_SETTINGS_MODULE: ebdjango.settings

U can use only their already premade infrastructure, deploying application + deploying S3 bucket. Everything else, DNS and etc is handled for you!

Decreasing complexity of deployment, in tradeoff of even less controlling our application and having less flexibility of its infra

Consider it is.... Application Deployment tool. Abstraction of deployment remains at Application level, without diving into what are Docker images/Linux servers. Same DigitalOcean offers too now.

P.S. scanning terraform code features of Beanstalk helped me to evaluate it faster
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/elastic_beanstalk_application

additional documentation for reading about fixed offered infrastructure solutions in beanstalk: U have only web app and worker applications (may be something else)
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/concepts-webserver.html

[tawdry needle]

@rapid sparrow EB also supports running arbitrary docker images too, so there's overlap in that respect

Everything else, DNS and etc is handled for you
is this not handled for you with ECS?

[rapid sparrow]

nice. DigitalOcean offers that too in their Application Deployment

[rapid sparrow]

Umm, technically u can configure VPC and subnets for beanstalk too. So if u want, u configure DNS too.
lets attempt to find another difference

ECS is a scheduling system of container orchestration, it gives flexibility to launch multiple containers in one EC2 instance. Requesting hardware resources where to launch them, limiting how much they can use

From what i see in Beanstalk = One EC2 instance is one app here.

[rapid sparrow]

Also, where did u find it can launch arbitary docker image?

there is no mentions of it anywhere in terraform docs

nvm, found

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker.html
it can have Docker instance

@tawdry needle https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_ecstutorial.html

Hmm

@tawdry needle

{
  "AWSEBDockerrunVersion": 2,
  "volumes": [
    {
      "name": "php-app",
      "host": {
        "sourcePath": "/var/app/current/php-app"
      }
    },
    {
      "name": "nginx-proxy-conf",
      "host": {
        "sourcePath": "/var/app/current/proxy/conf.d"
      }
    }  
  ],
  "containerDefinitions": [
    {
      "name": "php-app",
      "image": "php:fpm",
      "essential": true,
      "memory": 128,
      "mountPoints": [
        {
          "sourceVolume": "php-app",
          "containerPath": "/var/www/html",
          "readOnly": true
        }
      ]
    },
    {
      "name": "nginx-proxy",
      "image": "nginx",
      "essential": true,
      "memory": 128,
      "portMappings": [
        {
          "hostPort": 80,
          "containerPort": 80
        }
      ],
      "links": [
        "php-app"
      ],
      "mountPoints": [
        {
          "sourceVolume": "php-app",
          "containerPath": "/var/www/html",
          "readOnly": true
        },
        {
          "sourceVolume": "nginx-proxy-conf",
          "containerPath": "/etc/nginx/conf.d",
          "readOnly": true
        },
        {
          "sourceVolume": "awseb-logs-nginx-proxy",
          "containerPath": "/var/log/nginx"
        }
      ]
    }
  ]
}

literally EBE can run on ECS

what is difference?

difference in scale?

We are seeing here essentially... ECS Daemon(scheduling strategy) deployment, with ability to point numbers of created ECS2/Nodes of our cluster @tawdry needle

@tawdry needle conclusion: EBE is a very simplified ECS, out of daemon scheduling only deployment strategy of one task definition, across N replicated machines.

[tawdry needle]

right, there seems to be so much overlap

i guess with ECS you have to set up ELB yourself

whereas EB is a central place to set up ECS + ELB + Route53

but i find it really weird that you can't run Fargate inside EB!

so i want the convenience of EB but i don't want to do capacity planning... sucks to suck, you can't use Fargate, and you need to go back and DIY all that stuff again

[rapid sparrow]

EBE automatically configures Auto scaling group, with Cloud watch alarm to trigger scaling operations.
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.managing.as.html

[tawdry needle]

i see, but you're still stuck with "discrete" EC2 instances

[rapid sparrow]

EBE is kind of very opinionated ECS cluster

[tawdry needle]

as opposed to Fargate just setting a CPU usage

[rapid sparrow]

oh, like Heroku it allows to configure databases inside EBE!
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.managing.db.html

[rapid sparrow]

Ergh, the meaning of EBE to be simple solution i guess, that boilerplated everything

once people outgrew it, they are supposed to migrate to ECS at least

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/migrate-to-ec2-AL2-platform.html
at least so this guide suggests

[rapid sparrow]

anyway, thanks for saying about existence of EBE. Now i will now what i will recommend to novices to AWS.
Definitely good novice solution i think

at least if to skip AWS Lightsail existence

@tawdry needle quite heavy overlap with Lightsail too
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lightsail_container_service_deployment_version
it allows easier solution to getting server, deploying container.

Difference in lack of Platform as a Service approach with already available Python environts, and
that if u want load balanced solution, u will need to create load balancers on your own and attaching it

data "aws_availability_zones" "available" {
  state = "available"

  filter {
    name   = "opt-in-status"
    values = ["opt-in-not-required"]
  }
}

resource "aws_lightsail_lb" "test" {
  name              = "test-load-balancer"
  health_check_path = "/"
  instance_port     = "80"
  tags = {
    foo = "bar"
  }
}

resource "aws_lightsail_instance" "test" {
  name              = "test-instance"
  availability_zone = data.aws_availability_zones.available.names[0]
  blueprint_id      = "amazon_linux"
  bundle_id         = "nano_1_0"
}

resource "aws_lightsail_lb_attachment" "test" {
  lb_name       = aws_lightsail_lb.test.name
  instance_name = aws_lightsail_instance.test.name
}

More hands on controlled approach.

so, i guess without autoscaling group stuff then. Good old, single servers

okay, i think Lightsail is still better to get started in AWS then xD

[tawdry needle]

ugh wtf, i've never even heard of lightsail

i guess it's even more specifically optimized for websites vs. just "general apps"?

[rapid sparrow]

it is nice. It is like very simple providers DigitalOcean/Linode/Hetzner. You create single Linux server and u a already happy.

[tawdry needle]

it's insane how many overlapping features they have

i see, lightsail is like a traditional VPS built on top of S3, EC2, etc?

[rapid sparrow]

Yeah, traditional VPS on top of EC2.

• simplified Load Balancer for them

• simplified DB

everything is a separate very simplified entity to configure/launch

[rapid sparrow]

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lightsail_domain even simplified DNS

[tawdry needle]

makes sense

[rapid sparrow]

xD https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lightsail_static_ip static IP like in simple providers

[tawdry needle]

so it sounds like lightsail is the easiest thing if you just need to deploy a website or web app

[rapid sparrow]

No and Yes

[tawdry needle]

EB is the easiest way set up a multi-container app with docker compose, and/or to set up a few interacting apps in various languages and/or docker images

[rapid sparrow]

in Lightsail you go oldschool, u get one Linux server (or can deploy one docker image only https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lightsail_container_service_deployment_version), but usually u go with Linux server.
U need to learn how to work with Linux servers usually from CLI

[rapid sparrow]

EB is easier to deploy, because platform as a service, already configured server which you aren't supposed to configure

but Lightsail gives more oldschool flexibility... because in oldschool way u can install stuff in a manual way to server (or with Ansible xD)

[rapid sparrow]

is is still more recommended to advice novices to Lightsail, for the sake of people learning how to work with Linux servers from CLI xD

better them learning what is Load balancer and configuring as separate entity on their own

instead of jumping to EBE where everything is already premade

[rapid sparrow]

in a manual way, don't forget how much oldschool manual complexity is involved... u need to know how to open firewall, launching web server as a daemon task, configuring HTTPS certificates, working with Linux CLI. xD learning how to SSH connect to server and adding public keys

[rapid sparrow]

well... Lightsail is ideal (from AWS) solution at least for Homelab though. Because...

manual server is ideal for homelab, for the sake of maximum money economy, at the cost of u configure everything on your own
And launching 10+ services to the maximum of available hardware resources

[tawdry needle]

yeah that all makes sense

lightsail is the traditional VPS experience

EB is an easy all-in-one "multi server" experience

annoyingly you can't use Fargate with EB, but who the hell knows why

then there's ECS (+Fargate optionally, as well as various other options like EC2 and their mobile edge computing stuff)

or there's EKS if you really want to use kubernetes instead of whatever proprietary thing ECS is

[rapid sparrow]

except AWS EKS has 80$ per month+ starting cost for initialization of cluster 😐 for managing interface or something like that

[tawdry needle]

yeah i assume EKS is targeted at bigger orgs that want a lot of control over the kubernetes stuff

and/or are migrating from kubernetes on other platforms

[rapid sparrow]

AWS is just greedy.

starting cost of Kubernetes in other providers is way less

[rapid sparrow]

AWS prefers to trap customers into EBE/ECS

so they would not go away, which they can do with kubernetes

[rapid sparrow]

even Azure... M$, only makes u paying for VM instances of kubernetes cluster xD. Paying only for EC2 instances of cluster.

regretfully GCP asks for 72$ per month to start with GKE too

[rapid sparrow]

so... cheap managed kubernetes clusters options only.. Azure, DigitalOcean, CIVO

CIVO is nuts how cheap

5$ per month starting cost xD

[short peak]

have you used it?

[rapid sparrow]

no, only registered and knowing they have cost optimized because they run it on K3

they offer 250$ trial for 2 weeks or months or something

DigitalOcean
1 CPU 1(2) RAM = 12$
2 CPU 1(2) RAM = 18$
4 CPU 6(8) RAM = 48
8 CPU + 13(16) RAM = 96

CIVO
1 CPU + 1 RAM = 5$
2 CPU + 4 RAM = 20$
4 CPU + 8 RAM = 40$

difference... CIVO is 17% cheaper + u can use hardware resources not taken by fully fledged kubernetes cluster

• just less minimal instance cost due to ability for them to run weaker instance because of K3

https://registry.terraform.io/providers/civo/civo/latest
terraform access is present like in DO

[tawdry needle]

oof that's a good point. charge money to keep people away from using EKS unless they're really "serious" customers

[rapid sparrow]

Still sucks in comparison to Hetzner prices though. If u are capable to afford... installing microk8s/k3s on your own. (overhead in ansible scripts at least)

Hetzner is greater for homelab for the sake of having more hardware resources xD Around 2 times cheaper for same resources.

• when u use kubernetes volumes, i suspect CIVO will cost you additionallal price like it is in DO.
  So with manual kubernetes cluster is greater economy

[heavy knot]

bump

[rapid sparrow]

Epics are large bodies of work that can be broken down into a number of smaller tasks (called stories).
https://www.atlassian.com/agile/project-management/epics-stories-themes
googled for you. Epic is a mega fat tasks splittable into regular tasks

[heavy knot]

right but i can make subtasks with tasks too

i guess it's you can have multiple stories per epic

[tawdry needle]

epics are projects spanning a longer period of time. stories are top-level units of work. tasks and subtasks subdivide stories. that's how i think most people use it.

[arctic flicker]

why does pycharm run off with the execution and not stay in the location in my code where the exception was encountered? The result is that I can't check the parameter values in that place, which I always used to be able to do. Instead, I can only access the variables of the Xtree module function....

[heavy knot]

https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html#understanding-variable-precedence
I don't understand what 4. and 5. mean.

4. inventory group_vars/all
5. playbook group_vars/all

I tried:
inventory/group_vars/all and playbook/group_vars/all but it's not it

I can't find additional info about these two points. everything I find is copy/paste from original docs

where do I define group_vars for 4. and where for 5.?

[red oxide]

I'm trying to run my ansible playbook but ansible is not respecting values in hosts.ini

# hosts.ini

[nodeServer]
172.17.176.1 ansible_port=2200 asnible_user=vagrant

# ping
>> ansible nodeServer -m ping

172.17.176.1 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: avi@172.17.176.1: Permission denied (publickey,password).",
    "unreachable": true
}

as you can see, it's not using the user and port mentioned in the hosts.ini file

[real hedge]

I have a python application running behind NGINX. I'm constantly seeing this in my logs as a GET uri path.

\u0016\u0003\u0001\u0001H\u0001\u0000\u0001D\u0003\u0003<���i�Q����\u0004��I�/�����w\u0013\u001F�#�\u00116\tH x�Z�*�Z+�<=�\u00148��2M�\u0019K��'��\u001D׎�N\u0000b\u0013\u0002\u0013\u0003\u0013\u0001�,�0�+�/̨̩\u0000�\u0000�\u0000�\u0000�̪�����$�(�

Any ideas what that could be?

[gentle solstice]

You misspelled ansible_user

asnible_user

[red oxide]

oh nooooo T_T

I wish ansible threw a more accurate error 😦

[gentle solstice]

It did the best it could.

[red oxide]

thanks for pointing it out tho, I looked at eveything but that lol

[rapid sparrow]

U could try installing syntax highlighter/linter. May be it would have caught

[gentle solstice]

It's just an ini file. There's no schema for that

[red oxide]

I do have one but it only works on my mac, I can't vagrant there because of virtual box support on m1

and this sucks on windows

[gentle solstice]

ansible is designed so you don't have to write any code to do what you need to do. (most common tasks)

[red oxide]

Ansible VS Code Extension by Red Hat I use this, but it works on and off and never worked on windows

[gentle solstice]

it's mostly a jinja template in yaml

[rapid sparrow]

Technically there is, you can only in certain way building inventory at least in yaml format

Btw yaml format is superior xD

[gentle solstice]

yet another markup language 😔

have you ever tried parsing yaml? There's so many options, it isn't funny

Who knew yaml could be as dangerous as xml

[rapid sparrow]

U can be even yaml developer, you know

[rapid sparrow]

There is no better alternatives xD

[real hedge]

Hold my beer 😆

[errant bison]

heya! are there any good lists for http(s)/socks proxies?

searching for a bit more speed in order to scrape some data of seriously rate limited sites (<50requests per hour)

not sure if this would be better suited for #networks or #web-development instead, if im wrong here please let me know :)

[timber umbra]

Is there some python linter that can check whether the code is compatible with a number of python versions?

Got it: pylint's --py-version can do that.

[rapid sparrow]

https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs
A more modern and 99.9% having reliability way to handle this problem will be written CI pipeline doing verification in matrix of python versions

name: Django CI

on:
  push:
    branches: [ "master" ]
  pull_request:
    branches: [ "master" ]

jobs:
  build:

    runs-on: ubuntu-latest
    strategy:
      max-parallel: 4
      matrix:
        python-version: [3.7, 3.8, 3.9]

    steps:
    - uses: actions/checkout@v3
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v3
      with:
        python-version: ${{ matrix.python-version }}
    - name: Install Dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
    - name: Run Tests
      run: |
        python manage.py test

then all tests will be invoked automatically under different python versions

[timber umbra]

oh, I saw that done actually, though usually via tox and not just github actions

and yeah, that's far better than just static analysis, the latter can sadly catch very little

[rapid sparrow]

tox is local... i guess it can be useful for the sake of having test code less CI tool dependent and invokable locally

hmm but using CI tool things it will be better parallized

and faster finished

better going with CI things there, but compensating their awkwardness through building docker image with build arg perhaps for testing if neceesary

[lusty forge]

I've noticed a lot of experienced programmers choose Debian over Ubuntu for VPS' in some cases. What are the reasons for this? Are there other notable distros for a VPS?

[tiny jungle]

the general reasons that debian is chosen for anything is that it updates in a manner that places an extremely strong emphasis on reliability over newness. for servers, it's usually okay to have a few programs a year or so out-of-date if you can get more uptime and/or less errors

[wet valve]

debian has a reputation for stability

[indigo zenith]

Are there other notable distros for a VPS?

Red Hat, etc.

[red oxide]

# Ansible task

- name: Start ssh-agent.
  ansible.builtin.shell: eval `ssh-agent -s`

- name: Load local SSH key.
  ansible.builtin.shell: ssh-add ~/.ssh/id_rsa

# Problem
ansible executes Start ssh-agent task but for some reason the next step always fails.

This works fine when I run eval ssh-agent -s first before running the playbook.

# Questions

1. What could be the reason for this behaviour?
2. How can I execute eval ssh-agent -s the right way so I can simply run the playbook?
3. What eval does exactly? Cause if I run only ssh-agent -s then although it seems the the command was successful, the play book still fails at that second step

[tawdry needle]

@red oxide it helps to check the manual 🙂

 -s      Generate Bourne shell commands on stdout.  This is the default if SHELL does not look like it's a csh style of shell.

so ssh-agent -s generates a shell script. eval runs a string as a shell script. so you are generating a shell script, and then running it. that shell script sets up the current shell session so that ssh-add calls will be directed to the running ssh agent.

[red oxide]

i see, so why won't it run with ansible?

[tawdry needle]

probably because they run in separate shells

try ansible.builtin.shell: 'eval "$(ssh-agent -s)" && ssh-add ~/.ssh/id_rsa'

i know nothing about ansible but that would hopefully run the two commands in the same shell script

[thorny shell]

has something changed with how github does authentication? In the last couple of weeks? Now when I do "git fetch", even from a public repo, it pops up a browser page and asks me to auth; it never did this before.

This behavior is annoying, since I can no longer update all my repos non-interactively.

I cannot find anything about this at https://github.blog/changelog/label/2FA,authentication,securitykeys,SSH,tokenauthentication/

[tawdry needle]

git fetch in particular? this sounds like something in the credential helper you have on your system. maybe it's a git-for-windows feature or something

[thorny shell]

it does I guess, but I don't think I've ever needed credential helper before; I have a running ssh-agent that's properly set up, which I assume is necessary for private repos

also I'm using MacOS, not Windows, fwiw

[heavy knot]

I've been reading more about building tools and dependency management systems, what do people use on Python in the case of a monorepo? I've heard about Bazel which seems to be good enough but I can't seem to find much information on IDE integration, for example if my dependencies are managed by it, how would it work for development (in the case of for example typehinting)

shared/ -> shared Python files
services/
|- service1/
|-- ...
|- service2/
mainproject/
|- ...

[thorny shell]

Sounds like it's https://github.com/GitCredentialManager/ which I do indeed have installed

[tawdry needle]

are you trying to push to an https repo? i thought github disabled that anyway

[thorny shell]

naw I was fetching

and it seems to have been just a single private https repo

[tawdry needle]

ah yep. i always use ssh for any repo that i control anyway

[thorny shell]

so do I

this was somone's repo from here, that I was helping them with

[tawdry needle]

but it still brought you to a browser login page??

oh i see

maybe they switched it to private later

[thorny shell]

I saw the browser login page before I purged github-credntial-wozzit from my system 🙂

[tawdry needle]

from what i remember on windows, that git credential manager is actually pretty useful

i think it acts like the mac keychain where it can save your ssh key passphrases and such

[thorny shell]

I think I use it at work where it's all ADO all the time

[somber falcon]

is it common practice to include functions/methods that are only called internally within a library in official documentation?

i have methods/functions that i never intend any user to be worried about, see, or even call, but for contributors they should definitely be documented

[subtle quarry]

how do i auto tag and realease python package to github releases, like for this package for example: https://github.com/fastai/fastai/releases
edit: reading docs: https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases

[subtle quarry]

what's the purpose of these releases? can people install packages using pip from there?

[harsh linden]

You should not use camel case in Python but you should also not use a mix of snake and camel case if one of your libraries uses camel case. In my case, I will be building an application with PyQt. Black and Pylint both expect snake case. Is there some kind of switch to tell them that my project uses camel case. Do I need other tools for their jobs?

[tawdry needle]

arguably you shouldn't include these in the public api documentation. you might want to have a separate section of the docs for internal-only interfaces. consider moving such things to a separate module

e.g. the public interface goes in mylib/__init__.py and the internal stuff goes in mylib/_core.py. and the docs for the latter would include a stern warning that it's only meant for developers and to not rely on it as a public interface.

or if you define it all in _core, you can import and re-export the public stuff. there are various ways to handle this with tools like sphinx, although they aren't particularly elegant. the sphinx ecosystem i think is really lacking in good automatic api doc generation tools.

[timid haven]

im not able to install python3.7 with pyenv

configure:

  By default, distutils will build C++ extension modules with "clang++".
  If this is not intended, then set CXX on the configure command line.
  
checking for the platform triplet based on compiler characteristics... darwin
configure: error: internal configure error for the platform triplet, please file a bug report
make: *** No targets specified and no makefile found.  Stop.

i need 3.7 exectuable

Im new to poetry, I wonder how to dockerize application. I see some examples use this image ```js
FROM --platform=linux/amd64 ghcr.io/chrislovering/python-poetry-base:3.10-slim
Do u need to use this image?

[tawdry needle]

looks like your platform isnt supported but i find that surprising. how did you install pyenv? what command did you run to install 3.7? what version of mac are you using? intel or m1?

[tawdry needle]

no, you can install poetry yourself in the container if you want. i do that with pipenv in a current project i have

[rapid sparrow]

docker --version
By default u can only build for OS mentioned in your docker version under OS/Arch:
in order to build for other platforms additional configuration is required (module buildx can do this)

[timid haven]

I have macbook m1 MAX

I tried to install python 3.7 by doing ```js
pyenv install 3.7.0

First i checked if it was available with ```js
pyenv install --list

[rapid sparrow]

macbooks work on arm64 architecture. They can't build for amd64 without buildx/virtualization stuff

[tawdry needle]

ah, older python versions don't support m1. you might have to try using python-build or just use docker

[rapid sparrow]

FROM --platform=linux/amd64 ghcr.io/chrislovering/python-poetry-base:3.10-slim i mean he tried building docker image with amd64 arch as source

[tawdry needle]

yeah you will need to run the compiler under rosetta or something. messy

[tawdry needle]

well that should work with amd or arm base

[rapid sparrow]

Anyway, that's why using mac is not a great idea in the world where 98% of pcs and servers run on amd64 xD

[rapid sparrow]

it should not He is not supposed to be able building amd64 in default settings on arm64

[tawdry needle]

ive had zero trouble on my work m1 with py 3.10+

runs natively on arm64

3.7 could be an issue

[timid haven]

cant i just change to this ```js
FROM --platform=linux/arm64 ghcr.io/chrislovering/python-poetry-base:3.10-slim

arm64 instead of amd64

[tawdry needle]

for 3.10 it should work

but amd64 should also work, albeit slower

[rapid sparrow]

it depends on if they published docker image under different architecture in their docker registry

[tawdry needle]

oh true, i assumed they would

the underlying python debian images are

id just use that tbh, and pip install poetry in the dockerfile

[rapid sparrow]

https://hub.docker.com/_/python

Supported architectures: (more info)
amd64, arm32v5, arm32v6, arm32v7, arm64v8, i386, mips64le, ppc64le, s390x, windows-amd64

yeah u can start from python docker image at least

[tawdry needle]

or export a requirements.txt (i think poetry supports that?)

[rapid sparrow]

exactly

[tawdry needle]

pipenv does that, it's what i use

although pipenv also works fine in docker if you pip install it

[timid haven]

if they havent cant i just easily get the amd64 version and convert it to arm64

When i build a docker image I have used --platform before to specify arm64 instead of amd64

isnt it possbile to easily covert?

[tawdry needle]

is there a way to ask docker CLI to rebuild an image?

[rapid sparrow]

buildx plugin enabled or its alternative is required in order to build cross platform docker images.

[tawdry needle]

buildx is available on docker desktop and colima though

[rapid sparrow]

in default CLI in linux it is available too, it is just not configured

[timid haven]

buildx sounds familiar

[tawdry needle]

colima is awesome btw, so much faster and less memory. everything i ever wanted from podman

[tawdry needle]

it's the next generation "build backend", the thing that does the actual image building part. there is a particular environment variable you need to set in order to tell docker to use it

when i'm back on the computer i can look all the stuff up from my actual work projects, because that's exactly what i do

[rapid sparrow]

next gen is going to be running wasm images xD
https://docs.docker.com/desktop/wasm/

removing need for platform

build once, reuse everywhere.

[timid haven]

❯ pyenv --version
pyenv 2.2.5
❯ pyenv versions
* system (set by /Users/user/.pyenv/version)
  3.9.5
``` I only have python 3.9.5

Damn arm64 is hell

i cant install shit

😄

[rapid sparrow]

Now u become understanding what is Apple 😄

[timid haven]

Apple wants their own cpu instead of intel, so they force their users to adapt to get more monopoly

[subtle quarry]

I'm looking for a automated way of making python package release from GitHub repo. Do you have any pointers where i could look for some inspiration? Thanks!

[tawdry needle]

are you talking about automatically publishing a new release to pypi when you tag a new release in git and push that to github? if so, look into Github Actions

[subtle quarry]

Yes want to use GitHub actions. Want to Auto bump package version and publish to GitHub releases and potentialy to pypi

Can't find any inspiration i can use

🙂

[tawdry needle]

there are probably pre-existing "actions" for these, but github actions ultimately are just shell scripts that you define in a yaml file

[subtle quarry]

Yes creates some basic ones. This one struggling a bit. No preexisting templates for that if you click on actions create new

[tawdry needle]

im not sure how this works with pypi credentials though. maybe you use github "secrets" for that

[tawdry needle]

i mean, the actions marketplace probably has "upload to pypi" and "publish github release"

[subtle quarry]

Oh. Let me check marketplace. Thanks

[tawdry needle]

do audit the code and be careful that you trust the source. these are your pypi credentials after all!

[subtle quarry]

Ideally i want update version acc to semver on push to main

[tawdry needle]

should be possible to tag, release, build wheels, push to pypi

[tranquil slate]

Hey! Not sure if this is the correct channel, but relating to kubernetes, I just wanted to ask what the benefit of deploying a monolith to kubernetes is?
From what Im understanding, the main benefit comes when using a microservice architecture

[short peak]

hmm we use it for both. Not sure about clear benefits but a reason why we use it for monolith is that we use Django and using k8s you can setup CI/CD quick, in a startup, time to market, etc that's key. If you have no experience with k8s already the "quick" part might not apply. For microservices we use it to handle hundreds of microservices with different traffic load, for this I'm not sure about benefits we use it mostly because our platform was designed years ago and it was "the way to go" and now it might be expensive to move out of it (time wise). We don't see using k8s as a problem, once you set it up, maintenance is not crazy. Also, I don't pay the infrastructure bill 😅 but there're some providers offering good/cheap service for a managed k8s solution. I'd say, if you have infrastructure skills, budget and you want it "quick", k8s could be a valid option. Notice that you will need to deal with kubernetes file specifications, bash scripts to automate deployments, possibly docker registry, load balancer, etc

[tawdry needle]

you can deploy a monolith app in 3 different containers and route different requests to different containers. we did this at a previous company, we had one pod handling a webook and another pod handling a rest api. same exact codebase, same exact docker image. we used istio to handle the request routing to the right pod.

that is, the python app was "monolithic", but it was still being deployed as kind of a microservice. it did share a database with the other app instances handling the other workloads, but that made sense for our use case anyway.

it wouldnt have made sense to build these as two separate apps even if they were functionally separate services as seen from the outside, because the underlying code and logic was almost entirely identical

[rapid sparrow]

Besides stuff mentioned by salt rock lamp, i can add that with using Kubernetes, you gain access to its Monitoring/Logging/Load balancing/Or even SSL installing solutions, you get access to ecosystem that is not depended on cloud provider, and being done in declarative way at the level of containers.
Less infrastructure pollution basically.

Instead of using AWS Cloudwatch, u use Helm deployed Prometheus/Loki/Grafana for example

and there is no need to operate with Linux Servers (if u use managed kubernetes), everything is declarative

U can consider Kubernetes being... sort of... Cloud infrastructure ecosystem in itself.

[tawdry needle]

you get to avoid vendor lock-in this way too

(at least theoretically)

[mystic void]

opentelemetry can help with that decoupling as well

[tawdry needle]

meanwhile at my company literally everything is AWS and we'd probably go out of business if anything bad happened with that business relationship

[mystic void]

not even looked at new relic, datadog or chronosphere?

[tawdry needle]

actually sorry, we do use datadog and teamcity

but it's all kind of secondary to the AWS stuff

[thorny shell]

we're using datadog, and it's a hell of a lot better than what we had before

[wet valve]

Same here, at least most of the services are migrated 😄

It's a breeze to use

[red oxide]

- name: Clone or update *** repo.
  ansible.builtin.git:
    repo: [e-mail]:***/***.git
    dest: "{{ config_dir }}/***"
    clone: yes
    update: yes
    force: yes
    accept_hostkey: yes
  become: true
  become_user: "{{ ansible_user }}"

I'm trying to clone/update a private repository but I'm struggling with permissions.

Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists

It works well when I test it on vagrant VM but it's again having this issues when I'm trying to make it work on actual server.

and yes, I'm already adding id_rsa with this task:

- name: Add SSH key.
  ansible.builtin.shell: eval `ssh-agent -s` && ssh-add ~/.ssh/id_rsa

[thorny shell]

are the "Add SSH key" and "Clone or update repo" tasks "close together"?

not sure what I mean by that, but, if they run in separate processes, that might explain why the ssh auth info isn't getting from the "Add SSH key" task to the "Clone or update repo" task.

I'd see if ansible.builtin.git lets you specify ssh options, and if so, add one like -i ~/.ssh/id_rsa

[although I'd have expected ssh to grab that private key even without being told to]

another possibility: ssh is reading that private key, but it's the wrong one

again, if ansible.builtin.git lets you pass options to ssh, then pass -v, so you can see a bit more about what's going on

[red oxide]

I add ssh-key at the very beginning of the playbook, this is a very long playbook with many tasks broken down in separate files .
What amazes me is why it would work in a vagrant VM and not in an actual server, shouldn't it behave the same?

[thorny shell]

🤷

[red oxide]

there is an option for key_file, and I tried that already, no luck 😦

[thorny shell]

another debugging idea: have your playbook dump out the complete environment, both in vagrant and real-server land; sort those two lists, then compare them

[red oxide]

--verbose?

[thorny shell]

?

[red oxide]

how can I do this? I'm still learning ansible

[thorny shell]

dunno! 🙂

haven't touched ansible in years

probably easy though

at worst, have it invoke a shell command, and make that command be env | sort

[red oxide]

i see, oki, I'll try it out

[rapid sparrow]

File permissions are main pain
Let me find correct ones

[upper portal]

Hi, I have a question regarding private pypi servers. I am trying to setup a workflow that helps my team version and install pypi packages in the most lean way (not a devops here).
I found out about pywharf project, but it looks stale and I cannot find any contenders.

I have also found a couple blog posts explaining how it is possible to use pip install package from github and use it as a pypi server. However I find it a bit hackish and I am not so found of the fact that the repo needs to contains only the pip package code (AFAIU).

So, my questions are :

• What's the best way to use Github with private pip package ?
• If I have code relying on a particular pip package, does it have to be in a separate repo ?
• Does the index necessarily have to be public ?
• Are there any better (simpler but fully featured) solutions ?

(If I am in the wrong chan, I apologize and will post to the correct one after deleting this post)

[rapid sparrow]

https://unix.stackexchange.com/questions/257590/ssh-key-permissions-chmod-settings/257648#257648

Ensure correct permissions and correct user owner to your operations

Folder leading to .ssh can be affecting too

[tranquil slate]

@tawdry needle @rapid sparrow @short peak Sorry for the ping! Just saw your messages and wanted to say thank you all so much for your help, I really appreciate it 😄 Its very clear now!

[red oxide]

I checked just now, seems like I have correct permissions

I think I understand the issue

so this is what I did just now:

I generated a new ssh-key on the server and added it to my git and ran playbook, and guess what?! It works now!

So, for some reason, it's not respecting the key I'm passing

[tranquil slate]

Hi! I have a question regarding kuberenetes, specifically when used in GKE.

I think I am generally getting confused in terms of mixing the functionalities of what GKE and kuberenetes offer, and hence want to confirm my understanding and ask some questions:

From my understanding: GKE will host a kubernetes cluster on the cloud. GKE offers other services such as metrics, logging etc. and from what im understnading, these monitor the cluster as a whole. We can use kubectl to directly control the kubernetes cluster itself. Hence, GKE just focuses on hosting and managing the cluster as a whole, while kubectl will let us interact specifically with the cluster.

I've played a bit around with minikube, which I kind of see as being GKE In this case. Ie defining num of cores, nodes etc would be the job of minikube and GKE, and actually managing the cluster's internals (ie pods etc) would be the duty of kubectl/whats stated int he deployment config thats applied via kubectl

My questions are:

1. Relating to load balancing, would i still have to specify this is a deployment config, or does GKE handle load balancing within the kubernetes cluster?

2. Regarding replication of pods, does GKE handle this automatically depending on load? Or does this have to be explicitly defined in a deployment config?

[rapid sparrow]

1. Kubernetes has ClusterIP, NodePort and LoadBalancer Services, all of them allowing to load balance traffic.
   All kubernetes cluster usually support ClusterIP and NodePort, even if they are... some very local micro tools.
   LoadBalancer on another hand relies... on additional plugins installed to cluster. If u use GKE Cluster, then during its creation it should create some kind of cloud google load balancer for cluster
   If u use self hosted microk8s cluster, u can use metallb plugin to... achieve using hosts for similar purposes.

2. u define in whatever way you need amount of pods replicas, in any strategy you wish. be it Deployment-Replica / DaemonSet or anything else
   GKE brings autoscaling node groups for you for automated increase of resources for cluster at least.
   Does have kubernetes some additional ways to autoscale amount of pods? It does already in DaemonSet at least, making them equal to amount of nodes.
   May be GKE brings additional features / or may be something was added for regular ReplicaSet? i haven't seen it yet

[rapid sparrow]

2. Additional answer: Kubernetes made part of its defaykt syntax to autoscale replicasets of deployment too
   https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
   Or i missed it seeing before xD

Oh, it is present from 1.23 kubernetes version

[torn oar]

in terms of security is there a major difference between using a service like heroku vs barebones linux vps from linode?

[short peak]

at app level, probably there's not much difference. At operative system level, you might need extra work to secure your vps e.g. blocking undesired traffic, throttle incoming connections when attack is detected, setup user access, etc

[torn oar]

and would these services make this a bit easier or automate it?

[short peak]

usually yes. It's more expensive than a VPS but if you include the time you invest doing that for yourself, it might be cheaper. Unless your unix skills are not a problem but usually that takes time to setup and to maintain

[rapid sparrow]

Especially it is better from the point that... learning bareboned linux, you prepare yourself for usage of container deployment strategies (docker). Which are usually... the best modern approach today.

[torn oar]

that's perfect, I would love to learn all this and planning to but I have to do that in the future. Going live, safely takes precedence

[torn oar]

so you think going the barebones route with docker is the best approach?

[rapid sparrow]

umm, no. i recommend learning barebones route, in order to learn eventually going docker route at barebones, in order eventually going docker-compose route at barebones, in order eventually going docker-deployment-straight-into-cloud route and/or docker-deployment-into-kubernetes route.

[torn oar]

unfortunately time is not a luxury I have

I'll have it in the future though

[mystic void]

dokku is nice for beginners, at least those in between

[torn oar]

so I need something that covers all that as a service so I can run my app and then learn these steps at a later time

[rapid sparrow]

😆

[rapid sparrow]

or actually using then Digital Ocean Apps deployment or AWS Beanstalk

as the most beginner friendly ones. AWS Beanstalk is very scalable too

[short peak]

here is your later - https://roadmap.sh/devops

[torn oar]

tyvm, saving this for later

[torn oar]

checking those two out now too, thanks

[mystic void]

yeah these are nice too

[tranquil slate]

Thanks for your reply! I just wanted to confirm my understanding on ur answer:

So GKE already makes the load balancer for k8s, and hence i don't have to specify this in the deployment config (ie gke configrues this for us)? OR is it that GKE provides a load balancer for the LoadBalancer service in kubernetes. Hence, in the deployment config, I would just have to tell kubernetes to use the LoadBalancer service (and kubernetes would just use GKE's provided load balancer?)

For 2), so just to confirm, GKE handles the resources for the cluster as a whole, and kubernetes has an auto way of doing it via num of pods = num of nodes. Hence, If I want a more custom way of replication, I could specify this in the deployment config rather than GKE/kuberenets managing this automatically?

[rapid sparrow]

GKE provides a load balancer for the LoadBalancer service in kubernetes

Hence, in the deployment config, I would just have to tell kubernetes to use the LoadBalancer service

yes

For 2), so just to confirm, GKE handles the resources for the cluster as a whole, and kubernetes has an auto way of doing it via num of pods = num of nodes. Hence, If I want a more custom way of replication, I could specify this in the deployment config rather than GKE/kuberenets managing this automatically?
not understood

[tranquil slate]

"GKE brings autoscaling node groups for you for automated increase of resources for cluster at least."
Does this mean that GKE only handles the load for the cluster - Ie, GKE will provide more resources if the cluster as a whole requires it (and hence, GKE does not manage adding more pods for example)

"Does have kubernetes some additional ways to autoscale amount of pods? It does already in DaemonSet at least, making them equal to amount of nodes."
Does this mean that kubernetes already load balances automatically via creating the same amount of pods equal to the amount of nodes?

[tranquil slate]

AHhh ok its starting to make sense: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress#:~:text=In a GKE cluster%2C you,with a set of Pods.&text=Note%3A To use Ingress%2C you,Load Balancing add-on enabled. I read through this documentation and it links super nice to what u said.

[lusty forge]

I've got a ZMQ receiver running in a Docker container (using compose) on tcp://127.0.0.1:5555.
I'd like to be able to send and receive messages from this from outside the Docker network, i.e. the OS from another Python script, though I'm not sure how I'd go about 'connecting' these ports, and how the binding works.

[tranquil slate]

Hey! I have a question regarding gitlab CI/CD and GKE. ```yml
deploy:
stage: deploy
image: google/cloud-sdk
script:

Authenticate with GKE

• echo "$SERVICE_ACCOUNT_KEY" > key.json
• gcloud auth activate-service-account --key-file=key.json
• gcloud config set project GCP_PROJECT_NAME
• gcloud config set container/cluster GKE_CLUSTER_NAME
• gcloud config set compute/zone asia-south1-c
• gcloud container clusters get-credentials GKE_CLUSTER_NAME --zone asia-south1-c --project GCP_PROJECT_NAME
• sed -i "s/<VERSION>/${CI_COMMIT_SHORT_SHA}/g" deployment.yaml
• kubectl apply -f deployment.yaml ``` config is from: https://blog.searce.com/gitlab-ci-cd-to-deploy-applications-on-gke-using-shared-runner-47f8c42817ac .

my question is mainly about the last line, in which the deployment.yaml file is applied to the GKE cluster. Lets say I specify 2 things in deployment.yaml: 1) the deployment of my application 2) a load balancer service. Would it make sense to apply this deployment config each time for example a merge is made to master - I am mainly wondering this as wouldn't the load balancer be re-deployed each time as well?

Or, does kubernetes know that the loadbalancer service itself is the same, and hence does not re-apply the configuration for the load balancer each time this config is applied?

[rapid sparrow]

it knows by its name, hence will not reapply.
But i recommend learning stuff like helm
Because it gives ability to gather objects into packages, which u can apply to cluster
if u delete some objects/files form helm package => objects is surely deleted too during next apply of helm

may be there are alternative technologies to achieve the same i suspect another tech can do similar stuff

helm works great with skaffold together (skaffold allows at yaml level to set some of helm settings for applying + additional stuff)

[tranquil slate]

Gonna take a look into it (I remember using it to connect gitlab to the cluster i think), sounds super interesting! Thank you so much for your help 🙂

[tranquil slate]

Hey, I have another question related to deploying a django application to GKE.

I currently have a django app that uses sqlite3. I initially thought that I could just do this:

1. Put the entire app into a docker container
2. deploy this container to the kubernetes cluster on GKE.

Now, I am wondering about 2 things:

1. if I have 3 pod replicas for example, would these pods be able to share the sqlite database between them? Each pod would just run one container which contains the django app and sqlite.
2. https://cloud.google.com/python/django/kubernetes-engine according to the GKE docs, they use cloud sql, and I believe they deploy it as a separate resource. I know that sqlite isnt really the best option for deploying to kubernetes, but is there still a way to use it?
3. As stated above, everything from my django application is in a single docker container. Would it in any way be better/make kubernetes deployment easier if I split the sqlite3 database into its own container? Hence, each pod on kubernetes would have 2 containers: 1) the actualy django app, 2) the database.

[thorny shell]

1. tl;dr: not a good idea.
   https://sqlite.org/useovernet.html

[rapid sparrow]

1. • a bit better idea will be migrating to normal database like postgresql. (pip install psycopg2-binary for its possible usage in django)

[thorny shell]

sqlite is wonderful, but it's not really usable by multiple "machines", and in your case, a pod is basically a machine. You should really look into a real database like Postgresql, or some cloud-based system that Google offers

[tranquil slate]

Hmm alright thanks, imma take a look into that 🙂

[rapid sparrow]

feel free to give a look to ... (checking cheat sheet) Google Cloud SQL with Postgresql Engine too then.
Managed databases are way better idea than trying to fit db into kubernetes

u a always free to have db in kubernetes though, if u don't care about its resizing/backups/loss of data (basically it is temporal db)(ideal for redis for example)

[tranquil slate]

Ahh i see

So basically, cloudsql would be hosted outside of the gke cluster

But in that case, couldnt I just host an SQlite database on google cloud instead of a cloud sql database?

[tranquil slate]

Quick question about this - I may be miss understanding the page u sent, but isnt that related to if the SQlite database is hosted separately from the django app?

Rather than having django and sqlite running within the same container

[thorny shell]

uh, yes; isn't that exactly your situation?

you have three separate pods, all running django

they want to share the same data

therefore they must share a single database

which cannot, by definition, exist on all three pods

[tranquil slate]

Ahhh ok I see, yeah alright I was under the assumption that you could have a database in each and that they could somehow sync

[thorny shell]

not sqlite

fancier databases can sync with each other, I think, sorta, maybe; but I get the sense that that's out of fashion

the cool kids are all using nosql now (or, they were 10-15 years ago; there's probably something even cooler now 🙂 )

[rapid sparrow]

SQlite database is not cloud friendly (and not friendly to be used over network in general), that's why it will be not offered in managed Cloud SQL database object.
There will be options only for postgresql/mariadb and other friendly for usage over network dbs

SQLite is ideal for local usage...when no need to send network requests between different PCs
Application at user desktop using it for local stuff to parse and query results / keeping state of app locally.
Application at client side in user browser can be even using it probably in some form. (in frontend framework like React/Vue)

+Sqlite is too feature simple, and having lack of many validations normal database over network should have.

[rapid sparrow]

get book Docker Deep Dive and learn using docker. then u can get locally postgresql easily. with docker u can use in your dev environment locally(+in tests included) before deploying to managed db+GKE

[rapid sparrow]

Actually, conviniently, i am writing article at the moment... about how to dockerize python web applications
and as part of it... i provide code examples of running tests in postgresql over docker-compose, locally and in Github Actions + Gitlab CI
https://github.com/darklab8/darklab_article_docker_python

Examples for Django and FastAPI are available already
https://github.com/darklab8/darklab_article_docker_python/tree/master/example_frm_django
https://github.com/darklab8/darklab_article_docker_python/tree/master/example_frm_fastapi

U can see example of postgresql usage in docker-compose
https://github.com/darklab8/darklab_article_docker_python/blob/master/example_frm_django/docker-compose.dev.yml

• needed to install stuff + configuring settings.py file

CI examples to run tests automatically
https://github.com/darklab8/darklab_article_docker_python/blob/master/.github/workflows/build.yml
https://github.com/darklab8/darklab_article_docker_python/blob/master/.gitlab-ci.yml

[tranquil slate]

This is really great! Thank you both so much, I really appreciate all of your help 😄

[thorn portal]

repo looks pretty neat!

• you might also want to pin poetry version for increased confidence in reproducibility (still curling a 3rd party script 🤷‍♂️)
• I used to also use Makefile a lot like this, then i found https://taskfile.dev/ to be a better alternative for me personally, perhaps you will find it useful as well (i really hated the necessity of tabs in make)
• personally, i have my volume mount definitions placed in the docker-compose spec (the dev variant of course) instead of in ad-hoc docker-compose run , i believe that is a more consistent behaviour (i.e. if you spin up docker-compose base dev env and change a file on local fs, it's going to be reflected in your docker container 100% of the time)

[rapid sparrow]

• personally, i have my volume mount definitions placed in the docker-compose spec (the dev variant of course) instead of in ad-hoc docker-compose run , i believe that is a more consistent behaviour (i.e. if you spin up docker-compose base dev env and change a file on local fs, it's going to be reflected in your docker container 100% of the time)
  Good note. i thought to leave volume out of docker-compose file for the sake of reusing same compoe file for CI pipeline where it should not be used though

• you might also want to pin poetry version for increased confidence in reproducibility (still curling a 3rd party script 🤷‍♂️)
  yeah, i was lazy. I will fix it. Thanks

• I used to also use Makefile a lot like this, then i found https://taskfile.dev/ to be a better alternative for me personally, perhaps you will find it useful as well (i really hated the necessity of tabs in make)
  OMG, i can't believe it!

i tried to find tool like that but was not able

and i exactly look like that wished already to implement on my own!

The dream comes true!

This tool answers my prays in building CI language agnostic workflows!
(funny enough i thought building it in golang like the tool above)

[brazen forge]

there's also just
https://github.com/casey/just

[rapid sparrow]

nah. that's completely ugly in comparison

[brazen forge]

¯_(ツ)_/¯

[rapid sparrow]

I wished tool written in yaml with syntax being.. readable/user friendly and reminidng me about Gitlab CI syntax, easily chaining stuff, easily supplying environment vars

taskfile exactly like this

[rapid sparrow]

the only thing taskfile is missing from my dreams -> having ability to run some tasks in one off docker image runs

[thorn portal]

devops people are just yaml engineers tbh
yaml is quite nice 😛 love your yaml anchor usage btw

[thorn portal]

i think you can achieve this with CLI_ARGS in task

write {{.CLI_ARGS}} in taskfile.yml
invoke with task docker-exec -- sleep 999d e.g.
i think that works, but slightly clunky

[rapid sparrow]

i see, thanks.

yummi. having full specter of writing CI workflows locallly in CI agnostic language!
CI that can be running locally!

isn't it the dream, in the world of too many CI tools around, that can be run only when git pushing to cloud xD

[thorn portal]

you mean you don't enjoy git pushing to remote, then proceed to twiddle your thumb for 20 mins? (edit: and yes my old job indeed has 20 minutes CI)

shocking 🙀

[rapid sparrow]

for some time i was actually avoiding the problem of Makefile with just straight writing workflows in Python though

#!/usr/bin/env python3.10
import subprocess
import argparse
from enum import Enum, auto
from types import SimpleNamespace
from contextlib import contextmanager
from typing import Generator, Callable

class EnumNames:
    environment: str = "environment"
    action: str = "action"

class TestEnvActions(Enum):
    shell = "enable shell in test env of docker compose"

class DevEnvActions(Enum):
    run = auto()

class Environment(Enum):
    testenv = TestEnvActions
    devenv = DevEnvActions

def enum_name(enum):
    return enum.__name__.lower()

def register_actions(envs, env: str, actions: Enum):
    space = SimpleNamespace()
    space.environment_parser = envs.add_parser(env)
    space.environment_subparsers = space.environment_parser.add_subparsers(dest=EnumNames.action, required=True)

    for action in actions:
        setattr(space, action.name, space.environment_subparsers.add_parser(action.name, help=str(action.value)))
    return space

def parser(input_args) -> SimpleNamespace:
    parser = argparse.ArgumentParser()
    envs = parser.add_subparsers(dest=enum_name(Environment), required=True)
    
    testenv = register_actions(envs=envs, env=Environment.testenv.name, actions=TestEnvActions)
    devenv = register_actions(envs=envs, env=Environment.devenv.name, actions=DevEnvActions)

    args = parser.parse_args(input_args)
    args.environment = Environment[args.environment]
    args.action = args.environment.value[args.action]
    return args

def shell(cmd):
    print(f"{cmd=}")
    subprocess.run(cmd, shell=True, check=True)

@contextmanager
def docker_compose(file, build_args = "") -> Generator[Callable[[str],None], None, None]:
    try:
        shell(f"docker-compose -f {file} build {build_args}")
        def compose_executor(cmd):
            shell(f"docker-compose -f {file} {cmd}")
        yield compose_executor
    finally:
        shell(f"docker-compose -f {file} down")

def run_action(args):
    match args.environment, args.action:
        case Environment.testenv, TestEnvActions.shell:
            with docker_compose(file="docker-compose.test.yml", build_args="-- web") as executor:
                executor("run --rm -u 0 -v $(pwd):/code -w /code web bash")
        case Environment.devenv, DevEnvActions.run:
            with docker_compose(file="docker-compose.yml") as executor:
                executor("up")

def main(input_args=None):
    args = parser(input_args)
    run_action(args)

if __name__=="__main__":
    main()

The only little problem is quite a bunch of you know... code to boilerplate each time / not very consistent

and this decision is welcome only in python projects, and still not always

argparse + Enum + match-case, and it all worked nicely

[thorn portal]

well i don't think makefile is a problem per se, it's just that i prefer task 😛 and it's just a single binary to be dropped into wherever you need it and it's not that intrusive

i think if you are in a python-based shop, this is completely acceptable as well!
(iirc in my old job, we made heavy use of paver and went for a less homebrewed solution)

(also getting fancy with the match case already eh? - i haven't used it like at all yet)

[rapid sparrow]

(also getting fancy with the match case already eh? - i haven't used it like at all yet)
yeah, it is useful alternative to writing if, elif, or making hacky dictionaries containing lambdas

[rapid sparrow]

paver is like... click python library already almost

hmm, it can be installed without polluting pip packages? 🤔

well, i need to try task and paver i guess

[thorn portal]

i prefer task these days, yaml all the way haha