tools-and-devops

[red oxide]

I hope your ssh solution does the job

[rapid sparrow]

Everybody usually uses ssh. Quality is guaranteed.
https://media4.giphy.com/media/1hAxQTH0HEWS3L0oRF/giphy.gif

[red oxide]

@rapid sparrow thanks, I tested just now and it works flawlessly!

[red oxide]

@rapid sparrow I'm running this command so far:
killall run.py && cd worker-mobile-ip-auth/ && git pull origin main

which will kill all the process of the running old code and then update the code,
now I can't just do python run.py here right? Because the process will be killed when the ssh connection closes.
How can I run it so the script keeps running even when the ssh connection disconnects?

[rapid sparrow]

run it as daemon.
there is an easy way to do that
and a bit less easy way to do that.

[red oxide]

I know about tmux, but it requires CTRL + B, d to detach and I'm not sure how to do it though the way I'm doing it

[rapid sparrow]

wait a second, it requires almost nothing

[rapid sparrow]

nohup COMMAND TO RUN &

it launches program as daemon in background

Ctrl+C will not disrupt the program

so you can launch forever loop for example

BUT!

[rapid sparrow]

usually daemons are launched with systemd
https://medium.com/@benmorel/creating-a-linux-service-with-systemd-611b5c8b91d6

BUT!

[rapid sparrow]

this is entirely wrong silly approach. Funny for education though. I was thinknig in my beginning of the path to go same way

The right way is to utilize CI/CD pipeline

that redeploys your infrastructure at commit

[red oxide]

damn, I have no idea 😄

[rapid sparrow]

I would recommend to learn usage of Github/Gitlab CI (or any other 2nd generation CI tool)
and... Docker-Compose+Ansible for starters? It should be good enough to get working it

[red oxide]

okay but can we not build up on my "hacky" approach for the time being?

[rapid sparrow]

sure. https://youtu.be/FbFJZUwmUZE

[rapid sparrow]

btw, for your hacky way 🤔

I could recommend how to improve your hacky way in the most minimalistic approach

You need just Docker-Compose and Cron job

pack your application into docker-compose, it will have full rebuild and relaunch into just one command
git pull new stuff && docker-compose up --build -d
and place Cron job that goes to your project and launches the command as much often as you need

you will wish to run the command only when the new stuff is pulled though

[red oxide]

hmmm, I'll have to start with docker then

[rapid sparrow]

well, actually instead of Cron Job you could have the same shell script every N seconds, which i mentioned above

[rapid sparrow]

which runs in nohup

whatever. docker-compose will simplify reinstallation of dependencies / and will make easy clean start every time

less often maintance for your thing

[red oxide]

if I were to go nohup route then how would it look like with existing command that I have right now?

this
killall run.py && cd worker-mobile-ip-auth/ && git pull origin main

[rapid sparrow]

not sure for sure, but you could try it like
nohup $(killall run.py && cd worker-mobile-ip-auth/ && git pull origin main) &

i mean you need wrapper for loops anyway

nohup can be launching just another shell script that is looper 😉

or actually just use Cron Jobs already

https://www.howtogeek.com/101288/how-to-schedule-tasks-on-linux-an-introduction-to-crontab-files/#:~:text=Opening Crontab&text=Use the crontab -e command,the root account's crontab file.

lets not reinvent the weel for task scheduling

[red oxide]

okay, I'll look into all this. Seems like I got a lot for stuff to learn 🤧

[rapid sparrow]

I can promise you, that's just the beginning of a hell of a ride.
Which is worthy. Because it is the path to the world of automatizations!
Which speed up development tremendously. Well, and turn your infrastructure into code. And make software more reliable.
Time savings everywhere!

[red oxide]

okay, thanks for all the info... I'll go though everything

[rapid sparrow]

u a welcome and good luck 😉

[rapid sparrow]

btw just in case, if you will rethink to do it in the CI/CD
the most minimalistic set of technologies would be CI/CD tool + Ansible

those two should be enough to perform it

docker can be skipped, even if desired.

Well, actually. It is possible to execute ssh command remotely 🤔

Ci/CD tool + remote ssh execution technically is sufficient too

although much less elegant than ansible (with a touch of docker)

[red oxide]

Okay, thanks for the heads-up

[analog lava]

Is there some tool to plan code(classes etc.)? like diagrams or so?
I mean something to visualize my ideas, before coding.

[rapid sparrow]

https://app.diagrams.net/ i use this tool for this

it comes in offline version too i think

https://github.com/jgraph/drawio-desktop/releases/tag/v16.1.2

[analog lava]

cool

thanks

[heavy knot]

I run command
docker build -f Dockerfile -t photo_math_container .
and it is running for 10 mins maybe and it's still 5/10

This is my Dockerfile

[heavy knot]

Now I got this error

[heavy knot]

Now for some reason this is how my CMD looks like

[tawdry needle]

@heavy knot is it trying to compile tensorflow from scratch or something?

also, why not put all those in the same pip3 install command?

[tawny temple]

Wow really old comment. No it doesn't only do 2. It's heavily customisable.

I'm not sure cause I've never used it with VSCode. I always create the config file for each project anyway. I don't think there is a global config location it can use

[velvet spire]

maybe just your home dir?

it might scan upwards?

[rapid sparrow]

don't instlal all apt/pip3 things in one RUN

split every installation to different RUN commands

it would be easier and faster to debug and work with it
having installation of every pip3 thing in one go makes sense only when you are having them frozen in requirements.txt.
but since you are already having them written explicitely in dockerfile, lets evolve your idea further.

Btw. since you don't mention for pip3 exact version, you aren't really having frozen dependencies.

[sly sleet]

yeah it searches parent directories

according to docs

[tawny temple]

https://paste.pythondiscord.com/lesofofabe.rust unfortunately it cannot exactly format func calls like I want. There is a patch and PR with lots of discussion on it but I can't find it right now.

I suppose that's getting too off topic anyway

Oh it was https://reviews.llvm.org/D33029 and you're welcome

You'd have to build the program from source using that branch, and even then IDK if a custom version can be used with vscode

Probably cause you're using an older version of the tool. Also I don't really think it's on topic to be discussing clang format here since this channel is intended for python stuff. You can ask in one of the off topic channels if you have more questions.

[heavy knot]

Thanks for response. What you mean by that?

also, why not put all those in the same pip3 install command?
Yeah, I was unsure whether I can do it in Docker - this is my first time of building Docker imagee

[heavy knot]

Could you please edit my Dockerfile because I am not sure how would it look like?

FROM ubuntu:20.04
MAINTAINER "X"

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update && apt-get install -y \
    libopencv-dev \
        python3-pip \
    python3-opencv && \
    rm -rf /var/lib/apt/lists/*

RUN pip3 install tensorflow && \
    pip3 install numpy && \
    pip3 install  matplotlib && \
    pip3 install  jupyter && \     
    pip3 install keras && \
    pip3 install opencv-python && \
    pip3 install imutils

RUN ["mkdir", "notebooks"]
COPY conf/.jupyter /root/.jupyter
COPY run_jupyter.sh /

# Jupyter and Tensorboard ports
EXPOSE 8888

# Store notebooks in this mounted directory
VOLUME /notebooks

CMD ["/run_jupyter.sh"]

Btw. since you don't mention for pip3 exact version, you aren't really having frozen dependencies.
What you mean by frozen dependicies?

[indigo zenith]

That running your app on different environments may end up with different versions of the required packages. If you're lucky this won't matter but it can cause all kinds of unexpected errors

[tawdry needle]

Just write pip3 install tensorflow matplotplib ... once, listing all of the libraries you need

Also because this is an application you definitely should pin your dependencies at exact versions

[heavy knot]

Yeah, that's also what I assumed. Thanks for advice

[tawdry needle]

@heavy knot what is this dockerfile for, anyway?

[heavy knot]

You mean what project?

[tawdry needle]

what kind of usage?

is this just to serve a notebook in "read-only" fashion?

[heavy knot]

RUN apt-get update && apt-get install -y \
    libopencv-dev \
        python3-pip \
    python3-opencv && \
    rm -rf /var/lib/apt/lists/*

Guys, I am not sure why is above code needed. I don't understand what does rm -rf /var/lib/apt/lists/* and also why above that is needed.

[heavy knot]

To read and run notebooks

[tawdry needle]

@heavy knot the 2nd line just clears out the list of packages fetched by apt, which makes the container smaller

see https://askubuntu.com/a/179992/347888

APT is the package manager in ubuntu

apt update updates the list of known/available packages, apt install installs stuff

[heavy knot]

By 2nd line you mean? rm -rf /var/lib/apt/lists/*

[tawdry needle]

yes

where did you get this file?

btw this is what i was suggesting earlier (i also reformatted things a bit):

FROM ubuntu:20.04
MAINTAINER "X"

ENV DEBIAN_FRONTEND noninteractive

RUN    apt-get update \
    && apt-get install -y libopencv-dev \
                          python3-pip \
                          python3-opencv \
    && rm -rf /var/lib/apt/lists/*

RUN pip3 install imutils \
                 jupyter \
                 keras \
                 matplotlib \
                 numpy \
                 opencv-python \
                 tensorflow

RUN ["mkdir", "notebooks"]
COPY conf/.jupyter /root/.jupyter
COPY run_jupyter.sh /

# Jupyter and Tensorboard ports
EXPOSE 8888

# Store notebooks in this mounted directory
VOLUME /notebooks

CMD ["/run_jupyter.sh"]

[heavy knot]

Why it's needed to install python3-opencv if it's installed below with pip3 install opencv-python

[tawdry needle]

it isn't. there are some less-than-optimal practices on display here

where is this container being run? who is going to use it?

[heavy knot]

Well all I know about container is that they make possible that libraries mentioned in image will be run

I don't know what is this process here, but I got idea that Docker is about that people don't install libaries on their own

[tawdry needle]

I don't really know what you mean by either of those things

Back up here. What are you personally trying to do with this dockerfile, and who gave you this code?

[heavy knot]

So is it correct that they will have to run containers on their PC and just before that download image from somewhere?

I found it here https://hands-on.cloud/how-to-run-jupiter-keras-tensorflow-pandas-sklearn-and-matplotlib-in-docker-container/

[tawdry needle]

Kind of, yes. Are you trying to distribute an application to users?

[heavy knot]

I don't really know what you mean by either of those things
What part is confusing?

[tawdry needle]

You haven't explained what you are actually trying to do

It's hard to make recommendations without knowing your goals

[heavy knot]

Yeah, I just first wanted to make sure whether my assumptions are right

So what I am trying to do is to make it possible that someone who want to run my application run it without losing time on installing libraries

[tawdry needle]

OK, your understanding seems correct. If you build an image and upload it to Docker Hub or some equivalent "registry", users can fetch that image and run containers using it

So yes, the Dockerfile should set up the system in the image with all the required dependencies and files

[heavy knot]

@tawdry needle I built image but can't see in current directory

Should I also upload some other files except image to registry?

I have this file "run_jupyter"

[tawdry needle]

can you clarify this question?

what did you do, what did you expect, and what happened that differs from what you expected?

[heavy knot]

And I also have jupyter_notebook_config file in conf folder

[heavy knot]

This is what I did - just built image

I thought image would be in folder when I ran that command for building image

[tawdry needle]

no, images are stored in a central location

you shouldn't mess with those files directly

you should see it if you run docker image ls

also, did you read https://docs.docker.com/docker-hub/ ?

it says to use docker push to upload images to docker hub

[heavy knot]

So there is no difference if I push my image to GitHub hub?

[tawdry needle]

github?

[heavy knot]

I will read it not, I will check it out

[tawdry needle]

what does github have to do with anything here?

[heavy knot]

Well I was very sleepy yesterday when I was reading about that, but I think I read something about GitHub Hub or GitHub registry - don't know if those are same things

[tawdry needle]

there is no such thing as "github hub", and "github" is unrelated other than the word "hub"

[heavy knot]

Lol yeah

@tawdry needle Ok I thought that there are Docker registries provided by GitHub

[tawdry needle]

you can rig up docker hub to build images based on a dockerfile hosted on github https://docs.docker.com/docker-hub/builds/link-source/

but they aren't really "related"

[heavy knot]

What you mean by "rig up"?

[tawdry needle]

configure

you can put a dockerfile on github, and then tell docker hub to build an image whenever that dockerfile is updated

[heavy knot]

So what about files such as run_jupyter and jupyter_notebook_config?

[tawdry needle]

sure, yeah. but it seems unnecessary in your case

[heavy knot]

Why?

[tawdry needle]

Github is for hosting Git repositories. Git is a "version control system" for source code

because if you don't currently use Git, then you have to learn Git

[heavy knot]

Yeah I know basics of Git

[tawdry needle]

then sure, go ahead

[heavy knot]

So I will upload my notebooks and run_jupyter and jupyter_notebook_config to GitHub repo

[tawdry needle]

also the dockerfile of course

however i doubt it will work as a volume

in fact i'm almost sure it won't

[heavy knot]

Also I will push image to Docker Hub

[tawdry needle]

the user will have to download the notebooks separately

i think you need to read the docs and stop guessing

[heavy knot]

Why Dockerfile?

[tawdry needle]

this "automated build" system is a replacement for uploading images to docker hub

[heavy knot]

What you mean by "work as volume"?

[tawdry needle]

you have VOLUME in your dockerfile

i strongly recommend you stop what you are doing and read the docker documentation

i'm happy to help, but you have blindly copied and pasted a bunch of code that you don't understand. that is always a recipe for bad things

if you don't know at least on a basic level how your tools work, you won't ever be able to use them

i'm not trying to be a jerk, i'm trying to save you from getting neck deep in things you don't understand

it's also a lot harder to ask for and receive constructive help if you don't know the basics

[heavy knot]

Haha ok I was curious. I thought it's like this I upload all notebooks to GitHub repo and run_jupyter and jupyter_notebook_config. Then push image to Docker hub. Then someone who want to run my app will clone GitHub repo, take Docker image from Docker Hub and run container

[tawdry needle]

i recommend reading this front to back at minimum https://docs.docker.com/get-started/

[tawdry needle]

if you want to know about automated builds, read the docs and don't guess. https://docs.docker.com/docker-hub/builds/

software is written by humans: it's arbitrary and complicated

sometimes you can get away with following your intuition and/or familiar patterns, but you can't really know unless you have read the docs

that's assuming the docs are correct, but that's another story entirely

[velvet spire]

this is how I started with docker...

... ended with me asking every conceivable question about docker here

[heavy knot]

@tawdry needle Thanks for advices. Sorry for my wrong attitude, I have maybe stupid kind of behaviour with regards to coding - I love to make something and then understand how does that work.

[abstract jay]

I have a bunch of python scripts on Debian written in text editor without an IDE. I want to migrate them into an IDE - pycharm for now. But I want themt to work called from the linux shell. In this setting is it better to create a venv for them? How would invoking them from the shell deal with the venv?

[tawdry needle]

yes, imo it's better to create a venv. you can always write a shell script that internally uses the venv

[abstract jay]

Okay, makes sense. Can I share one venv for multiple scripts that import mostly the same group of libraries?

[tawdry needle]

sure

#!/bin/sh
exec /opt/scripts/.python-venv/bin/python /opt/scripts/do-important-stuff.py "$@"

save that to /usr/local/do-important-stuff or whatever

[abstract jay]

and can pycharm use a single venv in more than one project? or should I just make one big project containing multiple scripts (which do different things)?

[tawdry needle]

yes, pycharm can re-use the same venv for multiple projects, although things might get a bit messy keeping track of dependencies, e.g. if you have different requirements.txt files

[abstract jay]

okay. thanks very much

[tawdry needle]

you might also want to try using version control

Git is pretty much the standard VCS nowadays

[abstract jay]

yes, good idea. Our group has used rcs for VC mostly

but git is available and probably preferable.

I am just 2 years from retirement and I don't want to leave my group a plate of spaghetti

[tawdry needle]

that's kind of you!

[abstract jay]

I still have a couple of key scripts in perl, which I'd used for > 12 yrs before adopting python to "stay current"

I may start to tackle refactoring the perl scripts into python

The range of packages for python is just stunning

I'm loving the one called 'openpyxl' that can read and write Excel files. Really opens up choices for working with admin staff

[tawdry needle]

yeah it's quite amazing. but there's nothing wrong with perl imo

as long as you write readable perl 🙂

but i find that it's easier to write readable/maintainable scripts in python than in perl

[abstract jay]

yeah, i made tons of use of perl's associative arrays, now termed dictionaries in python.

[tawdry needle]

this one is well-loved in the data science world, it's been essential for me to do my job in the past

[abstract jay]

I used openpyxl to process hundreds of forms done in Excel that were not data-normal by any stretch

people would add a row near the top, so I had to key on the label fields to watch out for that.

a lot of my code was to steer around curve balls like footnotes etc that humans assumed another human would understand

Now I need to get my head around how pycharm handles packages vis-a-vis a venv that includes them (so the shell can use the scripts w/o the IDE)

[heavy knot]

"If a directory in the container is mounted" what does "mounted" means in this sentence?

[abstract jay]

Luka: hmm - it's a standard term in Linux (in Windows it would be like "map network drive")

sorry I'm not a docker user so I'm not sure how that env. applies the concept

[heavy knot]

To me it seems that if directory from particular container is made available outside container

[tawdry needle]

i think pycharm attempts to parse the requirements.txt file or setup.py file and then warns you when those packages are missing + offers to install them for you

otherwise there's nothing really to "migrate"

you just tell pycharm which python interpreter to use for your project

oh also pycharm lets you "attach" multiple projects together into a super-project

so maybe you will want to do that for your setup

[heavy knot]

@tawdry needle Can you please explain what is done with VOLUME /notebooks

[tawdry needle]

i recommend reading about what a docker volume is 🙂

it will be easier than me explaining it

[heavy knot]

I read this article https://docs.docker.com/get-started/05_persisting_data/ but it's still confusing

Is that once particular container is removed, then what was programmed will be lost?

[tawdry needle]

normally yes. if you use a volume, the stuff in the volume will be kept after the container goes away

[heavy knot]

Why did you say this?

[tawdry needle]

if docker hub is pulling from the git repo, how else does it know how to build the image?

[heavy knot]

Lol, I should stop assuming that my assumption is right. At least that's what I learnt today. I will read https://docs.docker.com/docker-hub/

[heavy knot]

@tawdry needle Is it like this in my case: push to GitHub notebooks, run_jupyter, jupyter_notebook_config and push to Docker Hub Docker image. Then when user want to use app, user clone project from GitHub, pull Docker image and run container. In my case Docker file will not be needed as I don't want to use Docker Pro. Basically, automated builds are that Docker Hub build image from an external repo.

[tawdry needle]

i don't think so. i am not an expert either, but my impression of the documentation is that the Automated Build system will build the image for you from a Dockerfile in your Git repo

but you don't have to use the automated builds either

you can upload your container images to docker hub, and then upload the other assets to github

then the user has to pull the container image from docker hub and the git repo from github

[heavy knot]

Yes okay, so it works like I mentioned

Btw what's your role in your job?

And also, why are there channels help-food?

[tawdry needle]

there is a large pool of channels with easily distinguishable names. they used to be atomic elements

[tawdry needle]

currently software engineer, formerly data scientist

[tawdry needle]

yes, but that's not how automated builds work!

[heavy knot]

they used to be atomic elements
What you mean by "atomic elements"?

Why did you stop being data scientist?

https://docs.docker.com/docker-hub/builds/

"Docker Hub can automatically build images from source code in an external repository and automatically push the built image to your Docker repositories."

[tawdry needle]

things in the periodic table of elements in chemistry, like argon and strontium

[tawdry needle]

right, but the point is that with automated builds you don't build the image and upload it. with an automated build, docker hub fetches the dockerfile and builds it for you, then puts it in your docker hub repository

[heavy knot]

I see

Why did you stop being data scientist?
@tawdry needle

[tawdry needle]

lots of reasons. if you look for messages sent last night in #data-science-and-ml you'll see me discussing it a little

mostly personal reasons + covid led to some good opportunities elsewhere

[ancient seal]

Working on a tutorial on Git for exactly that - I'll publish once I complete it

[half grotto]

Hi all, I lead a team of customer insight and marketing analysts. We are definitely not developers, nor sysadmins. We want to move our analysis/ETL away from SAS and towards Python. We've got some experience now playing around with Python/Pandas (via JupyterHub) and are wanting to set up a proper dev/prod environment. Our operations team has more experience in Windows (the corporate environment) than Linux, so we're going in a little blind. Where would I go to find out more about what I will need to do to move towards a local (not cloud), production-ready environment that will support Windows LDAP auth, and ODBC connections to our MS SQL servers?

[heavy knot]

my setup.cfg is wrong because of the install_requires:

[options]
[... other stuff ...]
install_requires=['mypy_extensions', 'backports.zoneinfo; python_version < "3.9"']

I know it's this part 'backports.zoneinfo; python_version < "3.9"' is stopping me from python3 -m building

    raise InvalidRequirement(
pkg_resources.extern.packaging.requirements.InvalidRequirement: Parse error at ""['mypy_e"": Expected W:(abcd...)

ERROR Backend subproccess exited when trying to invoke get_requires_for_build_sdist

is the syntax wrong?

[heavy knot]

it's def something wrong with install_requires=['mypy_extensions', 'backports.zoneinfo; python_version < "3.9"'] because it runs without it

[heavy knot]

hi! i want to access this directory. How can i?
AppData\Local\FiveM\FiveM.app

[rapid sparrow]

in which way /form / language and where do you wish to access it, and for what. Spill details.

[heavy knot]

Hi I have only a small question does anyone know a tool that automatically accepts the Discord recaptcha but without hcbypass?

[tawdry needle]

i've used odbc (specifically pyodbc) before on linux to connect to ms sql server and azure data warehouse, and i think the only special requirement was that you had to install a special driver on the system, https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver15

regarding ldap, i did some quick searching and it seems like you need to run OpenLDAP on the machine that you want to log into

https://www.linux.com/news/linux-ldap-authentication/ this seems like a decent tutorial, in the "Client Configuration" section

[half grotto]

Thanks for the reply. Unfortunately I'm not a sysadmin (nor a developer), more looking at locking down specifics of how we'd want our system to work then trying to find external contractors to help us set it up (as mentioned, our sysadmins are primarily Windows-based). We managed to cobble a VM together ourselves and it's working alright, but definitely not production ready.

However, I just don't know what sort of contractors we'd need to engage, so don't even really know what to Google for. That's mostly what I'm looking for.

Kinda looking for someone to ask questions such as "how should we package these mini automations?", "what's the best way to handle environment variables/credentials?", "what's the best way to go from testing something to deploying it, then monitoring it?", "how do we handle local/private libraries?", etc.

[tawdry needle]

the answers to most of these questions will be some variation of "it depends". but i think it sounds like you need some combination of IT and DevOps

how are you currently running jupyterhub?

is there any reason you can't just run it on a windows server for the time being?

"migrating to linux" seems at least somewhat orthogonal to "migrating to python"

[half grotto]

Currently running it on a Ubuntu VM. Running on Linux because JupyterHub doesn't exist for Windows.

That was the main reason for Linux.

[tawdry needle]

huh, i thought jupyterhub was just a python app that could run on any platform

ah https://jupyterhub.readthedocs.io/en/stable/installation-basics.html

JupyterHub officially does not support Windows. You may be able to use JupyterHub on Windows if you use a Spawner and Authenticator that work on Windows, but the JupyterHub defaults will not.

so who is using this system? what needs to be done with it? what kinds of "mini automations" are you talking about?

and importantly: how do you do all this currently?

[half grotto]

It's a small team of 3 analysts who run mostly ETL tasks on data stored in a MS SQL server for use with marketing automations. Currently these are run using SAS and Windows Task Scheduler.

[tawdry needle]

i see

i unfortunately don't know much about finding IT contractors, and my experience working with them in the past has been not-good

obviously linux and python experience are important

[half grotto]

Haha, yeah. It's been a strange journey so far - because we don't know exactly what we want yet which makes it even harder.

We just know we'd like the flexibility that Python provides.

[tawdry needle]

fwiw if you plan to do this long-term, it might be better to just invest in your in-house knowledge

and yeah i guess bring a consultant in to get things set up nicely

if all you need is to have a server run a python program intermittently, you can get by with a pretty minimal setup

[half grotto]

That was the plan. Part of the contract will be getting run through the setup and documentation on it. We'd want to be able to self-maintain.

[tawdry needle]

run each application in its own venv, run on a schedule with cron or systemd timers. not sure about monitoring solutions but a lot of people seem to like a tool called prometheus

[half grotto]

Perfect, that response is almost exactly what I was looking for!

(Having only just recently learned about venvs myself)

[tawdry needle]

environment variables in general... depends. credentials you can read from a file with your python code, or put them in env vars

[half grotto]

Any specific venv system/manager that is more useful than the other?

[tawdry needle]

my understanding is that if someone gets enough access to your machine that they can read env vars from another process' memory, you are pretty fucked anyway

[tawdry needle]

python -m venv might be good enough to start with

you could try using poetry or pipenv

personally i'm mehh on starting out that way though

[half grotto]

I was actually just looking at poetry. Does look pretty cool.

[tawdry needle]

especially if you want to learn and develop your team's knowledge, start with the simplest tools, because really that seems like all you need at this level

especially since it's such a small team

you all can figure out your needs as you go and learn along the way

[half grotto]

What about internally developed libraries? Create our own local pip repository for them?

[tawdry needle]

much much better imo than starting out with some complicated tool that turns out to not do what you want, have bugs, etc.

[tawdry needle]

yes, that's an excellent idea. and surprisingly easy to do

you can even set up a mirror for pypi so you don't beat up their servers so much

[half grotto]

We wouldn't want them to be public anyway, they would just be small helper functions specific to our use cases.

[tawdry needle]

https://packaging.python.org/en/latest/guides/index-mirrors-and-caches/
https://packaging.python.org/en/latest/guides/hosting-your-own-index/

at bare minimum, you just need an http server with "autoindex" ability

you might also want to read https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/

and https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/

really all the guides are good

[velvet spire]

poetry. 🥴

it's great for some things, much less so for others.

[half grotto]

Yeah, we kinda skipped a lot of (what I'm now seeing as) the necessary steps. Jumped straight into playing around with data in JupyterHub without any foundational Python knowledge (and basically no programming knowledge).

[tawdry needle]

meh

[velvet spire]

I essentially have come to use it just for dependency management. good for applications, but if writing a library, I don't think I would consider using poetry.

[tawdry needle]

python is easy enough that you can just jump in and start doing stuff

[tawdry needle]

i've been using pip-tools personally, good enough for me

[velvet spire]

!pip tools

[rancid schooner]

tools v0.1.9

Set of tools for web scraping projects

[velvet spire]

!pip pip tools

[rancid schooner]

pip v21.3.1

The PyPA recommended tool for installing Python packages.

[velvet spire]

!pip pip-tools

[rancid schooner]

pip-tools v6.4.0

pip-tools keeps your pinned dependencies fresh.

[velvet spire]

finally

[thin jasper]

Im a happy poetry user, but yeah, it's more useful for projects than for libraries

Don't use pipenv

[coarse anvil]

how can i activate venv in vscode?

[lime trellis]

Open terminal then activate it by running the "activate" script, bat, ps, or the executable file, depending on your environment, Inside vscode

[obtuse vault]

Need help with the AWS SDK "boto3"

[manic temple]

from py_ecc.bls import G2ProofOfPossession as bls_pop
from py_ecc.bls.g2_primitives import signature_to_G2, pubkey_to_G1, G1_to_pubkey, G2_to_signature

pk = G1_to_pubkey(11181692345848957662074290878138344227085597134981019040735323471731897153462, 6479746447046570360435714249272776082787932146211764251347798668447381926167)
sig = G2_to_signature([18523194229674161632574346342370534213928970227736813349975332190798837787897, 5725452645840548248571879966249653216818629536104756116202892528545334967238],
            [3816656720215352836236372430537606984911914992659540439626020770732736710924, 677280212051826798882467475639465784259337739185938192379192340908771705870])

print(pk)
print(sig)

I am not able to get public key and signature from G1 and G2 Points fron BLS

[left root]

Hello guys 👋
I'm trying to monitor jsp web-app ,the path mapped between my Jenkins container and my tomcat container.

I got "java.net.ConnectException: Connection refused (Connection refused)".
I'm fairly confused because I cant understand the cause of this error.
I will appreciate any help ,thank you in advance!

[halcyon dock]

I am looking to completely replace pandas with excel for campus utility budgeting, is this a good idea to solely use pandas or use it in combo with excel?

excel with pandas*

I am not a programming expert but I am learning and not sure if I am heading in the correct direction

[odd locust]

Hello guys, when i do git checkout origin/mybranch i get something with head detached

is this a problem?

or shouldnt I worry about it

[indigo zenith]

Really depends on what you're trying to do. If non-developers need to interact with your data, pure Python obviously won't be sufficient. What makes you think you want to switch to Pandas?

[indigo zenith]

It can be if you don't understand what's happening. https://www.git-tower.com/learn/git/faq/detached-head-when-checkout-commit

[halcyon dock]

Well, I am looking for good real-world projects

That I can use current/future

I am the only person who actually modifies the data

[indigo zenith]

Makes sense... Is there some specific task that you wish you could do in Excel but can't, or something that requires a bit more work the it should? In other words, I would focus on small but useful problems to solve

[halcyon dock]

Hmmmm, Advanced regression analysis

Thinking it might take quite a bit of time to really get good at in python but probably pay-off in the long run

Modeling mechnical systems (HVAC)

[indigo zenith]

Yeah, if you haven't tried loading your data into Pandas at all yet, start there, then maybe the next step once you do that would be some visualizations of the data, and running regressions like you say. Any data science tutorials you want to run through can be applied to your own data

You could also think about automating your data pipeline depending how the data is getting into Excel in the first place.

[halcyon dock]

So I have done data extration and automation between spreadsheets but wondering if using it for this specific application is going to be a good idea

Do you recommend sklearn, statsmodel, or any other regression tool over others?

I noticed that some make it difficult to force intercepts (if not impossible)

I am talking multiple linear (polynomial regression). Let me show you an example

[rancid schooner]

Hey @halcyon dock!

It looks like you tried to attach file type(s) that we do not allow (.xlsx). We currently allow the following file types: .gif, .jpg, .jpeg, .mov, .mp4, .mpg, .png, .mp3, .wav, .ogg, .webm, .webp, .flac, .m4a, .csv, .json.

Feel free to ask in #community-meta if you think this is a mistake.

Hey @halcyon dock!

It looks like you tried to attach file type(s) that we do not allow (.pdf). We currently allow the following file types: .gif, .jpg, .jpeg, .mov, .mp4, .mpg, .png, .mp3, .wav, .ogg, .webm, .webp, .flac, .m4a, .csv, .json.

Feel free to ask in #community-meta if you think this is a mistake.

[halcyon dock]

The example of data creation to be used in regression analysis above

[indigo zenith]

#data-science-and-ml might be better able to help you there

[halcyon dock]

thx

[vital coyote]

Can you simulate user inputs in Python, similar to AutoIt/AutoHotKey?
Is there a library that exists for this?

Or is the point moot?

[tawny temple]

Yes, with pyautogui. There's also a Python wrapper for AHK but I am not sure why that would be preferred over a pure Python solution.

There are other libraries too, but I don't remember them all. I believe keyboard is another.

[vital coyote]

Is pyautogui a cross-platform solution, capable of handling inputs in Linux?

Also keyboard sounds like a PSL module, and now I wanna see if it is lol

[tawny temple]

Yes it's cross-platform

[vital coyote]

Ooo, then perhaps that's just the library for me!!

I'm looking to build a layer of abstraction over debian's Scrot tool, and I wanted to make hotkeys that had customizable behavior for screenshots

It's mostly for private use, but I don't mind publishing the code for people that are curious 🤓

[vital coyote]

Also looks like keyboard is NOT from the PSL; it is installed via pip

[vital coyote]

Once again, cheers!

[tawny temple]

You're welcome

[left root]

Can someone explain to me why can't I access to my Jenkins through the Prometheus?

[light elbow]

so i want to program something where i do on a zip right click open with and then my program. how can i do that that that the open with works

[heavy knot]

why cant i upload a folder to github

[rugged hare]

show us what you did.

[heavy knot]

okay

[rugged hare]

think about it: how would we be able to help without more information?

[heavy knot]

i went there

inside of my repo

and clicked add file

idk how to add a folder tho

[rugged hare]

(my face is in the screenshot 🙂 )

[heavy knot]

😆

ew that laugh emoji

[rugged hare]

i don't know if you can add a folder on the website like that. you might want to use the command-line, or a gui client.

[heavy knot]

where can i upload my project folders thenm

[rugged hare]

you can upload them to github, just not through the github web site.

[heavy knot]

i thought github was a place where you can upload ur projects and stuff

how do i do that then

[rugged hare]

it is. but mostly people do it with the git command-line tool, not the web ui

[heavy knot]

it would be a lot simpler if they allowed you to upload it through the website.

[rugged hare]

i guess it would.

did you try "choose your files"?

[heavy knot]

yes but like it says i can only choose a file

and my projects are in folders so they are more organised

[rugged hare]

that sounds good. Now might be a good time to try out the git command-line tool

[heavy knot]

like i cant upload any of those

How do i use the git command line tool?

[rugged hare]

how did you make this repo on github?

[heavy knot]

just added it on the website a few months agho

ago

[rugged hare]

maybe you can try github desktop: https://desktop.github.com/

[heavy knot]

what email do i use

just the same one which i used for github?

[rugged hare]

sure

[heavy knot]

how do i open i

it

the repo

[rugged hare]

i've never used github desktop, i imagine it's straightforward. Double-click the name of the repo?

[heavy knot]

ive already tried that

[rugged hare]

i don't know, sorry.

[novel lotus]

Help me pls

i cant use it

[raw axle]

hello, if I am on the master branch and have to merge a dev branch and get a merge conflics error should I try to fix the problem while on the master branch or should I switch to the dev branch and fix using rebase?

[rapid sparrow]

Because it supposed to be not empty.
Put .gitkeep empty file into it

[glacial plaza]

Does anyone happen to know where pip-installed console scripts are actually saved on different OSs? Someone's been trying to install a package I have, but are getting Command not found, and I suspect it's the install location not being on the PATH, but I'm unsure what said location would be

[vale wedge]

little more context needed, some environments dont add user-site scripts dirs to the PATH

[glacial plaza]

I'm using flit and I'm using the scripts tag (which works similarly to entry_points in setuptools)

Just not sure where the resulting script is placed

I'm just not sure which directory the actual script is downloaded / put in by pip when the package is installed

[heavy knot]

i've been trying to open bash terminal in windows 11 and i've turned on developer mode and enabled linux terminal for windows and everything but everytime i try to open bash it just doesn't open, can someone help

[gentle solstice]

If using a virtualenv, venv/bin. If user, ~/.local/bin. If root, /usr/bin

Basically, it goes next to the python exe

[serene seal]

Any chance on a bit of help getting my Python project to work with the rest of my Docker-Compose? It isn't finding the files that I'm pointing to right now

I'm not sure why one of my python projects is being located just fine but the other isn't

[valid ice]

greetings. even though i have a file credentials in my .gitignore it is still showing up as modified. ```cmd
On branch daily
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: main.py
modified: auth/credentials

Untracked files:
(use "git add <file>..." to include in what will be committed)
status_git.txt

no changes added to commit (use "git add" and/or "git commit -a")

it does not show up on my main branch

my .gitignore looks like this ```.gitignore
.vscode
pycache
/data
credentials
auth\credentials

[forest stirrup]

because .gitignore prevents from adding new files to index
already commited files isnt ignored by gitignore

i think, you shouldnt commit this file at all

[valid ice]

i got frustrated and did a rmdir /S /Q .git and now i want to kick my self

[gentle solstice]

you can use git rm to remove a file from vcs

[native shadow]

so this tool is very handy https://vb-audio.com/Cable/, is there a way i could make some virtual audio cables in python. i have googled but with no luck

[winter spruce]

Guys do u recommend terraform + K8s, for managing infrastructure as code, instead of k8s yaml files and kubectl?

Terraform seems to be comfortable got me, I'll be having all my infrastructure in one place

Please ping me if reply, thanks a lot!

Moreover I find yaml very tiresome to write, compared to HCL

HCL = no camelcase

[serene seal]

Trying to get my Docker-Compose to play nice, having issues getting the various pieces to communicate properly, I tried setting up a network but it still isn't returning the expected results:

version: "3.8"

services:

  pgdb:
    container_name: postgresql_db
    image: postgres
    restart: always
    ports:
      - 5432:5432
    environment:
      - POSTGRES_USER=${DB_USER}
      - POSTGRES_PASSWORD=${DB_PASSWORD}
      - POSTGRES_DB=${DB_NAME}
    networks:
      - pg_database

  pgadmin:
    container_name: pgadmin
    image: dpage/pgadmin4
    environment:
      - PGADMIN_DEFAULT_EMAIL=${PGADMIN_EMAIL}
      - PGADMIN_DEFAULT_PASSWORD=${PGADMIN_PASSWORD}
    ports:
      - 5050:80
    depends_on:
      - pgdb
    networks:
      - pg_database

  fastapi:
    container_name: fastapi
    build: .
    command: bash -c "alembic upgrade head && uvicorn main:app --host 0.0.0.0 --port 8000 --reload"
    volumes:
      - .:/fastapi
    ports:
      - 8000:8000
    restart: always
    depends_on:
      - pgdb
    networks:
      - pg_database

  redis:
    container_name: redis
    image: redis:6.2-alpine

  celery_worker:
    container_name: celery_worker
    build: .
    command: celery -A celery_worker.celery worker --loglevel=info
    volumes:
      - .:/fastapi
    environment:
      - CELERY_BROKER_URL=${CELERY_BROKER_URL}
      - CELERY_RESULT_BACKEND=${CELERY_RESULT_BACKEND}
    depends_on:
      - fastapi
      - redis
    networks:
      - pg_database

  flower:
    container_name: flower
    build: .
    command: celery -A celery_worker.celery flower --port=5555
    ports:
      - 5556:5555
    environment:
      - CELERY_BROKER_URL=${CELERY_BROKER_URL}
      - CELERY_RESULT_BACKEND=${CELERY_RESULT_BACKEND}
    depends_on:
      - fastapi
      - redis
      - celery_worker
    networks:
      - pg_database

networks:
  pg_database:
    driver: bridge

Some of the error traceback when launching:

Cannot connect to redis://redis:6379/0: Error -3 connecting to redis:6379. Temporary failure in name resolution..

fastapi              | psycopg2.OperationalError: could not connect to server: Connection refused
fastapi              |      Is the server running on host "db" (172.18.0.2) and accepting
fastapi              |      TCP/IP connections on port 5432?

[indigo zenith]

I've not used Terraform but my impression is that it's more of a complement than a replacement. Terraform to establish the nodes, kubectl to manage what it manages

[restive crown]

2 things here. First, if all containers are on the same network and you dont care if its internal only, you dont need to define a network. they will all use the default network and intercommunicate
Second, i would recommend NOT naming the container redis if you are trying to connec tot redis://redis.. something feels off about the same name twice and I worry what that means. try redis_instance maybe? Also, based on your error for fastapi, i am not sure which container is "db" unless my slumbered state is missing something in your compose file

[tawny temple]

Naming the service redis is perfectly fine and will work.

Problem might be that you're trying to connect to redis/pg before it's ready

Depends_on doesn't solve this since that only ensures the container started. But container started doesn't mean the server is actually ready to accept connections

I think the issue with pg though is like the other person pointed out, your connection string is likely using the wrong host name

Once you get that solved, for pg you can ensure it's ready like this https://github.com/python-discord/site/blob/main/docker-compose.yml#L21-25 and https://github.com/python-discord/site/blob/main/docker-compose.yml#L42-44

[rapid sparrow]

that's right.
Terraform to establish the nodes + terraform makes some initial setup to them like... setting up node labels.
Small providers have just the most essential functionality at terraform level
Providers like AWS / Google / Azure have much richer amount of features at terraform level.

The rest of manipuations goes to kubectl 😉 or to helm charts ;b

[rapid sparrow]

So I would recommend Terraform + k8s Helm charts.

helm charts are quite nice to use once you got a hang of native manifests

basically helm charts bring Jinja2 (if we will take python experience) like language (actually there is used mutated golang templating language) into manifests.

[winter spruce]

But doesnt terraform offer templating in itself

[rapid sparrow]

Yes it does, but its templating language could be not be used for k8s manifests. Or at least I don't think it could be used in any nice way.

[winter spruce]

Why not? I could just use it while defining resources right?

Oh, and helm also makes it easier to install dependencies via charts

So I might as well as use it

[rapid sparrow]

Because terraform gives templating language for its own HCL language. Terraform just manages your Infrastructure / automating features that you can buy from cloud providers. Terraform removes the problem of super overbloated cloud providers services and making it as a code.
As to launch things at server targets, it can at best launch just shell script during server start up

in order for k8s manifests to work they need interactivity through kubectl

and kubectl already has comfortable external connection easily setup to cluster

[winter spruce]

Oh, I thought of orchestrating k8s with terraform

With terraform apply

[rapid sparrow]

manipulate your Cluster Nodes with terraform

operate with k8s objects like deployments / pods and e.t.c. at kubectl/helm level

terraform does only what for it was made. Infra provisioning

not more than that

[rapid sparrow]

this phrase makes no sense

regular pipeline to operate would be looking like

- Terraform apply
- Auth into cluster
- Kubectl apply

there are altneratives to using kubectl, but kubectl is the easiest out of them

[winter spruce]

I'm sorry if I got that part wrong, I thought that I could use terraform instead of kubectl. Now I've got an idea of what you mean, and it makes sense. Thanks a lot!

@rapid sparrow if I'm not wrong you recommended using yaml files with kubectl, instead of defining resources with HCL right?

[rapid sparrow]

for kubernetes resources? Yes I recommend using kubernetes native yaml manifsts in kubectl, with upgrading them to helm charts once you are ready

[rapid sparrow]

while you aren't using helm chart you can be experiencing trouble to have some variables in yaml manifests... as variables, to have more reusability of manifests

You can use envsubst
Just use like image: $IMAGE in your yamlz
and IMAGE=abc envsubst < deployment.yaml.tmp > newdeployment.yaml

as solution you could use the thing above before helm charts

that's for cases when you need different used image for test and prod environment for example

the problem with variables would be gone once you upgrade to helm charts

[winter spruce]

@rapid sparrow do you happen to know any examples which follow the pattern you mentioned? That would be helpful! Thanks a lot!

[rapid sparrow]

nope. Only having it made at work.

[raw axle]

Hello, I am new to Jenkins, I would like to know what is the best way to make a python pipeline that tests my code in 3 different environments where each one has a different python version and dependencies from the other.

[sharp adder]

What is the most advanced tool to develop an android app with Python?

[thin jasper]

@raw axle well the best is probably the one that works for you. Do you have an idea how to do it?

or something already running?

Or do you don't even know where to start?

[raw axle]

that one 😦

[thin jasper]

It's not something I currently do in my project (testing with different python versions), but the first thing that comes to my mind is running the tests in containers using different base images

probably and overkill though

you could also look into tox

[raw axle]

its a possibility 🤔

[raw axle]

i will, ty!

[sharp adder]

...

[crimson stump]

I'm attempting to Drain a K8s node, using the eviction API, however when I run the following

def drain_affected_node(availability_zone):
    load_kube_config()
    k8s_api = kubernetes.client.CoreV1Api()
    affected_node_ip = nodes_in_affected_availability_zone(availability_zone)
    response= k8s_api.list_namespace()
    for i in response.items:
        response = k8s_api.list_namespaced_pod(i.metadata.name)
        for pod in response.items:
            metadata_node_ip = pod.spec.affinity.node_affinity.required_during_scheduling_ignored_during_execution.node_selector_terms[0].match_fields[0].values[0]
            if (metadata_node_ip in affected_node_ip):
                k8s_eviction_api = kubernetes.client.V1Eviction()
                eviction_response = k8s_api.create_namespaced_pod_eviction(pod.metadata.name, i.metadata.name, k8s_eviction_api, dry_run='All', include_uninitialized='True', pretty='True')
                print(eviction_response)
                print(pod.metadata.name,i.metadata.name)

I get the following error

01-11 13:10:31 ERROR:   log:The exception is this: name 'k8s_eviction_api' is not defined

Any advice?

[rapid sparrow]

never used eviction API, but just as a note kubectl has an easy way to drain the node: https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/

[crimson stump]

Yeah it's trickier with Python client unfortunately doesn't integrate very well with API outside of CoreV1 methods

I've managed to get it working now though, documentation was actually incorrect, needed to use

                eviction_body = kubernetes.client.V1beta1Eviction(metadata=kubernetes.client.V1ObjectMeta(name=pod.metadata.name, namespace=namespace.metadata.name))

Rather than V1Eviction

[crimson stump]

I have a script I've written with a CLI tool, I would love to use classes but I never have, would someone be able to tell me if there's a potential to use classes here so I can try and incorporate one.

https://paste.ofcode.org/DfkTC4tgZwXe9fheTEtt6J

I plan on expanding out to further processes which is why ideally I think it's a good opportunity to learn about classes if possible

[wooden ibex]

maybe yes to Terraform, K8S is a deal with the devil, you would have to see if Terraform has Kubernete Modules or not

[wooden ibex]

most people recommend wrapping kubectl because Kubernetes API can break but Kubectl commands tend to be much more slower to shift

[tawdry needle]

what's the alternative to k8s? podman?

[wooden ibex]

native cloud tools

[tawdry needle]

or "k8s but you pay someone else to host it" like gke

[wooden ibex]

gke only gives up the host management

and kubernetes system updates

[tawdry needle]

what do you mean by "native cloud tools"

[wooden ibex]

if you running at a place that can only handle VMs or K8s (Linode/DigitalOcean/Vultr), then K8s it is!

[tawdry needle]

for example my company uses gke, and i think the reason they chose k8s was the ability to scale up or down as needed. i'm honestly not sure how our use case differs from "a few vms and a load balancer" though

[wooden ibex]

Elastic Beanstalk/Lamdbas on AWS or App Services/Functions on Azure

[tawdry needle]

ah i see

yeah it wouldn't make sense for us to do everything serverless

[wooden ibex]

If you got sucked into Google, then they have equivalent but crazy limited

[rapid sparrow]

kubernetes won the competion in container orchestration. I would say there is no alternative ;b
the really choice is only going easy way: using managed k8s like GKE and e.t.c.
or raising self raised kubernetes (there are quite easy to install ones like microk8s for example)

among managed k8s, better to use big providers like AWS/GCP/Azure
from small providers only DigitalOcean provides stable sufficient managed k8s.
I tried Vultr and Linode k8s, it was a horrible experience

[wooden ibex]

really, everything is really long running processes?

[tawdry needle]

ah, no

[wooden ibex]

then you likely easily could go serverless

[tawdry needle]

i always imagined that there would be a latency hit if you did that

is that not true?

we definitely have some backend services that just need to read from a db, do some processing, then return json

[wooden ibex]

My point is, I'm not a fan of K8s, it's Siren who will drag you over to rocks and drown you likely but song is too sweet to notice

[tawdry needle]

and can effectively "go to sleep" between short-lived requests

[wooden ibex]

So Lamdba tied to DynamoDB could easily handle that

write it off Blob Storage

back to sleep

[tawdry needle]

you can handle an http request with lambda and not take a huge latency hit?

not sure what you mean by "write it off blob storage"

i'm not talking about some asynchronous rpc use case

[wooden ibex]

If you have latency concerns, Beanstalks are kept alive all the time

[tawdry needle]

we do that for data etl stuff

i see, that's interesting... but it's still serverless? like you can't upload an arbitrary container image, you have to use the lambda framework?

[wooden ibex]

Beanstalks you just upload your code and it runs it, assuming it's sanely formatted, it should run fine

[tawdry needle]

my first job we just had 2 always-on VPS servers running centos + a load balancer in front of it

worked well enough

[wooden ibex]

and I think if you are doing some really really weird shit, you can upload container and beanstalk will run it

I'm Azure person myself but I like to throw out AWS since it's most popular, last job was GCP

[rapid sparrow]

k8s brings certain simplification to infrastructure code, less code is needed being written.
it gives internal DNS, that makes easy routing between services
self-healing of containers, if something died, it would be restored automatically (if necessary then in another server machine)
quite efficient tooling to use available resources as efficient as possible, k8s allows to set requests/limits to CPU/RAM/HDD in all the ways
its infrastructure code is just quite reusable in terms of configs
plus there is a feature for autoscaling node pools, you can save money further by having dynamic amount of machines for certain things depending on the current load
k8s allows to have always free loadbalancers for internal traffic (saving of money)
and even public loadbalancers could be minimized through usage of k8s Ingress
and another important feature of k8s, that people write for it reusable code like docker public images, it is easy to find a lot of necessary easily setup stuff that needs to work across multiple machines

[tawdry needle]

that all makes sense @rapid sparrow, thanks

interesting that there's no popular alternative

[rapid sparrow]

there are alternatives, but k8s won 😉

[wooden ibex]

Hashicorp is working on one, k8s sucked all the oxygen out of the room

[tawdry needle]

and yes being able to run our docker images locally + a local instance of our mongodb setup is nice

[wooden ibex]

you are ignoring the dumpster fire that is powering your pretty home

[tawdry needle]

i've worked on google cloud function stuff for the data engineering people, i prefer the docker images

(as a rank-and-file dev, not devops or sysadmin)

[rapid sparrow]

huh?

[wooden ibex]

functions are great if you have event based triggered, latency insensitive stuff, quick things

[tawdry needle]

right, like ETL jobs that trigger when something gets dumped in a GCP bucket

which is what we use it for

made total sense, i just didn't like the developer experience

[wooden ibex]

all this is available on most cloud native tools but doesn't involved slathering K8s with shit ton of plugins so it becomes SuperFund site like most node applications

[wooden ibex]

It's Google so........ (they are the worst)

[tawdry needle]

it might also be the case that our devops people are really good and are hiding all the k8s complexity from people like me 😆

i do really hate our ci system though

codefresh

idk what their reputation is

but i really do not like it

[wooden ibex]

or you are small enough it hasn't hurt someone yet

[tawdry needle]

apparently one of our senior people was doing a bunch of custom istio stuff at one point

idk if that counts as "hurt"

[wooden ibex]

oh god, that's type of shit I hate about Kubernetes

[tawdry needle]

he said that istio added features that made the customizations no longer needed

not sure what they were

[wooden ibex]

but still, you are now running ADDITIONAL network layer

[tawdry needle]

i admittedly have no goddamn idea where the separation between istio's job and k8s' job is

does k8s just hand off all the networking stuff to istio?

[wooden ibex]

most people don't

[tawdry needle]

like how x11 hands off window management to a separate window manager program

[wooden ibex]

I don't know exactly either but that's my complain about Kubernetes, Well it does all this stuff? Does it really do all this stuff? Well, not natively, we had to install ArgoCD/Isito/Enterprise Traefik/Security Tool

So this is better then VMs? IT'S WEBSCALE!

[velvet spire]

/dev/null is best

[rapid sparrow]

resistance is futile

[tawdry needle]

part of my issue is that i dont understand how it would work without istio

[wooden ibex]

Current job, I think we have derailed our K8s project

[tawdry needle]

my understanding is that istio is basically a big router for your cluster

[wooden ibex]

hopefully

[tawdry needle]

does k8s also have its own routing but istio does it better?

[wooden ibex]

k8s has network, most people layer Istio because they want some additional feature

[tawdry needle]

i don't hate that

at least in principle

[wooden ibex]

really, you wouldn't mind if I just started downloading libraries and shoving into requirements.txt?

[tawdry needle]

what do you mean by that

if i use django i need like 5 middlewares to actually make an app

[wooden ibex]

So you just randomly shove packages onto your work projects without a care in the world? Hip place

[tawdry needle]

i dont know what you mean by that either

i didnt choose kubernetes or istio, but a system that is extensible seems like a good thing 🤷‍♂️ i can't speak for how much of a clusterfuck it is to configure it all, i'm sure that part isn't good

[wooden ibex]

You are cool with people just randomly adding packages to your software projects? No care in the world?

[tawdry needle]

who is randomly adding anything to a software project?

[wooden ibex]

That's my POINT

If you wouldn't want it in your software, I wouldn't want it in my OS

[tawdry needle]

i don't really know what your point is

[wooden ibex]

Kubernetes encourages people to slather their k8s install with a shit ton of plugins because they need functionality

[tawdry needle]

i know that k8s has a reputation for being very difficult to configure, and that it seems to need istio in order to do more sophisticated networking stuff that k8s doesnt support natively

ah

[wooden ibex]

so your K8s project becomes superfund site just like any Node Project

[tawdry needle]

is that any different from python

i know node apps tend to have a lot more deps

[wooden ibex]

with X different plugins, hopefully no one stops developing them, Kubernetes API changes or they just start hating each other

[wooden ibex]

and it's serious problem for Node Apps

[tawdry needle]

true

and that's fair, if the plugin ecosystem is unstable and chaotic

[wooden ibex]

but instead of your application being unstable, this is the OS that powers EVERYTHING

[wooden ibex]

It's pretty stable but it's lurking issue

I've worked with K8S, it's best at what it does but very much a Google Product but because it's so hot in Ops world, so many people sing it's praises

[tawdry needle]

interesting

[wooden ibex]

it's usable tool but for alot of issues it takes away, it adds bunch of different ones instead

[rapid sparrow]

;b it removes much more issues than it adds

[wooden ibex]

how big was/is your k8s cluster?

[quartz dune]

Can I ask IDLE questions on this channel?

[tawny temple]

#editors-ides is more appropriate for IDLE questions

[quartz dune]

Ok, thank you

[distant tartan]

I'm on the struggle bus trying to get this package to work, my previous problem was figuring out how to get a resource included and now that I got that squared away I'm getting a file not found error.. but the file appears to be present. I suspect its something really simple, like I'm messing up the file path. any thoughts on the best way to include data and access that data in a module in python?

oh, the code is located at https://github.com/burnish3d/ACNH_pixel_color
and the error I get after installing it with pip into a venv and trying to run it with pyton -m is

    with open(r"res/ACNH_color_palette", "r") as f1:
FileNotFoundError: [Errno 2] No such file or directory: 'res/ACNH_color_palette'

but when I go to that directory, there is in fact that file. Is the issue an OS dependent way of talking about files or..?

[rapid sparrow]

I am using small clusters within 10 nodes

[odd dragon]

Not sure if the question belongs here or not, but rather than using docker and whatnot, if I have a simple django project that is small in scale (maybe under 1000 lines of written code?), can I simply share my virtual environment and have other devs simply activate it and install from requirements? Or should I keep my venv exclusively my own and have everyone else set up their own?

[tawny temple]

Don't check the venv into source control. Don't distribute it. Let users set it up themselves. You can pip freeze your dependencies into a requirements.txt and users should be able to reproduce the venv with that

Venv are not designed to be distributed. Upon being created they are tied to that location (if you ever tried moving a venv you'd see it won't work)

[rapid sparrow]

@tawny temple there is an exception regarding the venv.
if we freeze venv packed in a container with sending result to docker registry for reusage in QA / Staging / Production environments.
it is more reliable way to freeze dependencies, especially for python web programs 😉 I would say it is almost equivalent to built binaries

for Dev environment when it is developed futher it is of course not really valid, although sometimes containers are used for Dev env too

[rapid sparrow]

and Note: Venv becomes broken when it is moved into any other folder from where it was created. It uses absolute paths that rely on the paths being not changed

[odd dragon]

Thanks guys. I learned a lot today 🙂

[crimson stump]

Yeah i was initially just using sub procceses which made the whole thing incredibly easy but i thought it'd be a good challenge to just use the Python API client libs. Any ideas on the class suggestion? 🙂

[stuck badger]

https://github.com/YashIndane/pypi_uploader

end to end automation

[rain sedge]

Hey, just wondering if I'm not alone: my projects are usually set up with poetry as a dependency and package manager. Everything works, no bugs, good experience, has everything I need. Except the networking and resolution with pypi is unbearably slow on a regular basis. Because of these speed issues I intend to switch to mamba + conda-forge. Any thoughts?

Compared to conda or even better mamba+conda-forge is just no comparison and a real drag. If I set up a tiny project from scratch poetry/pip take tens of seconds if I just install black and pytest. Everything's working, it's just slow. This appears to be a known issue, and genuine. I.e., it's not my networking, misconfig, ISP or similar under my control. I.e., at work poetry/pypi is a bit faster than at home but still orders of magnitude slower than mamba/conda-forge. It's been way too slow with open github issues on the matter open for years and only keeps getting worse in my experience.

[thin jasper]

@rain sedge The reason why poetry is slow, is that it has to download lots of the packages in order to get the needed information. Does your alternative provide the same functionality?

just asking since I have no idea, but if not you're comparing apples and bananas

we use a local pypi mirror at work, where we cache all the package we download

that makes it way faster

[tawdry needle]

@thin jasper @rain sedge poetry and mamba/conda are two very different tools that serve different purposes. mamba/conda is more like pipenv; poetry is kind of like pipenv+setuptools.

so it depends on what those projects actually are

switching to conda means switching away from pypi entirely

usually it's only a good idea for data science and machine learning projects

i wouldn't recommend it for general purpose software development

poetry taking 10 seconds on a new project is strange, especially because conda is legendarily slow

although mamba is quite a bit faster

maybe you can set up a local mirror of pypi

[odd dragon]

Hi guys, back at it to learn more

I suck at this security thing, but I realized this is probably more a devops type question than a security one.

I spent a few days reading about ssh-keygen and how the cryptography works so I have a basic understanding of whats going on. Here's the issue:

I have an architecture that looks like this:

My computer -> VPS
VPS -> Docker Container (lets call it master)
VPS -> Docker Container (lets call it slave)

My goal is to be able to have the VPS and master communicate with the slave in a jenkins setup.

I ran ssh-keygen -t rsa -b 1024 on the VPS host, and logged into jenkins and put in the private key -----BEGIN RSA PRIVATE KEY----- ...

I then went into the slave docker container and made .ssh/authorized_keys and copy pasted my public key from my generated pair. Am I on the right track? Jenkins logs says:

[01/13/22 02:47:45] [SSH] Opening SSH connection to 172.20.0.2:22.
[01/13/22 02:47:45] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
ERROR: Server rejected the 1 private key(s) for [user] (credentialId:4c93498a-e608-45fe-85ad-fa776e2f53e8/method:publickey)
[01/13/22 02:47:45] [SSH] Authentication failed.
Authentication failed.
[01/13/22 02:47:45] Launch failed - cleaning up connection
[01/13/22 02:47:45] [SSH] Connection closed.

Any hints to where I may have went wrong?

[thin jasper]

Not sure if I understand correctly

You got a SSH key for system A, then put that key in system B as an authorized key and then try to SSH into system C with the same key?

that wont work since C needs to authorize a key of system B

[rapid sparrow]

I got confused by your explanations

Let's be clear, SSH keys are generated in pairs.
id_rsa - the private part
id_rsa.pub - the public part

We copy the .pub key to target machine authorized keys.

We use private key to perform Auth to any machine/system that has our public part of the keys

Same private and public keys can be in multiple machines

[thin jasper]

So, say my PC is system A and I have to servers B and C which have my public keys and let me SSH onto them without problems

I can always go from A to B or from A to C

but not from A -> B -> C

[rapid sparrow]

Yes

[thin jasper]

just wanted to make sure, that @odd dragon is not actually trying that

[odd dragon]

Oops sorry I’ll have to revisit my setup in like 12 hours lol

It’s a Jenkins master slave setup that I don’t fully understand so I’ll have to remember the above details while relooking at it

[dense grotto]

I'm struggling to get a Poetry project to work in Visual Studio Code. Some modules I add to the project appear to import fine when I use Run > Start Debugging. However, I have not been able to get numpy, pandas, or dask to work. I get ImportError for them. Any ideas on what else to try would be appreciated.

• M1 Mac using macOS Monterey.
• Python.org install of Python 3.9.9
• Poetry 1.1.12
• Venv in the Poetry project (./.venv)
• Venv's Python 3.9.9 is selected as VS Code interpreter
• sys.path shows that the venv is in the path
• modules are installed in the venv by Poetry
• have tried poetry shell && code .
• used a (new) terminal in VS Code to rebuild the venv to make sure modules were installed

[rain sedge]

The mirror is a good idea, thanks!

[rain sedge]

Thanks for your input! Just a note one can use flint which is a dedicated packaging library you can take care of packaging when using mamba. Otherwise didn't find anything missing, you can specify any pypi dependencies in the conda.yaml in case something isn't on conda-forge.

[velvet spire]

aaaa

[tawdry needle]

i haven't heard of flint. i just used conda-build in the past when i wanted to repackage something for conda. it's a bit hard to search for, can you provide a link?

i am aware of flit, which is an alternative to poetry. but doesn't seem to be very popular

i think flit is more like setuptools than anything

[red oxide]

I am trying to follow this blog (https://www.digitalocean.com/community/tutorials/how-to-install-jenkins-on-ubuntu-18-04) to install jenkins in linux machine but I'm getting this error:

root@racknerd-766cca:~# sudo apt update
Hit:1 http://us.archive.ubuntu.com/ubuntu focal InRelease
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]      
Get:4 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]     
Get:5 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]   
Hit:6 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu focal InRelease
Ign:3 https://pkg.jenkins.io/debian-stable binary/ InRelease
Err:7 https://pkg.jenkins.io/debian-stable binary/ Release
  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses expired certificate.  Could not handshake: Error in the certificate verification. [IP: 151.101.2.144 443]
Hit:8 http://ppa.launchpad.net/wasta-linux/cinnamon-4-8/ubuntu focal InRelease
Reading package lists... Done
E: The repository 'http://pkg.jenkins.io/debian-stable binary/ Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

[rapid sparrow]

I would recommend that you learn Jenkins only if you have a legacy project that already has Jenkins

otherwise learn some other CI tool

Jenkins is 1 gen CI tool and highly not recommended for new projects

better choose some 2 gen CI tool, like Github / Gitlab / Circle Ci, Drone Ci and e.t.c.

Quote about it from one wise person:

Jenkins was a first gen CI system. Everything is configured in the UI, which makes it hard to see/approve changes, things are live straight away so any larger changes can break things while you play around with them to see if they will work (as oppose to testing them in a branch first). They where designed to run agents directly on servers which bloated after a while with every tool ever single pipeline needed. Different versions of packages meant to needed different agents and managing it all is a huge pain. Often java focused and everything else is a second class citizen. This also goes for other older tools like teamcity, stash etc. Jenkins is also plugin hell, everything is a plugin and you need a whole bunch to make it a useable tool these days. Which increases the learning curve and makes the whole thing a mess.
I would not use a first gen CI system any more. They should just be left to die. They have tried to tack on more modern features but have all done it badly to keep support for the older ways of doing things.
Second gen CI systems are all config first, where the config is in the repo next to the code. As well as Docker (or at least ephemeral VMs for each job) so the agents are kept clean and essentially reset after each job and each job can install the tools it needs for that job. This includes things like travis, circle ci, github actions, gitlab ci, drone ci etc... so many choices there is no reason to use the legacy tools any more.
I think there is also a third gen popping up now - more abstract than just CI and starting to go down general workflow pipelines. Often kube focused and while you can build CI/CD systems out of them they are more general and no longer tied to only being triggered by Git event but rather any event. This includes things like Tekton, Argo Wrokflows etc type tools.

[odd dragon]

Do you know of any other CI CD tool that can be used in a way that abides by very strict compliance rules? As in, I cannot have anything living outside company premises which eliminated GitHub actions along with many other tools for me

Jenkins has been pretty tough for me

[rapid sparrow]

clarify, would be having Runner for CI at your company enough to be complaint
or you need it being fully in your servers, including controller part?

[odd dragon]

Ah I can’t clarify that myself. I’ll have to check. I’m assuming my boss wants 100% to be local—able to run if say the premises were entirely offline

[rapid sparrow]

For first case: Gitlab CI, it can launch Self Hosted Runners. Technically it has an option to launch its Server side as self hosted too. So it can be fully at your servers.
for completely fully at your servers (that has it as the only choice): Drone CI i guess, but I never worked with it so can't recommend beyond hearing recommendations from others

[odd dragon]

I’ll have a look at both solutions. Thank you!

[rapid sparrow]

depending on how you setup them, both choices are actually valid for being fully in your servers

[rapid sparrow]

better to clarify it, just Self Hosted runner should be light weight solution in terms of maintenance, while solving you 95% percent of security issues

you can forbid in Gitlab interface using anything but self hosted runner

if it is not enough, self hosting gitlab server instance is an option too for a full complaince
or perhaps Drone CI would fit your case better in this case, if is easier to setup for full server solution

[thin jasper]

some of this sounds wrong. The pipeling configuration (Jenkinsfile) can easily be stored in the project repository, so Im not sure what exacly theyre talking about. But I agree, its Plugin-hell

[odd dragon]

I’m super interested in this solution. If it’s cheap enough I may be able to sell it to my company haha

[rapid sparrow]

Too many settings can be configured in Jenkins GUI that affect stuff without being as code in repo

There are Jenkins mods that minimize this issue

[thin jasper]

sure, but this source says Everything which is a bit too much imho. Anyway, I would recommend looking for other tools as well

[tropic gulch]

could i know whats wrong here?

[rapid sparrow]

pip3 is not in the PATH of your console environment variables

use full path to where pip3 is installed
or add folder with its binaties to PATH

or actually I would recommend to use python only in order to create venv

and to use pip3 from venv

there is no any justification to use global pip3

https://docs.python.org/3/library/venv.html
c:>{FullPathToPython} -m venv venv

https://mothergeo-py.readthedocs.io/en/latest/development/how-to/venv-win.html
.\venv\Scripts\activate
to activate it

[tropic gulch]

im actually pretty new..

[rapid sparrow]

from venv u'll have fully working local pip3

[tropic gulch]

hmm ty

[sharp elm]

Any reason why this (taken from pyproject.toml) doesn't include my additional source files when doing pip install .:

[tool.flit.sdist]
include = ["docs/","profiles"]

[anton ~]$ ls -l /home/anton/.local/lib/python3.10/site-packages/archinstall/
total 16
-rw-r--r-- 1 anton anton 8578 Jan 14 10:37 __init__.py
drwxr-xr-x 1 anton anton  428 Jan 14 10:37 lib
-rw-r--r-- 1 anton anton   78 Jan 14 10:37 __main__.py
drwxr-xr-x 1 anton anton   96 Jan 14 10:37 __pycache__

profiles/ is missing. It used to work before i switched to flit. I suspect it's because flit can't follow symlinks?

[anton archinstall]$ ls -l archinstall
total 24
lrwxrwxrwx 1 anton anton   12 Apr  8  2021 examples -> ../examples/
-rw-r--r-- 1 anton anton 8578 Jan 14 08:46 __init__.py
drwxr-xr-x 1 anton anton  428 Jan 14 08:46 lib
-rw-r--r-- 1 anton anton   78 May 28  2021 __main__.py
lrwxrwxrwx 1 anton anton   12 Apr  8  2021 profiles -> ../profiles/
drwxr-xr-x 1 root  root    92 Oct 22 20:31 __pycache__

Moved them manually and it seems to be the case. Ignore the previous message, had to write it to realize that was most likely the issue 🙂 Bug report: https://github.com/pypa/flit/issues/509

[thin jasper]

I think what you try to prevent is harder to achieve than what you want

If i understand it right

Just commit locally and dont push

and dont branch the second branch from your first branch

[thin jasper]

git stash is for saving (and reverting) changes that you haven't committed

but that would be before commit and not push

but take care

to pick the right stash when you do that

I'd often rather commit than stash a lot

[tawdry needle]

@rapid sparrow do you know if there is some standalone self-hostable CI tool that isn't also tied to some "monolith" like gitlab?

[tawdry needle]

same, i would usually rather git commit and then git cherry-pick + git branch -f

[rapid sparrow]

gitlab CI can be launched with its monolith on your servers too.
but anyway, I would tried to check and run Drone CI. It sounds like a lightweight a nice solution
Never used it, but heard a bit of good words

[rapid sparrow]

And also I would have checked Argo (Workflow?) for self hosting

Because I work with kubernetes already anyway

(it is kubernetes first ultimate thing)

[tawdry needle]

thanks

[stable cloak]

Looking for some recommendations. What style/library/whatever-it-would-be-called do you folks use for generating docs from your code? The context is that it's for the following project: https://pglet.io/docs/ Quick and dirty explanation is that it's a quick way to make a GUI. It renders all this into a web app. Not sure if that all matters, but I figured context is key.

The Python wrapper library for it is rough at the moment. No docstrings, no type hinting, etc. So there isn't a current style guide or anything we have to follow at the moment, so anything goes

[rapid sparrow]

i used Sphinx and quite liked it

it can generate docs exactly from python functions docstrings written in a google something format
+at the same time allows writing normal documents in Restructurized Format (RST)

End result is easily automated and deployed to Github/Gitlab pages
https://dd84ai.github.io/darkbot/

example of deployed wiki for pet project

[stable cloak]

Oh yeah, I've seen several docs like that

[rapid sparrow]

https://github.com/dd84ai/darkbot/blob/master/sphinx/source/conf.py
if you wish having it working exactly from even your code functions, you need to enable its extensions (just adding lines to config)

extensions = [
    'sphinx.ext.autodoc', 'sphinx.ext.coverage', 'sphinx.ext.napoleon',
    'sphinx.ext.autosectionlabel'
]

autodoc feature scans python docstrings

this repository is not probably having an example of how to use it though

I could find though, a moment

[stable cloak]

I appreciate you taking the time

[rapid sparrow]

Gitlab CI auto deploying to pages

pages:
  script:
    - python scripts.py test unit -r
    - python scripts.py sphinx build
    - mv docs/build public/
  artifacts:
    paths:
      - public
  rules:
    - if: $CI_COMMIT_BRANCH == "dev"

@sphinx.command()
def build():
    "build sphinx"
    say(f"sphinx-build -b html {os.path.join('docs','source')} \
        {os.path.join('docs','build')}")

example of documented function

class CountryView(mixins.CreateModelMixin, generics.GenericAPIView):

    queryset = Country.objects.all()
    serializer_class = CountrySerializer

    @drf_required_key(settings.MONITOR_PUBLIC_KEY)
    @error_api_handler
    def get(self, request, *args, **kwargs):
        """:route: **Запрос данных по странам**

        | Выдает список доступных стран
        | и иконок к ним либо выдает подробности по конкретной стране

        Входные данные::

            [GET/POST]: http://dev.example.com:5555/country/all?api=FRONTKEY

        Returns:
            JSON: status_code: 200

        Запрос без ключа pk

        .. literalinclude:: write/country_all.json
            :language: JSON

        Запрос с ключем pk

        .. literalinclude:: write/select_output.json
            :language: JSON

        """

Sorry, not in English

[stable cloak]

Perfectly fine

[rapid sparrow]

.. literalinclude:: write/country_all.json
            :language: JSON

auto inserts at this place JSON formatted data from the file

[stable cloak]

Ooo

Nifty

[rapid sparrow]

Source code of Sphinx to document the code

docs/sounrce/index.rst

.. gotruenet documentation master file, created by
   sphinx-quickstart on Mon Apr 26 11:01:02 2021.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Добро пожаловать в документацию к gotruenet proffit!
======================================================

=============================================================================================================
`вернутся в главное меню <https://gtn_admins.gitlab.io/gotruenet_documents/>`_
=============================================================================================================

.. toctree::
   :maxdepth: 3
   :caption: Contents:

   country/index.rst
   farhub/index.rst
   language/index.rst
   proxy/index.rst
   vpn/index.rst

Description
################

MONITOR_PUBLIC_KEY = blablabla

Web service for controlling and monitoring of all gotruenet projects
   

Инструменты навигации
======================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

docs/source/country/index.rst

==================
Countries
==================

.. toctree::
   :maxdepth: 2
   :caption: Contents

   routes

docs/sounrce/country/routes.rst

API end points for countries
=============================

.. automodule:: country.views
    :members:

docs/sounrce/country/write

#FOLDER WITH JSON FILES

[rapid sparrow]

shortly speaking. index.rst is main file, starting point of wiki, ..roctree is auto generated menu

[stable cloak]

Sure sure.

[rapid sparrow]

as you can see routes.rst is quite small, all it has enabled auto discover of the documentation in my file country.views

it is imported python package

[stable cloak]

Yeah this seems to be what all I'm looking for

I didn't like the bare bones style you showed (I think requests has the same style), but there do seem to be some themes that'll fit the bill

[rapid sparrow]

i know that it is possible to choose at least between 10 inbuilt themes for it

I tried them all, I just like the default white/black one

[stable cloak]

This is the one that jumped out at me: https://github.com/pradyunsg/furo

[olive path]

Another one is MKDocs

[stable cloak]

For themes you mean?

[rapid sparrow]

Player Base Tracking
######################

**base commands**

Those are command to track status of player bases!
Example of output below.

check `the list <https://discoverygc.com/forums/bases.php>`_
of precised based names before using the commands

base add
***********

possible usages:

.. code-block::

    .base add MyStation

if your base have **spaces** in its name, better be using:

.. code-block::

    .base add "My Very Long Named Station"     

you can add multiple bases in one command

. code-block::

    .base add "My station #1", "My station #2", "My station #3"

also, you could add only sub string that the list of bases have

. code-block::

    .base add Depot

it will render all bases having Depot in its name

.. code-block:: JSON

    ["48.0885"]{"Copper Storage Depot"}["Freelancers"]
    ["100"]{"Wismar Shipping Depot"}["Kruger Minerals"]
    ["83.3811"]{"Bristol Depot"}["Freelancers"]
    ["100"]{"Shiojiri Storage Depot"}["Samura Industries"]
    ["4.79464e-26"]{"Malfunctioning Depot"}["No Affiliation"]
    ["54.6548"]{"Howler Depot"}["Junkers"]
    ["56.3603"]{"Aruba Depot"}["Freelancers"]

additional commands

base remove
**************

for deleting of one base names

. code-block::

    .base remove Depot

base clear
*************

to clear the whole list of bases

. code-block::

    .base clear

this is example of using RST to write files directly

without auto doc discovery

[olive path]

The benefits is that I get to write the docs in Markdown - a format I much prefer.

[rapid sparrow]

there is small advantage of RST in comparison to MD, you don't need to write \ for new line 😉

but at the moment I am actually writing my docs in markdown (no need to build them, and good looking from repository without building). With using Typora as editor

[stable cloak]

I do love me some markdown

[finite fulcrum]

Every mkdocs doc page I had to use so far felt like a mess compared to sphinx, perhaps because the md leads to it ending up feeling like tutorialy disconnected pieces of information

[stable cloak]

Would you not be able to pump out a Sphinx into markdown?

[rapid sparrow]

i don't think so. Perhaps there is module for that

[olive path]

I've actually quite enjoyed modifying themes that have been written in Jinja2 to fit my needs. I am not sure how Sphinx does theming

[rapid sparrow]

Sphinx is intended to build into html/css/js

[olive path]

Did you use mkdoc-docstrings or any of the other auto-generating extensions?

[rapid sparrow]

https://typora.io/

markdown editor, offline downloadable to linux at least

[finite fulcrum]

Not using it myself, just the experience I had when referencing docs of other libraries

[olive path]

FastAPI?

[rapid sparrow]

considering that markdown documents can create link and link files in repository between each other...
and that it works straight from repository web GUIs without hosting even to pages...

I like markdown

a bit limited in what it can express, but nice

[finite fulcrum]

I believe that was one of them, though I didn't use it too much. I can't really say which projects it was off the top of my head

[olive path]

Because you can use autodoc features

[stable cloak]

I've got my Sublime configured where I can edit markdown and it'll show the updated results

So that's not a huge deal

I think Sphinx for sure seems like a better option

[rapid sparrow]

for big stuff, sure

Sphinx is universal

[stable cloak]

Especially for the amount of work that's going to go into this project

[rapid sparrow]

but for API docs there could be alternatives, if documentating rest apis, OPENAPI/Swagger could work better

[olive path]

These doc build have probably been failing for weeks, but here's what I generated for a project of mine:
https://wumpy.readthedocs.io/en/latest/reference/interactions/commands/

[stable cloak]

Like it's going to be a full overhaul

[olive path]

Oh yeah HTTPX is another user of MKDocs

https://www.python-httpx.org/

Most people use the "Material for MKDocs" theme

[finite fulcrum]

I wouldn't say it's bad per se, just that (for me and in the ones I had to use) it was hard to use as an actual reference.

From the one you linked I would at least expect a TOC and permalink buttons though I'd guess those are available. Apart from those it does look fairly fine

[rapid sparrow]

@stable cloak tha's btw how it looks like

the documented function after building

[finite fulcrum]

I think it boils down to the approach taken, where mkdocs supports the style of writing out some tutorial pages and then leaving the reference up to auto generation (or nothing), which leaves the reference looking a bit bare while with sphinx I usually see them interwoven

[rapid sparrow]

and I am torn apart between trying to learn Drone or Argo. I think I should go for Argo, it is way more popular

[olive path]

Yeah I haven't set up a TOC yet (apart from the one you see to the right). I haven't bothered too hard looking into it right now but that was mostly a test 😅

Does Sphinx support the Google style docstrings?

[rapid sparrow]

yes

[rapid sparrow]

example

I used them

[olive path]

Ah okay cool, I've always kind of disliked the way some setup headers

[olive path]

Didn't even notice 😅. Thanks

[native shadow]

anyone know how i can make something in python that can read notifications from any app then do custom stuff with it

the end goal is to have python control a 10x10 led matrix and do cool things when i get a notification

[rapid sparrow]

it sounds like data extraction from app with API usage or with web scrapping

[rapid sparrow]

I don't understand even combinations out of two words out of it

[leaden tartan]

I have a problem that I need help with -
When a user clicks a button, I need to fetch some details from an API, and need to keep polling the API until the API responds with a success status. What can be a possible solution to this? The simplest solution can be setting up a do while loop and a few scattered sleeps, but I hardly think this is an ideal solution.
Another can be sending it off a redis queue, making the request in the consumer and if the API request was not successful, add it to the queue again.

What are you thoughts on this?

[red oxide]

So I have a Python Selenium Automation Script and I want to kill the running script and browsers properly, update the script, and simply run updated script again when I push new code. Nothing more than that, no tests, no nothing.
Darkwind here suggested CI/CD flow and it's seemed very interesting when I looked at it but I'm looking for something more simple. Something that just does that 3 simple steps when I push new code to my git repo.

I stumbled upon Jenkins recently and I'm guessing it can help me do this??
I looked on the internet but again I could only find blogs for people doing complicated things with it while I'm looking for something simple.

Can anyone guide me if Jenkins is the right tool for things and if yes then how can I simple use it to do the things I mentioned above?

Also, one more thing, geckodriver required executable permission before running so gotta do it too automatically after updating the script.

Thank you.

[rapid sparrow]

#tools-and-devops message
Jenkins is not the way

[red oxide]

oh shit, I'm sorry I missed your response @rapid sparrow
Github, interesting O_o
so can I achieve what I'm looking for just with github itself?
can you please elaborate on the subject whenever u are free?

[rapid sparrow]

according to your request you need
CI/CD tool + Ansible

CI tool keeps track your Git repository and fires to do jobs when something was changed

Ansible script makes ssh connection to your target machine, updates the code and restarts the processes to run the new stuff

[red oxide]

so basically I add my repo link or something like that to ansible (deployed on single machine) and it takes care of everything by connecting to other machines?

[rapid sparrow]

if u use public CI runners, you will not need to host your own thing to have it

if u wish to use your own self hosted runners for better security, then feel free to host them

I could recommend gitlab CI runners

easy to setup

[red oxide]

okay

[azure compass]

so i have two APIs running both written in python

deployed using docker and traefik

it was working fine for some time, but now I'm having this annoying problem where my containers would just randomly recieve a term signal

it'll run for about 2 hours before all the containers just stop suddenly

Here is my docker-compose for traefik:
https://mystb.in/HiredHebrewParticipation.typescript

Here is my docker-compose for my first API (call it "Hierarchy"):
https://mystb.in/OtherwiseVidExhaust.yaml

Here is my docker-compose for my second API (call it "Voicerooms"):
https://mystb.in/StudiesSyndicationSuffered.yaml

[distant tartan]

If some one knows of a better spot to ask lemme know, but, I need a little help with chapter 2 of the docker getting started guide. https://docs.docker.com/get-started/02_our_app/ I can get the first app to run, I can build (I don't see an error at least) the second app (same name, but we are building it from a dockerfile), but when I try to run it the container immediately exits with status code 127

errr maybe not an issue, just found a typo in my dockerfile

YEP. typo.

[sand spoke]

Hey, anyone knows here Terraform 🤔 ?

[rapid sparrow]

https://pythondiscord.com/pages/resources/guides/asking-good-questions/

What Not To Ask
Q: Is anyone here good at Flask / Pygame / PyCharm?
There are two problems with this question:

This kind of question does not manage to pique anyone's interest, so you're less likely to get an answer overall. On the other hand, a question like "Is it possible to get PyCharm to automatically compile SCSS into CSS files" is much more likely to be interesting to someone. Sometimes, the best answers come from someone who does not already know the answer, but who finds the question interesting enough to go search for the answer on your behalf.
When you qualify your question by first asking if someone is good at something, you are filtering out potential answerers. Not only are people bad at judging their own skill at something, but the truth is that even someone who has zero experience with the framework you're having trouble with might still be of excellent help to you.
So instead of asking if someone is good at something, simply ask your question right away.

and answer is yes

[sand spoke]

I am looking for someone to learn it with 🙂

[indigo zenith]

If no luck here, maybe try https://discord.gg/devops

[sand spoke]

Thankss

[rapid sparrow]

https://www.amazon.co.uk/Terraform-Running-Writing-Infrastructure-Code/dp/1492046906
I can recommend reading a book

[heavy knot]

what do you guys think of lua

100k people in the server and no one reply's damn

[crimson zenith]

I have no clue what that is

[rapid sparrow]

Good language to play ComputerCraft minecraft mod
(which adds computers inside the game that can be programmed with Lua)

In general I heard it is a scripting language that is used inside of the games ;b

[cobalt warren]

Does anyone know any good example repos that build a docker image and have tests? Trying to figure out best practices around repo structure and some examples would be good.

Or have any opinions on the above?

[thin jasper]

are you talking about testing the docker images? or running tests in containers?

[cobalt warren]

tests run outside of image as part of ci/cd (or locally)

[rapid sparrow]

xD I just run tests inside the container during ci/cd run

less code to write.

[thin jasper]

yep, running tests in containers as well

[cobalt warren]

ive done that before as well, but there are benefits to running it outside (easier TDD, exporting results to gitlab/sonar etc)

[rapid sparrow]

can't imagine how it would make easier TDD or why it would make easier result exporting

[cobalt warren]

well, you dont have to docker build every time you want to run a test + lets you use IDE features better. Exporting from containers would require volume mounting, and each dockerfile would need to write its own 'run test + export in this format' steps, compared to a generic CI/CD that runs pytest against your repo with coverage and reporting set up

this is at a company, so its more than just my repo here

so reusability etc has a higher weight

[rapid sparrow]

Ci CD run is not able to use IDE features anyway

why would I need need volume exporting to run my pytest? I don't need

I don't make additional dockerfile, I override CMD in docker run containerName pytest, which makes running it pytest instead of gunicorn

just to clarify

I use Docker to run tests exactly in CI CD pipeline

in dev environment (when I am in IDE) I run pytest without docker

those are two different cases.

when using celery I am having weird errors

2022-01-17T10:38:46.967267151Z stdout F ERRO[2022-01-17T10:38:46.967051638Z] copy shim log                                 error="read /proc/self/fd/14: file already closed"
2022-01-17T10:38:46.966866201Z stdout F INFO[2022-01-17T10:38:46.966619655Z] shim disconnected                             id=67884774eec5c0518afa8b631d7d33f96ec3f46c26f72fb0efd7a266648cc8ec
2022-01-17T10:38:46.965673437Z stdout F INFO[2022-01-17T10:38:46.965245237Z] ignoring event                                container=67884774eec5c0518afa8b631d7d33f96ec3f46c26f72fb0efd7a266648cc8ec module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
2022-01-17T10:35:52.044062923Z stdout F time="2022-01-17T10:35:52.043657679Z" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/67884774eec5c0518afa8b631d7d33f96ec3f46c26f72fb0efd7a266648cc8ec pid=1208

what could be wrong?

[cobalt warren]

obviously

[rapid sparrow]

So nothing stops to run tests in docker during the CI CD run

[cobalt warren]

coverage/pytest result reports, these generate files

these files need to be stored as job artefacts and uploaded to other services

[rapid sparrow]

okay, then there can be a problem to run dockerized tests if you have files like that as result of the test

coverage report can be outputed to stdout though

and pytest too

[cobalt warren]

not really a problem ,more just weighing up, what's the best way to do this ... pros and cons to every solution

[rapid sparrow]

it is not like there is a real need to have artifacts from tests usually

at least general case is surely not needing them

[mossy pawn]

Hey folks, I was wondering if anyone knows a good solution for the PyPi publishing problem I'm having: We generate bindings for python and publish it to PyPi whenever we have a dependency update or a new release of our core tool. With that we are running into issues with max project size of 10GB being used. What is the go-to solution for this, would we need to delete old versions on a cadence? Is there some tool for this? What are we doing wrong to run into this problem?

[heavy knot]

anyone using nektos/act to test github actions with dind?

I can't get actions/checkout@v2 to work

[cursive linden]

I build a project to generate docstring using ML and looking for thoughts. Available for VS Code at https://marketplace.visualstudio.com/items?itemname=mintlify.document

[heavy knot]

👍

[ebon estuary]

% python convert.py to-png  --help
Usage: test.py to-png [options] dirname

 Convert all files with extension in dirname to .png. Only
convert if result does not exists or is older.

Required arguments:
dirname: where to search for images

Options:
  -h, --help        show this help message and exit
  --outdir=out      where to store the results
  --extension=jpeg  file extension to convert
  -n, --dry-run     don't actually do anything
  -v, --verbose     be more verbose
  -q, --quiet       be more silent

Is there any python lib for making that kind of stuff in the terminal?

I found only scriptine but it looks abandoned

[vague silo]

pillow is probably your best bet

or a python binding to imagemagick

[ebon estuary]

Naaah

[ebon estuary]

I mean that help menu

It was only the example 😅

I want the python script to show that kind of help menu in the console

[vague silo]

Oh I see

argparse, docopt or click

all of those do it, argparse is built-in

[digital quail]

Will try it out!

[heavy knot]

I have Anaconda notebooks. I install Python virtual environment, but when I import numpy it's imported normally, even I didn't install it in virtualenv

Also, I don't know if this is right, but I made requirements.txt file and I proposed on my GitHub to clone repo, make Python virtual environment and then pip install -r requirements.txt - is that right approach?

[stuck badger]

https://github.com/YashIndane/pypi_uploader

check out this project. This is a single command to put your python module on pypi.org server, So it can be used by anyone.

[heavy knot]

Hey, quick and (hopefully) simple Docker question. I have a pair of containers which read an SQLite database file held in a volume. When the containers are set up with docker compose up, I need to initialise the database file if it doesn't exist. What's considered the best practice for doing something like this?

If anyone has a suggestion, just ping me. :)

[heavy knot]

Okay, so I looked into this a bit more. One option that comes to mind is to run a shell script which checks if a file exists and creates/initializes it if it does not, but the issue is that I can't do that in my Dockerfile, since those run before volumes are mounted

[heavy knot]

https://github.com/CrashAndSideburns/mailbox/commit/a17c6d1084d4d13c22d95b6f5a2f5f0ad29ea2aa wound up doing this.

[indigo zenith]

Yes, another option is to include a separate container for that purpose and run it with docker compose

[ebon estuary]

click looks great and updated
benefit of argparse is that it's built-in

Thank you 😊

[spark lion]

Hello can I asko about selenium?

[silent sedge]

If you had I would have answered now 🙂 #web-development might be more appropriate though

[heavy knot]

Ah, and then just let that container shut down?

[indigo zenith]

Yes, that's fine

[heavy knot]

I guess that’s probably more sensible than tacking that behaviour on to another container

[indigo zenith]

I'm very new to all this myself so I don't know if it's considered best practice, but I got it working that way without much fuss so 🤷‍♂️

[heavy knot]

Okay, cool!

[heavy knot]

Actually, I have another somewhat similar question. I also need to somehow maintain SSL certificates for this application. How does one do that?

[indigo zenith]

Some hosts like GitHub pages handle it for you automatically. Otherwise I've used Certbot + Let's Encrypt. I think Cloudflare may also do some free thing but I've not looked in to that one.

[heavy knot]

Second Certbot + Let's Encrypt

[heavy knot]

Yeah, but I need to somehow renew them, and I can't figure out how to do that.

[heavy knot]

Certbot can be scripted with crontab. I believe there is a section on it in the docs.

[heavy knot]

Yeah, but wouldn't I need to run certbot in another container or something?

Idk the docs. for using it with docker are not super detailed

[heavy knot]

Haven't personally done it with Docker, but yea, could run it in its own container and write to the filesystem you mount. Or just run it on the underlying server. I'm going to say something controversial...not everything needs to be in a container.

[heavy knot]

Yeah, but I don't really have a good handle on how my server is gonna be set up yet, so I think that containerizing it will be less of a headache

At least for me

[indigo zenith]

I've never done it that way, cron is probably easier if it's an option, but this looks helpful if you need it: https://pentacent.medium.com/nginx-and-lets-encrypt-with-docker-in-less-than-5-minutes-b4b8a60d3a71

[heavy knot]

how do you do it? host machine install?

[indigo zenith]

Yes, but obviously that's not always going to be an option depending on how you are hosting

Right now though I just have a personal domain pointing to GitHub pages which auto-renews the cert. I have a DNS record for a different server as a subdomain and the cert seems to work that way. So until I ever need a separate domain I'll just do that.

[heavy knot]

right

[rapid sparrow]

How to build python into binary for running at linux without any dependencies

[rapid sparrow]

nvm

FROM python:3.10.2-alpine3.15 as build

RUN pip3 install pyinstaller
COPY app.py app.py
RUN apk add binutils
RUN pyinstaller app.py

FROM alpine:3.15.0 as run
COPY --from=build dist/app app
CMD ["./app/app"]

app.py

print("Hello, 123")

i built it

but it is taking huge size 14 megobytes

how to build in a smaller size 🤔

[gentle solstice]

you still need to include python.

if you want a smaller file, you'd need to use a language with a smaller required runtime.

or one that uses dynamic links to pre-installed libraries

[rapid sparrow]

i mean, not really I need to install python

pyinstaller gives me dist folder with all necessary running files

with python included into it

[gentle solstice]

I meant python was already included in the dist

python isn't tiny.

[rapid sparrow]

golang is a nice option

[gentle solstice]

yeah, hello world is 1mb

[finite fulcrum]

14MB sounds like a bit too much though, you can look at what it included and may be able to delete some binaries

[gentle solstice]

If you want a small file, use a pya instead? That will compile your application into an archive you can execute with python.

at least i think. I haven't actually needed to use one.

[rapid sparrow]

base_library.zip libbz2.so.1.0 libcrypto.so.1.1 libexpat.so.1 libffi.so.7 liblzma.so.5 libmpdec.so.2 libpython3.8.so.1.0 libreadline.so.8 libssl.so.1.1 libtinfo.so.6 libz.so.1
_asyncio.cpython-38-x86_64-linux-gnu.so _codecs_iso2022.cpython-38-x86_64-linux-gnu.so _contextvars.cpython-38-x86_64-linux-gnu.so _lzma.cpython-38-x86_64-linux-gnu.so _opcode.cpython-38-x86_64-linux-gnu.so resource.cpython-38-x86_64-linux-gnu.so
_bz2.cpython-38-x86_64-linux-gnu.so _codecs_jp.cpython-38-x86_64-linux-gnu.so _ctypes.cpython-38-x86_64-linux-gnu.so mmap.cpython-38-x86_64-linux-gnu.so _posixshmem.cpython-38-x86_64-linux-gnu.so _ssl.cpython-38-x86_64-linux-gnu.so
_codecs_cn.cpython-38-x86_64-linux-gnu.so _codecs_kr.cpython-38-x86_64-linux-gnu.so _decimal.cpython-38-x86_64-linux-gnu.so _multibytecodec.cpython-38-x86_64-linux-gnu.so _queue.cpython-38-x86_64-linux-gnu.so termios.cpython-38-x86_64-linux-gnu.so
_codecs_hk.cpython-38-x86_64-linux-gnu.so _codecs_tw.cpython-38-x86_64-linux-gnu.so _hashlib.cpython-38-x86_64-linux-gnu.so _multiprocessing.cpython-38-x86_64-linux-gnu.so readline.cpython-38-x86_64-linux-gnu.so

[gentle solstice]

Are you using ssl?

[rapid sparrow]

it is a script file, not a web server app, it does not need ssl

[gentle solstice]

what imports?

[rapid sparrow]

right now it is just hello world in Alpine linux
wishing to add to it... 🤔 Jinja2

[rapid sparrow]

u know...

when i ran without those .so files included

it gives precise runtime errors what is missing

I can just add one by one, until the real minimum stack for helloworld is added 😉

[finite fulcrum]

Yes, that's what I do in my apps on the binaries that have a size that's not insignificant

Can't really know what's needed without just trying what works, as it may load some things even if they're unnecessary. But things like the libcrypto are fine to remove afaik

if it's still big you can run upx on the binaries, but that can also mess them up in some cases

[heavy knot]

d = json.loads('{"example1": "1", "example2": "2", "example3": "3"}')
pairs = d.items()
d2 = json.loads('{"example4": "4", "example5": "5", "example6": "6"}')
pairs2 = d2.items()
for key, value in pairs:
  string = value
  for key, value in pairs2:
    print(string, value)

Output:

1 4
1 5
1 6
2 4
2 5
2 6
3 4
3 5
3 6

how to make the output like this?:

1 4
2 5
3 6

[tawdry needle]

use zip

[heavy knot]

what is that? sorry i am new

[tawdry needle]

!e ```python
import json

d1 = json.loads('{"example1": "1", "example2": "2", "example3": "3"}')

d2 = json.loads('{"example4": "4", "example5": "5", "example6": "6"}')

for ((key1, ), (, val2)) in zip(d1.items(), d2.items()):
print(key1, val2)

[rancid schooner]

@tawdry needle :white_check_mark: Your eval job has completed with return code 0.

001 | example1 4
002 | example2 5
003 | example3 6

[tawdry needle]

like this?

[heavy knot]

YES

THANKS

[heavy knot]

resp = requests.get(url, headers=headers)
jsons = resp.json()
listings = jsons["listings"]

for ((key1, _), (_, listing)) in zip(d.items(), listings):
  print(key1, listing['id'])

is it incorrect?

Traceback (most recent call last):
File "main.py", line 87, in <module>
for ((key1, ), (, listing)) in zip(d.items(), listings):
ValueError: too many values to unpack (expected 2)

the listing['id'] is

1316890503
1137531552
332012716
1088297153
1695237113
107951986
993199154
1131679329
1778910556

[tawdry needle]

listing is a list, so it doesn't make sense to use (_, listing)

[heavy knot]

oh yeah thankn youu

[tawdry needle]

for ((key1, _), (_, val2)) in zip(d1.items(), d2.items()):
    print(key1, val2)

this is equivalent to:

for pair1, pair2 in zip(d1.items(), d2.items()):
    (key1, _) = pair1
    (_, val2) = pair2
    print(key1, val2)

[heavy knot]

it's work rn

[tawdry needle]

it lets you do multiple layers of "unpacking"

[heavy knot]

tysm

[tawdry needle]

also equivalent:

for item in zip(d1.items(), d2.items()):
    pair1, pair2 = item
    key1, _ = pair1
    _, val2 = pair2
    print(key1, val2)

zip produces an iterator of tuples

dict.items also produces an interator of tuples

but if you have a list, you don't have an iterator of tuples: the list itself is an iterator of the values in the list, which you don't want to "unpack", because they might or might not be tuples

(technically it's an "iterable" which is converted to an "iterator" by the for loop, but that's not relevant right now)

[heavy knot]

I just want to make auto making thread / creating list. I just checked in console, the error says

"{"error":"Unexpected token u in JSON at position 0","version":"1.3.0"}".

How can I resolve this error?

The Python Code:

import requests
from io import BytesIO

headers = {
    'authority': 'traderie.com',
    'method': 'POST',
    'path': '/api/adopt_me/listings/create',
    'scheme': 'https',
    'accept': '*/*',
    'accept-encoding': 'gzip, deflate, br',
    'accept-language': 'en-US,en;q=0.9',
    'authorization': 'Bearer ',
    'content-length': '722',
    'content-type': 'multipart/form-data; boundary=----WebKitFormBoundarypqN01X2kArSkOHBn',
    'cookie': '_ga=GA1.2.850822138.1642224893; _gid=GA1.2.556027755.1642224893; usprivacy=1---; ad_clicker=false; _pw_fingerprint=6818da724c1a866373d3104d2cd082e4; _gcl_au=1.1.628091182.1642224893; _hjid=cc106801-cff3-4156-afad-2b5845f96325; _hjSessionUser_2441420=eyJpZCI6IjQyMjdmMjVjLWEyMjItNTI1Ny04NjUwLWY3MmU1NjEyYjNlZiIsImNyZWF0ZWQiOjE2NDIyMjQ4OTMyMDAsImV4aXN0aW5nIjp0cnVlfQ==; _hjSession_2441420=eyJpZCI6ImVkODE1MDc5LTYwNWItNDIzMC1iODUxLWZkOGRlNmM2YjM3MCIsImNyZWF0ZWQiOjE2NDIyMjQ4OTM0MzEsImluU2FtcGxlIjpmYWxzZX0=; _hjAbsoluteSessionInProgress=0; _pw_audience_segments=["1","3","4","5","6","7","8","9","10"]; __stripe_mid=a848c4df-5112-4afd-8bdd-2235857cc00abc7ef9; __stripe_sid=e1c6337b-f03f-4b93-8441-eb1d118cee58001fb3; _pw_audience_categories=["games_hardcore"]; _pbjs_userid_consent_data=3524755945110770; pwUID=369796799639702; __gads=ID=dcb867edd8f2877b:T=1642224926:S=ALNI_MYArRYYa1oDgieG9f1YFUuJjVv-oQ; G_ENABLED_IDPS=google; G_AUTHUSER_H=2; _gat_UA-176465218-1=1; _gat_gtag_UA_176465218_1=1; cto_bundle=5p2Tj182VmVrbFhLNUFmS3lZNXllZnk2b1UzbkYwR0JhVUFUMSUyRnplYWd1QWh1VyUyQmlrJTJGYjlOTU1XdE1xb050NjhHJTJCajczQW1MeDNQSnRoRG9tM2UzeHN1blk5Z2pHYllDUlJLQUxVUHVDclp1JTJGT0N6cHluWjdjZzRGNVdRZFhqeUJiQmZSZlVORnI0dGlhdnV4MDQ2YnZFeVVnJTNEJTNE; playwirePageViews=16',
    'origin': 'https://traderie.com',
    'referer': 'https://traderie.com/adoptme/product/1276977709',
    'sec-ch-ua': '" Not;A Brand";v="99", "Google Chrome";v="97", "Chromium";v="97"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"Windows"',
    'sec-fetch-dest': 'empty',
    'sec-fetch-mode': 'cors',
    'sec-fetch-site': 'same-origin',
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
}

body = '{"acceptListingPrice":false,"currencyGroupPrices":[],"diy":false,"endTime":"","free":false,"item":"1276977709","itemType":"pets","makeOffer":true,"needMaterials":false,"offerBells":false,"offerNmt":false,"offerWishlist":false,"offerWishlistId":"","selling":true,"standingListing":false,"stockListing":false,"touchTrading":false,"wishlist":"","amount":1,"properties":[{"id":1,"property":"Flying","option":true,"type":"bool","preferred":null},{"id":2,"property":"Rideable","option":true,"type":"bool","preferred":null},{"id":9,"property":"Age","option":"Full Grown","type":"string","preferred":null}]}'

resp = requests.post(f"https://traderie.com/api/adopt_me/listings/create", data=body, headers=headers)

print(resp.text)

[tawdry needle]

this looks like it violates our rule 5

!rules 5

[rancid schooner]

Rules

5. Do not provide or request help on projects that may break laws, breach terms of services, or are malicious or inappropriate.

[tawdry needle]

oh wait, this looks like an api

yeah sorry, this probably isn't something we can help with

https://traderie.com/termsofservice

Additionally, you agree not to:

• Use any robot, spider, or other automatic device, process, or means to access the website for any purpose, including monitoring or copying any of the material on the website.

[oak ridge]

hi

can someone tell me some cool and interesting python libraries?

thanks

[red oxide]

@rapid sparrow Hey! Based on your recommendations I looked into GitLab, GitLab runner and Ansible. I kinda understand how everything will works but I have some doubts.

1. When the code will be pushed, a job will be triggered right?
   I looked into an example where it did echo into the console but It was only an output in GitLab
   So, I'm not sure how can I trigger it to run my playbooks to my other machines

2. They selected shell executor while creating runner, there were many other runners, docker and what not... Which one would be good for my use case?

3. As you know I have a selenium automation so I just gotta write a playbook that kills existing code and browser, pulls the latest code and runs it back again?
   Very confused about this part as I'm not sure what all I should do in playbook

[rapid sparrow]

1. Yes, when the code will be pushed, pipeline with jobs will be triggered
   You can make additional conditions when particular jobs will be triggered, based on file changes in certain file folder, or manual verifications and e.t.c.

1.2 In order to run playbook in your machine, all you need to supply SSH private key into your repository, so they ansible could use it to run its work against your machines.
I would recommend using Gitlab Secrets in order to store private key securely and injecting it during the job

2. I would recommend standard docker executor for starting. It allows better best practice with having all your pipeline jobs as a code within one repository.
   Shell executor is too much influenced by local machine configuration, it can lead to bad things in long perspective. Docker machine could be a good option too technically, but official docker machine died, i am not sure how well this option. I did not work with this technology due to its deprecation from original authors at least. Gitlab uses a fork highly likely

3. Ansible code is idempotent. All you need to write basically:
   Ansible tasks to install necessary dependencies to your machine
   Ansible task to copy the code of your current repository to server target
   Ansible task to launch the application
   Optionally adding somewhere task to trigger handler that restarts the application, based on one of the tasks above having status "changed" (it did job and not was just skipped)

Ansible tasks are idempotent. They do the same even if interrupted, they can be safely rerun from where they were stopped.
When new code would be supplied, they will be able to do exactly same job like they were first time installing it, they will just change what was changed. In order to apply new code and to relaunch as new app. As long as you keep Ansible philosophy of idempotency during its writing. Ansible language fully encourages you to do that with its syntax and modules that helps you everywhere for that

Obviously ansible will be all the time installing more and more polluted server target machine.

Use docker to minimize the impact

Or apply terraform or similar family of tools(Pulumi) to recreate server when pipeline is started

Or ignore the problem, it will be not really often causing the problems anyway

[red oxide]

I'm still having some trouble understanding how the playbooks will run on trigger.

Suppose I push code from my machine on gitlab so what instructions gitlab-ci.yaml should have? Do we specify something here that triggers ansible here?
cause I read that ansible is a master slave architecture so ansible has to be hosted somewhere

[untold crystal]

if im working in some big repo, and have some subdir that i want to make into a package.. for example
/a/b/c/mypackage
and all the imports in mypackage use that full directory tree
from a.b.c.mypackage import foo
is there any magic with setuptools/packaging that can fix those imports? I think i need to use sed and just edit them before packaging in some tempdir...

[rapid sparrow]

playbook will run on trigger from submitted code to repo

once Gitlab Runner will find the new job

it will execute them inside docker containers lined as pipeline

ansible in your code will be executed directly in them like a regular Shell Script

basically ansible is a Shell Script, but written in Python and has Yaml syntax to be perfect configuration management tool

It is not requiring to be hosted

Ansible is agentless and serverless

It works purely through ssh connection

Basically where it will be executed, ansible will transform its code into installing scripting actions and transmit them through ssh to execute at its target

[rapid sparrow]

https://docs.ansible.com/ansible/2.9/modules/docker_container_module.html
https://docs.ansible.com/ansible/latest/collections/community/docker/docker_image_module.html

- name: Re-create a redis container
  docker_container:
    name: myredis
    image: redis
    command: redis-server --appendonly yes
    state: present
    recreate: yes
    exposed_ports:
      - 6379
    volumes_from:
      - mydata

Example of Ansible code to manage docker containers

We declare the desired state, the container being present and recreated if it is already running. We declare its parameters.

The ansible will make it to the desired state during the task execution

[red oxide]

That's a lot of info to digest for sure, I'll go though everything and try it out. Thanks again @rapid sparrow :)

[rapid sparrow]

Read the book 😉

It helped me to get into the ansible

https://www.amazon.co.uk/Practical-Ansible-infrastructure-configuration-applications/dp/1789807468
Quite fresh one

[red oxide]

Okay, thanks 🤧
You are always so helpful! 🙂

[rapid sparrow]

😊

[woven jetty]

Hey guys,

So I am completely new to docker, and fairly new to Python aswell. I am trying to put a flask "app" in a docker container. In my project I am using notion-client (pip install notion-client) which works great on my own pc, however when I am trying to do the same on my docker I am getting the following errors:

ERROR: Could not find a version that satisfies the requirement notion-client (from versions: none)
ERROR: No matching distribution found for notion-client

My Dockerfile looks as the following:

FROM tiangolo/uwsgi-nginx-flask:python3.6-alpine3.7
RUN apk --update add bash nano
ENV STATIC_URL /static
ENV STATIC_PATH /var/www/app/static
COPY ./requirements.txt /var/www/requirements.txt
RUN pip install --upgrade pip
RUN pip3 install notion-client
RUN pip install -r /var/www/requirements.txt

I have also tried pip install notion-client

But getting the same error

[rapid sparrow]

pip and pip3 are two different things

also better to use python in docker of same version that u use for development

[woven jetty]

On my own environment I use python 3.9.0

How do I go about doing pip install notion-client

[rapid sparrow]

how about using pip3

and changing docker image to python 3.9

and preferably freezeing notion-client to specific version

like pip3 install notion-client=1.5.3

(randomly wrote version)

you get versions when you freeze to requirements.txt

[woven jetty]

I think the docker image I use only supports 3.8

as max

https://hub.docker.com/r/tiangolo/uwsgi-nginx-flask/

if I understand this correctly

[rapid sparrow]

still better than 3.6

[woven jetty]

True

What do you mean by freezing?

[rapid sparrow]

install python 3.8 to your dev env too

and test that it works in 3.8