#tools-and-devops

1 messages · Page 2 of 1

rapid sparrow
#

it is all SaaS web site hosters offer it

pine fern
#

it has that + the frontend stuff

#

like he can spin up AWS lambdas + other services easily

rapid sparrow
#

is it having CLI interface and code to control, or its pure GUI interface?

pine fern
#

the former

rapid sparrow
#

well, not bad pithink But also technically it could be adhieved with the mentioned tools terraform/Pulumi/AWS CDK
Just their code + Upload frontend

pine fern
#

yeah thats what im saying. im not sure why he made it tbh

#

he mustve been bored at the company or something?

rapid sparrow
pine fern
#

ikr?

#

im pretty sure its in .yml too

rapid sparrow
#

So, i hope he made without reinventing wheels for that

#

it is nice to do, for AWS that's complex stuff to write even if having infra provisioning tools

#

but without them it is kind of pointless to do

pine fern
#

and ECR

#

and he was like, 'oh our [in-house tool] doesnt have ECR compatibility. let me incorporate that feature'

#

then he did it

#

and i pushed my image to aws ecr the manual way

#

while he used the in-house tool

cursive grove
#

Anyone know how I could expose my postgres database to be accessed in my api image?

I have something like this

services:
  database:
    image: postgres:14-alpine
    volumes:
      - postgres_data:/var/lib
    env_file:
      - ./.env
    ports:
      - 5432:5432
    networks:
      - app

  api:
    build:
      context: ./
      dockerfile: api.dockerfile
    env_file:
      - ./.env
    ports:
      - 4560:4560
    depends_on:
      - database
    networks:
      - app

volumes:
  postgres_data:

networks:
  app:
    driver: bridge
#

Keep getting

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "0.0.0.0", port 5432 failed: Connection refused

Is the server running on that host and accepting TCP/IP connections?
Probably because it's trying to access my machine and not the container itself (?)

#

I try to put the local ip of the wsl container but also not working

indigo zenith
cursive grove
#

Yes

cursive grove
indigo zenith
#

That "bridge" might be causing the problem you describe of it going to host instead of container

cursive grove
#

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "127.0.0.1", port 5432 failed: Connection refused

Is the server running on that host and accepting TCP/IP connections?

#

If I start the API trough vscode task it connects normally

#

With the same .env

indigo zenith
#

You could also check the container logs in docker itself for any hints

cursive grove
#

If I run the API container and make a request that interact with the DB from outside it works indeed

#

From the inside it just gives me that error

#

2022-08-08 23:57:56.253 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432

2022-08-08 23:57:56.253 UTC [1] LOG: listening on IPv6 address "::", port 5432

2022-08-08 23:57:56.260 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"

2022-08-08 23:57:56.269 UTC [21] LOG: database system was shut down at 2022-08-08 23:27:00 UTC

2022-08-08 23:57:56.274 UTC [1] LOG: database system is ready to accept connections

done

#

DB container ir all good runing

half grotto
#

Poetry users! Anyone know if there's a way to always include a custom repository with every call of poetry new or poetry init?

deep estuary
half grotto
#

Yeah, scripting it is about where we're at currently. Thanks!

cursive torrent
#

so i have a tiny project which generates text but I would like to make it searchable. i'm using flask + python + pytorch as the main tech currently (+redis/rq). My documents are basically ~500 words long.

what do you recommend w.r..t adding a search engine functionality to this without too much trouble?

I just want to search by keyword and have a reasonable set of results returned... I don't want to get into implementing bigrams/tf-idf and the sort...and I need to be able to insert documents dynamically (i.e., at runtime) into the index...

I considered: elasticsearch (seems overkill), redisearch (it seems like i would need to build the index periodically as it's supposed to be for ephemeral searches)

deep estuary
#

but I know our eng team just has a huge collection of slightly jank bash scripts for this hahahahaa

half grotto
#

I mean, the duplication of repository information is already pretty redundant. Need to specify the repository in the config AND in the .toml file. There's a few things about custom repos which seem a bit lacking in Poetry 😦

cursive grove
#

Don't how to do otherwise, but it is what it is

sinful pollen
#

hey yall, I think that main and development branches have somehow switched

#

something is wrong at least

#

cause whenever I make a single commit on development branch

#

I always get "merge conflicts" when creating a merge request to main

#

Even when no commit have been made to main

#

and the wierd thing is that it says that main is 14 commits behind development

#

when it should be 1

tawdry needle
#

also show the output of this command:

git log --topo-order --graph --oneline --decorate=short --all
heavy knot
#

hi can someone pls help me encrypt my code yeyCat

frosty dawn
heavy knot
#

yes thats the problem i dont know how to

#

and its not just one file

#

its multiple but with cogs

tender current
# 
├── LDS
│   ├── __init__.py
│   ├── book_loader.py
│   ├── french_textrank.py
│   ├── gen_utils.py
│   ├── hf_summarizers.py
│   ├── nlp_utils.py
│   ├── pythonrouge_evaluate.py
│   ├── randomsum.py
│   ├── summarization.egg-info
│   └── summarizer_ios.py
├── __init__.py
├── dev-requirements.txt
├── experiments
│   ├── evaluate.py
│   ├── ex_french_textrank.py
│   ├── ex_hf_summarizers.py
│   └── ex_randomsum.py
├── poetry.lock
├── pyproject.toml

pylint complains about all imports of LDS from experiments in this fashion:

experiments/ex_french_textrank.py:6:0: E0401: Unable to import 'LDS.french_textrank' (import-error)
experiments/ex_french_textrank.py:6:0: E0611: No name 'french_textrank' in module 'LDS' (no-name-in-module)

from LDS.french_textrank import FrenchTextRank

The code runs fine. How can I fix this? 😕

tawdry needle
#

@tender current the linter is probably running with working directory experiments/ rather than the top level of your project

#

by default python searches for modules in the same directory as the script you're running

#

what directory are you in when you're running pylint, and how are you running it?

tender current
#

I'm in the project root (also named LDS), and I run it like pylint --output-format=colorized $(find . -maxdepth 2 -iname "*.py")

#

@tawdry needle

tawdry needle
#

the one that is sibling to pyproject.toml

#

__init__.py marks a directory as a python module

tender current
#

Holy cow that fixed it 😍

#

Thank you 🙏

tawdry needle
#

a "package" in this case is just a "module that can contain other modules"

tender current
#

That __init__.py was a leftover from before I moved my .pys into separate directories (LDS/ and experiments/)

#

That makes sense, thank you!

rapid sparrow
#

finally. I have read the book about DevOps cultural aspect

#

apperently i did not know significant amount of it xD

#

i knew only small aspect what was assumed under it

#

the book is trully amazin

#

you can read it while.... feeling almost like you are high on drugs from what is happening in it.

#

Recommending to read it to others

unknown.png
#

It is quite easy to read one. I have consumed it really quickly within few days. Its novel way to tell the story is super fast to comprehend

solemn ravine
#

hello, i am trying to schedule a daily git pull on my repo but for some reason it gave me this error

 * branch            HEAD       -> FETCH_HEAD
error: The following untracked working tree files would be overwritten by merge:
        item.ipynb
        picture.png
Please move or remove them before you merge.
Aborting

here's my code:

import subprocess
import schedule
import time
def goonline():
    subprocess.call(["git", "pull", "https://personalaccesskey@github.com/myusername/reponame"])
schedule.every().day.at("05:00").do(goonline)
while True:
    schedule.run_pending()
    time.sleep(1) # not sure how to do scheduling
wooden ibex
#

Why? Just setup a cronjob.

solemn ravine
solemn ravine
solemn ravine
# wooden ibex Why? Just setup a cronjob.

also, i have a question related to cronjob: if i have a python file running in the background and the cron activates and runs another python file, will the original process terminate abruptly and cause errors?

#

my current solution to this is that at the end of the cron python file, it will reboot itself and after reboot, the previous process will run automatically after startup. does my current solution work?

wooden ibex
rapid sparrow
rapid sparrow
solemn ravine
# rapid sparrow Btw, what u a trying to achieve here? It looks like pre CI CD attempts.

as my raspberry pi are going to be physically away from me when i implement it, to check for any code updates that i've written, it'll connect itself to the internet daily at this time to look for updates on my repo and download them if any. (i think the internet connection for this would be turned off for the rest of the day to conserve resources). the raspberry pi is also going to produce some outputs so my next step after the update checker is to implement a way to send output to a database or smth

solemn ravine
rapid sparrow
solemn ravine
rapid sparrow
solemn ravine
steep yoke
#

!code

#

Hi, can someone explain me how do I keep my program opened once he finish, to have time to show the answer.

#

And how to dont run another part of the code if the person choose the incorrect input.

indigo zenith
lethal yarrow
#

Want to learn microservices, I know there is Docker and Kubernetes. I like learning from videos, it's easier for me, so I went through some freeCodeCamp videos and found a lot of them. Now some videos include Docker and Kubernetes, like 2 concepts in 1, but I think it's more intelligent to learn one at a time. What do you think? Also, which video from here would you recommend? https://www.youtube.com/results?search_query=free+code+camp+docker Note I only know Python, so I'm ignoring JS related videos from that link. On the other hand, is there any other resource you would recommend for Docker/Kubernetes apart from one of those videos in that link? Thanks in advance!

Mention on reply

deep estuary
lethal yarrow
#

great 🙂 will do so. and as for where to learn it? what's your opinion/recommendation ?

deep estuary
#

when I was learning I just used the guides that Docker had put together as well as examples from people who had already built applications with docker containers https://docs.docker.com/get-started/

Docker Documentation
Orientation and setup

Get oriented on some basics of Docker and install Docker Desktop.

lethal yarrow
#

great. Well, then next thing to learn. tysm joe!

coarse rapids
#

I saw this posted on news this morning, thought I'd check it out and it's actually decent. It highlighted a few issues in my Dockerfile that I hadn't thought about and I'm trying it in my build pipeline.
https://gitlab.com/gitlab-org/incubation-engineering/ai-assist/dokter
A static analysis tool for Dockerfiles, think like flake8 but instead of checking formatting it outputs security problems, possible improvements and best practices for your Dockerfile
Examples:
dokter -d Dockerfile will show you any warnings for your file
dokter -d Dockerfile -s will print a rewritten Dockerfile with the changes it would make

rapid sparrow
obsidian marsh
#

is there a way to stop cibuildwheel from building for pypy

mystic void
manic siren
#

difference between conda and virtualenv?

rapid sparrow
subtle quarry
sand thistle
#

any tips on managing secrets using Digital ocean

rapid sparrow
#

Purely theoretically, Vault tech could be used for more. I haven't used though

sand thistle
#

Thanks @rapid sparrow

lusty forge
#

How would you guys run an endpoint for a Prometheus /metrics scrape? Would it just be a Quart server in your main.py and a create_task in the setup_hook with a Prometheus config file looking something like ```yml

  • job_name: "prometheus"
    scrape_interval: 1m
    static_configs:
    • targets: ["localhost:9090"]```(for a Discord bot)
rapid sparrow
#

it is really... messed up technology created to add async stuff to flask

#

with only half movement tech from flask to quart

#

not really even justified to exist now since flask supports already async too

#

I would have preferably used FastAPI though. It is a nice default option.

lusty forge
#

Oh I see, in that case wouldn't I have to run my bot in a create_task?

rapid sparrow
#

suprisingly i have same issue

#

except i need to have rest api info out of it

#

i have really trouble with async stuff, but managed to find my way with having it working with aiohttp

#

I would prefer to find a way how to it with fastapi though

rapid sparrow
#

i was recommended to use create_task too

#

but i don't get it for now xD

#

ergh, and it is ugly code, i did not clean it yet

#

i just tried different ideas and hoped to see them working

lusty forge
#

ah i see, in that case i'll hold off on prometheus lol

#

looks a bit sloppy, or i might have it go through my ZMQ socket and use another dedicated FastAPI / Quart API to query that socket

iron basalt
#

alternatively you can use prometheus pushgateway

#

you can post the registry to it in a background task, rather than running a server & having prometheus scrape your service

#

otherwise i'd recommend looking into aiohttp for serving metrics

#

i dont see a reason to try to run fastapi in a non-api process

mystic void
#

Please no ads! Thanks
( I also down voted it)

lusty forge
#

Is this a valid Prometheus config to scrape an aioprometheus socket? If I run docker compose exec grafana curl prometheus:9090 I won't find any bot metrics exposed, though if I run docker compose exec bot curl localhost:8000 it will display Prometheus styled data```yml
scrape_configs:

  • job_name: "prometheus"
    scrape_interval: 1m
    static_configs:

    • targets: ["localhost:9090"]

  • job_name: "node"
    static_configs:

    • targets: ["node_exporter:9100"]

  • job_name: "overseer"
    static_configs:

    • targets: ['bot:8000']```
tender current
#

Is Poetry both a build-system and a package manager? I don't need a requirements.txt nor a setup.py when using it?

lusty forge
#

Poetry is a glorified dependency manager, you don't quite need a requirements.txt nor a setup.py as Poetry does everything through its CLI, i.e. poetry add aiohttp would be how you add a dependency.
It can also help you generate a pyproject.toml, essentially a more readable setup.py and generates the actual setup.py for things like PyPi when you use poetry publish (which will automatically upload it to PyPi too).

#

Though every Poetry project requires a pyproject.toml which'll generate a poetry.lock file full of every dependency you need, preventing duplicates and whatnot.

sacred lodge
#

not sure if this is the right channel, but the Pillow module pings localhost when showing a picture on windows

#

and thats odd.... right?

finite fulcrum
#

it's most probably used for the delay

sacred lodge
#

yea, in the mac version, they sleep for 20

#

the programmer, Fredrik Lundh, wrote re and PIL though. Now hes gone. RIP. what a sad fucking rabbit hole that was

#

damn he also had his hand in tkinter

#

fucking legend

polar lava
#

I need some help with coming up with an initiative idea for data analytics related. Right now the scope is kind of broad because anything is possible, no particular direction yet. It could be for improvements of an existing framework/lib or project or using python to link another software to the database/warehouses/lake to either solve problems or improve cost or efficiency.

I mainly deal with python, tableau, aws, datalake, jenkins CI/CD, power bi and other dashboards.

Is there any recommendations or some references page I could look at for ideas?

hardy jay
#

Would it be dumb to have a python script to write powershell code

rapid sparrow
#

yes = why would you need writing powershell code?

#

no = because python is perfect glue and templating stuff to make other invoked stuff

rapid sparrow
#

!e

print('run --user 0 --rm -v "$(pwd):/code" {service}_base bash'.format(service="scrappy"))
rancid schoonerBOT
#

@rapid sparrow :white_check_mark: Your 3.11 eval job has completed with return code 0.

run --user 0 --rm -v "$(pwd):/code" scrappy_base bash
rapid sparrow
#

python can easily template stuff for other commands 🙂

#

togerther with argparse... python becomes ultimate glue

#

learned how to have sub groups in the right way, so that --help commands would work properly

#

when i invoke python3 make.py scrappy migrate
to run database migration test my makefile for example combines me next command

'docker-compose -f docker-compose.scrappy.yml  -p 6c7a83f5 build && docker-compose -f docker-compose.scrappy.yml  -p 6c7a83f5 run --rm scrappy_base sh -c "python3 utils/scripts/await_db.py --host=scrappy_db && python3 make.py shell migrate scrappy"'

which invokes sub command

'alembic -c scrappy/alembic.ini upgrade "head"'
hardy jay
#

@rapid sparrow I have an excel file full of details of new people I need to put into Active Directory, i could write a python script to read the excel and output the powershell code I could copy and paste

#

ohh, so a python script to template A powershell script would be useful

wooden charm
# hardy jay <@370435997974134785> I have an excel file full of details of new people I need...

I've worked with Powershell a lot, and I think you may be adding some unnecessary complexity by templating powershell script in python. If you're looking for a quick way to get users into Active Directory, I would suggest converting your excel file to CSV and google "powershell create users from csv"

But Python may be better suited to automate other stuff around that if you wanted it to. Eg. Automate the excel to csv conversion for you, generate secure passwords, send onboarding emails.

shrewd tendon
#

Hi guys, I've made a PYTHON MODULE for automating the downloading of pdf's from libgen (by web-scrapping).

We just have to enter the NAME & AUTHOR of the book and it downloads the best version of the PDF of the book from LIBGEN.

I'd love to work with someone for doing the same for PDF-DRIVE, ZLIBS and some more other websites.

Then combining all of them into one single module. So the user has to enter the NAME & AUTHOR of the book and then the code would download the latest and best version of the PDF of that particular book.

rapid sparrow
rancid schoonerBOT
#
Rules

5. Do not provide or request help on projects that may break laws, breach terms of services, or are malicious or inappropriate.

lusty forge
#

Is this a valid Prometheus config to scrape an aioprometheus socket? If I run docker compose exec grafana curl prometheus:9090 I won't find any bot metrics exposed, though if I run docker compose exec bot curl localhost:8000 it will display Prometheus styled data```yml
scrape_configs:

  • job_name: "prometheus"
    scrape_interval: 1m
    static_configs:

    • targets: ["localhost:9090"]

  • job_name: "node"
    static_configs:

    • targets: ["node_exporter:9100"]

  • job_name: "overseer"
    static_configs:

    • targets: ['bot:8000']```
sinful pollen
#

Hey guys. Whats the difference between "Checkout and Rebase into 'development'" and "Rebase 'development' onto 'main'".
I know what rebase is, but I'm struggling with differentiating the wording. Like I know "merge 'main' into 'development'" means creating a new commit in the development branch with all the changes from master (fixing conflicts when necessary). However when it says "into" vs "onto" I'm unsure what I have to use

unknown.png
thorny shell
#

I think the first means "check out your "development" branch, then fetch from upstream and rebase onto that" -- so that you have the latest copy of upstream's "development" stuff

#

the second means "rebase development onto main, so that you have the latest "main" stuff mixed in there too"

ruby linden
#

When should I use tags on Git?

thorny shell
#

when you want to remember a specific commit for some reason

#

like: this commit is probably a dead end, but I don't want to completely forget about it because maybe there's something valuable in there

#

if you were a big fancy software project that did "releases", you'd create a tag for each release, so that if a customer says "Hey I'm running version v1.2.3.4 and it's broken", you could find the tag named v1.2.3.4 and know exactly which code they were running

#

but that seems less common these days

foggy elk
#

Have someone tried to automate TikTok uploading? I see some already done Python bots on that, but they require that you login manually, which destroys 90% of the purpose of automatic upload.

thorny shell
#

never tried it myself

foggy elk
#

Yeah, might be smarter to try to use the actual documentation by them, rather than selenium chaos xD

glacial sand
#

Devops newb: Are you supposed to have your formatting, typechecking and testing as separate github workflows ?

thorny shell
#

beats me

#

I shoved all those things into the build step

#

(I use azure, not github; but it's probably roughly the same)

#

((and the two will probably converge at some point, given that Microsoft now owns github))

glacial sand
#

I'm leaning towards two flows, a format+typecheck and a unittest one. Just checking to be sure 🤔

thorny shell
#

I honestly don't know the pros and cons

#

I assume "two flows" is a bit more effort to create initially than one, but maybe not much

#

but I don't see any benefit either

rapid sparrow
glacial sand
#

@rapid sparrow do you cache your dependencies ?

full bronze
#

I had them in separate workflows but have recently put them into one using pre-commit as I can just add any other tools into that and it automatically gets added to ci

rapid sparrow
full bronze
#

*testing is in it's own workflow though

rapid sparrow
# full bronze I had them in separate workflows but have recently put them into one using pre-c... 
variables:
  IMAGE_PIPELINE_RUNNER: darkwind8/pipeline_images:kubectl-v1.23_v1
  DOCKER_HOST: tcp://dind-service:2375

stages:
  - test

.env_scrappy_staging: &env_scrappy_staging
  - echo "$env_scrappy_staging" > .env.scrappy

unit-testing:
  image: ${IMAGE_PIPELINE_RUNNER}
  stage: test
  before_script:
    - *env_scrappy_staging
  script:
    - python3 make.py scrappy test

linting:
  image: ${IMAGE_PIPELINE_RUNNER}
  stage: test
  before_script:
    - *env_scrappy_staging
  script:
    - python3 make.py scrappy lint

migrate-scrappy:
  image: ${IMAGE_PIPELINE_RUNNER}
  stage: test
  before_script:
    - *env_scrappy_staging
  script:
    - python3 make.py scrappy migrate

I am using my python makefile with argparse, to have simple docker compose commands same that I use locally

#

Simple CI easy to repeat locally

glacial sand
#

oh that looks pretty involved, can't you use github's cache in the workflow and skip docker ? since actions are containers themselves ?

rapid sparrow
rapid sparrow
#

In my current version I can migrate between different CI tools with zero effort

#

And able to reproduce locally without triggering CI and waiting for it to run

#

And I just dislike github actions syntax and features. In comparison to gitlab CI, I think it sucks.

glacial sand
#

oh cool, I suppose it also gives you unlimited testing for private repos since you are relying on your own host ?

rapid sparrow
glacial sand
#

tho I can't image it's easy to reach githubs limit

rapid sparrow
#

That is where I get dependency caching with zero anything usage

glacial sand
#

will keep in mind for any future big projects

#

tyty ❤️

rapid sparrow
#

xD currently I do it just for pet project

rapid sparrow
glacial sand
#

I just don't think I can be bothered to set up dockers and self host for a pet project

rapid sparrow
glacial sand
#

are you not losing some of docker's advantage when "polluting" the image with multiple projects ?

#

i mean for the dependencies

rapid sparrow
rapid sparrow
lusty forge
ember veldt
#

hi, am searching courses or tutos about web scraping using selenium WITH PYTHON. there is any recomondation?

wooden charm
#

Is anyone aware of a tool or service for scanning a database for personal identifiable information?
We have a test customer database where details have been obfuscated already with a SQL script by our developers, but I would like to be able to scan for future changes where new tables or fields with PII has been introduced

wooden charm
#

I almost found the exact tool I needed and its written in Python. Doesnt meet my need because it doesnt support MSSQL.

But still looks darn cool. Sharing here just in case others see a use.
https://github.com/tokern/piicatcher

GitHub
GitHub - tokern/piicatcher: Scan databases and data warehouses for ...

Scan databases and data warehouses for PII data. Tag tables and columns in data catalogs like Amundsen and Datahub - GitHub - tokern/piicatcher: Scan databases and data warehouses for PII data. Tag...

lusty forge
#

Is it possible to attach a Docker container to an already running compose network?

knotty basin
#

should i ask question related to flake8 here ?

#

nvm

tawdry needle
quick sparrow
#

hi, is it better for a commit message to be "create file.py" or "add file.py"?

#

in git

tawdry needle
#

maybe for something like pyproject.toml it's fine

#

both "create" and "add" are acceptable. if you use some particular standard (e.g. Conventional Commits) then follow that standard, otherwise do whatever you prefer

quick sparrow
soft axle
#

If I have a Svelte web app with FastAPI mounted on /api, all behind Nginx as a reverse proxy, what would be the best way to dockerise that? Should it all go in one image, or should I split it up somehow?

tawdry needle
soft axle
tawdry needle
#

you could also offer two versions

soft axle
#

In that case, the Svelte and FastAPI parts - should I have them in the same project, or separate them?

tawdry needle
#

or make two separate container images

#

how do you have it configured currently with nginx?

soft axle
#

I've tried to not jump the gun here, and am wanting to fully plan out the structure of this before I go about doing things

#

Is it possible to have two containers in a single project? Or is that a rubbish idea?

tawdry needle
#

of course it's possible

#

how do you currently run your project? e.g. just as a dev server

#

i assume the code is currently split into something like this:

README
frontend/         <-- svelte
  package.json
  src/
  ...
backend/          <-- fastapi
  pyproject.toml
  src/
  ...
soft axle
#

Currently I have only the FastAPI part implemented - but yes, that is the plan

#

I've also considered making the backend into a PyCharm project and the frontend into a WebStorm app, then putting that in a common directory with version control

tawdry needle
#

i think the easiest thing to do is to make one dockerfile for each, so frontend/Dockerfile and backend/Dockerfile, and to build a separate image from each dockerfile

#

this also gives you the most flexibility to deploy and run the two parts separately

#

you can put docker-compose.yaml at the top level of the project (alongside README) as well as a sample nginx config

# 
README
docker-compose.yaml
nginx/
  sites-available/
    myapp-frontend.conf
    myapp-backend.conf
frontend/
  Dockerfile
  package.json
  src/
  ...
backend/
  Dockerfile
  pyproject.toml
  src/
  ...
#

and you can use pm2 or whatever to run this all locally during development

#

i tend to use a makefile and/or shell scripts to encapsulate common operations like ./scripts/run.sh frontend or make run-both

# 
## Project top-level stuff
.gitignore
README
Makefile 
docker-compose.yaml
scripts/...

## Nginx example config
nginx/
  sites-available/
    myapp-frontend.conf
    myapp-backend.conf

## Frontend app (Svelte)
frontend/
  Dockerfile
  Makefile
  package.json
  scripts/...
  src/...
  test/...

## Backend app (FastAPI)
backend/
  Dockerfile
  Makefile
  pyproject.toml
  scripts/...
  src/...
  test/...
tawdry needle
#

feel free to set up a pycharm project and webstorm project, and include those in version control too if you prefer

#

does the intellij ecosystem have an affordance for "polyglot" projects? or do you end up using 2 separate ide's?

soft axle
#

Well, there is interoperability to a certain extent

#

it's just that each IDE specialises in a specifc type of project

#

for example, WebStorm will have better support for Svelte

#

You can open one project in two different IDEs though - just not simultaneously, afaik

tawdry needle
#

got it. does the layout above make sense?

soft axle
#

Yeah, I believe it does

#

So something like this could be done, right?

unknown.png
tawdry needle
#

pretty much

true vapor
#

You shouldn't inflict pycharm or webstorm on users. I've never seen anything in the .idea that was worth checking in to git.

And pycharm pro has pretty much all the features of webstorm

tawdry needle
#

you can also "attach" two projects together to a parent project

#

and yes that's true too, i don't normally suggest checking the project config files into version control

#

it can be useful to include things like run configurations, but you can also use a makefile for that

soft axle
#

yeah, I wasn't planning on committing that - that was to denote which parts of the project would be opened by which IDE

#

however, I think I'll probably cross that bridge when I'll get to it, and see if PyCharm has some sort of Svelte compatibility plugin

true vapor
#

Fair enough - but pycharm pro has all the js support that webstorm does

#

Some? What is it missing?

soft axle
#

You know what? On second thoughts, screw WebStorm :p

#

that's a simplification I definitely won't complain about

true vapor
#

Yeah, it took a couple of years for me to realise it was superfluous. I think the one thing it has that pycharm doesn't is super silly - mongodb support or something

tawdry needle
#

doesn't their "all in one" ide package not actually include all their ide products?

#

i wish i could pay like $50 for the base ide and $25 for each language "layer" that i use

#

the all-in-one package is pretty tempting, it would be easy to pop open any large open source project in any language and get to work, instead of configuring nvim to do a million things

#

plus i've been interested in datagrip and dataspell specifically

soft axle
#

JetBrains doesn't actually have an "everything IDE" yet - however, it's in development

tawdry needle
#

so sick of jupyterlab working in a browser

soft axle
#

There's JetBrains Fleet, which is real exciting

#

You could also turn off the smart language stuff altogether, and have a very lightweight frontend as a Sublime Text equivalent

true vapor
soft axle
#

I'm only just realising... I have no idea how to actually deploy Svelte here 🤦‍♂️

#

Rollup, Vite, Yarn... I have no idea what I'm actually doing lmao

#

I made the foolish assumption it'd require some sort of compilation/preprocessing like how Sass does, then I can just serve the end result

#

except it seems more intricate than that

gentle solstice
#

yarn build and serve everything to dist/index.html

tender current
#

I kinda wish pyright was less verbose when there is nothing to report. Like mypy and pylint do, less is better.

bash: https://github.com/microsoft/pyright/blob/main/docs/command-line.md: No such file or directory
No configuration file found.
pyproject.toml file found at /home/usr/LDS.
Loading pyproject.toml file at /home/usr/LDS/pyproject.toml
Assuming Python version 3.10
Auto-excluding **/node_modules
Auto-excluding **/__pycache__
Auto-excluding **/.*
Searching for source files
Found 11 source files
pyright 1.1.266
0 errors, 0 warnings, 0 informations
Completed in 3.182sec

So much to glance through for nothing

thorny shell
#

I suspect the designers figured it's gonna be run by an IDE, and that output is really just for debugging pyright itself

gentle solstice
#

Does it have a return code when there are errors?

#

Maybe a -q flag?

heavy knot
#

why is git giving me the noop error despite me giving the rebase command a range of commits?
git rebase -i master HEAD~2

tawdry needle
lusty forge
#

Is it possible to set environment variables in GitHub workflows (like running tests on commit) and keep them private so that others can't see em'? Maybe something like yml - name: Test Suite env: BUCKET_ID: ${{ secrets.BUCKET_ID }} KEY_ID: ${{ secrets.KEY_ID }} APP_ID: ${{ secrets.APP_ID }} run: | poetry run pytestAnd then be able to use os.getenviron['BUCKET_ID']

rapid sparrow
# lusty forge Is it possible to set environment variables in GitHub workflows (like running te...

https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions
you are looking for syntax with:, it makes variable accessable as environment variable
example from docs

on:
  workflow_call:
    secrets:
      access-token:
        description: 'A token passed from the caller workflow'
        required: false

jobs:
  pass-secret-to-action:
    runs-on: ubuntu-latest

    steps:
      - name: Pass the received secret to an action
        uses: ./.github/actions/my-action
        with:
          token: ${{ secrets.access-token }}
#

not sure how they handle security protection there

#

i expect giving access only to staging env variables

#

which leaking will not harm anything

lusty forge
#

Oh, but where does access-token get pulled from?

#

Is it just a secret key defined in the repo settings defined under the secrets tab?

rapid sparrow
#

so you could define for multiple repositories

lusty forge
#

gotcha, and access-token has to match the name of one of those secrets?

rapid sparrow
lusty forge
rapid sparrow
lusty forge
#

a step cannot have both the `uses` and `run` keys

#

Looks like my Git commit history is about to get destroyed

tropic olive
#

im running an imap server in docker, for some reason, everytime i start it it receives a SIGTERM and exits

warm tundra
#

yo

gentle solstice
tropic olive
# gentle solstice Try starting with `docker run -ti <image>` 
[   INF   ]  Welcome to docker-mailserver 11.1.0
[   INF   ]  Initializing setup
[   INF   ]  Checking configuration
[  ERROR  ]  Setting hostname/domainname is required
[  ERROR  ]  Shutting down
2022-08-17 16:49:36,515 WARN received SIGTERM indicating exit request```
gentle solstice
#

looks like you need to set a hostname/domainname

tropic olive
#

think i forgot something

tropic olive
gentle solstice
#

the github readme suggests using the provided docker-compose file, substituting mail and example.com as needed

tropic olive
#

alr

hybrid bridge
#

Hello there
I want to creat 5 painting windows with a button in each window by kivy library. Then connect windows with screen manager. In other words, when I'm clicking button, new painting window open and so on.
But I can't find any similar code like what I mean.
Do you have any idea that can help me to build such app?
When I put a button in painting window, I can't use on press of button to go to the next window.

wooden ibex
#

Don’t run your own mail server. It’s a ton of work.

stuck saffron
#

Hello folks... If you noticed.. in CI/CD pipelines like GitHub Actions... there is no way to determine.. what network calls were made by code ..
There is tool that does the same.. it monitors all the network calls and you can also set it to block unwanted calls... for security reasons..

https://github.com/step-security/harden-runner

GitHub
GitHub - step-security/harden-runner: 🛡️Security agent for the GitH...

🛡️Security agent for the GitHub-hosted runner to monitor the build process - GitHub - step-security/harden-runner: 🛡️Security agent for the GitHub-hosted runner to monitor the build process

rapid sparrow
obsidian marsh
#

my cibuildwheel is failing with NonPlatformWheelError: Build failed because a pure Python wheel was generated., i tried running pip wheel locally and everything worked just fine. any ideas?

heavy knot
#

Idk why that would change anything since i was just using git for the local repo but I think I fixed it

heavy knot
#

Do github action workflows run on every branch? It seems to be doing it with on: push but I can't tell if it's doing it with on: schedule'ed ones

cerulean bronze
#

Hey! I was deploying my website to heroku server but I'm getting code=H13 desc="Connection closed without response" this error. Can anybody help figure out how do I solve this issue?

tropic olive
#

i forget the exact syntax

cedar wyvern
#

yo anyone have some good resource(s) they can point me towards for atlassian/jira and incorporating github actions specifically to verify/validate deployment versions before allowing a release/push/merge?

tawdry needle
#

(i don't have good resources on it)

cedar wyvern
rapid sparrow
#

Books Docker deep dive

#

practical ansible 2

#

Terraform up and running

#

Well. That is if u a in web dev

#

For desktop/mobile dev, it would be different

cedar wyvern
#

sweet! thanks @rapid sparrow! That will work for what I need!

cedar wyvern
rugged depot
#

hey, is it possible to somehow import a commit from another repo in gitlab?

gentle solstice
#

add it as a remote, fetch, then cherry-pick it

rapid sparrow
heavy knot
#

anyone know what package to install to read windows system notifications? It’s not toast as that is used to create notifications, I want to read the windows notifications

rapid sparrow
#

Amount of functionality it provides in brain friendly way allow to have it scored as 10/10

While github actions can barely be 3/10

#

Gitlab CI has more human readable syntax, easy to remember

#

And it gives more than shell executor for self hosted runner

#

Gitlab CI allows to run jobs of self hosted runner in docker images and even kubernetes pods

cedar wyvern
shy yoke
#

yo, I have a quick question. A couple of my CI jobs have some of the exact same steps. I was wondering how I could easily eliminate the duplication and find a way to run those steps after checking out the repository for example (it doesn’t have to be after checking out the repository, but I hope you get the idea)

rapid sparrow
#

Also u can write Python glue scripts

wooden ibex
spare coral
#

I am looking for an advanced python and sockets programmer

thorny shell
#

where did you last see them?

#

what do they look like?

spare coral
#

anyone interested for a socket job?

#

what is the channel for post Jobs?

thorny shell
#

I don't think there is one

#

hence my snarky answer

spare coral
#

xd

thorny shell
#

recruiting seems to be frowned upon here

spare coral
#

oh

thorny shell
#

Rule 9
Do not offer or ask for paid work of any kind.

#

dunno why, but ... there it is

wooden ibex
shy yoke
shy yoke
wooden ibex
shy yoke
#

I’m relatively new to devops so enlighten me

thorny shell
#

If I may put words in @wooden ibex 's mouth ... I assume he/she is a pro, and pros likely use bespoke CI/CD systems like ansible, puppet, chef, salt, &c; which in effect have their own high-level programming languages.

#

writing CI/CD stuff in shell or python is vaguely like trying to write an app in assembly language: it's doable in theory, but it's so hard you'll either do a terrible job of it, or else you'll just give up.

#

So: I imagine Rabbit wants you to use the right tool for the job.

shy yoke
#

oh ok

#

that makes sense, yeah

#

I use GitHub actions. That is a CI/CD system right?

#

what would I use for that

thorny shell
#

Otoh, if what you're doing is simple, and only for fun, then maybe you should use shell or python:

  • it might do the job just fine;
  • if it turns out to be a horrible mistake, then you'll be well motivated to investigate something better 🙂
thorny shell
#

disclaimer: have never used github actions, nor even read about them

#

I'm just guessing what they are and how they work

shy yoke
thorny shell
#

beats me

#

I guess

shy yoke
#

ok, I will try to find a few things. also will wait for Rabbit’s response. Thank you

ornate surge
wooden ibex
graceful light
#

So for personal use I've been using GitHub Actions for automating building & releasing code, but pretty much every place I've worked at or applied to say they that they don't use that in their GitHub Enterprise stuff and instead use Jenkins
Is there something else besides Jenkins that big companies use for that kind of stuff? Not really a fan of making Jenkins pipelines in Groovy

mystic void
#

I would almost say that jenkins is legacy nowadays

graceful light
#

Sounds about right, places I've seen using Jenkins are Cigna and Cisco which are both using tech from at least 10 years ago
I know GitLab CI in popular as well but I just don't use GitLab enough personally to learn their CI stuff

mystic void
#

it all works on the same ideas, just different skin on it

graceful light
#

I interviewed with some startup that actually used Ansible for their CI which I thought was both interesting and weird

mystic void
#

that would definitely fall into the very creative category

rapid sparrow
rapid sparrow
#

Gitlab CI has docker executor for its jobs, and even kubernetes executor. official instructions how to install both

graceful light
#

I really should look into using GitLab instead of GitHub but I just can't bring myself to spend the time to migrate there, or to have mirrored hosting
And yeah all of it is hosted in house (at least where I work in Cigna) but that's a double-edged sword because the hardware they use for the pipelines is terribly slow

#

And prone to sooooo many issues

#

I understand that paying for a GH Enterprise license with CI capabilities would be expensive but also it would just run better

rapid sparrow
#

in gitlab there is a choice though: you can pay for enterprise, or just host entire instance of gitlab fully

#

or you can host only its self hosted runners

mystic void
#

Which is also interesting in itself because now, your choice of CI are also impacted by your choise of (d)vcs hosting

#

plus whatever other tools/bots that come on top of it

graceful light
#

This is more of a theoretical question on what I should focus on, I unfortunately do not have any say in what gets used at my work
If I had it my way we'd be using plenty of newer technologies

rapid sparrow
mystic void
#

What artefacts are you producing? How are you validating your builds? CI/CD? Blue/Green? Feature flags? Registries? How you integrate with the deployment?

#

beyond that, a shell script in gha will look like a shell script in gitlab ci which will also look like a shell script in jenkins

graceful light
rapid sparrow
rapid sparrow
#

and for stuff it is depenended, it is depended preferably on really good open source, instead of propietary stuff. ;b

graceful light
# mystic void What artefacts are you producing? How are you validating your builds? CI/CD? Blu...

I think I mentioned it to you before, but in the CI/CD setup for my personal projects, my pipeline goes like this:

  1. Users would have pre-commit installed to run black, isort, and flake8 on the code before allowing the commit to go through
  2. Use vermin to report the highest version of python that the code can run on, making sure that it doesn't break compatibility with older python versions unless specifically allowed
  3. Have the GH Actions script to set up environment for OS' Windows, Ubuntu, and MacOS, and for it to target the python runtime deduced from vermin
  4. Set up poetry in the GH Actions script to handle dependency management
    5A. Check if the project uses some GUI library (PyQt, PySimpleGUI, cli2gui, gooey, etc) and install the required files for them
    5B. Check if the project utilizes cython, nuitka, numba, pyinstaller, etc. and install the required files for them, and run their compilation / build / packaging as described in the relevant config file(s)
    5C. If the project meant to be configured into a pypi package, then use poetry to build and deploy it
  5. Create a release on GitHub with the messages from the commits / PR

It's still a heavy WIP because it's so big and varied but my end goal is to get that all working, and add support for the Pypy runtime

mystic void
graceful light
#

I guess I can go a step further and add unit tests and stuff like selenium for web-based projects
And as you guys said, I'll look into making a version of this for GitLab's CI and others once it's complete for GH

mystic void
graceful light
#

True
Maybe I'll make it universal with a dockerfile, although I guess that'd be limited in terms of what OS I could have docker images for

mystic void
#

That's the CD part of CI/CD 😉

subtle quarry
balmy veldt
#

Anyone suggest me the path to learn devops

rapid sparrow
rapid sparrow
#

Starting from here

914-sUgELZL.jpg
#

I always hear recommendation to read also
Data intensive applications book, but my hands did not reach it 🤔

rapid sparrow
wooden ibex
wooden ibex
#

Unless you see DevOps as Build Engineer in which case, learn YAML

heavy knot
#

.

thorny shell
#

!

balmy veldt
balmy veldt
balmy veldt
wooden ibex
empty radish
#

I need to download and store millions of images to a s3 bucket. My thought is I'd spin up a vm then upload using the aws api. Is there a better way? For instance passing the list of uris to a lambda instead?

thorny shell
#

a pile of lambdas, yes

#

parallelizing that work seems useful

#

off the top of my head: create an SQS queue, put each URL into a separate message on that queue, spin up a pile of lambdas that consume from that queue.

empty radish
thorny shell
#

so?

#

read 'em outta the sqlitedb one at a time, enque 'em one at a time

#

at this scale you might want to think about using nosql (dynamodb) instead of sqlite

empty radish
# thorny shell so?

Why would I use SQS to register components if I have the "messages" already.

#

read from db -> download -> convert to webp -> store to service (currently google drive, switching to S3)

#

After reading the API sqs isn't the solution here, great concept for consumable messages but not practical for this.

thorny shell
#

well it sounds like you're doing everything in serial

#

SQS+lambda sounds like an easy way to get some parallelization

empty radish
#

I am spinning up threads, but I agree, was looking for a good way, your suggestion is good, just not the best giving my current setup.

#

I did this with google collab to save on networking calls.

balmy veldt
#

Anyone suggest me a good resource for linux

thorny shell
brisk knot
#

Can you make pydoc ignore import errors from unknown libraries when making a doc?

raw crescent
#

Any recs on converting a py to exe for Windows?

empty radish
#

Thoughts on threading a update to sqlite?

empty radish
vague silo
#

still a fun book

loud jewel
stable coral
#

Hey folks, what would you recommend for a desktop app monitoring? Data are relatively sensitive so I'd like to stick to monitoring locally, in development (no datadog and friends). Log feels messy and I'd like to limit logging to error/warning display as much as possible.
My current idea is to plug an SQLite database to the app and store data in an organized fashion. For instance store time taken by an operation for a given git commit/release version. The output being a file, it's easy to back them up, I could even include it in the repo if it doesn't become to big. Then I can create a few queries to have a basic analytics tool.

#

what do you think of this setup? Maybe I am reinventing the wheel?

#

I am also thinking of generating CSV files directly, but what I don't like is that spreadsheet software might not let me have "clean" queries backup up (in Excel the processing is very tied to the actual file you use as far as I can tell, so it can make it hard to backup/delete/recreate the file without losing the analytics)

#

alternative could be CSV + a python script to agregate data

stable coral
#

For logs, I might setup a basic debug UI to handle the flags and logs only certain workflows, because it's not realistic to handle them by command line (need a server restart all the time, lot of typings, no way to tell the available flags etc.)

loud jewel
#

I commited crime

loud jewel
#

@stable coral I didn't even understand 99% of what you typed up here.

waxen gull
#

I want to get a MacBookPro. My current Windows 10 laptop can't upgrade to Windows 11 because of that stupid CPU Processor requirement.

#

I mean sure I could work around it but I'll probably lose support if something happens

#

I want to get a MBP because I want to go full stack cross platform dev.

#

Can't build for iOS if I'm on Windows or Linux

#

Should I get a MBP or is there a cheaper alternative that I should consider before buying one?

#

Like maybe use Github Actions or Jenkins (I've no experience) to build iOS apps?

loud jewel
#

Oh you want to upgrade to windows 11? It even changes anything?

thorny shell
#

fwiw, some years ago when I bought my first Mac, I went to the Apple Store and told the salesman "Sir, I am rich; I demand your best laptop!" He gently talked me down to a cheaper one [his argument was: the top-of-the-line is for heavy video editing and gaming, and I'd told him I wasn't gonna do any of that]

waxen gull
thorny shell
#

yep, and if you hate it, you can resell it and get back most of your money

iron basalt
#

I'm not sure about the development experience but GH Actions does have macOS VMs according to this

#

So building in CI shouldn't be a problem

calm ginkgo
#

Hey fellas - for those who have been using VSCode or PyCharm, what would you say is easier to use if you want to program Django Applications ? Is it worth it buying Pycharm subscription compared to VSCode?

true vapor
# calm ginkgo Hey fellas - for those who have been using VSCode or PyCharm, what would you say...

I use PyCharm pro, but the Django integration is a bit overrated. The main thing it's useful for is the integration with templates. Bullying PyCharm to actually turn on that integration can sometimes be a bit annoying.

If you can get it for free via a student license, or if you are already used to Jetbrains keybindings, then I'd recommend at least the 30 day trial - but VSCode is completely fine with pylance etc

#

I've used Django+Pycharm, and VSCode+Python, but not VSCode+Django - not sure if there's some killer extension, but I doubt it

calm ginkgo
#

I'm trying to relearn Python, knowing most of the syntax etc, but the environment, adding modules etc is still a bit new to me

#

I'd start with VSCode + Python and then when I found a job I'm going into Django+Pycharm.

true vapor
calm ginkgo
#

I do love me some magic green buttons but also learning new stuff.

thorny shell
#

Speaking as a caveman, I'm very wary of magic buttons.

#

They prevent you from understanding what's going on.

calm ginkgo
#

Having worked as a programming teacher, I understand your concern

#

Monke see button, monke click

#

Even with a tooltip to explain - it doesn't help

true vapor
# thorny shell Speaking as a caveman, I'm very wary of magic buttons.

I find the magic buttons quite helpful at this point, but I think there's a good chance they were detrimental back when I was first learning.

At this point I know enough that I just know how to do everything the buttons are doing manually, but I definitely didn't when I started out learning

calm ginkgo
#

monke like button

stable coral
#

in the web we would use loads of 3rd party tooling (Sentry, GraphQL specific solutions, Datadog etc.) but I am looking for more local alternatives

wooden ibex
#

but if you want something cheaper, you can try Mac Mini. Note, M1/M2 chips only support a single monitor which might be a good programming blocker

tawdry needle
#

well 2 + the builtin one

#

one over hdmi and one over usb-c

#

maybe they only support a single monitor over usb-c, i wasn't sure if that was the case or if the monitor combiner box thing i had just didn't work right

wooden ibex
tawdry needle
wooden ibex
#

Yep, Pro is 2

tawdry needle
#

oh i see

#

it says "Apple M1 Pro"

#

so that supports 2 external monitors whereas the other ones support 1?

wooden ibex
#

M1 Max is like 3-4

waxen gull
graceful light
#

My new job is letting me choose between some different laptops, the only specs they list are the OS (MacOS or Windows) and the screen size
I have an M1 Mac Mini for personal use, but as much as I hear that Mac's are the favored development machines, idk if I'm comfortable using them over Windows
Are Mac's really all they're cracked up to be? In my case I'd be doing Python development

rapid sparrow
# graceful light My new job is letting me choose between some different laptops, the only specs t...

I think they are just overrated shit. In reality apple is just a cult. https://www.businessinsider.com/nyu-professor-says-apple-is-a-cult-2015-9#:~:text=Robles-Anderson says that%2C despite,than its iPhone release days.

Nothing is more perfect for development than Linux OS. a whole OS is the most developer friendly ecosystem. Considering that also all servers run on Linux, you have minimal difference between platform target and developer machine. At least if your product is meant to be run on Linux servers./

#

All dev tools are having first support of Linux

#

the only exception when you develop for Windows/MacOs/Ios as platform target. In this case you should consider choosing Windows/MacOs
It would make easier life testing stuff

wooden ibex
wooden ibex
rapid sparrow
thorny shell
ornate surge
#

While I think Apple is a terrible company to consumers, in terms of power efficiency they can't be beat right now

flint mango
wooden ibex
flint mango
#

Oh I didn't see that you replied to someone, sorry.

true vapor
#

Macs have the advantage of working out of the box. No fucking around with network drivers, with getting the laptop to sleep properly, with not being able to use outlook without Wine. The hardware just works nicely with the software in a way that you don't get on Linux

thorny shell
#

that is exactly why I swtiched

#

got fed up with Linux never working

rapid sparrow
#

well. in the world of web development it is kind of opposite.

#

with linux everything works out of the box

#

in other OSes you would have to go through all the pains of getting some compatibility to use Linux tools (say hello to using VMs/WSLs)

#

also that's why i chose Lenovo for laptop. It works out of the box, no problems with network drivers 🙂

#

bad laptops do not really work out of the box for anything pithink installing Windows to HP laptop is going to be painful too. Nothing will work xD including attempt to install drivers. They will be rejected until you get correct month of release of your windows.

true vapor
rapid sparrow
true vapor
rapid sparrow
#

which is proprietary expensive system

#

it would be just stupid

#

obviously all tools are meant for linux first

true vapor
rapid sparrow
#

because with linux = they can be easily put into CI CD

#

and just chained in whatever wish you want

#

or launched in some stuff like Fragate if wished, whatever.
the first support to launch something or run for linux in web dev.

wooden ibex
gentle solstice
gentle solstice
clever jolt
#

I am guessing this has not improved since the posted date of the article.

rapid sparrow
finite fulcrum
#

I've deployed a simple django app with nginx, all in a single docker-compose but then I realized I also want to run znc behind nginx and the znc is ran on the host. Should I move the nginx that's currently managing the django server to the host? Don't really know much about what I'm doing, but I'm not sure whether that'd be the best course of action, and if it is how to get it to see django's static files

rapid sparrow
finite fulcrum
#

That seemed like it td be more trouble than worth it when the only additional docker setup there is a port, and I need to configure it through its config directory

#

Or is there some way to get a volume (or whatever) that can be read/written by both docker and the host

brazen forge
#

you can mount a host directory/file to a directory/file in the container

#

a bind mount to be specific

#

it's a two-way binding by default

finite fulcrum
#

I'll have to look into that later; the things I tried yesterday weren't quite working

brazen forge
# 
...
  znc:
    ...
    volumes:
      - /etc/config/znc:/etc/config/znc
#

something like that

#

with the proper paths

finite fulcrum
#

Would I just throw the znc service into the docker compose override I have for the web server or would there be a better way to do that while routing things through nginx?

#

I guess a custom network?

brazen forge
#

I've only used docker compose overrides for different environments and not services, so can't tell.

#

also, you can serve the host ZNC service behind an NGINX container, provided the service can listen in on just the localhost and docker interface.

#

though that's a bit complicated than having it containerized

paper pagoda
#

is there a tool to resolve import statements to system path?

#

for example:

import importutil

print(importutil.resolve("mymodule.submodule"))```
#

and then it would give like "/path/to/mymodule/submodule.py"

thorny shell
#

maybe just look the module's name up in sys.modules

# 
>>> import requests
>>> print(sys.modules["requests"])
<module 'requests' from '/opt/homebrew/lib/python3.10/site-packages/requests/__init__.py'>
>>>
#

or ```py

print(sys.modules["requests"].file)
/opt/homebrew/lib/python3.10/site-packages/requests/init.py

paper pagoda
thorny shell
#

ah.

finite fulcrum
#

importlib.machinery.PathFinder.find_spec seems to work

In [49]: Path("python_package").mkdir()

In [50]: Path("python_package/__init__.py").touch()

In [51]: importlib.machinery.PathFinder().find_spec("python_package")
Out[51]: ModuleSpec(name='python_package', loader=<_frozen_importlib_external.SourceFileLoader object at 0x00000140834AF430>, origin='.\\python_package\\__init__.py', submodule_search_locations=['.\python_package'])

In [52]: "python_package" in sys.modules
Out[52]: False
paper pagoda
green lagoon
#

So i have a problem with git

#

i have these files

unknown.png
#

and when i do

# 
git add .
git commit -am "Fix"
#

the new edit appears

#

but the items file gets reset back to its previous state

#

and does not save its previous data

#

which basically makes it empty again

unknown.png
#

how to i do git add but only add the changes from main.py

#

and completely ignore the items file

tame dagger
#

git add . will add everything in the current directory, if you only want to add main.py you can do git add main.py

green lagoon
#

ive tried that it does not work

#

Basically

#

and i want anything in items to save

#

i never want items to back to its previous state

#

items will never be edited

shy yoke
#

if you don't want to track it, just use .gitignore?

green lagoon
shy yoke
#

.gitignore is a file

#

where you put patterns and/or names for git to ignire

green lagoon
#

and i want to put it in my Git

#

without changing any data in items

#

what would i type in the command prompt

tame dagger
#

there is no way any combination of git add and git commit will change items if all you have done is change main.py

#

if items is being changed, it's because you ran something that changed it

green lagoon
tame dagger
green lagoon
#

in the main.py code it edits and changes items

#

adding and removing data

#

but in the beginning items is nothing

tame dagger
#

ok, you probably just want to add items to .gitignore then

shy yoke
#

if items file isn't edited, it won't come up as a change if you are tracking it. if items file isn't tracked, it will never be part of the changes

green lagoon
shy yoke
#

make a .gitignore file and put items in it

brazen forge
#

make a file with that name

#

if it's already tracked, untrack it using git rm --cached items

green lagoon
#

like this?

unknown.png
shy yoke
#

no

#

a file named .gitignore

green lagoon
#

how do add items into it

shy yoke
#

type items inside the file

green lagoon
#

ok ive done that if i now do

#

git add .

#

then git commit

#

it will all be fine?

tame dagger
#

that should work

shy yoke
green lagoon
#

Do i need to put anything in command prompt

tame dagger
#

I find it useful to use git add -p, which shows you all the changes one by one for individual confirmation

#

less chance of accidentally committing stuff that way

green lagoon
#

so ive done that

unknown.png
#

items in the beginning is just {} and in main.py it edits it

#

so will it accept changes by main.py

#

also the gitignore does not work

tame dagger
#

yes, the file will still be changed when the script changes it...

green lagoon
#

what ive done is

#

im basically connecting this git to heroku

#

and type the commands

# 
git add .
git commit -am "Fixed"
git push heroku master
#

it updates the change i added to main.py

#

but items gets reseteed

#

All i want it do is not to touch items

tame dagger
#

none of these commands is editing items

green lagoon
#

During the code

tame dagger
#

right

green lagoon
#

Then I off the code and edit main.py

brazen forge
#

when you push to heroku, it restarts the app with the new changes. this means starting from a fresh slate. so items will be deleted as well.

green lagoon
shy yoke
#

you don't. heroku has an ephemeral file system

#

you need to store stuff in other places like an s3 bucket if you dont' want to lose everything

brazen forge
#

if you want the changes you've made to items (via main.py on heroku) store it somewhere other than Heroku. like, an external DB

green lagoon
#

But how do I connect it to main.py if it is not with main:py in Jerome

#

Heroku

brazen forge
#

there are libraries to interact with S3 (or whatever storage you choose)

green lagoon
#

O bruh so I can’t use json anymore

brazen forge
#

you can still use JSON. but it'll be stored somewhere other than with your code

green lagoon
#

Can I still do with open(file name) in the code

#

So with open(items)

brazen forge
#

probably not exactly that. but similar

green lagoon
#

Where can I learn how to do this

tame dagger
#

I'm confused about what the original problem was now, because it sounded like you didn't want git to pick up changes to items and now it sounds like you do want git to pick up changes to items

green lagoon
brazen forge
#

they want to persist the changes that happen on the hosted version

#

not the local version

tame dagger
#

ohh

green lagoon
#

So how would I do it these stuff seem way to complicated to me

brazen forge
#

.rp aws s3

signal cloakBOT
#
Search results - Real Python

Here are the top 5 results:

Python, Boto3, and AWS S3: Demystified

https://realpython.com/python-boto3-aws-s3/

Migrating your Django Project to Heroku

https://realpython.com/migrating-your-django-project-to-heroku/

Deploying a Django App to AWS Elastic Beanstalk

https://realpython.com/deploying-a-django-app-to-aws-elastic-beanstalk/

Deploying Django + Python 3 + PostgreSQL to AWS Elastic Beanstalk

https://realpython.com/deploying-a-django-app-and-postgresql-to-aws-elastic-beanstalk/

First Steps With PySpark and Big Data Processing

https://realpython.com/pyspark-intro/

Click the links to go to the articles.
brazen forge
#

check the first link

green lagoon
#

I don’t understand anything in that :(( I feel hopeless

green lagoon
#

Like a duplicate

#

If it was to edit it

#

Right ?

#

Can’t I just find that file copy and paste it into the main items and do git add

#

It is longer but easier for me

brazen forge
#

on the version hosted on Heroku, you can't rely on those changes to be persisted

green lagoon
#

Wdym?

brazen forge
#

but sure, there is a roundabout way to do this

green lagoon
#

So how do I find this version

brazen forge
#

before you push to Heroku, take the content of the items file from the hosted version and paste it in the local copy

green lagoon
#

How do I do that

brazen forge
#

lemme check

green lagoon
#

Ty

#

I have to go so can you please dm me the steps Adios

brazen forge
#

Heroku file system is ephemeral: applications can write on the file system but any change is discarded when the Dyno (the application host) restarts, making this option only suitable for temporary data.

It is also important to remember that Heroku implements cycling: every Dyno reboots (at least) every 24 hours and all local filesystem modifications are deleted.

#

well, nevermind. it'd be tedious

green lagoon
#

So how do I find this version

#

Or data

brazen forge
#

the thing is you can't reliably tell when the dyno will restart

green lagoon
#

So what do you recommending do

#

It will be hard to change databases

brazen forge
#

learn how to use databases and/or cloud storage like S3

#

I'd also recommend switching from Heroku

green lagoon
#

:(

#

To what

brazen forge
#

since they're discontinuing free plans

#

PythonAnywhere, DigitalOcean, etc.

green lagoon
#

:(

#

How thou

thorny shell
#

welcome to cloud comupting

green lagoon
#

Bruh

brazen forge
#

go to their site, they have Get Started pages

green lagoon
#

I need to change 5 different projects now

#

That are heavy around json

brazen forge
#

using JSON as a database is also not recommended

#

learn to use an actual DB

green lagoon
#

It’s only for a small group of people

brazen forge
#

still a valuable skill to have

green lagoon
#

If I switch from verily

#

Heroku

#

Can I still use JSON and will the problem be fixed

#

Also Wdym heroku is switching plans

brazen forge
brazen forge
green lagoon
#

So what software should I go to

#

To host

brazen forge
#

¯_(ツ)_/¯

green lagoon
#

:(

#

Have you ever hosted 24/7 if so what did you use it will help

thorny shell
#

if you don't mind spending ~US$30/month, you can get an EC2 instance or an azure equivalent

#

been up for 7 years, either continuously, or if it did crash, it was entirely my fault, not AWS' 🙂

brazen forge
#

I don't host anything currently, but I prefer local hosting services

green lagoon
thorny shell
#

makes it tough

#

weird that there's no easy way for a small group of people to share hosting costs

green lagoon
thorny shell
#

there are tons of sources for 24/7 hosting, but I am guessing you want free hosting

#

that makes it a lot harder

green lagoon
#

I need a cheap reliable one

#

Since ye

thorny shell
#

🤷

open forge
#

How do I publish a release as a pre-release? I've done everything I can with the version identifier, but nothing's marking as pre-release

#

1.4.3-beta-1 is supposed to mark as pre according to pep 440 (unless i misunderstood) but it just hasn't

brazen forge
#

!pep 440

rancid schoonerBOT
#
**PEP 440 - Version Identification and Dependency Specification**

Link

Status

Final

Created

18-Mar-2013

Type

Standards Track

velvet spire
#

!pypi pep440

#

what

#

!pypi pep440

rancid schoonerBOT
#
pep440 v0.1.1

A simple package with utils to check whether versions number match PEP 440.

brazen forge
rancid schoonerBOT
#
pep440 v0.1.1

A simple package with utils to check whether versions number match PEP 440.

green lagoon
#

So I guess I can’t do anything I’ll just stop doing python hosting

green lagoon
#

Is this for 1 app only

thorny shell
#

no idea

green lagoon
#

My friends paid me to make a discord bot for them 24/7

#

Idk what to do if 24/7 stops

#

On heroku

thorny shell
#

give your friends their money back, or else move it someplace else that can stay up 24/7

#

if it's just a discord bot, it should be feasible to set it up on a cheap computer in your house

green lagoon
#

Is your hosting 30$ for one app ?

green lagoon
thorny shell
#

it's $30 for one EC2 instance, which I currently only use to host a single app, but I could host as many as I wanted (until they all ran out of ram or CPU or whatever)

open forge
green lagoon
#

I’m just a beginner coder using heroku as my hosting service

thorny shell
#

and I think there's one plan that's cheaper

green lagoon
#

Why does heroku have to do this

#

Lots of people will stop using heroku when they make it payment

#

Since there are other better payment softwares

#

I used to use replit

thorny shell
#

amazon lightsail claims to be cheap

unknown.png
green lagoon
#

But for replit it constantly makes discord bot go down

green lagoon
#

They are relatively small files

thorny shell
#

dunno really but more than one, I'd think

#

IME discord bots don't do much of anything

green lagoon
thorny shell
#

"in my experience"

#

I'd try it. AWS are cool about letting you cancel your subscription

#

[their billing is incomprehensible, but they're pretty cheap so it doesn't much matter]

green lagoon
#

Is it similar to setup like heroku

green lagoon
thorny shell
#

likely, but you'll never know until you try.

#

it will cost you $3 to try, if that web site is to be believed.

green lagoon
thorny shell
#

well, I doubt you'll find anything cheaper than $3/month

green lagoon
#

my pc is a literal toaster

thorny shell
#

find some friends and pool your money

green lagoon
#

i have no friends

#

the people i made the bots for gave me some ingame currency worth a lot

#

and i feel like i cant scam them

#

when i dont have those items anymore

green lagoon
thorny shell
#

huh, that's twice the lightsail price, but ... whatever

green lagoon
#

ye but im used to heroku

#

i want to get hobby

unknown.png
last nest
#

Hello guys, question related to database data. We have hundred million records on 3 tables. We wanted to delete the older data but we also wanted to safely store it somewhere where we could query some useful information in the future. Any idea what is the right approach to achieve this?

rapid sparrow
# last nest Hello guys, question related to database data. We have hundred million records o...

https://www.digitalocean.com/community/tutorials/understanding-database-sharding
Consider reading about database sharding. The point in already having patterns/related technologies/algorithms existing for that, splitting database to multiple ones. Engine querying support for situation of querying across multiple databases

Understanding Database Sharding | DigitalOcean

Sharded databases have been receiving lots of attention in recent years, but many don’t have a clear understanding of what they are or the scenarios in which…

mystic void
tawdry needle
patent sparrow
#

I made my own Csv library for python
I know that there is a library for csv aldready, but i tried to make a better one. Pls try it and post bugs on the bug report!

My Module: https://pypi.org/project/ShanCsv/

Bug Reports: https://github.com/Ninjago77/ShanCsv/issues

Docs: https://github.com/Ninjago77/ShanCsv#readme

PyPI
ShanCsv

A Python Package for CSV File Parsing and Rendering

GitHub
Issues · Ninjago77/ShanCsv

A Python Package for CSV File Parsing and Rendering - Issues · Ninjago77/ShanCsv

GitHub
GitHub - Ninjago77/ShanCsv: A Python Package for CSV File Parsing a...

A Python Package for CSV File Parsing and Rendering - GitHub - Ninjago77/ShanCsv: A Python Package for CSV File Parsing and Rendering

graceful light
#

Trying to write some local scripts to handle some cython compilation, currently I have a script in bash for Linux & MacOS, and a PowersShell one for Windows
But I want to write a singular script to handle this so I don't have to maintain multiple ones in multiple languages - should I just do this in Python by using the subprocess module to run the individual setup.py's I have or is there a better method to doing this kind of stuff locally?

graceful light
patent sparrow
# graceful light Can you describe why your CSV library is better than the built-in one? Does it h...

Firstly its easy to use and follows the formats of other commonly used libraries:- pickle/json.loads/load/dumps/dump
Automaticly converts data values from/to a 2dimensional list as float, int, bool, none, str unlike the normal library.
Completly customizable by giving the specified kwargs to the Csv Class directly or throught the other func indirectly
Much more Simple and readable docs... uhhhhh idk what to tell more @graceful light

graceful light
#

The README would need to be updated a lot for you to say that's it's much more readable, but definitely mention those other things as a selling point for your library

patent sparrow
#

hmm.. nice idea but i don't know how to put this explination simple to the readers in the description

patent sparrow
graceful light
#

I mean you just did that for me, put it into more detail / a bulleted list in your README or make a proper docs page with Sphinx / ReadTheDocs

patent sparrow
patent sparrow
graceful light
#

Again, probably best to list the good stuff in bullet points, not a paragraph
Also you don't have to tag me each time

loud bear
#

i cannot override mkdocstring template style.css

tawdry needle
#

why are you parsing csv manually instead of using the csv module?

#

and why are you doing the parsing in __init__? i strongly do not like libraries that do nontrivial calculations in __init__, it seems like you really just want a function for that

patent sparrow
patent sparrow
tawdry needle
#

like how json lets you specify custom encoders and decoders

#

e.g. a lot of people would be happy with data = myscv.load(fp) returning a list of lists or a list of dicts

patent sparrow
tawdry needle
# 
# default, list of dicts: [{"a": 1, "b": 2}, ...]
data = mycsv.load(fp, row=dict)

# list of tuples with column names: [(("a", 1), ("b", 2)), ...]
data = myscv.load(fp, row=tuple)

# list of tuples with no column names: [(1, 2), ...]
data = myscv.load(fp, row=lambda pairs: tuple(value for key, value in pairs))

# same as above, but you could provide some tidy "combinators" for constructing row factories
data = myscv.load(fp, row=mycsv.row_tuple & mycsv.row_values)
tawdry needle
#

you're of course free to design your own library!

#

i'm just describing how i think a lot of users would want to handle something like this in the simplest possible manner

#

this load/dump idea with csv is a great idea. i wish i thought of it first!

patent sparrow
mint parrot
#

I'm having to give myself a crash course in ansible for a work project and I hate this ```yaml
- name: Create the build's log file
ansible.builtin.file:
path: "{{shared_file_path}}/{{ baseline}}{{tstamp}}"
- name: Save log's filepath
ansible.builtin.set_fact:
log_path: "{{shared_file_path}}/{{ baseline}}{{tstamp}}"

pliant isle
#

guys anyone here have ever worked with AWS ECS?

#

and gRPC

rapid sparrow
pliant isle
#

hey dark, we talk about gRPC some time ago kkk

#

now i'm trying to follow this tutorial
https://aws.amazon.com/blogs/aws/new-application-load-balancer-support-for-end-to-end-http-2-and-grpc/
to connect my containers via a ALB from AWS

Amazon Web Services
New – Application Load Balancer Support for End-to-End HTTP/2 and g...

Thanks to its efficiency and support for numerous programming languages, gRPC is a popular choice for microservice integrations and client-server communications. gRPC is a high performance remote procedure call (RPC) framework using HTTP/2 for transport and Protocol Buffers to describe the interface. To make it easier to use gRPC with your appli...

#

i can get all things working, but i can't call the ALB api or DNS or anything

#

i have no idea if i'm calling it right, if i block me in some config

#

and i know what i'm doing its trivial and common, so i'm trying to found someone that could help me in this

rapid sparrow
pliant isle
#

everything looks fine

#

i just can't call the ALB

#

i have no ideia why, he is on, have 2 ips in diferent zones

#

i try the DNS

#

the DNS from the network interfaces and nothing

#

@rapid sparrow could u help me?

#

i know it's probably a simple issue

rapid sparrow
rapid sparrow
#

try to have it working with simple example in http first

#

then just switch to grcp

#

if you don't know any infra provisioning code tools, recomendation to learn, this book can get you started

unknown.png
pliant isle
#

;-;

#

man i just want to do something simple and fast

rapid sparrow
#

welp. i believe simple and fast includes infra code. clickops is not simple and fast in my opinion, when we deal with big three providers xD

#

they have too much overbloated interface. easy to break legs.

pliant isle
#

if i send my ips or dns could u help me to test?

rapid sparrow
#

am i what, manual QA? get yourself postman or something, or telnet to check port

pliant isle
#

i know is better code everything, but i just want test things u know

#

i have nothing ready to production

#

i really just want to do a simple call kkk and see it working, so i can switch all the code from PHP to PY

pliant isle
#

i just need a little bit of your expirence

rapid sparrow
#

i am not hired to spend a lot of time on your problem. debugging things like that is kind of nightmarish.
with as a code infra solutions it is easy. without it, it is not in my opinion. too many settings in big providers to configure them manually

#

use at least this or something book

unknown.png
#

it has recipes for ECS

pliant isle
#

man i just follow the tutorial i send earlier, and it's really simple, i follow every instruction, but what i'm thinking is that i'm doing the calls wrong

#

i'll look this book, ty

#

if u could help me understand this one, i'll be glad, here he says he create A record, but what IP he uses to it? because i have a domain, so i could do it in the google domains right? and after that he says he will use credentials, but what credentials? something about the AWS cert?

unknown.png
#

i know this questions are probably dumb as fuck

#

but now i'm confused as hell kkkk

rapid sparrow
#

i think it is still A record i think? from you rdomain to AWS domain of LB

#

it is kind of easy with HTTP, but kind of get a bit more confusing for gRPC

#

shrugs. i did not use DNS in such situation to know for sure.
possible it can be working more easily when using AWS route53 thingy

#

it should be able resolving AWS internal name things

#

the desired functionality can be missing in regular DNS

pliant isle
#

the IP i should be calling it's supposed to be where?

#

in network interfaces?

#

because the ALB it self have no IP

rapid sparrow
#

networking knowledge is required to understand it fully

rapid sparrow
#

they have in one of code example, networking when

#

LB has public IP

#

and then it is rerouted into private network

#

and communicates in it to services within

#

yeah, here you can see how they configured with nat_gateway/network stuff to make it happen

rancid schoonerBOT
#

modules/alb/main.tf line 22

subnets         = var.public_subnet_ids```
rapid sparrow
#

which leads to id obtaining public IP/exposure

#

private subnet
then public subnet
nat_gateway for public subnet
aws route for public and private
hmm.

#

i don't know networking enough to understand how was made rerouting from public to private xD

#

oh wait. it can be seen in code

#

public Nat is defined with public subnet

#

and in AWS route of private subnet, it is declared as nat gateway of it

#

this connects them

rapid sparrow
pliant isle
#

ok some news @rapid sparrow

#

i found the problem, i have no inbound rule in the target-group, so now i can call the task IP directly

#

but when i call any IP from the newtwork Interfaces or the ALB DNS i got this error

#

"14 UNAVAILABLE: Trying to connect an http1.x server"

#

i know it responding http1.x and the gRPC accept only http2.x

#

but u know what i could been missing? because my target-group is using gRPC as protocol version

#

i need to create a Client? to return info in Http 1?

#

but if it the way, how would i specify the function or the service i'm calling

mint parrot
#

can someone explain to me what I'm doing wrong here? so I have this copy task ```yaml
- name: Extracting the build scripts from the workspace
ansible.builtin.copy:
src: "{{ build_script_location }}/"
dest: "{{ build_script_dest }}"
mode: '0777'
remote_src: yes

#

with this starting condition :

unknown.png
#

however after running the command I have this:

unknown.png
#

for the life of me I can't figure out why the mode is not being set to 777

rocky yew
#

Hi guys! I'm in a bit of a bind here and I'm looking for pointers as to how to best manage my situation:

So, I have a Flask app with uwsgi and I'm deploying this to Kubernetes with 2 pods. UWSGI configuration is 1 process and 4 threads are we're really expecting something like ~50 requests/min.

Why 1 process? Because we're loading a lot of data in memory (don't ask me why, I know it's wrong and I finally won my battle to introduce a DB) and UWSGI doesn't guarantee sharing of data across all requests - which means if I had two processes - pid x and pid y - if x has all the data loaded and y decided to service the request, it can return the wrong result. With just one process and multiple threads, I can hope to establish the same result every time.

With 2 pods(/replicas) - if say one pod goes down (coz memory issues, maybe), the second pods should be able to service the traffic, right(until pod comes up again and loads the data into its memory)?

^ Does this setup make sense?

mystic void
#

There are a few other things I would look at to maximize availability, but that's the idea

rocky yew
mystic void
# rocky yew what would you suggest?
  • Making sure you have monitoring and alerting in place
  • Setting some (anti) affinity to make sure the pods are on different nodes
  • Making sure the nodes are on different Zones
  • Setting limits/requests on the pods
  • Setting the k8s checks on the pods
rocky yew
mystic void
rocky yew
#

okay, makes sense

mystic void
#

I also assume your pods are setup as a k8s deployment

rocky yew
rocky yew
mystic void
#

If you are worried about scaling and availability, you could also set up autoscaling with some reasonable limits

rocky yew
#

OH sorry - I meant ~50 requests/hour

rocky yew
#

(all requests basically just look up data in the in memory dataframe and return a result)

mystic void
# rocky yew apologies if this is a dumb question - but why are limits on requests a good ide...

I will have to refer specifically to the k8s doc for that.
But for the requests/limits, it uses that to schedule pods on appropriate nodes. Failure to do so will mean that either a new node is started (and cost $$$) or the pod will not be started and remain in a failed state (and no more requests handled).
So requests is the the min amount of resources the machine needs available for the pod and limit is a hard limit that will kill the pod if it tries to go beyond that

rocky yew
#

My p95 is between 2-3 seconds (locally), hopefully more threading can shave off a second ( 🤞 )

mystic void
rocky yew
#

But yes, thanks for validating what I have so far

#

I just hope we move to a database quickly - complexity is greatly reduced

short peak
#

I recommend locust for load testing. Also having built-in k8s healthcheck setup will help cluster to know when pods are ready/alive

rocky yew
#

found another tool called hey-hdr which lets me plot my p50s and p99s nicely as a graph 🙂

tawdry needle
#

funny, i was just thinking about http load testing. i've never done it before, is there a good place to read about it?

#

i can check out both hey and locust

#

i know that you're not supposed to load test all on the same machine, since in-memory networking is pretty different from actually making requests to another machine

brazen forge
mystic void
# brazen forge any thoughts on <https://home.robusta.dev/blog/stop-using-cpu-limits/> (sorry fo...

I believe they misunderstood how kubernetes works, especially their third point (or story). So just a case of someone being wrong on the Internet :p
The requested CPU does not guarantee that this will be enforced. Requested CPU is only used at scheduling time. So that means that, yes, a misbehaving pod can screw up the other workloads running on that node.

To back that up:

The CPU request typically defines a weighting. If several different containers (cgroups) want to run on a contended system, workloads with larger CPU requests are allocated more CPU time than workloads with small requests.

(which is not a guarantee)

The way that worked for us was to set the requests as the expected usage of the resource and limit to be used for the bursts but at a reasonable level.
Unless I recall wrong, the sum of the limit does not have to be less or equal to the physical available resources

heavy knot
#

hi, I have a question regarding dependency management.

In Maven, used a lot in the JVM world, comparable to poetry, there is the notion of a parent projects with a DependencyManagement section that contains the version definitions for 3rd party libraries for the current team-/company-/system-wide tech stack. These definitions get inherited by child projects and applied to any actually referenced dependencies.

So basically, the version of a potential dependency is defined in the parent project. But whether or not a child project actually has the dependency, is determined by the project itself.

Does poetry have a comparable concept or a known approach to centralize defining 3rd-Party-Library versions?

rocky yew
#

familiarize yourself with P50 & p95, p99

#

hey & ab are good tools. I haven't had the chance to use locust tho! If you use hey, check out hey-hdr for plotting your load stats 🙂

heavy knot
brazen forge
tawdry needle
tawdry needle
#

i guess also that i would strongly prefer an open source option that doesn't require web hosting

brazen forge
#

Gatling comes in both forms. OSS and Enterprise

tawdry needle
#

dang there are a lot of http load testing tools out there... i just found one of those "awesome" lists with like 20 tools on it

brazen forge
#

yeah

#

try some out. it's relatively easier to get started with Locust and even Gatling to an extent.

#

k6 is also a good one.

rocky yew
tawdry needle
#

i see, k6 is by grafana

brazen forge
tawdry needle
#

sounds good

wise spoke
#

Help with setting up @rancid schooner using DOCKER

mystic void
rocky yew
#

Has anyone extensively used hey ? I have some questions about the output

tawny temple
#

Not sure if poetry supports constraints files in any way

#

They're basically like requirements.txt files in format, but used with -c instead of -r

rapid sparrow
#

As advantage, poetry, Pipenv builds readable tree in text files, which secondary deps any dep needs, which can be useful to have more data for resolving dep problems

tawny temple
rapid sparrow
tawny temple
#

I use the constraints feature when I use pip-tools and need to have multiple sets of dependencies e.g. one set for testing, one set for linting, one set for doc generation, etc.

rapid sparrow
tawdry needle
tawny temple
elder solstice
#

I am developing a package that supports multiple Python versions. To simplify the testing process, I would like to have a single requirements.txt that specifies a specific version of each dependency, such that it can be installed for each Python version. However, it is quite troublesome to go through each package on PyPI manually and check which version(s) support all of the Python versions allowed for the package. Is there a good way to automate this process?

brazen forge