networks

[wise jungle]

Client sends data to server, server sends data to all clients

[cloud spruce]

if you have enough clients it will take quite a bit of time to send the data to every peer
in that case you might want to check if you are over a specific threshold number of currently connected clients and if you are you could fire off a thread that takes the data and addr as input and sends that data to every client except the client it came from
long term you could even have worker threads in a pool ready to work off a queue, where a worker would take one of the messages and send it to all clients and the next thread in the pool would do the same with the next message in the queue
but i think you can start of and make it work without threads to start with, it will probably be able to handle enough clients to start with, then you can add threading and thread pools and such later on

@wise jungle it's generally not good to start optimizing long before it's even a concern
premature optimization makes development of the software takes much longer and many times totally unnecessarily (a lot of the time it can even kill projects before they are done)
as long as your general design is correct i would suggest optimizing the code when you need it or when you got the core functionality working and not before then
when then is done you can probably do some refactoring of the code and then you can use performance measurements of your code to pinpoint what part of the code needs urgent attention to remove performance bottlenecks

[wise jungle]

im not trying to optimize the code im trying to fix it and make it do what i want

if you try to use the app

you will notice a fat latency

when u use more than 2 main.py

sometimes it even bugs out

and you cant hear anyone

and its just choppy and messy

[cloud spruce]

i understand, but having threading in there right now is probably one of the sources of your problems

[wise jungle]

the most recent code i sent was just to test

but im essentially using what i put up on the repository

[cloud spruce]

it's the code in the repo i'm looking at

[wise jungle]

yea

when you use

serverObj.udp_server.sendto(d, c)

does that block it till its sent or

@cloud spruce

[cloud spruce]

on a blocking socket it could, but i think it will only do that if the kernel transmit buffer for the socket is full already, in all other cases i think it should be quick

[wise jungle]

AttributeError: module 'socket' has no attribute 'SOCK_NONBLOCK'

[wise jungle]

@cloud spruce

[cloud spruce]

that's strange, it works on my linux, but you can use your original line instead and then add another line with udp_server.setblocking(False), the end result should be the same

[wise jungle]

I get the other error as shown earlier

.

[cloud spruce]

hmmm, sounds like it's not supported under windows for udp then 😞

[cloud spruce]

or wait, i think i know what it is 😆

our code is probably just too different, i'm doing something along the lines of (simplified and adopted to your constraints):

import socket
from select import select
udp_server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
udp_server.setblocking(False)
udp_server.bind(('', 5555))
sockets = [udp_server]
while True:
    ready, _, _ = select(sockets, [], [])
    for sock in ready:
        data, peer_addr = sock.recvfrom(4096)
```where the `select()` will be blocking, waiting for any data to arrive on any socket that we are monitoring (right now only one)

[earnest blaze]

The windows IOCP api is light years better than select.
You have no excuse for not using it, aside from the fact the only built-in library that exposes it, is asyncio.
(As the proactor loop)

select is pretty much an artifact of the past at this point.

[dim vigil]

how to join a vpn with python + create a local website on that vpn

so when anyone join that vpn will enther local website's ip

[earnest blaze]

What?🤨

[dim vigil]

yea

[cloud spruce]

sounds like you are talking about a captive portal, but for users of a vpn when they first connect to the vpn and try to access anything after that

[dim vigil]

lol?

no

[cloud spruce]

do you want to try to explain again what you want to do?

[dim vigil]

i want to live camera in local website

and by vpn i can watch that local live camera

[cloud spruce]

so, just be able to access an internal website inside a lan using a vpn from the general internet?

[dim vigil]

not lan

vpn from general internet

like hamachi

[cloud spruce]

i'm not sure if i understand you right
but if it's like hamachi, every user that should participate connects to the vpn and they can reach each others computers and other computers connected to the vpn, right?

[dim vigil]

yes

[cloud spruce]

so, the computer you want to access the website on that has the webcam streaming would also connect to the same vpn as you would be connected to

[cloud spruce]

this is more or less a standard feature a vpn, you just have to make sure clients has the right to access each other through the vpn gateway, client isolation features must be turned off
then there might be a problem if the computer you are trying to access is running a host firewall that would stop this traffic from getting through

[dim vigil]

idk python so can u say a github page this have?

[cloud spruce]

your question doesn't have much to do with python, it's self hosted VPNs in general and configuration of such a VPN

the other part is the web page that you want to display, that can be everything from a basic web server just hosting static pages to a complex web app that can be written in for example python using one of its many excellent frameworks

[wise jungle]

any examples of dat

[wise jungle]

this is what i tried to do and didnt work either

i thought it might be my network thats ass

and hosted it on google cloud and got the same problem

[cloud spruce]

you really need to rearchitecture your code, right now in your server code it looks like you will be spawning one thread per datagram that you receive on the socket

[wise jungle]

huh

are u talking about the github?

[cloud spruce]

yeah, the repository you posted

[wise jungle]

but im not

:?

server creates one thread

and handles all the data sent

[cloud spruce]

i'll take another look at that part and read it more careful then, i kind of skipped over parts of the threading since I think that should just be removed for now

[wise jungle]

the crazy threading shit was just yesterday

self.acceptConnectionT = threading.Thread(target=self.acceptConnection)
        self.acceptConnectionT.daemon = True
        self.acceptConnectionT.start()

the only thread created

so that the server is able to do commands/send messages

[cloud spruce]

what are you planing to do with the import of wave, mix together several peoples audio streams on the server side?

[wise jungle]

nah

wave i was testing with other shit

[cloud spruce]

aha 👍

[wise jungle]

the main things needed are these

import socket
import threading
import time
import traceback

for server.py

but i was just testing listening to incoming data

[mystic onyx]

snyone that knows how to create a software that allows to trace ip's from friends to troll ?

[cloud spruce]

how do you mean with trace?
would you have the ip address of the person you want to "trace" or is that what you want to find out
or is it information like isp, location or even person (which you generally won't be able to) that you want to find?

[mystic onyx]

nah i mean i want to know his ip

[coarse heath]

do i have to port forward if i want to make a server which is accessible from the wider internet

[wise jungle]

yes

[coarse heath]

like say if i wanted a multiplayer game in python which me and my friends could access not just on my LAN

k

can you port forward using python

[prisma cobalt]

no, its a router setting

[cloud spruce]

it all depends on the environment, is it on discord or what are we talking about here?

[mystic onyx]

yup

discord

[prisma cobalt]

if you can get them to run a python script then its possible however this kinda thing sounds sketchy lol

[mystic onyx]

yeeah thats kinda like a token grabber

[coarse heath]

ik but my friends are stupid and dont know how to portforward but still would want to host a server so im trying to automate the procces

[coarse heath]

i know a python script for that if u want but

its a little sketchy

[mystic onyx]

dont care

my friends are stoopid enough

plz send ?

[prisma cobalt]

only one person needs to host the server, the rest of your friends can then connect to that without port forwarding if that helps

[coarse heath]

ik

[prisma cobalt]

what would you do with their ips?? @mystic onyx

[mystic onyx]

mess around

they are my friends

[prisma cobalt]

@coarse heath either your friends port forward when they want to host a server or you host the server permanently on a raspberry pi or something for them to connect to or find a service to host it for you lol (aws or something)

[mystic onyx]

educational purposes only

[prisma cobalt]

what would you be learning 😂

[coarse heath]

ethical hacking

making your friends aware of the risks

[mystic onyx]

yup

[lost dagger]

sock.connect((socket.gethostname(), 1234))

while True:
 msg = sock.recv(1024)
 print(msg.decode("utf-8"))

 if socket.close:
    break
```Is this valid?

[cloud spruce]

sure, it will work just fine
you just have to add the import socket line above that, but i guess you just left out the imports from the code you posted here

[lost dagger]

import time
#AF_INET - IPv4, SOCK_STREAM - TCP
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((socket.gethostname(), 1234))
s.listen(5)

while True:
    clientsocket, address = s.accept()
    print(f"Connection from {address} has been established!")
    clientsocket.send(bytes("hi!!", "utf-8"))

    if s.close:
        print("Connection closed")
        break
    
    
    ```
Server.py.   ```
import socket
import matplotlib.pyplot as plt
import numpy as np
import time
#create a socket object
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((socket.gethostname(), 1234))

while True:
 msg = s.recv(1024)
 print(msg.decode("utf-8"))

client.py. Can someone run this code and tell me why that error is happening. I would explain it, but I can't. The terminal just bugs.

nvm I was looking at the wrong terminal

but

when I run both files

and I look at the client.py file it says connection closed? how come?

help pls ❤️

[earnest blaze]

if s.close:
    print("Connection closed")
    break

Take a moment to let this sink in.

Doesn't anything seem ... off?

[lost dagger]

uhh...

[lost dagger]

I don't see it... 😭

[earnest blaze]

First off, what is s.close?

[lost dagger]

No idea I asked this above:

[lost dagger]

^

he said it works

[earnest blaze]

They didn't pay enough attention.

Now, back to the question

[earnest blaze]

^

[lost dagger]

Not sure. I guess I thought it was to close the cocket

socke

socket

[earnest blaze]

So s.close closes the socket?

Is that what you're saying?

[lost dagger]

Yes I guess so.

[earnest blaze]

But if it performs the operation of closing the socket, why are you treating it like a state indicator, a true or false value?

[lost dagger]

I'm not?

[earnest blaze]

if s.close

You are.

[lost dagger]

Okay wait I'm confused, my thought process was like: if the socket closes then it will say connection closed

I am sorry for the bad questions. I just want to get directed to the right path and I will start research

[earnest blaze]

"If the socket is closed, do something".

But who or what closes it?

This isn't a straightforward issue, there are many tangled problems here.

[earnest blaze]

First off,
socket.close is a method that closes the connection instance when called.
socket.close is just the method object. You are not calling it therefore it is not closing the connection.
In addition, even when called, it does not return true or false, indicating the connectivity state of the socket.
Therefore you can't use it in an if statement.
However since you do anyway, python implicity tries to convert the method object into a boolean value.
Every empty/null or zero value in python is falsy.
Therefore socket.close is truthy, but that doesn't communicate any real information.
Since the if executes anyway despite nothing being correct, the break executes, and by extension the program ends since there's nothing else to do.
When the server process is done, of course the client loses its connection to it.

[lost dagger]

Okay, thank you so much.

But you see what im trying to do right? Any idea how I can do it?

[earnest blaze]

Maybe, there's another issue, that is the socket whose close method you're referencing.

You want to check if the server is closed?

What are you trying to do?

[lost dagger]

When I close my client socket I want the print("Connection closed") to show

[earnest blaze]

When the server or client is closed? Are you sure you're not mixing up the two?

I want to verify that's really what you want.

[lost dagger]

client* mb

[wise jungle]

im still clueless @cloud spruce

[earnest blaze]

Most likely a code 10054 WSAECONNRESET will be raised, if you're on windows.
https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2
But eitherway in python that'll be the ConnectionResetError exception.
You have to wrap all IO performed on the socket in a try/except to catch that exception.
When catched, meaning that client has disconnected, handle it accordingly.

[wise jungle]

quick update still lagigng like crazy (when more than 2 clients start sending data)
https://github.com/e1pupper/VoiceChat

[thorn stratus]

You aren't using multithreading or asyncio, so the clients communicate sequentially

e.g. you have to wait for one client to finish sending or receiving data before moving onto the next one

[wise jungle]

How do i solve that issue

Do i just make multiple threads for each client thats connected

[thorn stratus]

yeah generally

you'll have to rework your code a bit though

[wise jungle]

I tried it and still lagged

X)

[thorn stratus]

I'd need to see your code for that to give you any advice

it also looks like you are recieving from the server socket??

some weird stuff is going on in your code

oh this is UDP, not as familiar with that

multicast might do the trick?

for sending the data back at least

and then you could have the sending run in a seperate thread than the recieving

[cloud spruce]

unfortunately multicast doesn't work out on the broader internet
most ISPs will not support it other then within their own networks, and even then it is often restricted to their own services like IP TV and such

[trail basin]

hey guys. does anyone know how to import QEEG data? where should i start in programming to learn this?

[cloud spruce]

in what form do you have your qEEG data and what do you want to do with it?

[cloud spruce]

and i'm not sure this is the channel for that unless you fetch the data over the network as we are in #networks right now

[trail basin]

it's in the form of mat files

[cloud spruce]

something like this https://mne.tools/stable/index.html ?

[trail basin]

oh i didn't realize that. i only asked here cause i heard python would be the text editor to use for this.

Channels.mat :like this

[cloud spruce]

is this matlab files?

[trail basin]

omg this is super cool thanks!

[trail basin]

i think so. i'm not even sure! i'm so clueless dear god

considering how truly clueless i am , if anyone could recommend any books or websites i'd appreciate it!🌸

[cloud spruce]

if you just want to crunch numbers like https://towardsdatascience.com/how-to-load-matlab-mat-files-in-python-1f200e1287b5?gi=7ed4c6b55244 then #data-science-and-ml might be a more appropriate channel for that (i hope i'm not referring you to the wrong channel now)

[cloud spruce]

it does help if you elaborate what kind of data your working with and what you want to do with it (the end goal, not any intermediate step, to not fall in the "xy problem" trap)

[trail basin]

should i ask the same question there?

[cloud spruce]

maybe a slightly refined version of that question in that case, where you say what kind of result you are looking for or in what way you want to process the data

[trail basin]

ok thank you so much i'll try

[cloud spruce]

a lot of the times when working with data sets you will be using the python libraries "numpy" and/or "pandas"
and depending on your task "jupyter notebook" may be applicable to you: https://jupyter.org/

[trail basin]

so i should learn to use this one? thankyou 🌸 🫂

[cloud spruce]

it depends on what you want to do, i don't have enough information on what you're doing with the data to give you a better answer

[cloud spruce]

@lost dagger: sorry for botching that, @earnest blaze was totally right, i was not paying enough attention to your code 😧

[lone latch]

hey guys, I've just started using twisted as the base for a local proxy I'm running. Works great so far. Until now, I've been able to specify my proxy address and port in the client to connect to (e.g. 0.0.0.0:1234) instead of the real server. Proxy listens on this address and forwards to upstream server. Now, my scenario is that I can't tell the client what ip/port to connect to, it chooses itself. I know the ip/port it tries to connect to. If I do netsh interface portproxy add v4tov4 listenaddress=<server_addr> listenport=<server_port> connectaddress=<proxy_addr> connectport=<proxy_port> , I think this wouldn't work cause it would apply to both the client and the proxy itself, so the proxy would redirect to itself?

[cloud spruce]

sounds like you need to intercept and redirect the connections either on the client computer, the server or somewhere on the network in-between where you have control to do so
what kind of protocol and client application are we talking about here?

[lone latch]

it's a tcp chat client. I want to run this proxy on the same box as the client

Only concrete things I could have is knowing the process id that tries to connect, and the ip/port in advance

so instead of 123.249.142.6:3600 I want it to connect to my proxy that's listening on 0.0.0.0:1234 which is running on the same machine

but I don't have control of the client (well to rewrite it I mean)

[cloud spruce]

if it's encrypted communication you will have a hard time doing anything with it anyways

[lone latch]

it's not encrypted, I've seen the traffic over wireshark

but that's another question also

If you can intercept the handshake for the encryption, you can get the key material to decrypt the following packets no?

[cloud spruce]

so, you want to be able to modify the traffic in some way as opposed to just see it like with wireshark?

[lone latch]

exactly

it's a reverse engineering project

[cloud spruce]

if the encryption is not plain rubish you will not be able to decrypt it with only the data in the handshake
they would use asymmetric encryption and/or something like diffie-hellman for the handshake or on rare occasions pre-shared keys

[lone latch]

ah okay, so the traffic that is sent to the server is encrypted with the server's public key? But the downstream traffic, the client needs to have information on decrypting that right? So I could decrypt 1 way?

not that this applies here, just curious to know

I guess I would need the private key of the client, and the client doesn't send that over the network either

[cloud spruce]

no, for TLS/SSL they exchange one key for traffic client->server and another for server->client that is used with symmetric encryption, asymmetric encryption is only used for the exchange of the keys

[cedar forum]

diffie hellman my beloved

[cloud spruce]

the client will only have a long term private key if it is using client certificates (like with mTLS)
otherwise it's just a temporary session keys used for the symmetric bulk encryption

[lone latch]

so the symmetric key then must be resident in the memory of the client

[cloud spruce]

"my precious" 💍

[cedar forum]

yeah, the symmetric components are stored in RAM

[cloud spruce]

yes, it will be for the duration of the session or until it's rotated to a new one for long sessions
well written applications will try to make it hard to find and extract the key from memory, so your mileage will vary

[cedar forum]

I wonder if TLS 0-RTT affects that though, I've been meaning to talk to my prof about how 0-RTT actually works

0-RTT isn't great though, risk of replay attacks

ahh yeah operates off stored PSKs, that makes sense

TLS is pretty great stuff, lots of exciting developments in coming years

new quantum-safe algos will be fun (and maybe terrifying for performance, lol), encrypted ClientHello is also exciting stuff which should manifest in the near future

[cloud spruce]

yes replay attacks is a concern for 0-RTT

[cloud spruce]

it's already all here in TLS 1.3

[cedar forum]

yea they're all extensions to 1.3, but no wide rollout of any of them yet

ECH is in firefox now though, and Cloudflare is starting rollout iirc

hence https://crypto.cloudflare.com/

[cloud spruce]

still waiting for DTLS 1.3 to be finalized...

[cedar forum]

DTLS 🤔

oh TLS for UDP?

well, for any datagram protocol

that's pretty cool, I've never seen that before, makes sense that it exists though

[cloud spruce]

to defend against replay attacks when using 0-RTT i'll imagine we will need to take measures looking a bit like those used to defend against csrf and where the tokens would be one time use only

[cloud spruce]

exactly! 🙂

[cedar forum]

yeah -- the appendix to 0-RTT says that you need to handle it at the application level -- which frankly I'm not sure is great

Cloudflare were very big on the rollout and don't really mark anywhere the risks -- I'm unsure if they have mitigated them

[cloud spruce]

sounds like a disaster waiting to happen for customers that are unaware
financial transactions anyone?
sure, and the same one again and again and again... 😆

[cedar forum]

hahahahaha exactly

ahh seems they have mitigations in place -- that's nice at least

well, their mitigation seems fairly sane, they only allow 0-RTT on GET requests with no query parameters

which providing people follow idempotency of GET should be fine

nginx does a similar thing to retry idempotent requests

and they attach a header when their edge connects to origin which uniquely identifies the RTT PSK

so their deployment is fairly sane, but I do fear that people off Cloudflare are going to do silly things with 0-RTT

[wise jungle]

yo what was the thing the other guy said when he said select is outdated

he said something else that i forgot

[cedar forum]

IOCP

[cloud spruce]

that will limit the usefulness of 0-RTT severely for complex apps, but still great for CDN:s and such

[wise jungle]

ty

[cedar forum]

yea that's true

[cloud spruce]

select() may be "outdated" on windows, but not on unix/linux platforms where it is more versatile and performant

[wise jungle]

also rnd ive tried select but no luck with that

i ran the server script on a unix machine

still got the same problem not sure

slowly losing hope

not sure what to do next

[cloud spruce]

it's because of how you're implementing it (not meant to be offensive)

[wise jungle]

i dont mind slandering im trying to be better

just be 100% honest its not personal

[wise jungle]

im not sure how the way im doing it is bad to be honest

i feel like its just as simple as it is

[cloud spruce]

it's not, it's easy to go wrong with these things

[wise jungle]

after all the main thing i want to do is just forward the data to all the people "connected"

[cloud spruce]

are you sure the problem isn't in the client code?
it too needs to be either running the reading from the socket in it's own thread or be non-blocking (or have very short timeouts)

[wise jungle]

its reading from the socket on its own thread (client side)

[cloud spruce]

and from the audio device and then sending it through the socket in another?

[wise jungle]

no the sending data part isnt threaded i think

[cloud spruce]

it doesn't need to be as long as it's a different thread from the reading operation, as the main thread is more or less just another thread

[thorn stratus]

I'm aware, but we also know nothing of his usecase

[wise jungle]

Just a simple voicechat i sent the github link multiple times

[wise jungle]

Its on a different thread

[cloud spruce]

with "usecase" i think @thorn stratus was referring to the environment it will run in, like lan or wan
i just assumed it was a requirement to be able to run over the general internet as well

[wise jungle]

Yea

My aim is to have atleast 20 people talking to each other

[cloud spruce]

in that case multicast is out

[thorn stratus]

I think the main thing you need to do is send the data out on a seperate thread than you recieve it on

[wise jungle]

Both lan or wan

[wise jungle]

I tried that

I created a thread class that sends data to all the clients

Still choppy and slow

When two people are using it its absolutely perfect

No problems no latency

[thorn stratus]

can you show the code?

[wise jungle]

.

Github link

Its short nothing crazy

[thorn stratus]

that doesn't have any code that sends on a seperate thread than you recieve on

at least for the server

[wise jungle]

Let me scroll a bit and find it i sent it here before

[prisma cobalt]

whys that?

[wise jungle]

.

[thorn stratus]

because if you do it on the same thread you have to wait for the data to be sent to each client before you can recieve any more

[wise jungle]

Yup i tried that

Still same issue

I think recieving the data is the bottleneck

As its a constant stream

[cloud spruce]

you will probably end up with almost the same problem when trying to re-transmit the data you received anyways, just queuing at a later stage

[thorn stratus]

Modern connections should all be full duplex, I don't see how you would have the same problem

[wise jungle]

Idk if you would want to try it you should

You can try and run 2 main scripts and see how it doesn’t lag

Then when u add a third its starts going crazy

[prisma cobalt]

only if you've hit the sockets send buffer size right? i see what you mean tho

[cloud spruce]

as this is udp and all the connections are using the same socket and you need to send all data you get from each client to all the other clients you can sill build up a queue for outgoing traffic and huge buffers in the application if you can't send the data fast enough

[thorn stratus]

It still takes time to write the data to the buffer

[wise jungle]

The data sent is size 2126 bytes

Its always 2126

[thorn stratus]

True, but I doubt he is hitting a bandwidth limitation with 3 clients

[wise jungle]

I thought i was but i hosted the server script on google cloud and had the same issue

[cloud spruce]

but not much with non-blocking sockets

[thorn stratus]

So in the code you linked you aren't sending in a seperate thread than you are receiving in

[wise jungle]

Ya i guess but even when i updated it to what i sent earlier had the same issue

[thorn stratus]

I'm referring to what you sent earlier

[wise jungle]

I sent a github link and updated code

[cloud spruce]

the code in your github repo is four days old

[wise jungle]

Thats not what i meant

The github link is the og code

I then sent the updated code in discord

[wise jungle]

Look at these replies

[thorn stratus]

in that code, you send and recieve in the same thread, and there is some missing code as well

[thorn stratus]

True, but you are still gonna be blocking when you call select

I haven't really worked with non-blocking sockets at all though, I'm not sure if they would help here or not

[wise jungle]

When i go home ill see

Cause the formatting is messed up

On my phone

[cloud spruce]

no, select() doesn't block unless you give it a timeout value as well which if given will only "block" until the timeout if there isn't any activity on any of the sockets select() is monitoring for reading within that time

[thorn stratus]

you mean if you give it a timeout of 0 it won't block?

https://docs.python.org/3/library/select.html#select.select

The optional timeout argument specifies a time-out as a floating point number in seconds. When the timeout argument is omitted the function blocks until at least one file descriptor is ready. A time-out value of zero specifies a poll and never blocks.

[cloud spruce]

or if you don't give it a timeout at all

[thorn stratus]

in that case it will block

[cloud spruce]

not if you only select on non-blocking sockets, at least on linux

but you'll burn cpu cycles that way, so you probably want to block or have a low timeout value if there isn't any activity on any of the sockets anyways

[cloud spruce]

no, sorry, your (and the docs) are right, it blocks without a value even on only non-blocking sockets, i was using a 0 timeout in my code when i tested

[wise jungle]

im back

class serverHandler(threading.Thread):
  def __init__(self, mainServer, data, addr):
    threading.Thread.__init__(self)
    for c in mainServer.connections:
      if addr != c:
        mainServer.upd_server.sendto(data, c)

[cloud spruce]

but it doesn't matter much if it's blocking on select() if all actions that is initiated when there is any data available
otherwise you can have a timeout to be able to take other actions at intervals even in the absents of incoming data, low enough but not so low to run at 100% cpu

[wise jungle]

@thorn stratus

[thorn stratus]

that doesn't run the code in another thread

when you create the object it sends all the data, not when you start the thread

you need to implement the run() method

[wise jungle]

in the main thread i do

d, a = self.udp_server.recvfrom(2126)
self.clients[a] = time.time()
#stuff here
serverHandler(self, d, a)

[thorn stratus]

yeah that does not run it in a separate thread

[wise jungle]

yes it does

[thorn stratus]

no, it just creates a new thread object, which runs the code as part of the object creation, and then throws it away

to run it in another thread you need to put the code you want in the other thread in the run() method of the object and then call .start() on the thread object

[wise jungle]

it

really

i swear when you do

class name(threading.Thread):
  def __init__(self):
    threading.Thread.__init__(self)

starts a thread

for that class

and its seperated

[thorn stratus]

no that creates a thread object

start() starts the thread

[wise jungle]

breh

no fucking way

ima test

i mean even tho it does dat i still got da same problem

cause i also do tried this

                    self.sendDataT = threading.Thread(target=self.sendData, args=(d,a), daemon=True)
                    self.sendDataT.start()
            except Exception as ex:
                print(traceback.format_exc())
                    
    def sendData(self, d, a):
        for c in self.connections:
            if a != c:
                self.udp_server.sendto(d, c)

[thorn stratus]

you can't use the same socket

sockets aren't thread safe

[wise jungle]

so i need to create a new socket object

to send the data

[thorn stratus]

you could use something like asyncio if you wanna get around that limitation

or yeah use another socket object

[wise jungle]

what the hell

but you can still send

to that address even when not in lan

right

[cloud spruce]

you don't think @wise jungle can get away with that when it's a udp socket?

[wise jungle]

thats what i thought

or was thinkign till now

[thorn stratus]

the python docs specifically say the socket object isn't thread safe. I can't speak to it's implemenation so I'm not 100% sure though

[wise jungle]

alright

lets say we got 2 socket objects then

one to send and one to recieve

on the send socket

you can just send to address that the other socket has recieved from

correct ?

do i need top have 2 ports for each server if thats not the case

one for receiving data and one for sending

[thorn stratus]

you'll have to send back to the source port it came from also but yeah I believe that should work

[cloud spruce]

can you really have to different socket objects for the same socket/port like @wise jungle needs?
i don't think you can

[wise jungle]

i think just having 2 different servers

with 2 different ports would do it

[cloud spruce]

no, how would you pass the data between them?

[wise jungle]

server objects

just add a q

[cloud spruce]

as you want to receive data from any client and send it back to all other clients that is connected to the server port

[wise jungle]

on each one

create the receiving server object first

then create a sending server object later that is referenced to the server

[cloud spruce]

oh, okay, you could but then you need one unique port for each client, it doesn't scale well

[thorn stratus]

I think asyncio is really the answer here

[wise jungle]

no(to rndpkt)

let me try and sketch it out

[thorn stratus]

tcp would also greatly simplify this

although there is quite a bit more overhead

Having some type of buffer on the client would also help resolve stuttering issues, rather than it being 100% real time

[wise jungle]

damn

i just closed ms paint

im so dumb

[wise jungle]

it would but its a voice chat

[thorn stratus]

voice chat can be done over tcp

[wise jungle]

if u use tcp it would be very laggy and choppy

[wise jungle]

its better to use udp

[thorn stratus]

that's not true

[cloud spruce]

really quick and ugly hack that doesn't follow best practices but still demonstrates the core logic for the server:

#!/usr/bin/env python3

import logging
import socket
import sys
import time

from select import select

udp_server_port = 5555
read_buffer_size = 4096
client_timeout = 5

logger = logging.getLogger()
logging.basicConfig(level=logging.INFO)

# socket.SOCK_NONBLOCK does not exist on windows
#udp_server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK)
# for Windows we can do this instead
udp_server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
udp_server.setblocking(False)

try:
    udp_server.bind(('', udp_server_port))
except OSError:
    logger.critical(f'Error: Could not use UDP port: {udp_server_port}')
    sys.exit(1)

sockets = [udp_server]
clients = {}
expired_clients = []

logger.info(f'Server started on UDP port: {udp_server_port}')

while True:
    ready, _, _ = select(sockets, [], [], 0.1)
    now = time.perf_counter()
    expiration = now - client_timeout
    for sock in ready:
        try:
            data, peer_addr = sock.recvfrom(read_buffer_size)
        except BlockingIOError:
            pass
        if peer_addr not in clients:
            logger.info(f'New peer detected: {peer_addr}')
        clients[peer_addr] = now
        logger.debug(f'Received {len(data)} bytes from ({peer_addr})')
        for client_addr, last_seen in clients.items():
            if client_addr == peer_addr:
                continue
            if last_seen < expiration:
                # Mark expired client for deletion
                expired_clients.append(client_addr)
                logger.info(f'Peer has not been seen for {client_timeout} seconds, removing: {client_addr}')
                continue
            udp_server.sendto(data, client_addr)
        # Remove expired clients
        for client_addr in expired_clients:
            del clients[client_addr]
        expired_clients.clear()

[wise jungle]

its true i tried it

[thorn stratus]

tcp will have more overhead, so you might run into bandwidth limitations faster, but as long as you implement threading properly you shouldn't have issues

a buffer would eliminate any jitter effects

[cloud spruce]

i'm guessing you want to have missed packets to just be dropped instead of waiting for retransmits in this case, right?

[wise jungle]

yup

its voice chat so doesnt matter that much

[cloud spruce]

hence udp

[wise jungle]

if it lags here and there

[cloud spruce]

same for video

[wise jungle]

yes

ima try what you sent

a bit different use of it for when i tried to use select

but maybe just because ur on unix

[thorn stratus]

that's really only going to have an impact when your network link is congested either way, which would cause udp to show issues too

if latency is important for your use case udp makes more sense though

[wise jungle]

it is

but ofcourse latency would increase the more people connect

but the amount of latency increased was immense

[cloud spruce]

for real-time voice and video streaming latency is everything

[wise jungle]

from 25 ms to 150ms

[wise jungle]

didnt work sadly

blocking isnt the issue i guess

got the same results as my own code

x)

[cloud spruce]

yeah, doing this with udp where there is only one socket is quite different from when using it with tcp

[cloud spruce]

just as i said before; i think your main issue is with the client code

i just modified my code above to be more windows friendly

[cloud spruce]

@wise jungle i figured performance was more important than exact timing for client timeouts
that's why i do this in the beginning of the loop after detecting incoming data to not have to get current time and redo the calculation for each client that we are sending out data to:

    now = time.perf_counter()
    expiration = now - client_timeout

and will probably still be more granular/exact using time.perf_counter() then (time.time() - timeout)

i have not timed it, but in general for this use case i think that this:

        for client_addr, last_seen in clients.items():
            if last_seen < expiration:
                # Mark expired client for deletion
                expired_clients.append(client_addr)
                continue
        # Remove expired clients
        for client_addr in expired_clients:
            del clients[client_addr]
        expired_clients.clear()

will be faster and use less memory then:

        for addr in clients.copy().keys():
            if clients[addr] < (time.time() - timeout):
                clients.pop(addr)

[wise jungle]

im back sorry was having iftar

[wise jungle]

yea this is what i was doing

feels better for me to use time.time()- timeout()

[cloud spruce]

why? the loop will not take long for each iteration even with thousands of clients and time.time() is not granular enough anyways

[cloud spruce]

which part?

[wise jungle]

will be faster and use less memory then:

i want to maximize preformance latency and memory

[cloud spruce]

time.perf_counter() will also get you the most precis time that the platform can handle if you don't need exact time of day and can do with just time elapsed (i think you should be able to in this case)

[wise jungle]

i see

[cloud spruce]

the you don't want to use copy() and also do the (time.time() - timeout) calculation each time

[wise jungle]

hoy come

[cloud spruce]

i'm quite sure my longer code will use less memory and cpu than your sorter code

[wise jungle]

maybe

[cloud spruce]

now = time.perf_counter() will only get the time once for every received packet since i assign it to a variable
expiration = now - client_timeout and then only need to calculate the expiration cut off time once per such loop as well
if last_seen < expiration: just compares the time when we last received data from that client with the pre-calculated cut off time

if clients[addr] < (time.time() - timeout): does that for each client that you are going to send data to for each packet you have received

[wise jungle]

but at this point in time i dont think it matters

jsut about yet

[cloud spruce]

maybe not yet, but just wanted to highlight the point
also if clients[addr] < (time.time() - timeout) does yet another hash lookup for addr in clients, which my code doesn't do

[wise jungle]

i see

yea that makes sense

[cloud spruce]

copy() is also unnecessary slow here if you have many clients as it needs to copy the howl hash once more in memory which both takes time and takes up more memory

for client_addr, last_seen in clients.items(): will avoid doing any further hash lookups and you can compare directly with last_seen

but without copy() we can't modify the length of the hash within the loop
so instead i build an list of those few clients that will have timed out during the iteration of the loop and just after the loop i iterate through the list and remove them from the hash and then clear the list when i'm done

all the small optimizations adds up

[wise jungle]

yessuh

i was using .copy() cause i was modifying the dict in the loop

[cloud spruce]

yeah, i figured that was why, i'm just showing you another way to do it that is more performant and light on memory

[wise jungle]

fair i appreciate it

gonna use it on the next pull request x)

[cloud spruce]

also, i think you can do without storing both self.connections and clients, one of them should be enough
and you're not even using curclients anywhere either

[wise jungle]

after furthor looking into it i still have to use .copy () @cloud spruce for the way im implementing it

    def connectionTimeout(self):
        #credit to rndpkt#4407 from discord.gg/python
        timeout = 5
        while True:
            now = time.perf_counter()
            expiration = now - timeout
            for client_addr, last_seen in self.clients.items():
                if last_seen < expiration:
                    # Mark expired client for deletion
                    self.expired_clients.append(client_addr)
                    print(f'Peer has not been seen for {timeout} seconds, removing: {client_addr}')
                    continue
            for client_addr in self.expired_clients:
                del self.clients[client_addr]
            
    def acceptConnection(self):
        while True:
            try:
                d, a = self.udp_server.recvfrom(2126)
                now = time.perf_counter()
                self.clients[a] = time.time()
                if a not in self.clients:
                    print("Getting connection from ", a)
                    if d == b"connection":
                        self.udp_server.sendto(d,a)
                self.clients[a] = now
            except Exception as ex:
                print(traceback.format_exc())
                    
    def sendData(self, a):
        for d in self.serverObj.data_q:
            for c, lastSeen in self.serverObj.clients.items():
                if d[1] != c:
                    self.udp_server.sendto(d[0], c)
                    self.serverObj.remove(d)      

all these are gonna be threads

[cloud spruce]

expired_clients can be a local variable in the connectionTimeout(self) function
no need to have it be part of the class instance, you just have to create the empty list before the first for loop in that function
if you have that function running as a separate thread you will have trouble
for one it will run over and over as fast as it can eating cpu unnecessarily due to the while True loop and your not pausing/sleeping for a short while at some point during or in-between iterations
second and much worse is that it's not thread safe as you are changing the dictionary while other threads my use it in a way that is incompatible with the dictionary changing during those operations

[wise jungle]

you see if someone forcibly closes there connection

they magically get removed

also i tried to do the 2 ports thing and it didnt work surprise surprise @cloud spruce @thorn stratus

[errant bay]

Hey @wise jungle!

You either uploaded a .txt file or entered a message that was too long. Please use our paste bin instead.

[wise jungle]

breh

https://paste.pythondiscord.com/soxazokubu

[cloud spruce]

not at all surprised 😁

[wise jungle]

yup yup

what did i do wrong

i got 3 braincells left

[cloud spruce]

up late again or just tired since yesterday? 😆

[wise jungle]

both

[cloud spruce]

as this is udp you don't really have connections, just peers/clients, which are identified by the tuple (protocol such as tcp or udp, local address, local port, remote address, remote port) those five need to be unique for you to be able to identify a client, but you can mostly ignore the first three and just work with the last two for that
so you aren't really accepting new connections in acceptConnection(), you are just reading data from the listening socket that will be packaged and associated with that socket and the remote address and remote port, again big distinction when you are doing socket programing with udp compared to tcp where you have the abstraction of connections

[wise jungle]

i think i am "accepting" new "connections"

but

i kind of understand what ur saying

ima try something really quick

[zinc crow]

hey guys

[cloud spruce]

your not, because you don't know if it's a new client or a datagram from a current client at this point because you have a line
self.clients[a] = time.time()
that breaks your logic of the if statement just after it as you have just set it
so the if statement will never evaluate to true
also, that line where you read time.time() just looks like an accidental left over of your old code as you have my line
self.clients[a] = now
after the if statement, which is where it must be for it to work

[wise jungle]

what

where is the ti.etime()

are you looking at this
https://paste.pythondiscord.com/soxazokubu

this is the most recent version

where i use 2 ports

[cloud spruce]

i was looking at the code that you pasted above in the channel

[wise jungle]

no no look at the pastebin

also i think @thorn stratus was write about multicasting

im looking at https://www.youtube.com/watch?v=LnvxObLYO-o

[cloud spruce]

yeah, multicast is great for networks that support it, otherwise, not so great 😉

[wise jungle]

you see cant i pretty much have 2 "recievers"

but one of them is actually just a sender

[thorn stratus]

You would only be multicasting the voice data back to the clients

[wise jungle]

is that what u meant

[thorn stratus]

It would be unicast from client to server

[cloud spruce]

with multicast you send one packet and everyone that is subscribing to the multicast group address will receive it and you will never know how many that is

[thorn stratus]

But if you want to do it over the internet you would have to set up tunneling or something similar

[wise jungle]

hmm

[cloud spruce]

and the one sending the data would receive it as well and must know how to ignore it's own traffic/audio stream

[wise jungle]

multicast sounds like a huge way to easily breach any voice chat

[cloud spruce]

it was created to send out radio and tv transmissions over the network as well as stock market information

[wise jungle]

i see

[cloud spruce]

it's a one->many protocol, much like a radio transmission or if you talk in a room

[wise jungle]

hmm

would there be a way to check who is in the multi cast

you know have a initial connection to check who is in it

[cloud spruce]

and you can't run it between isp:s out on the internet (without tunneling)
even crossing routers will be problematic if they aren't configured to support multicast explicitly

[wise jungle]

but would this be

a server side issue

cause it sounds like it would be

[cloud spruce]

what is easy is just to use broadcast on the local lan, but that isn't too exciting for your application i'm guessing as people will probably not be sharing the same lan and voice chatting just amongst them self with no other external participants

no, it's a network issue and how routing works

[wise jungle]

so its just both client and server

[cloud spruce]

in multicasting you send traffic not to a clients ip address but to one virtual ip address called a multicast group address

[wise jungle]

ah

and everyone can read that data

[cloud spruce]

the client then needs to subscribe to that address using a protocol called igmp (not icmp) to tell the switches and routers in the network to give it a copy of all the traffic destined to that multicast group address

between routers it gets more complicated since they need to relay that information amongst each other to tell the next router in the chain about that they have at least one subscriber somewhere behind them and when the last subscriber leaves it signals its peer router that it can stop sending them packets destined to that multicast group address since there is no one interested in them any more... PIM such a protocol, but not the only one

you also have DM (Dense Mode) and SM (Sparse Mode), then you have RP:s (Rendezvous Point:s)

if you where to run it between larger network segments such as AS:s (Autonomous System, you can view them as different ISP:s even if they aren't always) that exchange routing using BGP you got special purpose protocols just to deal with multicast routing between those which almost no one configure, and hence you have no multicast on the general internet

@wise jungle TL;DR don't focus on multicast, use unicast for now unless you have a very good understanding of multicast and can accept all its constraints

[cloud spruce]

i would delete line 39 self.expired_clients = [] and rewrite this function like this (without doing to big structural changes):

    def connectionTimeout(self):
        timeout = 5
        expired_clients = []
        while True:
            now = time.perf_counter()
            expiration = now - timeout
            for client_addr, last_seen in self.clients.items():
                if last_seen < expiration:
                    # Mark expired client for removal
                    expired_clients.append(client_addr)
                    print(f'Peer has not been seen for {timeout} seconds, removing: {client_addr}')
            # Remove expired clients that has previously been marked for removal
            for client_addr in self.expired_clients:
                del self.clients[client_addr]
            expired_clients.clear()

and i removed the continue at the end of the if block since you are not doing anything else in that for loop after that line, so it's superfluous in this particular case

that said, i still wouldn't run it in it's own thread as that code isn't thread safe even by a long shot
and if you insist on running it in a thread i would defensively add a micro sleep within the while True: loop to keep it from taxing the cpu unnecessarily by running that piece of code as fast and as many times it possible can over and over again

[wise jungle]

yea ima do that

this shit really got me thinking like crazy

networking needs to be reworked

for sure

[cloud spruce]

i enjoy thinking, "i think therefore i am" 😉

@wise jungle you might want to add a requirements.txt to your project (and the github repo) so that one can do a pip install -r requirements.txt to get all the dependencies install (preferably in a venv of one type or another)

[rigid copper]

Ok it was indeed a caching issue. When I said "didn't help" I meant that the error temporarily went away but could occur again on subsequent loads. Just now, I thought of outright disabling the cache (through DevTools), which appears to suppress the issue. Obviously, this isn't ideal as regular users wouldn't be doing that. Is there a way I could go into the details and figure out the cause of the error?

[prisma cobalt]

Is it always that resource? If it is then there might be some wrong with the HTTP cache headers in the response from the server which you can see using the networking tab

[rigid copper]

No, it's a random file each time

[rigid copper]

It can even happen for my endpoints

[wise jungle]

yuh ima do that

[indigo fossil]

Does anyone know how to solve

Using broadcast. ```
On scapy? It happens every so often when I try to send an IP packet with ICMP (constructed as such:)
IP(dst=...) /ICMP(id=...)

[wise jungle]

hey @cloud spruce @thorn stratus after trying to fix the code i found the issue and it "works" but its very choppy now

not sure what i have done wrong now 😄

https://paste.pythondiscord.com/odizururul

i got 2 ports

one for listening and one for sending

[cloud spruce]

so, what was the issue and was it in the client?

[wise jungle]

nah the server

i forgot to add a while loop LOL

[pastel pine]

I want to create a monitor of some sorts that pings multiple hosts anyone know where to start?

[ember ledge]

Can I can ask Here about firebase

??

[cloud spruce]

you can try ask the question here or if you find a channel that is more appropriate, perhaps #web-development if it pertains to that

[ember ledge]

How i can use firebase firestore to know who is sending msg to whom and how to deliver the chat to the receiver??(want to make a chat app)

[cloud spruce]

what os are you going to run it on?
if you are going for a pure python implementation you need to run it as the superuser (root on linux, administrator on windoes and so on) as it would require raw sockets
otherwise you just have to run the systems own pingcommand from python, but i think that is a pretty ugly solution to call out to external binaries like that

[cloud spruce]

oh, and it's free too 😲
but then you probably want to use end-to-end encryption so that google can't read all the messages
but they will still be getting meta data about whom communications with whom at what time and how much, ip addresses and more probably too

[ember ledge]

We cannot use socket for sending data
to long distance and and server should be open all if client want to coneected ??that's the prop i can't use socket??

[ember ledge]

Many people uses firebase for creating chat app.

[cloud spruce]

maybe, i just didn't know it was free until now, it didn't use to be

[cloud spruce]

i wasn't saying you should use sockets
the thing about raw sockets that i wrote was a reply to another users question

[cloud spruce]

and distance has nothing to do with anything here either

[ember ledge]

Oh yes

[ember ledge]

Okay

[cloud spruce]

on that note, it's a battery drain on a mobile device to have a socket open all the time from an app and continuously monitoring the socket for incoming data or connection errors such as disconnects and handling them by reconnecting, so a platform specific way of getting notifications from the cloud is much preferred since the platform has it implemented in a way that doesn't drain the batteries too much

[ember ledge]

Guys if you make a web server in python then do you only need to port forward the port the server is running on for other people to connect?

[cloud spruce]

yes, and you can port forward tcp port 80 (or what ever you are using) on the router/firewall to a totally different port on the server running behind it, like 8080 or what ever you are using

[ember ledge]

thx

[cloud spruce]

you might need to configure the server so that it knows which port and under which DNS name or public ip address that it's reachable from outside on, otherwise redirects that is generated by the server can point the client to the wrong address and/or port

[ember ledge]

ight

[cloud spruce]

how many different ip addresses would you imagine to be the maximum number to ping and how often?

[reef nexus]

what should I do if I see an error socks range must be 0 to 255?? I set proxy and test differents proxies

[pastel pine]

Between 5-6 and 60s

[cloud spruce]

oh, that isn't too bad, then you could do it ugly by using the systems own ping command and you could do all in parallel
this way you don't need superuser privileges to be able to ping

[pastel pine]

Is their a way to make it pretty like
ping hostname | Interval 60s | Response:

[wise jungle]

anymore ideas for my problem

[cloud spruce]

what os are you targeting, or are you aiming for cross-platform?

[pastel pine]

Linux Debian

[cloud spruce]

we'll see if I have the time to write my own implementation of a simple voice client to test it out my self

[wise jungle]

alright

[pastel pine]

Would I use pythonping or ping3

[topaz sorrel]

guys is it true that c++ is more widely used for networking than python?

[cloud spruce]

it depends on what you mean with more widely used for networking, for what purpose?
what kind of network programming are you intending to do?
if it's web sites and services then the answer is absolutely no
as memory has to be so carefully managed in c and c++ to avoid introducing security vulnerabilities
if you on the other hand are talking about client applications or server software such as nginx and apache, they are traditionally mostly written in such a language
then you have the "new" kids on the block with golang and rust, where rust can be used as a more memory safe and security focused replacement for c and c++

[topaz sorrel]

OK thanks a lot

It was really helpful

[cloud spruce]

for web sites and web services a lot is developed in python and to some degree also in golang (googles own programming language also just called go a lot of the time)
but one of the most used languages is still php, unfortunately... probably because there is so much legacy applications written in it
php has seen a lot of security vulnerabilities, but mostly not in the core language but in the frameworks and libraries used and a lot of old unsecure code being reused because people are reusing old code and solutions they find on sites like stack overflow that can be really bad and without really understanding the code before the use it
just to take one example; php got support for placeholders/bind variables for sql (which is a security critical feature when using sql) very late in it's life
so there is still a lot of php developers that doesn't know about them or how to use them, hence a lot of security problems with that kind of code
of course there is stellar php developers as well, but the are dwarfed by the share numbers of junior developers that just use the above mentioned legacy stuff which leads to an even worse reputation for the language

[topaz sorrel]

Yeah my school is teaching php and I can feel the difference between a modern one and this one

[cloud spruce]

too bad for you that they haven't chosen better 😞

[torpid robin]

Hmm...

[daring hill]

@random roost voice chat 1

oh nvm

[astral bough]

im currently trying to ping my mc server an get current statistics with the code of

from mctools import QUERYClient


query = QUERYClient('play.bobbot.uk', "2894")
stats = query.get_full_stats()
print(stats)

with the error of
OSError: [WinError 10040] A message sent on a datagram socket was larger than the internal message buffer or some other network limit, or the buffer used to receive a datagram into was smaller than the datagram itself
is anyone able to help?

[bronze anchor]

seems like a bug in mctools. probably report an issue to them

[astral bough]

okay thank you

[proven vale]

how can people from outer web connect to my websocket server?

i was testing a websocket server today with a friend and he tried to connect but it didnt work.

i have the corresponding ports forwarded, and filtered in the firewall. i know this cuz we tested it today too and i had a mc server awhile ago. plus regular sockets work just fine.

a small javascript-embedded html file (messenger) works fine for me on localhost, but it doesnt work for my friend.

[proven vale]

https://portchecker.co/

will this not work?

or is that site any different

yeah but

the port is 5050

the open port i mean

how will this site know which port to check

not really

no

i have the port open

how can that ping

determine it tho

oh

hm

but he could

lol

he did

with nc

and regular sockets

yes

lol

we tried it literally today

its a websocket thing

it works on localhost

its literally open

and ncat works

has anyone else had this issue?

i mean

[proven vale]

the same thing here doesnt work with the websockets thing, but that could be because its.. well websockets and not regular sockets

its ws:// something

poof

now it look like im talkin to myself lol

[cloud spruce]

tcp is still tcp, the underlying socket won't "care" if you run something called websockets (they are two completely different things and only related in name, kind of like java and javascript doesn't share much relation) ontop of that or any other protocol which could also run ontop of tcp

[proven vale]

oh true, what could be the issue then?

[cloud spruce]

protocol inspecting firewall could be one thing, even though unlikely
local host firewall in your os that will not permit external connections
and quite a few more reasons

[proven vale]

i even disabled firewall competeley

and i have like everything allowed om 5050

on*

tcp and udp

[cloud spruce]

how does your bind line look like?
and what is the value of the variables used in that line if any?

[proven vale]

do you want to see the client or the server?

[cloud spruce]

just the relevant lines of the server will probably suffice for now

[proven vale]

ok ill see

server.py

...
from websockets.legacy.server import Serve
from websockets.server import WebSocketServerProtocol as Wssp

class Server(object):
    def __init__(self,
                 host: str = "localhost",
                 port: int = 5050,
                 *,
                 loop: asyncio.AbstractEventLoop = None) -> None:
        self.__loop = loop or asyncio.get_event_loop()
        self.__host = host
        self.__port = port
        self.__clients: set[Wssp] = set()
        self.__server: Serve = websockets.server.serve(self._client_handler, self.__host, self.__port)


    async def _client_handler(self, ws: Wssp, path: str):
        self.__clients.add(ws)
        rgbprint(f"[+] client connected! {ws.local_address=} {ws.remote_address=} {path=}", color="green")

        while True:
            try:
                msg = await ws.recv()
                rgbprint(msg, color="magenta")
                for client in self.__clients:
                    if ws is not client:
                        await client.send(msg)
            except websockets.exceptions.ConnectionClosedOK:
                rgbprint(f"[-] client {ws.local_address=} {ws.remote_address=} disconnected", color="red")
                break

    def run(self):
        rgbprint(f"[+] listening for connections on {self.__host}:{self.__port}", color="green")
        self.__loop.run_until_complete(self.__server)
        ...

part of the client

ws = new WebSocket(`ws://${host}:${port}`);

WS.onopen    = () => { alert("connection success") };
WS.onclose   = () => { alert("server closed connection") };
WS.onmessage = (evt) => {
    console.log(evt);
    add_message(evt.data, "Stranger: ");
};

[cloud spruce]

oh, abstractions... but i think i see the issue and it's my more likely guess i was getting at
don't set host to "localhost", then it will probably just listen to 127.0.0.1 and not be able to accept external connections on that

[proven vale]

AJJJJ

im so dumb

how did i miss that

[cloud spruce]

just set it to None, the empty string or even 0.0.0.0, i think any one of them will work

[proven vale]

itn needs to be 0.0.0.0 yea

i see

thanks! hahaha

[cloud spruce]

those three are in most cases equal

some people like to bind to socket.gethostname() but then they're just listening to one of the interfaces and it might not be the right one if there are multiple (virtual ones included)

[cloud spruce]

your welcome, hope things works better from now on 🙂

[wise jungle]

yo @cloud spruce any updates

[modest siren]

I'm using the builtin socket library and how do I know if a client has disconnected?

It needs to handle any disconnect. So I can't send a disconnect message to the server

There must be a better way then pinging the client every n seconds

[storm saffron]

udp or tcp?

[modest siren]

tcp

[cloud spruce]

sorry, still haven't had time to sitt down and write that amount of code that isn't work related 😞

[cloud spruce]

there really isn't, just think about it, how do you really defined "disconnected"?

the os will notify your program that is holding the socket as soon as it knows, but it will usually take time before it knows unless it has gotten explicit disconnection or some other error from the peer or a device anywhere along the path

also consider how long time your computer will attempt retransmissions, many times you'll want to set up a shorter timeout than that your self and declare the connection to not be fit for use anymore if it takes longer than that timeout

[earnest blaze]

There must be a better way then pinging the client every n seconds
From the way you phrased it, it seems you're making your own "keep-alive" packets, even though you're using tcp.
It already handles that for you.

[cloud spruce]

with udp you might not even know depending on the protocol being used ontop of udp and udp is connectionless, take syslog over udp for example, it is a one-way protocol so you will never know if anything is even received on at the destination

[bronze anchor]

I mean, if the client explicitly disconnects (FIN/RST), a .read() will return empty once

[prisma cobalt]

not strictly true, TCP doesnt implement this (and a tcp connection can theoretically be open forever without sending any data) however your operating system can be (and usually is by default) configured for TCP keepalive to check the connection is active by sending a packet and waiting for an ACK

https://stackoverflow.com/questions/1480236/does-a-tcp-socket-connection-have-a-keep-alive

[cloud spruce]

tcp built-in keep-alive is very unreliable as it has a system wide interval that the standard strongly suggest should be two hours and it is very discouraged to ever change/tweak it, with retransmission timeout ontop of that you are looking at more then two hours and fifteen minutes

in today's networks there are often many network middle boxes that keep state for NAT (firewalls, load balancers, transparent proxies and more, sometimes even routers with more functionality then just basic routing) or just firewalling in general which many times silently removes its connection entry after a much shorter period of time, 60 minutes is very common and even 5 minutes

yeah, you can easily disable tcp keep-alive for your sockets and they many live for as long as the system does if it's not used to send any traffic, as it can't detect being disconnected any other way (other then being notified by another system which i already mentioned above)

[earnest blaze]

I'm probably conflaing one thing for the other, but then how can one end detect the other went down instantly (for all intents and purposes) and propogate the error up to the application?

[cloud spruce]

it doesn't without the system going down notifying the it's peer about the fact before it goes down or if you are sending packets and you don't get acknowledgments from the peer that it has received the traffic within the timeout you have set in your application, or if you haven't it's the tcp re-transmission timeout for all retries, that will take quite some time

[cloud spruce]

normally when one peer wish to end a connection it sends a FIN (finished) packet, that only tells the other peer that we are done transmitting data on the socket but we are still listening for incoming data, hence we will still be sending ACK (acknowledge) packets (but no data on the connection) to the other peer that we have received it's packets, then the other side should do the same when it is done sending

if one wants to terminate a connection one sends a RST (reset) packet to the peer, that means that we will not use the connection anymore and we won't be listening to it either, this is typical for critical errors and such but used for a lot more than that

[earnest blaze]

Hmm, are you saying that if the os detects that a process that's using a connection encountered a fatal error / was killed, it will send an RST packet since the underlying connection still holds?

Otherwise I don't see how going down hard and still managing to send a packet can mix up.

[cloud spruce]

that is up to the os implementation, but generally yes

[earnest blaze]

Damn, I love operating systems.

[cloud spruce]

it's the tcp stack in the os that is actually managing the connection, not the application unless you use raw sockets and implement tcp or another protocol on those in your application

[bronze anchor]

surprisingly, it sent FIN when I SIGKILL-ed my program:

$ nc raylu.net 80

$ killall -9 nc

$ sudo tcpdump -nvi wlp2s0 'port 80'
tcpdump: listening on wlp2s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:18:14.790816 IP (tos 0x0, ttl 64, id 56227, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.1.211.41500 > 172.67.179.42.80: Flags [S], cksum 0x4e18 (correct), seq 2039171453, win 64240, options [mss 1460,sackOK,TS val 314664151 ecr 0,nop,wscale 7], length 0
15:18:14.811747 IP (tos 0x20, ttl 52, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    172.67.179.42.80 > 192.168.1.211.41500: Flags [S.], cksum 0x28f9 (correct), seq 949651015, ack 2039171454, win 65535, options [mss 1400,nop,nop,sackOK,nop,wscale 10], length 0
15:18:14.811836 IP (tos 0x0, ttl 64, id 56228, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.1.211.41500 > 172.67.179.42.80: Flags [.], cksum 0x679c (correct), ack 1, win 502, length 0
15:18:17.643954 IP (tos 0x0, ttl 64, id 56229, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.1.211.41500 > 172.67.179.42.80: Flags [F.], cksum 0x679b (correct), seq 1, ack 1, win 502, length 0
15:18:17.665118 IP (tos 0x20, ttl 52, id 53436, offset 0, flags [DF], proto TCP (6), length 40)
    172.67.179.42.80 > 192.168.1.211.41500: Flags [F.], cksum 0x6950 (correct), seq 1, ack 2, win 64, length 0
15:18:17.665188 IP (tos 0x0, ttl 64, id 56230, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.1.211.41500 > 172.67.179.42.80: Flags [.], cksum 0x679a (correct), ack 2, win 502, length 0

[cloud spruce]

there is so much that is going on in the different layers of the network stack that most users aren't aware of, and that's kind of the point, it should just work, but as a developer using network communication it's really powerful to know how the different layers work and interact with each other

[bronze anchor]

no idea who shichao is but this page that I just found is great https://notes.shichao.io/unp/ch7/#so_linger-socket-option

[earnest blaze]

Yeah it is fascinating, I'm kind of looking for a resource that touches on all those nitty gritty details, but all I can find are acursed abstractions and high level overviews.
Would you happen to know of any?

[bronze anchor]

...my link? https://notes.shichao.io/unp/ch2/#tcp-connection-establishment-and-termination

[cloud spruce]

this is the RFC standard and authoritative source that implementers must follow if they want to be complaint: https://datatracker.ietf.org/doc/html/rfc793
but one might want something a bit lighter like: https://en.wikipedia.org/wiki/Transmission_Control_Protocol

there is a lot of additional RFC:s that should also be implemented on top (extensions, added features, etc) of that to function in today's networks, but that one is the main one for tcp

[earnest blaze]

Read that, maybe I'm just not letting the information sink in, but i'll check out the other recources.
Thanks anyways.

Oh boy that site doesn't have presentation on their priority list I'm telling you.

[cloud spruce]

if you are talking about ietf, no look at the age of that (1981), it easily predates html, it was just text files and they have mostly just published them on the web

[bronze anchor]

also,
https://eklitzke.org/how-tcp-sockets-work
https://beej.us/guide/bgnet/

[cloud spruce]

yeah, i never said it was light reading 😆

[cloud spruce]

@earnest blaze that's also another thing, sockets are not the same as tcp, original bsd unix (not the flavors of today) use to be the reference for that

sockets is just an api that generally the os exposes to its applications for them to use those services that the os handles

[normal gazelle]

I'm a little confused with how python and networking works. I'm making a chat server and looking at reference from other peoples work, they tend to send their messages with some sort of escape character such as \n or >> for example. But my main concern is how can someone like a user type those without cutting off the rest of their message? It probably works very differently but thats why I came here for help. Any information or resources to start are needed. Thanks!

[bronze anchor]

@normal gazelle if you delimit messages with a newline like IRC, you just don't allow messages to have newlines
otherwise, you need to specify the length of the messages (and don't use a delimiter)

realistically, in 2022 if you build a chat service, you're gonna want to use websockets for all your communication so you can support a browser client. websockets take care of framing for you so you don't have to worry about delimiting/length

[normal gazelle]

hmmm alright, I'm not going to use a delimiter then, I'll just make a custom protocol. I would use websockets, but it would be a good experience doing this anyways, it would give me a better understanding and practice. Also, I want to check the message length from incoming messages but I am having a few issues, the first one mainly being packet fragmentation, currently I am sending data as a json, checking if the first and last index are brackets, then i am converting the string into a dictionary. I then check if it has the right keys to prevent any errors etc. This works pretty well so far but I was wondering if you are anyone had any input.

[bronze anchor]

JSON is not a great wire format. I'd use the struct module, which helps you encode strings with length

you can't just check for a closing brace at the end. if you have a nested dict, you might find that brace

and remember, tcp is a stream. nagling means 2 send calls from the peer may get grouped into 1 recv call on the other end. fragmentation means 1 send call may result in multiple recv calls. you must always delimit or specify the length of your packets to frame them (or use something higher level that takes care of framing, like websockets) @normal gazelle

[deep surge]

should i keep this where i use a new socket for every message or just have one or a few sockets staying open for each machine i'm connected to?

class Server:

    def __init__(self):
        HOST = socket.gethostname()
        PORT = 8080

        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.sock.bind((HOST, PORT))
        self.sock.setblocking(False)

        self.conns = []

    def poll(self, addr):
        self.sock.listen(1)
        read = [self.sock, *self.conns]

        readable, w, e = select(read, [], [])

        for sock in readable:
            if sock == self.sock:
                conn, addr = self.sock.accept()
                self.conns.append(conn)
                print(f"connected to {addr}")
            elif sock in self.conns:
                data = sock.recv(1024)
                print(f"received {data} from {sock.getsockname()}")
                sock.close()
                self.conns.remove(sock)
                del sock


if __name__ == '__main__':
    s = Server()
    while True:
        s.poll((socket.gethostbyname(socket.gethostname()), 8080))

||disclaimer : this is not production code don't worry||

[normal gazelle]

I mean if you are going to continue talking to the other clients then I would just keep them open, or else they would have to constantly reconnect etc

it just depends I guess..but I would just keep them open until you are done

[deep surge]

ok thx

[bronze anchor]

@deep surge in addition to not reconnecting, I'd suggest not recreating the read list for select and using selectors instead

also, super weird to call listen more than once and dangerous to set the accept buffer to something so low

[deep surge]

huh? but don't you need to call y = n*listen(k) for y amount of connections?

[bronze anchor]

you need to call accept for each incoming connection but you only need to call listen once (and I'm surprised you don't get an error calling it the second time)

[deep surge]

ok but how much to i set the accept buffer to then?

[bronze anchor]

oh, lastly, use SO_REUSEADDR. the socket docs have an example of this

some number at least add high as [the rate clients can connect to you] - [the rate you can accept connections at without falling over]. probably at least 16?

[ember ledge]

Anyone here currently enrolled in the WorldQuant data science lab?

[normal gazelle]

raylu, would you mind giving me some examples. I am totally lost

Im honestly almost at the point of using web sockets but I want to try and solve it without it first and try to understand it

[true oxide]

What are the best python libraries for finding and using proxies?

[bronze anchor]

@normal gazelle ok.... the struct module is not nearly as helpful for variable-length strings as I thought

https://stackoverflow.com/questions/3753589/packing-and-unpacking-variable-length-array-string-using-the-struct-module-in-py

so I think your options are

1. the above ^
2. something like protobuf (at which point you might as well just use gRPC)
3. websockets ;)

I guess
0. use fixed-length strings

[normal gazelle]

I'm going to try the packing / unpacking, and if that doesn't work. Web sockets it is

thanks

[bronze anchor]

(I got my start doing this sort of thing for battle.net 1: https://bnetdocs.org/packet/307/sid-chatevent)

[deep surge]

ok so i read this self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
but what does it do? it says it has something to do with timeouts?

[bronze anchor]

without it, try stopping your server and then restarting it immediately

[deep surge]

ok i think i see what you mean

[deep surge]

ok so i just read the selectors doc page, and why is it better? isn't it just more boilerplate?

[bronze anchor]

you can reuse the selector across iterations of the loop (and just add/remove client connections as needed)

[deep surge]

or well i mean

um less direct code

[bronze anchor]

also, it can use a better select implementation under the hood

[deep surge]

it's not more boiler plate

[bronze anchor]

it's an abstraction, yeah

[deep surge]

i don't get it

oh wait i think i get it

well is the performance difference major? i kinda prefer it like it is now

[bronze anchor]

I mean, the real perf problem is gonna be blocking/slow calls in whatever you use to handle your connections, so no

[deep surge]

ok well i think i prefer staying on the base select module. anyways thanks for your patience

dunno what kinda time i'm bothering at but here it's 3.5am

[prime notch]

Has anybody worked with p2p networks that utilize a rendezvous server? I'm looking to work with py-ipv8 but I'm not sure if there are good alternatives before i dive.

[static oracle]

hi, any good resources to learn distributed systems? tutorials/books etc...

[errant bay]

Hey @ember ledge!

It looks like you tried to attach a Python file - please use a code-pasting service such as https://paste.pythondiscord.com

[coarse heath]

Hello how would I establish a connection over the internet using socket. I would like to make my game a multiplayer game. Thanks.

[cloud spruce]

you would probably need to create some software to act as a game server and then the actual game would be the clients that connects to the game server, otherwise you will probably get problems with NAT on the players home routers

[high phoenix]

okay, can anyone help me with sockets?

[normal gazelle]

what do you need help with?

[ember ledge]

my_id = '844926742317367318'
url = 'https://www.repscore.app/'
xpath_search_bar = '//*[@id="id_input"]'
xpath_confirm_button = '/html/body/div/div[1]/div[1]/div[4]/div[2]/a'```
 
Hello, I have a quesion about automating scripts. 

I would like to do the following:
1️⃣ open https://www.repscore.app/
2️⃣ insert my ID: "844926742317367318" in the search bar.
3️⃣ store the result as a variable list_1

[wind oriole]

So what's the problem?

What are you expecting to get for a result too @ember ledge ?

[ember ledge]

@wind oriole first of all, as you might see on the code; I don't know how I can get Python to control my browser(Chrome, Firefox...).
Second; the code is just wrong, because I don't know what the commands are to do the steps (1,2,3).
Lastly; after step 2 there will be a search result that should be stored in the variable list_1.

[wind oriole]

K so, bear with me, on mobile

To control a browser you need Selenium

From Selenium you would import webdriver

Then...

driver = webdriver.Chrome()

driver.get('https://www.repscore.app/')

That will send a physical browser to that page

To find the search bar....

search_bar = driver.get_element_by_xpath(xpath_search_bar)

Then you need to send your text to it

search_bar.send_keys('blahblah')

Find the button the same way

And use button_variable.click()

But if the value you're looking for isn't on the page that follows, you're gonna have to use Selenium-Wire instead

[normal gazelle]

so...I decided to test out my chat server latency and I came across an issue, for some reason once a certain amount of messages are being sent everything stops, all connections are closed and it stops making any new coroutines. When the actual clients themselves try to connect they can, they just cant do anything since no functions have been started, this is my start server function. ```py
server = await asyncio.start_server(handle_client, 'localhost', 9090)
print('Server is listening...')
async with server:
await server.serve_forever()

[ember ledge]

Join me in this fun interactive session that will help you in exploring Data Analysis and also walk you through the details of the Microsoft Learn Student Ambassador Program.

Key Takeaways:-
-Introduction to Microsoft Learn Student Ambassador
-Overview of what is Data Analysis
-Creating Microsoft Excel Dashboard
-Introduction to Microsoft Power BI
-Quiz and Giveaways
-Q&A

EVENT DETAILS -
Date - 8th May 2022
Day - Sunday
Time - 5:00 PM IST
Duration - 1 Hour
Platform - Microsoft Teams
Event Host - Aditi Gulati (Alpha Microsoft Student Ambassador)

If anyone is interested then DM for registrations

[cloud spruce]

!rule 6

[errant bay]

Rules

6. Do not post unapproved advertising.

[cloud spruce]

@ember ledge please follow the rules

[night perch]

Why is a network splitted in WAN-zone, LAN-zone and Guest zone??

[cloud spruce]

you can split a network in as many or few zones as you want or need to, a few of the most common names for such zones are:
WAN (Wide Area Network) is just what you usually call the internet side of a gateway or firewall, that connects to the wider network
LAN (Local Area Network) is often the network within a building, apartment or what ever smaller area like that where you have your client desktop, laptop, tablets and phones on, even some internal servers that you don't want anyone from the internet to be able to reach
DMZ (DeMilitarized Zone) is where you would put your servers on that you want people both on the LAN and WAN (i.e on the internet) to reach, but still have them on a separate segment protected by a firewall
Guest zone is mostly for people that you want to be able to give internet access to through your internet connection and often they can reach the resources on the DMZ as well, but usually they are not able to reach anything on the LAN

[normal gazelle]

Does anyone have any information on application protocols? I'm working with Sockets and Asyncio using TCP as my network protocol and I need to be able to check if the message has been split / fragmented, re join or separate the messages in the stream if something were to happen. Not sure what I should do. I would use Websockets or something like HTTP , but Im not, so that's out of the question. Maybe I should just take like a 2 month break and study networking? Idk, any information is helpful

[cloud spruce]

you will never know using a stream socket the os is handling the heavy lifting in that departement, that's kind of the point
you just read a stream of data over and over until you think you have at least enough to process one message, action or what ever
if you want to invent your own application protocol ontop of tcp the two most used methods is to either use a header which encode a length field for the following message or you use some kind of delimiter which doesn't need to be just one byte, http for example use a combination of both approaches

[normal gazelle]

hmmm okay, So can you tell me if this sounds right? This is what I plan on doing (maybe)...Add a header of the length to all messages. When the server / client receives those messages , read up to the specified header length. If there is more information after that, check if there is another header and read up to that data and so fourth.

or I could use the delimter method, where I could just replace a certain character with an off limit one, then add the original as a seprator and revert it once recieved

[normal gazelle]

nvm, I'm gonna do some research

https://stackoverflow.com/questions/1046637/writing-a-stream-protocol-message-size-field-or-message-delimiter found this

[cloud spruce]

yeah, but I would read a larger chuck from the stream and then work with an internal buffer in the application, but other than that I think that is how I would do it using that method

[normal gazelle]

alright thanks, also. Do you think you could help me implement it? If not it's fine, I'm just not sure the most efficient way considering I'm using asyncio

[cloud spruce]

almost, think about how you use \ your self in strings in code, that illustrate this method very well, there \ is the special code point followed by what ever byte comes next and if it's the same character again then it represents that literal character
if using that character (or any other uncommon charachter) you could scan the input for it and doubled it, then after the encoding/escaping of the string you add your special end of message combo at the end, let's say \e in this example
when receiving you just reverse the process (in the reverse order)

[normal gazelle]

oh, that makes sense...and you would just probably use something like a for loop or something with a step of 2 so you can check every other pair

hmmm...so should I make something like you suggested or something like netstring?

I also don't want a ton of try catches hindering performance so I'm not sure

[cloud spruce]

i think using a length header would be the most performant of the two, while using delimiters would be the simplest to implement

[normal gazelle]

okay, also for the length header, would it be better to add aome sort of padding..like : 013 or something...or should I just use no padding and check for a colon or something

[cloud spruce]

for a header you also have two choices, either go with a constant length header or delimiter which in that case can never occur inside the header (this is just how http do it with two consecutive new lines to form one empty line, but you could use what ever you want for your protocol)

but padding as such should not be needed

of those two, the constant length header would be the most performant but add a restriction of the length of one message due to the maximum length of bytes that would be able to be encoded in the header

this was for a chat application, right?
unless it's a end-to-end style where you connect directly to the other party, i'm guessing you would like to encode the recipient and sender of the message in some way as well
the client first has to encode the intended recipient of the message somewhere
if using a server you might want the server to add the username or user id of the sender (doing this server side to avoid spoofing of senders by the client) to the message before relaying it to the recipient

[normal gazelle]

it's a chat server yeah, and for spoofing, the only thing they can spoof is thier ip

on initial join their username is mapped to a writer that I check each time before broadcasting information

[cloud spruce]

so that you can tuck there username or user id on to the message before broadcasting it i assume, right?

[normal gazelle]

their username, yes

and the message of course

[cloud spruce]

where are you adding the username of the sender?
you got a header field for that already or do you just append it to the start of the message?

[normal gazelle]

Well, currently I have it where it requires a username to start the loop that accepts user input, so just a single receive, then once that gets added it starts up the loop eyc. I will probably change it though

[onyx willow]

Hey,

We are looking for an addition to our team. We previously were some of the top performers in online Poker and have transitioned into trading and managing cryptocurrencies early last year.

Our goal is to find someone code-agnostic who can help build and support various apps, ranging from simple automation to penetration-testing to full-stack-development. But, most importantly, we are looking for someone eager to learn, a problem-solver who is excited to take on new challenges and grow together with our team.

Preferably you have experience with reverse-engineering tasks and participated in relevant events/competitions. If this sounds interesting to you and suits your set of skills, please reach out on Telegram @onyx willow12345.

Thank you!

[cloud spruce]

!rule 9

[errant bay]

Rules

9. Do not offer or ask for paid work of any kind.

[cloud spruce]

@onyx willow this is not a job advertisement board

[normal gazelle]

okay, so I haven't really made any thing invoking headers with streams yet so how would ingo about this? Would I just create a receive call that waits for the length being sent first, then on the second accept that many bytes?

since the streams have guaranteed order

length = (await reader.read(5)).decode()
data = (await reader.read(length)).decode()
``` it's probably not right,  but better to ask than be confused

[cloud spruce]

i guess that would be for a constant length header if you chose that as your solution?
and when you read you could get that many bytes or less, so you need to put the read in a loop that ends when you have gotten that many bytes that you are expecting, unless you make your own abstraction of read and do that part in there

[normal gazelle]

would i have to use a delimiter for a non constant length?

[cloud spruce]

yeah, a kind of delimiter but for the header in that case

[normal gazelle]

I plan on limiting messages below a certain limit so I'm not sure if it would cause more issues having to try and re assemble streams etc

a message cap on the server side

[cloud spruce]

if you are going to have a permanent limit for the maximum message length you could go with a constant header as you might not need the flexibility of a variable length header, but it does limit the future extensibility of the protocol, which might be fine for your application

[normal gazelle]

well, I might as well make it flexible in case I wanted to use it for something else...so....what should i use for my delimiter?

[crystal trellis]

guys anyone here who knows how to connect pyzmq client to a websocket server?

[cloud spruce]

just remember that flexibility also means much more complexity and at this point i think the complexity might now be worth it

[normal gazelle]

I agree...well...thanks for the help today. I'm going to do some research and figure out a way to implement this

[cloud spruce]

just remember how recv() on a socket works, that it doesn't always return the number of bytes that you have requested, you have only specified the maximum number of bytes you want to read
or are you using some other library then socket?

[crystal trellis]

guys anyone know how to fix this Traceback (most recent call last): File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\websockets\legacy\protocol.py", line 750, in transfer_data message = await self.read_message() File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\websockets\legacy\protocol.py", line 819, in read_message frame = await self.read_data_frame(max_size=self.max_size) File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\websockets\legacy\protocol.py", line 895, in read_data_frame frame = await self.read_frame(max_size) File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\websockets\legacy\protocol.py", line 971, in read_frame frame = await Frame.read( File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\websockets\legacy\framing.py", line 85, in read data = await reader(length) File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\asyncio\streams.py", line 723, in readexactly await self._wait_for_data('readexactly') File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\asyncio\streams.py", line 517, in _wait_for_data await self._waiter asyncio.exceptions.CancelledError

[crystal trellis]

client ! failing OPEN WebSocket connection with code 1006
client - event = connection_lost(None)
client - state = CLOSED
client x code = 1006, reason = [no reason]
client - aborted pending ping: d26ad5c7
client x closing TCP connection
Traceback (most recent call last):
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\websockets\legacy\protocol.py", line 750, in transfer_data
    message = await self.read_message()
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\websockets\legacy\protocol.py", line 819, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\websockets\legacy\protocol.py", line 895, in read_data_frame
    frame = await self.read_frame(max_size)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\websockets\legacy\protocol.py", line 971, in read_frame
    frame = await Frame.read(
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\websockets\legacy\framing.py", line 85, in read
    data = await reader(length)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\asyncio\streams.py", line 721, in readexactly
    raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 7763702 bytes read on a total of 16797496 expected bytes

[coarse heath]

hey how would i allow multiple clients to connect to one server using socket

and each send their own stream of packets

[dark cliff]

Can you send an example file about printing the value from the html websocket api to the console with python socket?

[coarse heath]

hey i want to make an online turtle game what would be the best way to implement the multiplayer part? have the server send position packets to the clients or maybe have it send directions to the client. I have already made a server and a client and can send messages to and from them i have also already implemented a username system but i dont know how i should go about syncing the movment of turtles across clients any help would be much apreciated

[normal gazelle]

yeah have the server keep track of players x,y...and eventually you could have it only send the ones on the actual screen

instead of all of them

[ember ledge]

Hi I'm trying to send files through sockets but it keeps saying

No such file or directory: ''"

  File "C:\Users\scrip\AppData\Local\Programs\Python\Python38\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\scrip\AppData\Local\Programs\Python\Python38\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\scrip\Desktop\SpotifyGen\main.py", line 26, in client
    readfile = open(f, 'w')
FileNotFoundError: [Errno 2] No such file or directory: ''

[ember ledge]

nvm fixed it

[normal gazelle]

asyncio, but im pretty sure it uses sockets deep down

since it has almost all the same methods in the loop

[cloud spruce]

yes, it does, but i think the code looks quite different using that abstraction

[normal gazelle]

what do you mean by the code looking different? Like how the sockets are used?

[main condor]

Hello!
i am sending a broadcast packet out but do not know how many devices will respond back. How do i count how many UDP packets were received?

[normal gazelle]

received on the sender or receiver side?

[main condor]

I send out 1 broadcast packet, 0 or more devices will respond with a broadcast packet

but i wont know ahead of time how many devices will respond

[shut shuttle]

Use the wireshark for packet capture at source and destination end.

[main condor]

but i am automating this and dumping the data into prometheus

i can make it non blocking and then catch the exception but it would be best to just know if their are packects in the socket ready to be processed

[normal gazelle]

What are you looking for? Just a basic counter in total? or something like a dictionary with the amount form each one etc?

[main condor]

ill work with what ever is recommended

[normal gazelle]

you could probably just get by using a while loop and a recv call

[main condor]

ive made the socket non blocking as i am sending out data to select of interfaces

i am unable to tell if the socket as a packet to be recieved

i get this error if i call recvfrom when no data is available
BlockingIOError: [WinError 10035] A non-blocking socket operation could not be completed immediately

[normal gazelle]

well yeah, it might get dropped etc

if you are looking for confirmation I would use tcp

[main condor]

i have no control over tcp or udp

i am using an existing protocol

its called NSDP

[shut shuttle]

Check with your network team that it could be possible if the ACL is not allowed for that particular udp Port.

[main condor]

i kinda thought i could read the buffer size

i am hobby coder

no network team

i recieve the first packet just fine

it comes in as a byte array, but then i am unable to know if additional packets are avilable

that error comes up when i recvfrom and no packet is in the buffer

but sometimes i will send broadcast and no devices will be online to respond. so i will want to check for that

i am doing this ```
data, addr = sock.recvfrom(1024)
print(data)

b'\x01\x02\x00\x00\x00\x00\x00\x00l\xb3\x11@>\xd2 \x0c\xc8=\xb8\xde\x00\x00\x00\x00NSDP\x00\x00\x00\x00\x00\x01\x00\tJGS524Ev2\x00\x03\x00\x08switch01\x00\x04\x00\x06 \x0c\xc8=\xb8\xde\x00\x06\x00\x04\xc0\xa83\xf0\xff\xff\x00\x00'

data, addr = sock.recvfrom(1024)
Traceback (most recent call last):
File "<pyshell#32>", line 1, in <module>
data, addr = sock.recvfrom(1024)
BlockingIOError: [WinError 10035] A non-blocking socket operation could not be completed immediately

i think i will just put it in a try, give devices a second to respond and then recfrom untill i get an error :\

[cloud spruce]

you don't want recvfrom you want recv as you don't know from which ip address you will get the response packet from

[main condor]

i thought recv was tcp and recvfrom was udp

yup i read recv is for tcp

The recv() call is normally used only on a connected socket (see connect(2)). It is equivalent to the call:

recvfrom(fd, buf, len, flags, NULL, 0);

[cloud spruce]

your right, i mixed up the two

[main condor]

i am going to keep playing with selector to see if it will notify on packet

it is late though i need sleep

[cloud spruce]

you're on windows too, aren't you?

[main condor]

i am writing this on windows but will run it on ubuntu once finished

honestly if it does not work i am thinking of reading more into twisted

i try to not rely on try/catch

[cloud spruce]

aha, because i saw that you wrote selector, which is kind of a abstraction of select
select apparently doesn't work very well on windows but on linux there is no problem

[cloud spruce]

for what you are doing asyncio sounds like it would be a good fit

[whole basin]

what kind of netwroking can you do with networking in python

[earnest elk]

That's very general- all sorts, I suppose

[whole basin]

mhm

[cloud spruce]

rather, what are you after?

[weak folio]

anyone used netmiko before ?

[jovial trellis]

hello someone here knows about socket programming?

[bronze anchor]

do you have a question?

[jovial trellis]

yeah this dont work but idk why about it

i got this error

[crisp trout]

Try taking out the space after the comma in line 5 in between ((serverName, serverPort))

It's just a guess, but all examples I've seen don't have the space after the comma

[bronze anchor]

@jovial trellis hola wapos server is definitely not a valid hostname, which is what the error is trying to tell you

[jovial trellis]

F

[jovial trellis]

well i did the changes and now works thx 😄

[crisp trout]

What was the fix exactly? Was it the server name? because my suggestion was downvoted lol

[jovial trellis]

commas

server name worked

[cloud spruce]

that space in the code does nothing other than make the code look a little better (increases readability), but it will execute the same

[earnest elk]

that has absolutely no effect on anything
it's just styling

[fossil fiber]

Hey there. Ive been working on a game framework in pygame and have began working on the server application. I see a lot of tutorials for simple programs but since my framework is quite robust im having difficulty figuring out how I want to set all this up. My current idea is to have the server program be a modified game client, one that does not render sprites or take controller input. This way, I can spawn game objects on the server the same way I spawn them in the client. My plan is to use pickle to send entire game objects between the server and back. This "server client" runs 2 threads, one for accepting and managing client connections and the other is for game logic. I've never designed a network framework like this before so this is new to me, am I taking the wrong approach to this?

[bronze anchor]

@fossil fiber pickle is not generally suitable for network communication. I'd probably come up with some other representation for objects on the server and some other protocol for serialization (https://docs.python.org/3/library/struct.html or protobuf). only use pygame for rendering

[fossil fiber]

Oh I see, thank you. I will take a look. Im guessing struct.pack is the equivalent to pickle.dumps, as struct.unpack is the same as pickle.loads?

[ember ledge]

Struct won't serialize objects for you the same way pickle would, so it doesn't make sense to compare them as equivalents. I'd refer to the example at the end of the page for a better idea of what struct does do

I guess the reason it was recommended to you was to make it simpler to encode and decode data over the network connection, but you'll still have to manually decide how to encode game objects into a transmissible representation, and decoding them back into game objects

[fossil fiber]

So sending all game data is not a good idea?

on a similar note, what if my player object has an attribute weapon for example, and its of type game object. I cant send a whole other game object within the attributes of an outer game object can I?

although it still is in the master list of game objects so i guess I would just package that seperately

It's a shame there arent more complex examples for this kind of thing, and if there are, someone please show me, like maybe a write-up or something

[ember ledge]

Ideally for a smoother experience you minimize how much data you send in general, because the more data, the more likely it is you'll run into problems.

A blind encoding and decoding is also a lot more susceptible to being compromised because you don't check the data or what it's doing at all. The docs for the pickle module says as much.

[ember ledge]

This is hard to answer without knowing what you end up using to encode your data

If you use pickle, it'll encode everything from the top down, so any nested objects will be encoded

If you write your own encoder which doesn't send as much data, that'll be entirely up to you

[fossil fiber]

I think writing my own encoder is the better direction from what you tell me. Since nested objects are still in the master objects list, sending it from both the parent object and the master list would probably send it twice, so thats no good. Any documentation on custom encoders?

If my experience from other engines ive used is anything, im guessing some sort of replication system is in order? I guess im just having a hard time understanding how I would specifically send object data from client to server and back, but that all depends on my framework so its not something I could really just ask in a public server

[ember ledge]

For the sending part specifically, there are many ways. As Raylu mentioned, structs are designed for this purpose (writing and reading binary data into a known format), but there are many methods.

In terms of your encoder, this is kind of hard to say because it'll depend on what you want, and what you want is even more complicated because you're trying to build an engine. You'll have to make some calls here about what you want your engine to do. Sending only metadata (such as what weapon is equipped instead of the entire weapon) can be useful, but it can be limiting (can't make more complicated leveling/upgrade systems without creating many copies of the same weapon for instance).

[fossil fiber]

Sending metadata is probably too limiting, since its funny you should mention it because complex upgrade/level systems are the main point of my current project. Although I am working on this infrastructure to be broad for use in other projects, I imagine it will change drastically depending on said project, so specifics can be tuned to the game i'm working on now.

So the general stance I (think) I'm understanding here is initialize game object data on the server, send it to the client. Client modifies that data, sends it back to the server, and spreads that data to the rest of the clients. Does that mean things such as physics calculation (for things like projectiles) and game logic (scoring, player management) should be done on the server? in that case, is the client only really for rendering and taking input?

[ember ledge]

Unfortunately another case of "that question depends on context" haha. In some cases, you'll want to do those calculations on the server, either to reduce client load, or prevent cheating, or w/e else. Some things you'll want to do on the client. For a game engine, it might be good to provide the user with ways to do either, so they can choose whatever is best in their given context

[fossil fiber]

I see, this has all been very helpful to me so far, thank you. The point of the server being a modified client is to allow for the freedom to do whatever, wherever, but the unfortunate reality of me doing that is I don't know where to go or what to do after lol. Differentiating what data goes where is the real learning curve for me here. I've already established a network <-> client relationship with a simple socket server, but what data needs to go between them is a challenge in itself. I will find out as I go. Thanks again for your help.

[bronze anchor]

in general, networked games send only a delta rather than updating the entire state @fossil fiber

FWIW, this is a pretty complicated topic. like, how sensitive is your game to latency (do you need to do interpolation?)

[ember ledge]

Hey, Any recommended practical tests on Netmiko or Napalm? Been recommended - https://www.packetcoders.io/

[fossil fiber]

To be honest, I'm not sure. It runs 60 fps well on my worst device, but I'm not familiar with that term

[bronze anchor]

not frame time. I'm talking about network latency

[fossil fiber]

Again, I'm not really sure. I havent programmed a way to send game data back and forth yet, all I have are clients connecting to a server and sending a confirmation back and forth.

And, a game framework

like there's a "finished" game but its purely client side

[brittle burrow]

Guys, I kinda really need help with a project.

Anyone can help me please? I am starting in this language and it has been complicated. maybe someone can give me a light

Thank you so much.

[earnest elk]

Describe your problem.

[brittle burrow]

So, I need to create a roulette bot. Which will look for the past 4 rolls, and send a message to telegram. That's it basically.

But with my skills of webscraping using selenium. I was not able to even get the information from the page

I tried using pyautogui to look for images( when its 4 same rolls) But I could not run it in a AWS like

forever

24/7

So I gave up for now.. I have no idea how I will be able to do that

Does it make any sense anything I said

[timber zodiac]

Hello, I'm no programmer but I'm familiar with fundamentals such as data types, variables, array, loops. I want to learn how to view packets and read them, what steps should I take? I searched on google and there's thing called "Wireshark" about packets, should I learn how to use packets? Am I in the right track about my goal?

[brittle burrow]

Wireshark gets stuff going through your network

What packets? U mean packets going through your network?

[timber zodiac]

Yes

Coming to my pc

Packets coming to my pc

I want to collect those packets and read them

I just have an idea to track something and analyze it

Am I in the right track?

If I'm going to learn wireshark?

[brittle burrow]

Yeah! I think that wireshark is a good idea

If you are able to filter the thousands of packets running

xD;)

[timber zodiac]

I want to collect data from a game

I dont want to do it manually

So I researched and there's thing called packets

So I look about packets

And that's when I discovered wireshark

[lucid garden]

hello, how can I make client and server side sockets communicate on different networks?

I've triend putting my ip found on whatsmyip, but it doesn't work

[small plinth]

can you ping your IP?

[prisma cobalt]

whoevers running the server needs to port forward

[wise jungle]

when u run the server just use 0.0.0.0

and then to connect to it you use the public ip and port

[bronze anchor]

@timber zodiac wireshark is good if the structure of the packets is simple (and unencrypted). otherwise, you might be better off disassembling the game (assuming windows, using x64dbg or ghidra)

[lucid garden]

yes

[lucid garden]

how can i do that?

[lucid garden]

client.py

[wise jungle]

did you port forward

on that ip

[lucid garden]

i dont know

[wise jungle]

you port forward on ur ipv4 address

of the server

do u know how to port forward?

[lucid garden]

no

[lucid garden]

i asked

[prisma cobalt]

its a setting in your router, open your router settings and their should be an option

[wise jungle]

you should probably look up a video

[lucid garden]

okay

[wise jungle]

on how to port forward on your router

[lucid garden]

is it risky leaving it opened?

can I just "port forward" my device?

my specific local ip

[prisma cobalt]

if you dont have a process running behind the port listening, connections will bounce off anyway (not a security risk)
if you do have a process running then the security is based on the implementation of the process

[lucid garden]

I am not talking about my program

Im talking about the router

(I am new to networking)

[wise jungle]

no dont worry

[lucid garden]

okk

is it "remote access"?

[brittle burrow]

Like, no one answered xD

Anyone works with python already?

[wise jungle]

No it should be on advanced settings

And then just port forwarding

If your router/network is new you probably dont have a static ip and probably have a cgnat

Where you cant port forward and should just ask your isp to get you a static ip

[vital plume]

Hey community! Iv been looking through docs but cannot find any specific examples thus far.
Has anyone used Python2 or Python3 to iterate through a subnet and list each individual subnet within that supernet? I.E pulling a list of /24's from 10.0.0.0/20.
TYIA!

[cloud spruce]

first of all, don't use python2 unless you are on a system that doesn't support anything else
you are looking for the ipaddress modules which is part of the python standard library

import ipaddress

for subnet in ipaddress.ip_network('10.0.0.0/20').subnets(new_prefix=24):
    print(subnet)

[lucid garden]

I have done it, thanks

[vital plume]

Thankyou!! I will try that

[prisma cobalt]

I know, I was explaining the risk of port forwarding. The risk entirely depends on the process (app) you have running on the port, in of itself port forwarding is harmless

[lucid garden]

okay

[jaunty marten]

Would this be a proper channel to ask for recourses on things like network automation and SDN

[ember ledge]

Hello I'm not entirely sure if this is the correct channel to post my question, please let me know if I should forward my question to another channel.
I'm currently developing on a Ubuntu 20.04 server which is connected to a proxy. I created a conda environment with python 3.8 and requests library so that I can consume an API.
The thing is that when I make an API call outside my virtual environment it works perfectly fine wget <api_endpoint>, but when I call the API using requests library within the conda environment it prompts me an error saying:
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1045)

[bronze anchor]

what do you mean "connected to a proxy"

[ember ledge]

Basically a gateway between the server in which I write my code and the Internet

[bronze anchor]

could you explain... less basically? how can your entire server be "connected to a proxy"?

is it... a SOCKS proxy? an HTTP proxy? what?

[ember ledge]

HTTP proxy

[bronze anchor]

and how is it connected to it?

[ember ledge]

Is it posible to figure out how is it connected cause I didn't set it up so I'm not entirely sure about how it's been set

[bronze anchor]

there is generally no such thing as having a whole server connected to a proxy. individual connections can choose to go through a proxy

so you've probably misunderstood what's going on

[ember ledge]

Yeah probably

I feel like when I try to call the API within my conda environment the API server requires a certificate that I don't have stored

I feel like instead of getting the certificates from the OS it looks for certificates within /lib/site-packages/requests

And there is a certificate that is missing

[bronze anchor]

what's the host?

[ember ledge]

Honestly I have no idea about how to solve it. I wonder if there is a way to tell requests library to look for OS certificates path instead of the ones that it has stored within the virtual environment or I should probably try to obtain the certificate from the API and paste the certificate into my virtual environment certificates path

[bronze anchor]

I still think you're mis-diagnosing the problem. what's the host?

depth=1 C = US, O = Let's Encrypt, CN = R3
depth=0 CN = www.afedegi.com

it's an LE cert, so it seems pretty unlikely that requests doesn't recognize it

show us some code

also, the whole traceback