networks

[cloud spruce]

don't you have this is line in your code?

observer.timestamp = target.timestamp

[elder cobalt]

doing that for some reason made my car teleport everywhere around the target

doing target.timestamp = observer.timestamp does not do anything to the observer car for a while but then suddenly teleports to my target car without lagging behind

seems weird lol

my guess is that maybe timestamp is proportional to ping?

[cloud spruce]

that line will not do anything really, so you can just skip that line

then just comment out the line:

observer.timestamp = target.timestamp
```it will have the same result as reversing them will not do anything at all

[elder cobalt]

huh strange maybe placebo effect?

😅

[cloud spruce]

haha, yeah, probably
either way, reversing that line will just update the target timestamp which is then never used for anything

[elder cobalt]

o wait forgot something if we delete

        ("gas", ctypes.c_uint8),
        ("performance_delta", ctypes.c_int16),

won't it apply the struct wrong?

unless there's a way to just skip those two lines 🤔

[cloud spruce]

yes, but we have to correct for that in our code by simply changing the line:

offset += update_record_size
```to:
```python
offset += update_record_size + 3
```as our update record size will not be 57 octets instead of 60 octets
but the custom batch updates from a modded server will still be 60 octets long per record

[elder cobalt]

A I see

didn't notice that

[cloud spruce]

you could even take care of both the normal batch updates (called "mega" in that server code) and the custom batch updates from a modded server with most of the same code

[elder cobalt]

Yeah pretty much we can parse info if the server is stock, has mega packets enabled or disabled

[cloud spruce]

after removing the two last fields from the data structure, try this code out:

if packet.is_inbound:  # Server packet
    metrics.packets_in += 1
    if packet.payload[0:1] == b"\x46" and len(packet.payload) == 61:  # single update
        metrics.updates_in += 1
        if packet.payload[1:2] == session_id:
            metrics.matches_in += 1
            target_data = packet.payload[1 : 1 + update_record_size]
            target = None  # invalidate cache after update
    elif (
        packet.payload[0:1] == b"\x48"  # batch updates
        or packet.payload[0:2] == b"\xab\x03"  # custom batch updates from modded server
    ):
        offset = 3
        for _ in range(packet.payload[2]):
            metrics.updates_in += 1
            if (
                len(packet.payload) >= offset + update_record_size
                and packet.payload[offset : 1 + offset] == session_id
            ):
                metrics.matches_in += 1
                target_data = packet.payload[offset : offset + update_record_size]
                target = None  # invalidate cache after update
            if packet.payload[0:1] == b"\x48":  # batch updates
                offset += update_record_size
            else:  # custom batch updates from modded server
                offset += update_record_size + 3
```you'll have to indent all the code three steps (12 spaces) to the right
it should take care of all the three types of packet from the server

[elder cobalt]

sure just changing it now

[cloud spruce]

you'll also have to remove the line:

observer.gas = target.gas
```further down in the code, as we don't have that field anymore

the "batch updates" refers to the "mega" packets as i'm guessing they are present in the stock server as well
and the "custom batch updates" are the modified version of those from the modded server, which do include those two fields but in a swapped order compared to the single update packets 🤦

[elder cobalt]

is this formatting fine?

[cloud spruce]

i think it should be

oh no, just saw something

the elif line and all the code under it should be indented one more step (4 spaces) to the right

but only the new code, the old code for the verbose output is correct as it is in that screenshot, so don't indent that code more than it already is

[elder cobalt]

how about now?

its under the if packet.is_inbound statement

[elder cobalt]

oh just a notice wanted to point out that version B uses offset = 8 and

for _ in range(packet.payload[1]):

I guess it's better if we just separate them

[cloud spruce]

how does the full payload of such a packet look like?
if you could print it with:

if packet.payload[0:1] == b"\x48":
    print(f"raw bulk update: '{packet.payload.hex()}'")
```you just have to put that code right after the line saying:
```python
metrics.packets_in += 1
```with that `if` being at the same indentation level as that line, don't alter any other lines
it's just for me to get some sample data to play with if you could paste one or two such outputs as text (not screenshot) here in the channel

[cloud spruce]

and oh yeah, i see it now

[elder cobalt]

O hold on I can share you the struct

[elder cobalt]

https://drive.google.com/drive/folders/1-pGAbibrpxJTl60gXdMWFv7LtZwxm5TB?usp=sharing

[cloud spruce]

or you could just change the two lines:

                    offset = 3
                    for _ in range(packet.payload[2]):
```to:
```python
                    offset = 8 if packet.payload[0:1] == b"\x48" else 3
                    for _ in range(packet.payload[offset - 1]):

my whole elif now looks like this:

                elif (
                    packet.payload[0:1] == b"\x48"  # batch updates
                    or packet.payload[0:2] == b"\xab\x03"  # custom batch updates from modded server
                ):
                    offset = 8 if packet.payload[0:1] == b"\x48" else 3
                    for _ in range(packet.payload[offset - 1]):
                        metrics.updates_in += 1
                        if (
                            len(packet.payload) >= offset + update_record_size
                            and packet.payload[offset : 1 + offset] == session_id
                        ):
                            metrics.matches_in += 1
                            target_data = packet.payload[offset : offset + update_record_size]
                            target = None  # invalidate cache after update
                        offset += update_record_size if packet.payload[0:1] == b"\x48" else update_record_size + 3

so, it's now the same number of lines of code as the original, just changed the first two lines in the elif and the very last line of the for loop

[elder cobalt]

Btw in

for _ in (packet.payload[2])

any reason behind why it was the 2nd index? on 0xab,0x03

[cloud spruce]

that is the third octet of the packet, now with offset - 1 it will read third octet for custom batch updates from modded servers and 8:th octet for normal batch updates (the "mega" packets)
that's what:

                    offset = 8 if packet.payload[0:1] == b"\x48" else 3
                    for _ in range(packet.payload[offset - 1]):
```does

[elder cobalt]

Ohhh I see

Makes more sense now

[cloud spruce]

as the bytes (and arrays in general in python) start index is 0, the third octet is 2 not 3, hence we take - 1 from the offset and then use offset to start reading the first update and so on

[elder cobalt]

@cloud spruce is this fine?

[cloud spruce]

you can change the four last lines into just:

                        offset += update_record_size if packet.payload[0:1] == b"\x48" else update_record_size + 3
```and it needs to be aligned with the inner `if` statement, not the `for` loop statement

[elder cobalt]

also is there anyway to get rid of these spaces in between?

looks kinda empty

😅

[cloud spruce]

the spaces was just to get the coordinates to align to the same position with the comma, decimals and stuff when multiple entries were printed and regardless where on the map the coordinates were
but if you don't like it you can remove the colon (:) and everything after it within each {} of the f-strings in the print() statements

f"  target pos(x: {target.position.x:>10.3f}, y: {target.position.y:>10.3f}, z: {target.position.z:>10.3f}), "
```would simply become:
```python
f"  target pos(x: {target.position.x}, y: {target.position.y}, z: {target.position.z}), "
```and so on

[elder cobalt]

a that's not the issue you see how there's space between the first target pos and the second target pos? like an empty newline

[cloud spruce]

oh, now i understand, i think i would need to see the code to find that

unless there's something obvious like an empty print() statement somewhere, but i would think there is something else or you would have taken care of it already

[elder cobalt]

on the previous code without taking those megapackets taken into account it used to print just fine without those newlines

                if verbose and target is None and target_data is not None:
                    # convert partial packet payload to struct and cache it
                    target = UpdateIn.from_buffer(bytearray(target_data))
                    print(
                        f"target pos(x: {target.position.x:>10.3f}, y: {target.position.y:>10.3f}, z: {target.position.z:>10.3f}), "
                        #f"rot(x: {target.rotation.x:>6.3f}, y: {target.rotation.y:>6.3f}, z: {target.rotation.z:>6.3f})"
                    )

[elder cobalt]

Ah when you mentioned this

went to check and it looks like that was the case

this was empty lol

[cloud spruce]

oh, you have commented out the second f-string, then you can remove the comma än the space from the end of the first f-string as well

[elder cobalt]

nah just commented it out since I might use it if in-case I need to debug things out later

[cloud spruce]

okay, if you commented out all the f-strings front a print statement you would need to comment out the whole print statement or else it will just print a blank line

[cloud spruce]

iI guess you've fixed it, right?

[elder cobalt]

yep pretty much

after you pointed out there might be a print statement somewhere

[cloud spruce]

about your other issue with the other day with intercepting traffic from your mobile device on your computer
you would need to configure your computer to forward and route traffic that hits it with another ip address as the destination and also setup SNAT (Source Network Address Translation) for that traffic on the computer
then on the mobile device you setup the computers ip address as your default gateway to route all your device traffic through the computer, to do this and keep those settings you would probably also need to set a static ip address and netmask on the mobile device as well

then you should be able to observe the traffic both to and from the mobile device for all traffic that is to and from an address not on the local network

i don't know if pydivert/windivert will be able to handle the traffic going through your computer or just to and from it

[cloud spruce]

@elder cobalt have you worked on or even solved you issue where you wanted to filter out some of the requests based on method and/or url in the request from the client to the game server?

[elder cobalt]

Yeah it was blocking scripts based on a wordlist filter

[cloud spruce]

have you solved it?

[elder cobalt]

Yeah 👍

[ember ledge]

https://paste.pythondiscord.com/xuvojowetu

[sweet rampart]

what would the best response code be if someone sends a post request without a body?

[misty elbow]

maybe 400 (bad request) or 422 (unprocessable entity)

[sweet rampart]

Sweet thanks

[ocean bay]

So when exactly does the TCP protocol and 3 way handshake take place?

I have two mental models that Im thinking of
a. functionality within the TCP header, or
b. a separate transmission.

a. Within a LAN, device A wants to send a file to device B. device A will send device B packets of the file. Now, these packets have a TCP header with the SYN (which I think is the sequence number) that device B receives. Device B then sends back a packet with the syn and ack, and device A sends device B an ack.

b. device A wants to send a file to device B. Before sending packets of the file, device A and device B want to establish a TCP connection first. They send each other packets purely to establish a TCP connection. After establishing the TCP connection, device A then sends packets of the file to device B.

Feel free to correct me at any point here thanks!

[cloud spruce]

almost b
whenever station A (a computer or other end node device, but does not include network devices) on a network wants to connect to station B (can be on the same network or another network, doesn't matter much) using TCP for whatever reason
station A that wants to initiate a connection first sends a TCP packet (typically without TCP payload, just TCP headers and the headers of all layers underneath) with the SYN flag set in the TCP headers to station B, saying "i would like to establish a communication channel from me to you" (the SYN)
station B will respond to station A with a similar packet but with the ACK flag set as well in addition to the SYN flag, saying "acknowledged, i have received your packet" (the ACK) and "i would like to establish a communication channel in the other direction as well" (the SYN)
station A sends back another such packet to station B but this time without the SYN bit/flag and but with the ACK bit/flag set, saying "acknowledged, i have received your packet" (the ACK)
that is the three way handshake in TCP, now either side can start sending data whenever they like
which one sends the first piece of data of the new connection depends on the above protocol used
for HTTP it will be station A (the client) sending the request
for SMTP it will be station B (the server) sending a greeting message introducing it self (almost like a person answering their phone)

right now we are ignoring all the layers underneath and what might need to happen there for this connection to be established

[ember ledge]

https://paste.pythondiscord.com/xuvojowetu

[ocean bay]

I see, and what is the actual data within the TCP-establishing packets? Or is it just no data since only the TCP header is relevant

[cloud spruce]

on the third line of the above message i wrote:

a TCP packet (typically without TCP payload, just TCP headers and the headers of all layers underneath) with the SYN flag set in the TCP headers
no, no data (i mean no payload within the L4 packet, just the TCP headers and the headers for the below layers) at all typically, unless the implementation is using "fast start" but it's not widely supported, so you can mostly ignore that corner case

[ocean bay]

Ahh ok thanks

[ocean bay]

Could I clarify that final part after the three way handshake is established? How does the TCP connection handle data transmission then? Say device a and device b have established a TCP connection. device A now wants to send a file to device B. What is the process?

Im confused because videos ive watched claim TCP can guarantee delivery through the ACK etc. yet the three way handshake which has the ACKs is only for establishing the connection not for sending the data through the connection.

(Ive also been unable to find many resources the complete process of a packet being sent through the TCP protocl if someone could send a link or two)

[cloud spruce]

after the three way handshake has completed either side is free to send packets with data, the other side should then send a reply packet back with an ACK for received packet to announce to the other party that it has received it, the station that sent the data will not consider the data delivered until it has received the ACK and will until such time keep that data in it's send buffer so that it can retry sending that data again after a period of time without an ACK

[ocean bay]

thats actually cleared up so much haha

I assume this reply ACK packet has no payload and just has the ACK flag set in its header yea?

[cloud spruce]

a packet with the ACK bit/flag set can have payload if the other party got some data to send it can combine the two
if it got nothing to send but need to acknowledge that it has received packets it sends a packet without payload in response

[ocean bay]

Cool

[ocean bay]

After the connection has been established, say a client sends multiple packets. How can the client distinguish which packet was acknowledged by the server, or better phrased if reply ACK packets all only have the ACK flag set, how can the client know which client-sent packet the reply ACK wants to acknowledge

[cedar forum]

alongside the ACK flag, the packet includes an ACK number

[ocean bay]

Ah that makes sense

[cedar forum]

where all those fields are defined in the TCP header

https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_segment_structure

TCP is a very very very smart protocol

it's built to be very much aware that the network can drop packets at any time, but also to maximise transmission speed

this is what transmission rate looks like with TCP, as long as it's receiving ACKs it'll slowly ramp up the transmission speed, but as soon as it loses a packet it'll suddenly drop that transmission speed

smart smart stuff

bandwidth probing

[cloud spruce]

for packets you send you include a sequence number and the other party replies with that sequence number as the ack number in it's own packet and uses its own separate sequence numbers

[ocean bay]

Hmm is this diagram just the three way handshake for establishing the TCP connection, or is the three way handshake also always done whenever youre sending a packet through an already established TCP connection

[cloud spruce]

until lots of udp traffic comes flying which doesn't have such good manners and doesn't back-off it's transmissions, but makes tcp back-off way down and eats tcps lunch 😉

[cloud spruce]

that first diagram was just for the three-way handshake, nothing else, then the transmissions just continue

[ocean bay]

Yepp makes sense

[ocean bay]

So the unique ACK number joe mentioned, is the sequence number from the client-sent-packet yea?

[cedar forum]

yeah the full thing looks more like this:

that's handshake, transfer, termination

[ocean bay]

4 way handshake is termination?

[cedar forum]

yep

FIN for finish

[ocean bay]

Perfect

Thank you all very much!

[cloud spruce]

it doesn't have to go in that order though, sending a packet with the FIN bit set just says "i'm done sending data from my end, i promise" but it will continue to listen and the other side is free to continue sending data, all you can do is to send empty ACK packets back or a RST packet if you want to tell the other side that you are abandoning the session fully, in such a case the other side should stop sending traffic as soon as it has processed that RST packet and close the socket on it's own side as well

[ocean bay]

Hmm sending a packet with the FIN bit seems kinda useless then

if the other device keeps listening

[cloud spruce]

no, you are telling the other party that you are done with sending data, then you just wait for the other party to finish as well and then you're done, it signals the other side that we are intending to be done with the connection "soon"

in practice you rarely see much traffic after the first FIN, other then the FIN sequence

[ocean bay]

What exactly does ‘wait for the other party to finish’ entail?

How would the other party decide to finish

[cloud spruce]

no, after you send a FIN you will not send any more traffic but you will keep listening until the other side sends a FIN as well
the other side doesn't have to keep listening since you have said you are done sending data and hence the tcp stack on the other side will just need to process ACK packets (and any possible RST packet) for that connection, but the application doesn't need to read anything more after that

[cloud spruce]

after you send a FIN, the other side doesn't have to wait for you to send any more data, you have said you are done already, it's just you that needs to wait for the other side to do the same and while doing so process any data that you receive

[ocean bay]

Ohhh ok yepp that makes sense

[cloud spruce]

as you can see it's not unnecessary at all 🙂
it's even very necessary or otherwise you would just cut the other party off if you just disconnected without signaling anything to the other side

[ocean bay]

Yeah haha

[ember ledge]

https://paste.pythondiscord.com/xuvojowetu

[ocean bay]

Im also a bit confused on when TCP is used. Is it just that whenever you want to send traffic between two devices, you need to set up a TCP/UDP connection beforehand?

[cedar forum]

more or less, whenever two devices want to have data transferred between they'll establish either TCP (for lossless data) or UDP (for best effort high speed)

which one is used depends on the use case

[ocean bay]

Cool

[cedar forum]

the lines are a little more blurred nowadays but TCP has inbuilt congestion control and resending, UDP is very very basic (more or less "I want this packet to this address")

UDP won't care if it gets there or not, so it's useful for low latency data transfer of things like live stuff where time is sensitive

and you don't want to have to wait for a retransmit if something is dropped (would rather just skip past it and carry on)

for example, voice and video stuff

[ocean bay]

For sure, that looks consistent with what ive read

[cedar forum]

UDP also supports cool things like multicast where you can send one packet to a bunch of different computers all at once

since UDP requires no handshake, multicast just works™️ https://en.wikipedia.org/wiki/Multicast

[ocean bay]

That makes sense since you can just fire off packets to multiple addresses simultaneously as the wiki article says.

But shldnt that also work with TCP? Just have a client establish connections with multiple servers, and THEN multicast?

[cedar forum]

the idea with multicast is it's one packet to multiple servers, that's inherently incompatible with TCP because you need to perform individual handshakes for the TCP connection

it's stateless, UDP is fire and forget, whereas TCP requires you to know who you are sending to and establish a handshake with them

[ocean bay]

Can a client establish TCP connections to multiple servers at once?

[cedar forum]

it can fire off a load of handshakes all at once, but each handshake and connection requires an individually addressed TCP packet

https://en.wikipedia.org/wiki/Multicast_address

a pretty common use of multicast: https://en.wikipedia.org/wiki/Multicast_DNS

[ocean bay]

Oh yeah, I meant that after you establish individual handshakes with all the servers and have connections to multiple servers. Then you can sorta multicast, or am I still missing sth

[cedar forum]

in theory yes, in actuality the TCP specification doesn't make any provisions for that since it's not something that has been necessary

tcp specification is strict on handshake, data, finish, there isn't a whole deal of flexibility outside of that, there are extensions which do a few things but nothing really large

[ocean bay]

This is actually rly cool

[cedar forum]

yeah mDNS is neat

[ocean bay]

Yeah I guess so

[ember ledge]

Joe is always in networking

[cloud spruce]

no, you can't, since tcp is stateful you would need to send the same payload in separate packets for each connection as you have different syn and ack sequence for each connection and are expecting acks in reply or to retry sending that data

with multicast you just set a destination port and a special multicast address as the destination ip address and then you let the switches and routers of the network take care of the heavy lifting of figuring out where they need to send that packet, which for them means keeping state of which next hop destinations has a active subscription for that multicast address, and if that is on different interfaces it needs to copy that packet to each interface (on layer 1) that has such a subscription

[ocean bay]

Mm I see, an individual packet has one sequence number.

So say a client has established TCP connections with multiple servers, if a client fires a packet off to multiple servers, the client cant differentiate which server sent which ACK since the ACK number is the same for all the ACK packets since the sequence number of the client-sent packet is the same.

[ocean bay]

Say I have a web browser and a web server, and I want the web browser to make a GET request to the server for the web page (btw assume this is using HTTP1/2 which still uses TCP). Ignoring DNS lookup, the steps are:

1. The web browser initiates a three way handshake to create the TCP connection with the server
2. The web browser, then sends a request to the server, and the server sends back a response

In step 1, How does my browser know to initiate the three way handshake to start the connection. Is it that when encapsulating the SYN packet, at the application layer, the HTTP Protocol is encapsulated which tells the SYN packet to start the handshake? Might be a silly question haha

Also, ive seen two different HTTP models, HTTP1 and HTTP1.1. In HTTP1, the connection closes after every request-response, and in HTTP1.1, the connection remains open allowing many request-responses through before closing. Now, how does the client and server know how to handle the TCP connections in these ways? Like, is the respective HTTP1/HTTP1.1 protocol encapsulated in every request-response packet, telling the devices how to handle the TCP connection, e.g. for HTTP1.1, the response packet has the HTTP1.1 protocol encapsulated that tells the client it should close the connection?

Thanks!

If my questions don't make sense please lmk so i can clarify

[cloud spruce]

yeah, those steps are correct if you ignore dns lookup and the work of lower layers such as arp (on layer 2) and so on

as for your question, if the browser doesn't have a connection to the server the code requests a new TCP connection from the OS with something like (if it was python):

import socket
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.connect((HOST, PORT))
```which will make the OS setup a new TCP connection (as it's requesting `SOCK_STREAM`) to the server `HOST` on `PORT` by doing a three-way handshake

[ocean bay]

That makes sense, thanks

[ocean bay]

I guess another way to phrase the question is WHERE is HTTP actually located...if that makes sense

[stuck mantle]

Hi meg hidey, the http protocol number is sent as part of the HTTP request, so the server can decide what it wants to do with it

the request line starts like this "GET /images/logo.png HTTP/1.1"

sorry, I think I misinterpreted the question a bit actually : (

[cloud spruce]

the original HTTP 1.0 specification was very basic (even tho it had an even more basic predecessor nowadays called HTTP 0.9 and which HTTP servers should still be compatible with but in reality not many are), according to the standard it would close the connection when after the reply, but as that was very inefficient many servers and clients (such as web browsers) started to implement an unofficial extension to the protocol with the Connection: keep-alive HTTP header, that would later in HTTP 1.1 be part of the official standard specification

the request line of the first (pre-standard) HTTP (now called 0.9) protocol looked like GET /path\r\n, HTTP headers weren't allowed as there were no blank line in the request and GET was the only method available and the server would just send a simple response with just the data, no status line and no headers
in the first standardized version HTTP 1.0 they had added the protocol name and version number and would look like METHOD /path HTTP/1.0\r\n\r\n where METHOD could be GET, POST a few others, HTTP headers where allowed between the request line and the empty line
HTTP 1.1 is similar but has HTTP/1.1 as the version number in the request instead and contains much more features

[cloud spruce]

it's a bit murky, in the internet protocol model it's easy, there it's simply on the application layer as that is the only layer above tcp layer, for the OSI model i think it's officially part of the application layer (layer 7), but some attribute it as the session layer (layer 5) which for parts of the http protocol would make sense, but it's not really part of that layer officially as it does more than that in the same protocol specification, so you can see it as handling all of the layers above layer 4 (5-7)

[ocean bay]

‘according to the standard it would close the connection when after the reply’,

sorry could you complete this line

[ocean bay]

I think my main question is how does the connection know to close after the reply

Oh wait 1 sec

Oh nvm I get it

So for HTTP1, the HTTP protocol is part of the reply and it tells it to close,

and for HTTP1.1, the HTTP header tells it to keep alive

Yeah sorry I didnt fully grasp your post before replying

[cloud spruce]

the pre-standard HTTP would just send the response data and disconnect when it was finished
so would the first standard HTTP/1.0 do as well but could (but didn't have to) add a Length: HTTP header telling the client how many octets (bytes in this context) to expect before the connection was closed so that the client could check that it had gotten the whole response and know that the connection had not been severed too early, but only if there was such a header

to be able to reuse connections implementations extended the standard to add a Connection: keep-alive HTTP header, if such a header existed in the request and the response both the server and client was in agreement to use this unofficial extension and then it was required by the server to also send the Length: HTTP header so that the client would know how many bytes to read from the server until it could send the next request over the same connection, this behavior was later part of the HTTP/1.1 standard specification

there is also such a thing as chunked encoding that extends this even more and is part of the HTTP/1.1 standard as well

[ocean bay]

Thank you so much for the detailed posts, rly cleared things up

[cloud spruce]

just found this, it's more detailed in some areas and less so in others, but still think it would be a good read for you: https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Evolution_of_HTTP

[ocean bay]

Where are the parts of the HTTP request, response e.g headers, body located? Is it this post im replying to?

[cloud spruce]

the HTTP protocol is the payload in the TCP packets
that is, the bytes directly following the end of the TCP header

[ocean bay]

Cool

[iron elbow]

When the Internet is disconnected, except does not work, although errors are displayed without try-except

try:
    https = urllib3.PoolManager()
    request = https.request("GET", "https://google.com/")

except:
    print("Text")```

[cloud spruce]

works perfectly fine for me on a laptop that i disconnected from the network 🤷

[iron elbow]

So the problem is elsewhere

I want to make a thing that monitors the state of the network in my discord bot

[ember ledge]

How can i stop this little twat from DDoSing my network?

My ISP supports dynamic IP’s but when i reset the router he still manages to target it, probably using some sort of DNS method

[cedar forum]

put your stuff behind cloudflare

[ember ledge]

The router is one of those issued sky ones, i don’t think i can do that

[cedar forum]

so are you hosting stuff on your network

[ember ledge]

No, i just can’t connect to the internet on any of my devices

It’s kinda depressing honestly

[ocean bay]

I think im mostly having trouble thinking of HTTP as a protocol. With TCP, I can understand it as a protocol due to its overarching nature over the transfer of data from one device to another. When a handshake is performed, a connection is created and all packets sent through this connection must adhere to the TCP protocol. However, with HTTP as a protocol, being located in a portion of the packets, L5-L7, I find it hard to visualise as this overarching protocol

[cedar forum]

right, how sure are you it's a (D)DoS, I'd contact your ISP

[cedar forum]

TCP is good for transferring data, but makes no provisions on the format of the data within, obviously when surfing webpages it's necessary to formalise how that data transfer happens, so HTTP implements things like:

• HTTP methods to tell the remote server what is going on (GET for fetching content, POST for creating content)
• HTTP headers to attach additional metadata to an outgoing request or incoming response

so on so on, it's a protocol which defines specifically the transfer of web documents

[ocean bay]

So what exactly is HTTP? Is it just a thing that sits on L7~ of every packet intended for HTTP transfer? How is it a protocol

[cedar forum]

because a protocol is just any standardised format for data

HTTP is the protocol used for transferring websites, to make a HTTP request you first create the HTTP packet (e.g. GET google.com/abc), then you wrap that in a TCP segment after performing a DNS resolution on the host from the HTTP packet, then IP packet and so on

[cloud spruce]

while ethernet, ip, tcp, udp and import are all binary protocols, http up until http/2 was a text based protocol just like smtp and pop3 and are easier for humans to read and write

http/2 changed that with compressed headers and channels (so that you can multiplex multiple separate requests and responses in parallel over the same tcp connectin)

[ocean bay]

So is HTTP protocol the whole process of making an HTTP request? e.g. creating the packet, wrapping it, etc.?

[cedar forum]

no, HTTP is the set of rules for laying out the text in a HTTP request/response

that's the protocol

it's just the structure of the text

[ocean bay]

Ohhhhhh

Oh okokok

So when you say an HTTP request, it means the content-type, header, body and wtv else is part of a request, that is structured in an HTTP format

[cloud spruce]

yes

[cloud spruce]

a protocol is just a set of instructions and rules about the format and how to use it

[ocean bay]

Real quick, what's the difference between an HTTP request and HTTP packet here?

[ocean bay]

Yessss it all makes sense now haha

[cloud spruce]

you can see a protocol much like a recipe

[cedar forum]

HTTP can have requests and responses, they're both part of the HTTP packet/format/protocol specifications

HTTP requests are specifically from client->server, HTTP responses are the reply from server->client, both of them are defined in the HTTP protocol

[ocean bay]

Nah I mean isn't the request the url,body,method etc. if so what's the HTTP packet

[cedar forum]

HTTP packet/payload is a general term for any HTTP communication, request or response

a HTTP request is a form of HTTP packet, in the same way that in TCP you've got the SYN, ACK, FIN, etc.

[cloud spruce]

what goes in to the body of requests and responses is outside of the http protocol standard (mostly, chucked encoding is kind of infringing on that and should probably be considered more a part of the headers)

[ocean bay]

Ohh ok cool

[cloud spruce]

html, css, javascript, images and other document formats are what you transport as payloads in the request and response bodies

[ocean bay]

Yepp

Now what is it about the HTTP protocol that creates this request<->response. Why is it that when you specifically format some data (HTTP Packet) with HTTP formatting, that the server must send back a response.

If I send any random data to a server, it won't send back a response

[ember ledge]

They told me, and all they said was turn off your router for 15 minutes

[ocean bay]

but if I format some specific data (the HTTP packet) with HTTP formatting, the server will send me back a response

[ember ledge]

Is there any way i can hide my router IP without using a VPN?

[cloud spruce]

because then it doesn't understand what the client wants, it's like a foreign language and the language that a http server knows is http

[cedar forum]

your router IP shouldn't be automatically known to said person if you aren't doing something which is intentionally going to expose it

if you're not hosting services, are you using a specific app which is revealing it?

and generally no, without a VPN you're not going to be able to hide that

[ember ledge]

i play a lot of multiplayer games

[cedar forum]

most multiplayer games are client-server, that shouldn't leak IPs

[ocean bay]

Ohh, but an HTTP server can receive an HTTP request and choose not to send back a response right?

[ember ledge]

you mentioned something about using cloudflare for the router?

[cloud spruce]

then that might reveal your ip depending on the protocol used in the implementation of the game

[ember ledge]

I play a lot of older games that don’t use servers to hide IPs so yeah

[cedar forum]

yeah, if you're hosting stuff off your router you can, if you're not hosting stuff and your IP is still getting found then it's something else

[ember ledge]

Oh ok

I might invest in a 3rd party router that supports ssh server so that i can configure my own firewall

[cloud spruce]

there are a certain amount of games and game protocols that does client to client connections and that would reveal your ip address

[ember ledge]

Yeah, unfortunately

[cloud spruce]

yeah, sure, but what good would such a server be if it's unreliable with responses?
mostly it will respond with something even if it's an error message saying that it doesn't understand the other party or that you are not allowed to make requests for some reason (not being properly authenticated or coming from the wrong ip addresses might be reasons for such error responses in the 400 to 599 range)

for other types of blocking where you don't send a response you typically use a firewall instead to achive this on layer 3 (ip) and 4 (port)

[ocean bay]

Alright perfect, thanks so much for the help!

[wind oriole]

I've been looking through the code of different crawlers and downloaders like archivebox and Scrapy, and I notice none are using requests

All urllib or, in Scrapy's case, Tornado

Is there a reason for this? I thought requests was supposed to be the best for, well, requests

[cloud spruce]

i think they are trying to avoid bringing in a external third party library as a dependence

[wind oriole]

That’s it? Huh, interesting

[cloud spruce]

i think so at least

[wind oriole]

Was wondering if something about them made it easier to spoof a legit browser request

[cloud spruce]

no, i wouldn't think so, it's just as easy if not easier with requests or urllib3, but none of those two is part of the standard python library and needs to be downloaded and installed to use them

[ocean bay]

In the transport layer, data gets segmented, and each segment has its own TCP header. Is the HTTP request placed in every TCP segment?

[cloud spruce]

no, it's just chopped up in to pieces if it doesn't fit in one segment and each piece gets it's own tcp header

[ocean bay]

Ah alright thanks

[cloud spruce]

the applications that uses L5-L7 on top of TCP will just see the data as a stream of data, but the data will be delivered in chunks with possible delay between chunks when receiving and when transmitting (if the TCP window is full and the other side has not acked enough data yet to be able to transmit more)

[ocean bay]

Could you define ‘applications’

[steady horizon]

They said urllib, which is part of the standard library

[quasi juniper]

when using ansible with VMware, does it matter if i use it on the esxi itself or on the vcenter? in the end it should be the same result, with some variance in how i specify the VMs, right?

[cloud spruce]

yes, i know, i was just saying that the libraries probably are using urllib instead of requests and urllib3 because the first is part of the standard library while the two later aren't, i just didn't say it in so many word

[cloud spruce]

i'm no expert when it comes to vmware, but i would imagine that if you have a vcenter instance all configuration should probably go through that, but i'm just guessing here so take it for what it is

[quasi juniper]

yea i assume so too. its a bit annoying that there is so little info online on vmware with ansible, especially with older esxi versions.

[brittle ruin]

yo wassup guys anyone know of any groups outside of the servee that has meetups or etc. Im new to this all and learning self taught so its kinda lonely learning this all by myself.

[ember ledge]

I can't say that I do

[iron bison]

Is there a site where we can finance the projects of software developers who have ideas about artificial intelligence and deep learning?

[ember ledge]

Is this something you should ask in #networks ?

[errant dome]

why does every flask app i deploy to elastic beanstalk not work

even when i literally copypasta code from someone else i have the same issue

[night shadow]

Do they provide any detailed errors?

[errant dome]

can't find any

this is basically immediately on launching the instance with the files

[static orbit]

Heyo I am trying to make to get a response from a page which is just purely a video. the url is something like .com/123456.mp4. But when I try to make a request from this page using requests, it takes too long to get the response. Why?

[acoustic patrol]

If i use https://www.footlocker.com/api/products/search?query=men&currentPage=1&sort=newArrivals as endpoint there is not every Sneaker listed that are instock. someone know a better endpoint

[somber swallow]

nvm wrong channel

[ember ledge]

what does it mean here when it says Ports: port 8888? like i get the obvious meaning but what are ports and why/how are they listed publically?

[ember ledge]

https://paste.pythondiscord.com/xuvojowetu

yo

how do i get the domain of an ip

I believe its called reverse dns?

[ember ledge]

iirc there was a downloader library

so prob requests loads it into cache

[cedar forum]

you need to look up the PTR records hosted by ARIN, RIPE etc.

they're under the .arpa domain

joe@IVO:~$ dig -x 1.1.1.1

; <<>> DiG 9.18.1-1ubuntu1.1-Ubuntu <<>> -x 1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64744
;; flags: qr rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;1.1.1.1.in-addr.arpa.          IN      PTR

;; ANSWER SECTION:
1.1.1.1.in-addr.arpa.   0       IN      PTR     one.one.one.one.

;; Query time: 0 msec
;; SERVER: 172.24.224.1#53(172.24.224.1) (UDP)
;; WHEN: Thu Jul 21 19:29:58 BST 2022
;; MSG SIZE  rcvd: 87

using dig you can do dig -x

in Python, use this method of dnspython: https://dnspython.readthedocs.io/en/stable/name-helpers.html#dns.reversename.from_address

[fresh vector]

i need to redirect/reverse domain.com to my localhost
to control the api of special program.exe for testing bug on it
with an easy way
Just only 127.0.0.1 domain.com
inside hosts file
the issue is the program will be run as administrator so it's reading hosts file first when it find "domain" program will close..
how can i fix this issue or there is any way to change the default name of hosts for Windows to hostsV2 for example? so the program will read hosts file and windows will change ip of the host from hostsV2 file instead of hosts file
or there is any way to change ip of domain to localhost without using hosts file ?

[twin flame]

im tryna figure out where to find these things

in that

[twin flame]

these are used for this thing

[twin flame]

session id is a constant actually

not the other two tho

also wtf is this stuff

is this stuff relevant to getting the info i want to get cause they are encoded?

[next jasper]

Discord doesn't allow to reverse-engineer it's api or use its user api from outside the provided app. Looking for session id looks like you want to do something shady with it

[next jasper]

The url literally says it's zlib. It's compression, not encryption

[twin flame]

i just want to test commands through code
the session ids seem like constants anyway
and thats not within the app but on the website version

i just wonder what nonce is and the content of the first pic
would all the relevant info be found in the response section and response headers of the Get thing?

[next jasper]

No. You're not allowed to use user accounts to "test commands". Or otherwise manually send any requests with user account. Bot accounts? Sure, you have whole documentation and rules for it - go read them and use a bot account. We won't help here with any ToS-breaking behaviour

What does nonce mean in cryptography?
A nonce is a random or semi-random number that is generated for a specific use. It is related to cryptographic communication and information technology (IT). The term stands for "number used once" or "number once" and is commonly referred to as a cryptographic nonce.

[twin flame]

interesting thnks

[cloud spruce]

just know that reverse dns is unreliable, there is nothing that says that one need to create a PTR record for a A record or the other way around or that the information is correct in any way
also, there isn't many protocols or applications that use them for anything useful on the wider internet (anti-spam for e-mail being one of the few, and some badly designed microcrap protocols/applications)

[cloud spruce]

if you have a raspberry pi or othere computer on the network you can setup a dns forwarding server which can override the responses for some domains depending on configuration (dnsmasq, pi-hole, adguard home are some solutions), some home routers even offer this functionality so it could pay off to check if yours does

[static orbit]

oh so basically it is downloading the entire video? Can I make it not do that?

[ocean bay]

Why is the TCP connection called a TCP connection. To my understanding TCP is just a protocol meaning it just defines a certain format for data. For example, in L4 the TCP 'formats' data by segmenting it and adding additional information to the data in the TCP format aka a TCP header. Yet, in a TCP connection there is no such formatting. My only view on this is that since the 3 way handshake that creates the connection mainly uses the SYN and ACK, which is data located within the TCP header, and TCP is the one that adds TCP headers, it kinda makes sense to call a connection created by this handshake that uses SYN and ACK's from the TCP header, a TCP connection

[cloud spruce]

when an application requestes a TCP connection to a peer the OS will initiate the three-way handshake with that peer and after that the OS will maintain state for that connection, such as syn and ack sequence numbers, window size and scaling, position within the window and other state information and take care of acknowledging packets from the peer as well as retransmission of packets that hasn't been acknowledged within a timeout, unburdening the application from all that housekeeping

[ocean bay]

Thanks, I think I understand the gist of the how TCP connection works.

However, my question is, which may sound kinda dumb, more specifically asking why the TCP connection has TCP in its name. TCP is just a protocol, so it’s just a way to format data, e.g segmentation and adding TCP headers.

Yet with a TCP connection…I can’t seem to understand how TCP is relevant here

[ocean bay]

How does a Layer X device work? For example, with a router as a L3 device, is it just only able to respectively decapsulate and re-encapsulate data until and from layer 3, such as in the image attached? (TCP/IP model)

[cloud spruce]

because there are other protocol that might be used in place of TCP (such as SCTP or UDP to just name a few, even though the latter is not as much of a connection) you might want to signify which protocol is being used for the connection so that one knows what kind of reliability and other characteristics one can expect from communications over that connection

[ocean bay]

But TCP as a protocol (as a way of controlling how data is formatted) is not necessarily relevant within a TCP connection right?

Oh wait actually, is it because you can only transfer TCP formatted data over a TCP connection e.g. data with TCP headers

[cloud spruce]

it might not be very relevant to the network devices in between (even though routers might peek at the L4 information as well if they wish and are smart enough to understand the protocol and for firewalls and many times load balancers it is crucial information as they act upon it) but it is very important to follow the protocol for the other peer to understand the communication