Latency not adding properly

[maiden elk]

Hi there, do you know if the same is happening in a non webgl test?

[random meadow]

Hey yeah I’ve tested both in Unity Editor & with dedicated builds. I am not using a network transform for sync: - I am replicating / reconciling position and rotation in my own CSP functions, which are called every tick. But this still shouldn’t cause any issues, replication latency should still be ~100ms from client 1 —> client 2. I am using state forwarding.

[maiden elk]

When you say your own CSP functions, do you mean using the FishNet CSP system or a custom one?

[random meadow]

Fishnets CSP system

CSP 2.0 specifically with replicate/reconcile attributes

[maiden elk]

Ah I see. Do you have a picture of your TimeManager by any chance?

[random meadow]

Yeah looks like this

[maiden elk]

Hm, that all looks correct to me

[random meadow]

me too 😭 haha

Appreciate your help here 🙏

@maiden elk do you know best way I could go about debugging this myself? Is the Transport for Bayou place to look? This latency definitely seems larger than 100ms right?

[maiden elk]

I would try printing out exactly when it starts to move on both sides in a tugboat build to see if the numbers are adding up

It does look longer than 100ms to me too

[random meadow]

Okay will do. Thanks 👍

[random meadow]

Yeah numbers are not adding up. Same latency with Tugboat.

[random meadow]

How can I debug? Should I be using UNIX time or is there some other way of measuring time a packet was in flight?

[maiden elk]

I was just printing the date time now including the milliseconds

[random meadow]

Any other thoughts on how I could get this latency issue figured out @maiden elk ? My best attempts to debug have proven futile

[maiden elk]

Not really, I don't have much experience with CSP. Perhaps @silk pumice would know?

[silk pumice]

I'm actually taking off for awhile, probably no support from me for the remainder of today. Been working all day, and wrapping up with a release.

Feel free to ping me earlier tomorrow.

[random meadow]

Beautiful thank you my friend - enjoy rest of your day.

[coarse radish]

im doing that for Andor as im having a similar issue on 4.3.7R on Tugboat

[random meadow]

Hey @silk pumice just sending morning ping 😄

[random meadow]

You get same latency issue?

I’ve tried w/ Tugboat and also get the issue so idt it’s a transport issue.

[coarse radish]

its showing around 50-70 and it feels like 300-500

i havent measured it yet

but yeah thats not 50ms

[random meadow]

Wow yeah good to hear someone else is having

[coarse radish]

xD

and with tugboat so a differnt networking plugin

[random meadow]

4.3.8R (released 18 hrs ago) says this:

Fixed PredictionManager.Appended causing overshooting and running non-owner inputs out of order.

I wonder if this has anything to do with it. When I get back to computer I’m gonna install newest version and see if any luck.

[silk pumice]

When using CSP how long until you see others updates will be the following...
ClientB to server time, server to ClientA time, State Interpolation * TickDelta time.

[coarse radish]

but that should be around 50-70ms

which its showing me

but it feels more like 500ms

[silk pumice]

so if you are running 50 tick rate thats 20ms per tick, with a state interpolation of 1, thats 20ms from the interpolation.
If clientB takes 35ms to send to the server and clientA takes 35ms to receive the packet thats 35 + 35 + 20 = 90

make sure youre on the latest then confirm its actually that long between when a tick is sent/run

[random meadow]

I'm on the latest released (4.3.8), and still experience the same issue. I added a Debug statement to my [Replicate] function call to print when "CurrentCreated" was called for a particular replicate data sent from client 1. The ping of both clients is 50ms/70ms respectively, simulation running at 30 ticks per second. For this particular ticks input, it was recognized with "CurrentCreated" state on each device with the following timestamps:

CLIENT 1  - CURRENT CREATED —— 23:56:26.332
SERVER - CURRENT CREATED —— 23:56:26.443 
CLIENT 2  - CURRENT CREATED —— 23:56:26.507

The biggest latency appears to be from client 1 -> server (110ms), but even then I feel like its weird that there is a 64ms latency between server & client 2. delay between when client 1 first processes the input, and when the server gets it. I've run these tests several times and keep getting similar values in that range. For Ping, I am determining its value using "TimeManager.RoundTripTime".

I call 'ProcessInputs' each tick -- looks like this.

        [Replicate]
        private void ProcessInputs(MyInput input, ReplicateState state = ReplicateState.Invalid, Channel channel = Channel.Unreliable)
        {
            if (input.networkedButtons.reloading && state == ReplicateState.CurrentCreated) {
               Debug.Log($"Processing Tick for Replication: {OwnerId} --- {input.GetTick()} --- {DateTime.Now:HH:mm:ss.fff} -- {state}");
            }
        }

I also ran a similar test with RPCs, and still get a latency much larger than ping. Again, not sure what it is. The 992 is just a random value I generated for helping identify the messages. Again here, about 72 ms till server gets message, and then from there almost 90ms till observers receive update.

OUTPUTS:
Client 1
Client Sent Message: 1 - 992 - 00:13:05.294
Client Got Message: 1 - 992 - 00:13:05.458

Server
Server Got Message: 1 - 992 - 04:13:05.366

Client 2
Client Got Message: 1 - 992 - 00:13:05.458

// ------------------ TEST -------------------------

[ServerRpc]
public void ServerRPC_Test(int randVal) {
    Debug.Log($"Server Got Message: {OwnerId} - {randVal} - {DateTime.Now:HH:mm:ss.fff}");
    ClientRPC_Test(randVal);
}

[ObserversRpc]
public void ClientRPC_Test(int randVal) {
    Debug.Log($"Client Got Message: {OwnerId} - {randVal} - {DateTime.Now:HH:mm:ss.fff}");
}

// ------------------ ON TICK --------------------------

// Called every tick.
public override void NetworkUpdate()
{
    MyInput myInput = GatherInputs.GetInputs();
    if (myInput.reloading && IsOwner) {
        // Send a random value as RPC call to server.
        int randVal = UnityEngine.Random.Range(0, 1000);
        Debug.Log($"Client Sent Message: {OwnerId} - {randVal} - {DateTime.Now:HH:mm:ss.fff}");
        ServerRPC_Test(randVal);
    }
    ProcessInputs(tripInput);
}

I'm sure its something that I'm just missing and nothing with FishNetwork itself... but I am calling replicate every tick & reconcililng each tick, and doing everything else according to guides. @silk pumice in any demos you personally have do you get this issue of high latency?

I also initially thought it was my dedicated linux server, but I upgraded to a way more beefy server and still get the same issue.

[silk pumice]

rpcs still send on tick

[coarse radish]

Im running 144

[silk pumice]

so there is also that potential latency I had forgotten to mention

well, thats probably excessive and nuking your server but..

[random meadow]

Right but regardless the numbers still add up rlly wonky. Like in this video the latency is large

[silk pumice]

you can test the rpc approach just by setting timing to variance inside the time manager

[coarse radish]

Eh its an epyc and im playing around

[silk pumice]

also, if both of the clients are not client only you can be looking at more latency

because the tick fires on frame when time is met, but if client sends out as clientHost server 100% wont get it until next tick

just as if server gets something on a tick, clientHost wont get it until next.

[random meadow]

Oh yeah not doing client host. Just 2 clients & a server.

[silk pumice]

this is something that ive been debating changing but for the time being something to keep in mind

[random meadow]

I'm doing build rn w/ Variable time will lyk if changes things

[silk pumice]

I can tell you Ive run tests many times on the latency and it all checked out. the ping isnt 100% an accurate number persay, pings are just estimates after all

test using variable timing + rpcs, that will get you an idea

[random meadow]

Word okay - but I mean do you know of any reasons why latency might be so large here? B/c changing timing to variable would still only save max of like 60ms, but theres still all that additional latency in above video. Following guides from online 1:1 -- not using a Network Transform though, I'm reconciling position & rotation in my own CSP

As in, the Fishnet CSP system.

[random meadow]

And do you get snappy feedback? Or do you also get delay similar to what I have in video above

[silk pumice]

that video is definitely nowhear near 500ms

Not even close

probably closer to 100ms if I had to guess

[random meadow]

Like here is 170ms between when client 1 first creates input and when client 2 receives it, despite each pint being 50-70ms range

I'm just trying to figure out where extra ms are coming from

b/c its not by a small amount, its almost 2x latency i would otherwise be expecting

[silk pumice]

make sure you are on the latest first, as that resolved some order stuff with latest appended feature

[random meadow]

Yeah I upgraded this morning

Or do you have any pointers for the best way I can go about debugging myself? Scripts I should be focusing on?

[random meadow]

Okay yes! Latency issue solved when I use variable time, but only solves it for RPCs. I get about 78ms between when client 1 fires off request and when client 2 receives it.

Is there a way to get similar behavior with CSP functions?

With RPC and Variable Time
12:12:19.116 is when its sent out on client 1
12:12:19.194 is when its received on client 2
78 ms latency in total from client 1 --> 2

With CSP system and Variable Time
12:12:19.120 is when its sent out on client 1
12:12:19.273 is when its received on client 2
153 ms latency in total from client 1 --> 2

whats going on mechanically here?

[random meadow]

anything I could change in my PredictionManager? Any thoughts on this all @coarse radish @maiden elk

[random meadow]

When do State Forwarded replicates get forwarded? Is it as soon as server becomes aware of the replicate, or after it processes the replicate itself?

[silk pumice]

Soon as it's aware

[random meadow]

Right --- so why is there such massive delays then with it replicating data from client to client?

The RPC with variable time doesn't have that issue

[random meadow]

As seen with this ^

[random meadow]

@silk pumice hey -- just submitted a pull request for Bayou Transport if you want to take a look (for some reason it visually added a diff for parts of code I didn't change.... didn't realize that till after I made the pull) 😄 Disabled Nagle's Algorithm though, got a massive reduction in latency on TCP client. Think its a must have for Bayou.

[silk pumice]

Ah, yeahhhh that will often improve high traffic latency for TCP

No worries about the other bits changed probably just IDE formatting.

What sort of improvement did you see?

[random meadow]

I did tests with 100 replicate datas sent. With Nagles algorithm enabled, I got an average of 112ms from client 1 —> server —> client 2. With it disabled, I got that down to 72ms.

Got the biggest dopamine rush when I saw the results haha, been 2 days of trying to debug

And then also, to save on some about 30ms (I was initially getting around 150ms latencys), I set state interpolation to zero on the PredictionManager. I don't think this will cause any issues b/c CreatedStates will never arrive out of order on tcp/ip? I haven't seen any issues with setting it to zero so far.

[silk pumice]

You kniow honestly, I forgot all about the nagles alg

Its been so long since I've worked with TCP sockets

At least and especially in the sense of speed optimization.

[random meadow]

Yeah totally haha. It's all these hyper niche things, easy to overlook especially when your juggling bunch of other things and building the beautiful asset that is FishNetworking

Also the other guys pull request abt case sensitive web socket headers, I also experienced an issue with. HAProxy automatically makes all headers lowercase, NGINX does similar thing, so you get issues when it forwards from your Proxy to your game server backend in the handshake.

[silk pumice]

Yeah that PR should also go on the SWT git. I'll forward it to them.

[coarse radish]

i found out what my problem was btw, my server took like 220ms to give out the answer, idk why but just creating a new serverbuild fixed it

[silk pumice]

that's... kind of unusual.

[balmy parcel]

looks like disabling nagle's alg might also increase total packets sent - @random meadow did you happen to track this? :o

[coarse radish]

prolly some unity garbage or compilation error

[random meadow]

Hey what’s good yeah it does send more packets, that’s the trade off. But for a game you want that ultra low latency state forwarding

if your running a Linux server, there’s also a ton of optimizations you can do Linux side to save milliseconds. There’s a “latency” of its own that exists from when the clients packet arrives on the servers network interface card, and when it gets to the game application running itself. Optimizations there would also save ms latency too

[balmy parcel]

hey thanks for the reply! I am indeed looking for that ultra low latency :)

Do you happen to know what sort of server-side linux optimizations are possible? i'm using edgegap hosting, which means i upload a docker image of the unity linux dedicated server build there

[random meadow]

Oh hmmm not to sure in terms of edgegap. Do you have Kernel access? Also are you building with Bayou or Tugboat?

I got some CRAZYYYY good Linux optimizations to share if u have kernel access 😎

[balmy parcel]

Eyoo sounds epic 😎

How much ping saving are we talking? 10ms? 20?

[balmy parcel]

Both - multi pass! Not sure about the kernel access, let me ask edgegap rq

[random meadow]

Over the last few days I’ve gotten from 150ms latency (150ms from time client 1 creates input to when client 2 receives), to now a consistent 65ms. I am quite certain I can get it lower though. The ping/pong to my server in NY is only 44ms, so half of that is 22ms. So I can get that 65ms to about 45ms latency from client 1 to client 2, there’s a few places to save.

[random meadow]

I just took a Quick Look at edge gap doesn’t look like you do, they provide infra out of the box. I’d assume they probably have taken most of the Linux optimizations into account though with their infrastructure. It’s things like isolating CPU cores, making sure the game isn’t sharing CPU time with other services on the machine, tuning the machine for low latency

[balmy parcel]

Dangggg that’s SICK!! Crazy latency difference

[balmy parcel]

Awww i see, that makes sense though maybe it’s already optimized there

What kind of game are you making?

[random meadow]

Most of my Ms saved came from the TcpNodelay fix (only applies to Bayou). That saved me about 30ms on average. And then another big save came from setting the interpolation delay on Prediction manager to zero. Packets arriving out of order doesn’t matter with Bayou bc with TCP/IP all packets are reliably sent in order. So having the interpolation delay of 1 tick doesn’t make any sense for Bayou, and adds an additional 33ms of latency.

and then I saved ~15 ms from Linux optimizations but it’s variable between 5-15ms saved

[balmy parcel]

Out of curiosity, are you checking ping via TimeManager.Roundtriptime or a rpc call? I applied your PR for bayou (nodelay) and roundtriptime didn’t decrease D:

[random meadow]

Oh no I made my own RTT method — the RRT for FishNet is weird. It’s not super accurate bc it does it according to ticks, not actual system time.

[balmy parcel]

I’m also experimenting with 0 buffer, though due to multipass with tugboat it might cause issues there

[random meadow]

Yeah with Tugboat it definitely might - when u try does it cause issues? I’m curious

You can make your own RTT by adding an extra variable to time managers ping/pong functions. When it sends ping to server, record the DateTime.Now. When server responds, calculate the difference

Much more accurate RTT measure than built one that uses ticks

[balmy parcel]

Ima try today :)

If buffer is client side only though I might be able to change it if the build is exe versus webgl

[balmy parcel]

I’ll try this, thank you!

[random meadow]

My pleasure 😄 Glad I could help

[random meadow]

First person shooter. Migrated from Photon Fusion about two weeks ago.

[balmy parcel]

Sickk why’d you migrate?

[random meadow]

But I love how much more control I have over my networking with Fishnet

[balmy parcel]

^^^ true!

[random meadow]

Fusion is bloatware

Was adding ridiculous DLLs to my game

Which for WebGL game you need to be super performance aware

and Fusion just didn’t gaf about webgl. Super lazy. Bad customer support. Paying way more than I needed to be.

Saved 60% $$ using my own Linux servers vs fusion.

And better cleaner performance. More gradual control. And now I have a better latency.

So worth the switch.

[balmy parcel]

Fairr, fishnet is the only multiplayer solution I’ve tried but I definitely am noticing and enjoying everything you’ve mentioned :)

[random meadow]

Great place to start my friend 👍 It’s one of those situations where open source always beats closed source. Reminds me of when HAProxy came out as open source load balancer and it is just infinitely better than any closed source one.

and being able to code in an edit the fishnet files to tweak things to my games need is a 10/10 feature

Best of luck with things 🙏

[balmy parcel]

So true! So good to be able to look into the source and experiment as needed

[balmy parcel]

Ty, you too!! 🙏