archived-networking

[graceful zephyr]

or in lockstep? no

neither of them use a server generally, and if you do it's only to get the final result of the game out, not for any direct communication with the clients

[glass bobcat]

aight thanks

[split crescent]

also because deterministic lockstep generally means sharing more information with your clients than you'd like to

[glass bobcat]

👌

[graceful zephyr]

@split crescent hmm.. not sure if I agree

Delta Snapshots suffer from the same issues

In that everyone needs all data

I mean sure you can mask data if you really want to

But honestly it's not a huge deal really

[split crescent]

I think the difference is you can do server side culling?

but will admit I'm a little out of my depth there

[weak plinth]

i second going over Unity's FPS Demo (took a team of guys ~2 years), i spent probably a good week going over it. it's a pretty fun learning experience but also a big eye opener 😃

[graceful zephyr]

@weak plinth it's very purpose built

But it's basically a standard delta snapshot

Doesn't detract from it at all, it's very well built

[weak plinth]

gotcha, i'm speaking more from no experience developing a networking lib and getting a view into how much is really involved, even if it's specifically for your game

it certainly woke me up 😃

[graceful zephyr]

Yes it's a great learning resource

@split crescent you can do server side culling, absolutely

I mean it depends on the game if it helps

[graceful zephyr]

@jade glacier o/

[jade glacier]

heyo

[jade glacier]

I can barely get FPS Demo to open, much less run on my laptop... so can't really poke at the net code they have set up in there

[graceful zephyr]

@jade glacier if you don't want to run it just open the code folder in visual studio code or some other smaller text editor which let's you open a folder as a project

[jade glacier]

I can open the source in VS, and the project already has done its 2 hour load up in the editor... so I can "look" at it... I just cant see it in action

Just makes the detective work a little harder without being able to have a running scene

[graceful zephyr]

Honestly I never played it

It doesn't help much TBH

If you want to learn what they did

[jade glacier]

I mostly need to locate the player object and sort out what classes are on it

The networking folder was full of lower level stuff, I am more interested in their tick engine, and where they handle their simulation

[graceful zephyr]

2 sec I'll link it

[jade glacier]

or just the file names of interest if you happen to remember

[graceful zephyr]

https://github.com/Unity-Technologies/FPSSample/blob/master/Assets/Scripts/Game/Main/ServerGameLoop.cs

There's a client one also

But that's the main loop

[jade glacier]

Odd seeing Update() ... are they not making use of an update manager for this stuff?

[graceful zephyr]

They're not using any other update calls

Everything originates from this call

[jade glacier]

Ahh, so this is acting as the update manager of sorts

[graceful zephyr]

Yes

[jade glacier]

Cool, will have to set aside some time to follow these methods around and see what kind of choices they made

[graceful zephyr]

Ping me if you need help with where something is or how it's structured

[jade glacier]

Will do

I am a little confused about Unity's networking roadmap... since their actual networking code seems to be making little to no progress, but this project has working networking.... are they struggling to staff the actual UNET replacement project or something? @graceful zephyr

I assume FPS Sample is running on their replacement transport code, or is that not the case?

[weak plinth]

i believe they wrapped the transport (from the new networking alpha) in their SocketTransport class

i kinda recall hearing in a talk on the fps demo that some of the things they did in the fps demo might find its way into the actual networking lib

[jade glacier]

yeah, thats what I am thinking is likely going to happen. They will be asked to turn their net code into more generalized HLAPI stuff

[weak plinth]

but yeah i've had the same concerns about the networking lib, since the advice they gave people writing games was to wait if they were releasing post 6 months out mark (i think that was 3 months ago?). but i only see the lowest level and no updates in the repo

[jade glacier]

Are they talking about yet another HLAPI?

All of the HLAPIs keep looking about the same... all RPC/Syncvar/Message based, and nothing that really focuses on giving developers a solid tick synced frame based HLAPI that can easily be made to work for determinism, input syncing, rewind, resim... etc

Unity doesn't need any more Syncvar and RPC libraries 😃

[graceful zephyr]

@jade glacier so... the fps sampel can use both unet transport and the new transport

it doesn't care, it just needs unreliable sends

literally

[jade glacier]

They abstracted it, yeah... as everything really should be now

[graceful zephyr]

Afaik their are not going to do a new HLAPI

but more 'dedicated' solutions

to various game types

[jade glacier]

most of our libraries as we wait for these things all are coming down to byte[] in and out

but, thats not solving the shortage of a proper state engine starting point for new developers

we have 1000 guys like myself hacking together our own

[graceful zephyr]

its incredibly hard to make a generic state engine that is easy to use and flexible enough

[jade glacier]

true, but some of the stepping stones can be made at the MLAPI level I would think

not the usual RPC Syncvar level crap

[graceful zephyr]

rpc/syncvar is just... the worst model imaginable

[jade glacier]

but a "here are the ticks and here is the byte[]" .... serialize your heart out

[graceful zephyr]

or at least the way it was implemented in unet

@jade glacier if you look at the fps sample in detail, you'll see why it's not as easy as just letting people serialize into byte buffers

[jade glacier]

I know it opens the doors for massive breakage

[graceful zephyr]

it needs a lot of meta data around

[jade glacier]

you can do safety bloat like PUN does

[graceful zephyr]

which isn't easily... made generic

[jade glacier]

but PUN too... NO tick engine concept, just messages

[graceful zephyr]

i think bolts general model is great

but it hides to much behind magic

[jade glacier]

yeah, Bolt is what I still say is the only proper tick engine available

but that library is sealed up tight

[graceful zephyr]

i mean i started bolt in unity 3.x

so... 😄

[jade glacier]

yeah, not a slight to you at all, its still stands

as somehow the only fucking tick engine

every one in networking chat is trying to sync inputs and work out how to resimulate....

its super needed

[graceful zephyr]

import component CustomComponent;

compressor GameVector3 : Vector3 {
    Owner {
        
    }
    All {
        X {
            Min = -1024;
            Max = 1024;
            Accuracy = 0.01;
        }
        Y {
            Min = -32;
            Max = 128;
            Accuracy = 0.01;
        }
        Z {
            Min = -1024;
            Max = 1024;
            Accuracy = 0.01;
        }
    }
}

component replicated Transform<V, Q> {
    Vector3<V> Position(All);
    Vector3<V> Velocity(All);
    Quaternion<Q> Rotation(All);
}

component replicated Inventory(CanSeeAll, OnlyItemType) {
    InventorySlot[20] Slots(CanSeeAll, OnlyItemType);
}

object InventorySlot(CanSeeAll, OnlyItemType) {
    Int32 ItemId(CanSeeAll | OnlyItemType);
    Int32 ItemCount(CanSeeAll);
}

archetype Player {
    use predictable Transform<PlayerVector, PlayerRotation>;
    use Inventory(Friendly, Enemy);
}

archetype Chest {
    use Transform;
    use Inventory(CanSeeContents, None);
}

filter Transform {
    has Transform;
}

here's my prototype DSL for specifying state/compression/filters/etc.

it's not finished at all, it needs a lot more work

[jade glacier]

Out of context, not sure I get most of that

[graceful zephyr]

its a 'custom' language

[jade glacier]

I recognize what they hint to

[graceful zephyr]

for defining networked state

basically

[jade glacier]

but not how it fits into anything 😃

[graceful zephyr]

it compiles into a .NET DLL

[jade glacier]

gotcha

[graceful zephyr]

so i wrote a custom parser/trans-compiler for it basically... so it can compile into a .net dll

still not decided if i'm gonna emit C# and then run the C# compiler, or emit a DLL directly via Cecil

[jade glacier]

That's coding and operating at a level above me 😃

[graceful zephyr]

the idea is pretty simple

you define components

like transform, inventory, etc.

and then you define entities/archetypes/state (whatever i end up calling them) which are groups of said components

[jade glacier]

Defined in code I assume, since this isn't inspector based

[graceful zephyr]

yeah, define in the DSL i linked

the DSL is then compiled into a .NET DLL via a tiny editor tool

and it will generate code like this into the DLL:

    [StructLayout(LayoutKind.Sequential, Pack = 4)]
    public unsafe struct Player {
      void* _ptr;

      public Transform Transform {
        get {
          Transform transform;
          transform.Ptr = (__Transform*)(((Byte*)_ptr) + Maths.RoundToAlignment(sizeof(__Entity)));
          transform.EntityState.Entity = (__Entity*)_ptr;
          transform.EntityState.ComponentIndex = 0;
          return transform;
        }
      }

      public Inventory Inventory {
        get {
          Inventory inventory;
          inventory.Ptr = (__Inventory*)(((Byte*)_ptr) + (Maths.RoundToAlignment(sizeof(__Entity)) + Maths.RoundToAlignment(sizeof(__Transform))));
          inventory.EntityState.Entity = (__Entity*)_ptr;
          inventory.EntityState.ComponentIndex = 1;
          return inventory;
        }
      }
    }

[jade glacier]

Something I can read finally 😃

[graceful zephyr]

which can then be used like this:

     Player p = new Player();

          var slot = test.Slots[0];
          slot.ItemCount = 1;
          slot.ItemId = 1;

          var transform = p.Transform;
          transform.Position = Vector3.one;
          transform.Rotation = Quaternion.identity;


          p.Transform.SetPosition(Vector3.one);

          var slots = p.Inventory.Slots;

          for (Int32 i = 0; i < slots.Length; ++i) {
            var s = slots[i];
            s.ItemCount = i;
            s.ItemId = i;
          }

[jade glacier]

Following in a general sense yeah

[graceful zephyr]

Still not decided on syntax fully yet for any part

[jade glacier]

Totally different world than I live in, so mostly looks alien. I never compile anything so that all looks like voodoo

[graceful zephyr]

I would like to do it without the dsl and compiler

But it's so many things you'd need to do by hand then

In code

Or you need to trade away executing speed

[jade glacier]

I don't see my networking skills ever getting past giving objects Serialize() Deserialize() Interpolate() and Rewind() methods, but thsi stuff is fascinating

[weak plinth]

so if it's a replicated transform type and you do p.Transform.SetPosition the replication is taken care of for you?

[graceful zephyr]

@weak plinth yes

[weak plinth]

oh my god i want it now

[jade glacier]

yeah, the end result is magical

[graceful zephyr]

That's what photon bolt does if you want something like that now

i know you know that @jade glacier that was for @weak plinth

:p

again this is super early, and i'm still changing things all over the place

[jade glacier]

I own bolt, I just hit walls too quick with it and exits maintenance of it and its docs.

I tell everyone how great it is, I just don't use it LOL

[graceful zephyr]

Here's the FPS we built in Quantum

just asset store/programmer art for now

[jade glacier]

I can't judge an FPS really without actually playing it - since the devils of netcode are in the details of playing

but I assume its rock solid

[graceful zephyr]

absolutely

[weak plinth]

what's the decider between PUN & Bolt?

[jade glacier]

PUN and BOLT have photon in common, and that is about the end of it

[graceful zephyr]

@weak plinth Bolt is aimed at basically a specific target: authoritative fps games, or games in that style... it can be used for other games of course (and it has)

[jade glacier]

PUN is client to client messaging through a relay

[graceful zephyr]

but it has a lot of stuff which you wont find anywhere else, like systems for authoritative movement, lag compensation for shooting, etc.

but bolt has ONE way of doing everything, and you can't deviate from it easily

[jade glacier]

Bolt is a full on engine

[graceful zephyr]

PUN is much more open ended where you just serialize stuff and deal with everything yourself

[weak plinth]

ahh ok, overkill for my mostly text party game atm :), but i'm gonna make a note of it

[jade glacier]

Would be more of a hindrance than a help for that kind of game

[graceful zephyr]

@jade glacier we're mostly going for mobile fps/tps games with quantum, not trying to do PC fps games.

[jade glacier]

What the Unity world needs and I am not quite good enough to produce in a timely fashion is a input sync + tick engine that is library agnostic

[graceful zephyr]

Yeah... it's a lot of work

and a lot of hard hard work

to get that right

[jade glacier]

@graceful zephyr Is there a reason to not make PCs a target? Seems if it works on mobile, it will slay on PC?

[graceful zephyr]

@jade glacier PC fps games are built in a certain way, and people expect it to work in a very very specific manner

[jade glacier]

@graceful zephyr Lot of work, but its also where 90% of Unity new networking projects are mired

[graceful zephyr]

mobile fps is easier since it's slower paced, slower aiming, etc.

there are some very very technical/specific considerations which makes quantum a good target for mobile and console fps, but not pc fps ... right now, we're going to solve those also

but initially we target mobile

[jade glacier]

Ahh, this probably goes back to my previous questions about client prediction in a deterministic game.... seems like its going to be tricker

[graceful zephyr]

like the current game plays just FINE on PC, but yeah... some very very technical details we need to figure out to make it rock solid even for PC

[jade glacier]

The whole reason non deterministic works so well is you can have the players all be in different timeframes and disagree

[graceful zephyr]

and very few high end FPSes are built in Unity

for PC

but for mobile... almost all of them are built in Unity

so a much bigger market

[cosmic atlas]

heh

[jade glacier]

is full determinisim really doable for a fast paced FPS ? I can't get my head around how client prediction works when the player locally has to agree with the server 100%, the whole favoring the shooter thing kind of goes out the window no?

@graceful zephyr

[graceful zephyr]

yeah it is

there is no server :p

but all clients have to agree

[jade glacier]

sorry, with the "reality" of things

[graceful zephyr]

and you can still do favor the shooter

[jade glacier]

Locally I will be ahead of time though still no?

so my local position in the world isn't the true position

[graceful zephyr]

no, you will all be at the same time, but it will be predicted time

but you will all be at the same point in time

[jade glacier]

so you predict the entire world

[graceful zephyr]

yes

[jade glacier]

gak

[graceful zephyr]

and roll back the whole world also

[jade glacier]

well that explains the missing thing I wasnt getting... but holy dick is that going to be tough to make tight

extrapolation = noise

[graceful zephyr]

this video

is one client watching another client

with ~230 ping

full prediction of 13-14 ticks (so 14*16 milliseconds)

0.22 seconds of prediction

[jade glacier]

The extapolation aspect is why you were saying slower moving tends to be desirable

[graceful zephyr]

but there's no extrapolation/prediction errors visible

[jade glacier]

though it seems the real concern is fast changes in user input

[graceful zephyr]

even though the other client is moving erraticaly

watch the video

im moving super erratic, still looks perfect

both clients have 230-240 ping there

[jade glacier]

Would be helpful to see that with a green ghost showing where it missed the prediction?

There must be some rubber banding happening

[graceful zephyr]

absolutely, but the goal here was just to show that we can correct massive prediction errors without any visual errors

this was just a video i recorded for internal showcase thing

[jade glacier]

And so in this... if you shoot the bad prediction... how does that get passed along as a hit when others will disagree?

[graceful zephyr]

it uses the same type of system as battlefield

but instead of the server doing a plausability check, all clients do it

[jade glacier]

Ahh, gives the client a little more authority to make up for it

not a hack risk giving it that much wiggle?

[graceful zephyr]

nah, it also has the benefit of providing pixel accurate shooting for the clients

which games like counter-strike or CoD don't do

so it's a fairly big benefit also, in that shooting always is as accurate as possible

plus we have the ability to do full rewind checks due to how quantum works internally, we can go back in time to say 10 frames ago and shoot there if we need to

[jade glacier]

But it can't check your shot based on your view angle across the network... it has to accept a "No seriously I shot this guy in the head" as fact across the network if he was pointing in the general direction?

Or is the plausibility test accounting for extrapolation?

and it actually tests objects as they would have extrapolated?

[graceful zephyr]

it accounts for the extrapolation, it will send the view angle also, so it's not just that you were facing the general direction

your crosshair has to line up

[jade glacier]

ahhh, ok

then now I get your prediction model

[graceful zephyr]

quantum has a buffer of all states for the past second in the memory, for the entire world

so we can go back in time on all clients, etc.

to check for hits, etc.

[jade glacier]

didnt realize you were extrapolating for starters, and didn't realize you were giving client that kind of authority with checks based on possible extrapolations

pretty advanced stuff to code, but I get the concept now

[graceful zephyr]

it's a lot of technical details, there's other things which are harder to explain without a fairly deep understanding of what quantum does internally in relation to game state, input, etc.

[jade glacier]

I'm not looking to create anything like it, just couldnt get my head around the prediction model in determinism

hadnt ever considered extrapolation to be viable

[graceful zephyr]

it's just not straight up dumb extrapolation either

it doesn't just forward the player based on velocity

or something 'stupid' like that

[jade glacier]

yeah, I assume you have all kinds of catmull equations, slerping, acceleration lerps etc

[graceful zephyr]

no i mean, the extrapolation is much 'bigger' than that

it's the complete game state that is extrapolated

got a meeting now

[jade glacier]

Ahhh

[graceful zephyr]

bbl

[jade glacier]

time to make bfast for the woman

lates man

[graceful zephyr]

o/ always enjoyable talking to you

have a good day

!

[jade glacier]

u2

[jade glacier]

Networking is a really hot topic on this server....

[stuck void]

it is with you and fholm around 🐸

[weak plinth]

I usually pass my questions to the Photon Server forums though 😁

[compact bramble]

This is fholm's house though, it's his channel so make sure you leave latency at the door.

[graceful zephyr]

@compact bramble 😄 lol

swings his axe around

[last hornet]

don't talk about how you serialize uncompressed values though

they'll bitpack shame you

[graceful zephyr]

@last hornet I don't :(

[last hornet]

nah that might just be @jade glacier 😆

[jade glacier]

That's me

And I will :) if it's a release.

[compact bramble]

Lets have a little bet on when Unity's own new networking will be mature enough for a released 16 player FPS title. I'm guessing early 2020.

[split crescent]

that seems pretty optimistic

is it clear that its not vaporware yet?

[compact bramble]

It's clear by the multi million investments into server infrastructure - the same servers that power pubg and other AAA games, as well as buying the most widely used VOIP solution for AAA games (pubg/fortnite/LoL).

buying all that indicates some degree of seriousness

Either that or Unity has much more money than any of us dare imagine.

[jade glacier]

The work on FPS Sample will likely help drive that particular goal

[graceful zephyr]

@compact bramble well that's more of a hosting and services thing tho

Not multiplayer code

[compact bramble]

Sure but it implies the code is a priority or Unity just blew millions and someone got fired so hard that they exploded in 0 ms

Cos no code, no cash back.

[graceful zephyr]

@compact bramble all of the services so far have explicitly been state that it doesnt rely on their networking code, that it's agnostic

and they get no additional rev just because you run on their networking stack

[compact bramble]

They can't rely on third party to use their own infrastructure, so they will have to provide their own solution which while will still be optional, will also be pretty much the only way they can monetise, so they have no choice but to roll out a quantum-killer

it will be delta compressed pistols at dawn

or would be if you didn't stand over them as they slept with an axe

[graceful zephyr]

@compact bramble well quantum is hardly a generic solution for every game 😄

[compact bramble]

so posh... I used to think you were a viking. Perhaps I was wrong

hardly a generic solution! only for posh people 😉 ❤

avoids an axe

[weak plinth]

would player auth, BaaS / custom, that sort of thing be out of scope for #archived-networking and more #💻┃code-beginner ?

mostly looking for suggestions for a good BaaS that can support friends (not just 1 way, like "following"), a party system (long-polling hopefully for party updates, so i can queue people together and keep groups synced across clients), and good auth w/ social auth. (i tried PlayFab, but that's what got me started writing my own)

[warm jungle]

If anyone has time to look over a noob, I can't figure out my issue. Using local host I can connect fine, but now when I try to use my ip i cant connect to the server being hosted and i cant figure it out. If anyone could point me in the right direction id appreciate it.

Also fair warning, code is pretty bad cause im testing right now
If you do want to glance at it, in all its horror, here are the key files
Client: https://hastebin.com/ixuxobinen.cpp
Server: https://hastebin.com/ulivegoqan.cs
UnityNetworkHandler: https://hastebin.com/davuqiguyi.cs (This one is preeeetty bad)
NetworkCanvas: https://hastebin.com/okufisewev.cs
Player and PlayerInfo: https://hastebin.com/iloniyoheq.cs

[cosmic atlas]

Any recommended compression classes or methods to use? Most of my network messages are small, but wouldn't mind profiling a couple just to see if there's any significant bandwidth gain (and CPU processing)

[warm jungle]

@jade glacier has a good one on the asset store

[jade glacier]

https://assetstore.unity.com/packages/tools/network/transform-crusher-free-version-117313

[jade glacier]

@cosmic atlas

[cosmic atlas]

Thanks, I'll take a look

[jade glacier]

The example scenes should make usage pretty clear, of not just ping me @cosmic atlas

[cosmic atlas]

Thanks, yeah just skimming the docs between other google searches haha
Probably am interested in serializing byte[] arrays, but could see if there's any optimisations for .PutFloat, .PutUShort kinda methods too, or for strings as well
I don't do any MonoBehaviour/Transform/Component magic syncing

[jade glacier]

its core is a float compressor

doesn't do strings, but it has a bitpacker in it, its an extension for byte[]

so you can just myByteArray.Write(valueOfAnyType, ref positionPtr, numOfBits);

there are three bitpackers in it actually, one for arrays (like byte[]) one for packing into primitaves like ulong, and a struct type called Bitstream that is a universal adapter between all of those

but the byte[] extension sounds like what you are after

for the float compression though, you want to make use of the Float Crusher

[cosmic atlas]

kk ty 😃

[graceful zephyr]

morning o/

@warm jungle well you shouldn't be sending strings as the event 'keys;

[warm jungle]

yes i know that lol, i figured that out and im redoing all my systems cleaner now using enums @graceful zephyr

[graceful zephyr]

so there's nothing wrong with your structure technically, it's like a good first thing

but it deals with stuff to.... manually

[warm jungle]

yeah, im changing that up now

otherwise its too annoying to add more objs

[graceful zephyr]

well... that's not really what im getting at

hold on, i'll show you an example

@warm jungle so like this... i can define an entity like this:

    public class Character : EntityAuto {
      public VectorProperty Position;
      public QuaternionProperty Rotation;

      protected override void SetupProperties() {
        Position = new VectorProperty(this, nameof(Position));
        Rotation = new QuaternionProperty(this, nameof(Rotation));
      }
    }

Then register it like this: _context.RegisterEntityAuto<Character>();

and create and use it like this:

        var character = _context.CreateEntity<Character>();

        character.Position.Value = new UnityEngine.Vector3(0, 4, 0);
        character.Rotation.Value = UnityEngine.Quaternion.identity;

and it doesn't have a concept of like 'a position update message', etc.

it just deals with the entity as a whole

position and rotation will automatically replicate when they are changed, etc.

no need to send like 'oh i want to update position'

it's too hard to deal with a system that does that

[warm jungle]

I.. dont quite understand

[jade glacier]

I should consider switching to interfaces like that rather than derived network classes

[warm jungle]

Is EntityAuto a monobehavior? is this supposed to go on an obj youre spawning? To track those values? Or is Character a variable inside a MB class to track them or something? Does it somehow do auto syncing? I had some ideas for that. Im just not sure exactly sure what you mean, and by like "Replicate", and how I would do it other than "updateposition" and shit @graceful zephyr
Im going to bed tho, so you can reply tomorrow or tonight and @ me here or in loaf and ill reply or look at it then

its 2am so

[graceful zephyr]

@warm jungle lol we can do this when you wake up

[warm jungle]

dope

[jade glacier]

For you auto serializing wrappers like vectorProperty, where does the codec end up residing? Does every instance store it's own multiplier and De-multiplier? Our do you have a means for sharing like codecs?

@graceful zephyr

[graceful zephyr]

@jade glacier the serializer is only generated once per entity type and shared

[jade glacier]

Cool, was wondering at a glance where that was happening

[graceful zephyr]

@jade glacier it happens inside the constructor of the property types

[jade glacier]

I HAD to jump through making hacks to make that work for my asset because of the inspector aspect

Major

Christ mobile typing is trash

[graceful zephyr]

All this code is outside of unity doesn't even care if unity exists

[jade glacier]

Dreamy

90% of my time is spent hacking around the unity inspector

[graceful zephyr]

my new system will use a DSL and code generator/compiler instead

there's too much hoop jumping to make this work from straight C#

[jade glacier]

I can imagine

[graceful zephyr]

i mean it's not tha dbad, i'll show u hold on

    public VectorProperty(EntityAuto entity, String name = null) {
      _entity = entity;
      _offset = entity.ValueCount++;

      if (entity.AutoSerializer.PropertySetup) {
        entity.AutoSerializer.RegisterSerializer(new VectorPropertySerializer(name, Axes.All, default(FloatCompression), default(FloatCompression), default(FloatCompression)));
      }
    }

that if case only executes when it creates the first entity of that type

its that simple really

[jade glacier]

I've actually gone the passcode route with mine

[graceful zephyr]

how do you mean

[jade glacier]

All codecs produce a hash based on settings, so similar codecs result in the same hash

[graceful zephyr]

ah

[jade glacier]

So I see if a hash exists yet for the particular codec... If so what gets used as shared

[graceful zephyr]

for the DSL im looking at a syntax like this:

compressor GameVector3 : Vector3 {
    Owner {
    }
    All {
        X {
            Min = -1024;
            Max = 1024;
            Accuracy = 0.01;
        }
        Y {
            Min = -32;
            Max = 128;
            Accuracy = 0.01;
        }
        Z {
            Min = -1024;
            Max = 1024;
            Accuracy = 0.01;
        }
    }
}

component Transform<V, Q> {
    Vector3<V> Position;
    Vector3<V> Velocity;
    Quaternion<Q> Rotation;
}

entity Character {
  Transform<GameVector3, Default> Transform;
}

[jade glacier]

Unless it's not a static codec and the seeing can be changed.... Then it's a whole other mess

[graceful zephyr]

the generic arguments are the compressors basically

[jade glacier]

Christ I would love to return to that clean a code

[graceful zephyr]

i mean this is a custom language that i have to compile to a .NET DLL by hand

so it's not like this is something C# supports or can be done in C# by default easily

it can also have different compressor clauses based on say which team your on, if you have control of the character, or if you're further away

[jade glacier]

https://gdl.space/tuxehifiha.cpp

as a reference of how convoluted things have gotten... this is JUST the float compressor

[graceful zephyr]

#define WORLD_MIN -1024
#define WORLD_MAX +1024

#define WORLD_MIN_Y -32
#define WORLD_MAX_Y +128

#define POSITION_CLOSE_ACCURACY = 0.01
#define POSITION_FAR_ACCURACY = 0.1

compressor GameVector3 : Vector3 {
  Close {
    X { Min = WORLD_MIN; Max = WORLD_MAX; Accuracy = POSITION_CLOSE_ACCURACY; }
    Y { Min = WORLD_MIN_Y; Max = WORLD_MAX_Y; Accuracy = POSITION_CLOSE_ACCURACY; }
    Z { Min = WORLD_MIN; Max = WORLD_MAX; Accuracy = POSITION_CLOSE_ACCURACY; }
  }
  Far {
    X { Min = WORLD_MIN; Max = WORLD_MAX; Accuracy = POSITION_FAR_ACCURACY; }
    Y { Min = WORLD_MIN_Y; Max = WORLD_MAX_Y; Accuracy = POSITION_FAR_ACCURACY; }
    Z { Min = WORLD_MIN; Max = WORLD_MAX; Accuracy = POSITION_FAR_ACCURACY; }
  }
}

[jade glacier]

You do your level of detail stuff in the cormpressor itself?

[graceful zephyr]

so you could do this for example

and it will switch those depending on which flags you set on an entity for a specific connection, etc.

yeah that's a lot of code lol

[jade glacier]

that used to be my simple base class, looked more like what you have above

but then... the inspector

[graceful zephyr]

inspector code... sucks

why im doing all text files this

and code driven

[jade glacier]

It could use a housecleaning, but it works so I am afraid to poke at it

[graceful zephyr]

i know the feeling 😄

the simulator in quantum is like that... the one that actually does the predict/rollback

it works perfectly

but its like 2000 lines of REALLY complicated C#

and i'm so scared to poke it 😄

[jade glacier]

yeah, you forget writing most of it

and it was written so quickly it didnt get documented... so you know it works.. but you forget why

[graceful zephyr]

i changed it a few months ago and introduced a prediction bug which would cause the simulations to drift... but only like a 1/100000000 chance

that was fun to find

[jade glacier]

ack

does anyone else understand that code base?

[graceful zephyr]

that specific code, i dont think so

like that class

no i dont think so

[jade glacier]

or if you die in train wreck is that whole project boned?

[graceful zephyr]

commonly called Buss Factor

i mean my colleague understands most of it i suppose, or at least knows what it does but maybe doesnt have a full understanding of the code itself

hes really good so

[jade glacier]

We had to have key man insurance on ourselves when we owned our old company for that reason

[graceful zephyr]

i have that on myself in my company

[weak plinth]

lol what is key man insurance?

[graceful zephyr]

@weak plinth basically the company cant survive without this person so we're gonna insure him so that we get a payout incase something happens so we get money to rescue the company

[jade glacier]

insurance for if a key member of a company is lost... that it WILL wreck the company and cost a lot to clean up

[weak plinth]

holy crap

[graceful zephyr]

my company is dead without me

like it's just... gone

[jade glacier]

Pretty common in small companies

[graceful zephyr]

yeah

[jade glacier]

where usually there are a couple of people the whole thing ride on the backs of

[graceful zephyr]

how much insurance did you have on yourselves?

[jade glacier]

million each I thnk

[graceful zephyr]

almost same as mine then ye

[jade glacier]

it was required by some clients I recall

I forget why we did it, but it was required

[graceful zephyr]

for me it's more a way to make sure that if osmething happens, wife/kids are okey, since wife is the benefactor to my company

[jade glacier]

yeah, different benefactor... same concept

[graceful zephyr]

and its much cheaper to take a keyman insurance than a private life insurance for that amount

[jade glacier]

Though if its just the family, can you just do life insurance instead?

I guess you could survive...

[graceful zephyr]

i have a private one also, but that's just to cover the house basically

[jade glacier]

and be alive but not producing

[graceful zephyr]

yeah

the keyman clause covers a lot more

then just 'u ded'

[jade glacier]

makes sense, key man I would expect to be better for most stuff

[graceful zephyr]

but yeah i have private life insurance also

but not as much

[jade glacier]

since your work = who you are for the most part

[graceful zephyr]

as that'd be stupidly expensive

at least here

im trying to figure out a way to override the compression with user code

incase people want to do weird stuff (they always do)

[jade glacier]

make a standardized compression interface that your compression uses as well?

I don't know your entire structure... so really I got nothing 😃

[graceful zephyr]

well so

[jade glacier]

and nothing I suggest you wont have already though of LOL

[graceful zephyr]

so one of the reasons i want to use code generation

is that i can generate serializers like this

      public static void Serialize(void* data, DirtyInfo info, Connection connection, BitStream stream) {
        // Transform
        var transform = GetTransform(data);
        if (info.IsDirty(transform->_PositionId)) {
          stream.WriteVector3(Transform->Position);
        }

        if (info.IsDirty(transform->_RotationId)) {
          stream.WriteQuaternion(Transform->Rotation);
        }

        // Inventory

        // ... snip
      }

and that is so much more efficient

than having some generic loop which checks property types, etc. calls into virtual methods, etc.

[jade glacier]

yeah, clearly

[graceful zephyr]

my old serializer code, which is written in C# which uses the structure form the character entity i showed before

the core loop is this

        // write properties into packet
        while (_propertyHeap.Count > 0 && written.Count < _propertyPacketCountMax) {
          // start position
          var s = packet.GetHandle();

          // peek property
          var p = _propertyHeap.PeekValue();

          // gotta hold
          Assert.Check(p.Index <= _propertyIndexMax);

          // write property index
          packet.WriteInt32(p.Index, _propertyIndexBits);

          // write property data
          _properties[p.Index].SendProperty(connection, entity, entity.Values, packet);

          if (packet.Overflow) {

            // restore handle (we could not write this property)
            packet.RestoreHandle(s);

            // break out and dont attempt to write anything else
            break;

          }
          else {
            // pop it
            _propertyHeap.Pop();

            // clear from dirty set
            link.Dirty.Clear(p.Index);

            // clear out priority
            link.Priority[p.Index] = 0;

            // add to written list
            written.Add(p);
          }
        }

[jade glacier]

yeah, that looks more like what I am used to

[graceful zephyr]

but the problem is that there's a ton of just... shitty housekeeping to do

because i dont know the properties, etc.

[jade glacier]

You are trying to sort how to introduce user code/custom serialization into your codegen?

[graceful zephyr]

yes

because i've been doing this long enough that i know people are going to want to do weird shit

that i didnt think of

and providing a way for them to do that is necessary

[jade glacier]

Can it just slip a callback method into that serialize method if they flag it as custom?

[graceful zephyr]

even if it's just custom compression

or they want to serialize stuff themselves completely

for various things

[jade glacier]

yeah, they probably will

[graceful zephyr]

also with my new method of doing delta compression

i realized that i can just write the data for an entity once

[jade glacier]

I recall with bolt that being like a question I had for you 10 mins into it

[graceful zephyr]

and then memcpy it to all connections that need the same update for the same entity

so i only need to actually serialize once (in most cases)

[jade glacier]

Was that not already the case?

[graceful zephyr]

no

[jade glacier]

memcopy requires byte alignment rather than bit correct?

[graceful zephyr]

correct, but that can be solved

[jade glacier]

yeah, the bookends just need some masking is all if I am reading you

[graceful zephyr]

even if i need to manually loop over it to insert the bytes, it's still faster than re-doing the serialization by hand

as i can copy it in 8 byte blocks

[jade glacier]

still slower with everything shifted though no?

yeah, thats what I recall from making my ulong[] parts of my bitstream

not using memcopy, but same concept

[graceful zephyr]

its about 2.5x faster using ulong[] than byte[], but even then compared to re-running the whole serialize method

just being able to go into one method, copy all data

is going to beat the serialization even if its in byte[] form

even if i cant use memcpy due to wanting bit instead of byte alignment

[jade glacier]

I was assuming you were doing that already, wasn't that the point of the two step process?

[graceful zephyr]

no all my old libs have done individual serialize calls for each entity for each connection

as they had no easy way of saying "this entity has the exact same update for client a and b"

but the new system does

[jade glacier]

ahh, that part I missed, the ability to identify entire connections as getting the same everything as another

[graceful zephyr]

yeah

[jade glacier]

seems like that is getting pretty deep into edge cases though?

[graceful zephyr]

i mean they needing the same update for a specific entity , not the whole world

[jade glacier]

ah, I was just wresting with that if I understand you

my issue is with keyframes

[graceful zephyr]

i suppose for people that want to overwrite the serializer i could do this (C#):

    public static class MyCustomSerializer {
      public static void Vector3(Entity* entity, Vector3 position, Connection connection, BitStream stream) {
        ... write custom serialier code ...
      }
    }

and then in the DSL:

  Vector3<MyCustomSerializer.Vector3> Position;
}```

[jade glacier]

and how delta frames (culled data frames, not delta like you are talking) get sorted when you aren't sending everything

[graceful zephyr]

yeah that'd work

[jade glacier]

Just let them hardcode it in, seems reasonable

since they are not newbs anyway

[graceful zephyr]

thats for compression/custom serializer at least

but not for custom data

[jade glacier]

If the code for like Vector3Property is exposed to them, I would think they would make their own variations with their own names?

[graceful zephyr]

the properties wont exist, that was the old system

[jade glacier]

Ooh

[graceful zephyr]

like the Vector3Property stuff

that's from the old system i scrapped

[jade glacier]

I'm not much help here, would really have to use the whole system as a developer to really give you constructive feedback on usability

[graceful zephyr]

yeah for sure, im just throwing around ideas

[jade glacier]

I would just think of it from the workflow perspective is all... like if they were using your system as you designed it.. and then decided they wanted to custom serialize something.... what code would they be looking at already

[graceful zephyr]

that's what im trying to figure out, exactly what they would do in the case they want to serialize something completely bespoke

like a fucking Dictionary<String, String> or something

[jade glacier]

My normal expectation would be I would derive from a class of yours with T being Dictionary<S,S> and fill in the overrides, but that isn't what you are doing

[graceful zephyr]

no, also my core network state system only supports primitive types, or structs of primitive types

since it will exist in native memory only

i cant have C# reference types in it

[jade glacier]

people serializing dicts over the network need a beating anyway 😃

[graceful zephyr]

yeah lol

but still

need some flexibility

[jade glacier]

For sure, the Dict example is a good edge case

if you can make that intuitive, you have a winner

Though really not even intuitive, since you are dealing with advanced users there... just doable at all would be good

[graceful zephyr]

i'm thinking if you need to do something like that it's gonna be streamed on the side using a separate data channel

[jade glacier]

I assume people have access to the network stream?

like before or after all of your auto stuff

[graceful zephyr]

not decided that yet, and it's not really needed to write into the main stream

[jade glacier]

As long as raw stream access exists in some form... everything is possible

[graceful zephyr]

it would be more like "okey i want to send this big update of string dictionary to this client"

and then it streams that over time to the user

[jade glacier]

ahh, yeah thats should not be part of the firehose

[graceful zephyr]

which would be completely reliable

since dealing with resends, etc. is too annoying in this case

[jade glacier]

Along those lines, a dict sync really doesn't seem like it would belong on an unreliable entity ever

[cosmic atlas]

and then you alter the dictionary before it finishes sending

[graceful zephyr]

@cosmic atlas well it'd be serialized in one go 😛

[cosmic atlas]

yea but then having to resend what's changed and track that

don't mind me, just musing as I read lol

[graceful zephyr]

yeah there's tons of edge cases that needs handling

im just thinking of the general architecture

of how to deal with when people want to send 'larger' updates

of some random crap binary data

which doesn't fit on the regular game state

[jade glacier]

I assume there is a concept of ontick and offtick for that kind of stuff?

like loading a level

has no concept of tick

[graceful zephyr]

correct, there's the game state goes in sync with the tick

and then whatever-else-crap-you-need which is ... whenever it gets sent/arrives

[jade glacier]

Can events be flagged that way as well? A tick event/trigger vs just some dumb stuff you want to broadcast?

[graceful zephyr]

the events are most likely going to be entities also

as they are 'one shot' entities basically

[jade glacier]

Seems like just a parallel messaging system for async stuff would cover the rest

[graceful zephyr]

ye

the parallel channel would be 100% reliable+ordered

just to simplify for the user, so you can send like "Add this string to this entities string,string dict"

[jade glacier]

with some metering system that breaks sends up into reasonable bytes per second and such

[graceful zephyr]

yes

as this data is not considered time critical, it's fine with full reliable/resend crap

[jade glacier]

I would use this system 😃

Where do I preorder? 😛

[graceful zephyr]

lol

also all of this is being done in a way so that I can squeeze as much single thread performance as possible out of it

i dont need multi core stuff, i will just have 2 threads

unity main thread, and background thread which does delta compression + socket plumbing

[jade glacier]

compression being the entity self-serializaiton I assume?

[graceful zephyr]

ye

[jade glacier]

Can the stitching also reside on the second thread?

for lack of knowing the real term

[graceful zephyr]

stitching?

you mean unpacking the updates on the client?

[jade glacier]

packing up all of the connection specific sends

[graceful zephyr]

yes

[jade glacier]

though I guess that would result in a lot of buffers

rather than one reusable in the main thread

[graceful zephyr]

it needs to for the server at least, for performance reasons

[jade glacier]

What do you call the process of the server creating each byte[] for each client?

That I am calling stitching atm

for lack of a term

[graceful zephyr]

packet serialization

or something, idk

packet sending?

o.O

[jade glacier]

vs entity serializtion

[graceful zephyr]

well they go hand in hand

entity serializes into a packet

[jade glacier]

entity serializes itself

[graceful zephyr]

yes

[jade glacier]

packet collects them

[graceful zephyr]

of sorts, this new system probably will work like that

[jade glacier]

I've labed it as Stitch for now so I don't have a bunch of methods called Serialize

[graceful zephyr]

the reason i want to use native memory is that i will have duplicate buffers of the networked state, i can just memcpy a few buffers from the main thread and send to the background thread and the background thread has a completely isolated copy of the game state, for each tick

[jade glacier]

so I can tell the internal methods for serialization on the entity from the writer to the conn.... but its not a good name

[graceful zephyr]

so server will say: "tick 100 done, copy it to background thread which takes care of sending it to clients"

[jade glacier]

ahh, gotcha

[graceful zephyr]

and then server can keep mutating his state for tick 101

[jade glacier]

its not breaking that into threads, its just one giant send to all conn thread

[graceful zephyr]

well it's basically a few memcpy calls to duplicate the whole world state, which it hands of to background thread

[jade glacier]

I was thinking parallel threads for each conn, but yea... absurd overkill

[graceful zephyr]

no no, i cant use that

as it doesn't scale well at all

[jade glacier]

yeah, hadn't considered copying over the whole connection serialization process... seemed like that would be hefty if there were a lot of entities

[graceful zephyr]

it's not, because they are all laid out in a few manually allocated blocks

it takes like 0.001ms to copy all of it, it's nothing

[jade glacier]

Yeah, I forget you live in the world of aggressive memory management

[graceful zephyr]

imagine you have a few byte[] arrays for varying entity types, etc. and you split them up with various indexes+length into the array is each entity, so then to copy the whole world state all you need to do is duplicate the arrays

by copying to another array of the same size

[jade glacier]

thats very ECS like

Hmm, actually thats not as crazy as I was thinking

you are't aligning all of the fields that make up the entire world state.... ? You are just aligning all of the byte[] buffers used for the entity serialization?

[graceful zephyr]

ye

[jade glacier]

ahh, smart and gives you some ECS like benefit without making an unreadable mess

Though, how do you align buffers like that when entities are created over time, like as players join and such?

Or are they not all aligned into one giant block like that?

I am thinking I am totally reading more into what you said than what it actually is doing

[graceful zephyr]

so it uses buffers that are sized and chunked based on entity sizes basically

rounded to some common values

[jade glacier]

Ah, I think I just got it

[graceful zephyr]

it wastes a bit of memory but that's fine

but say all entities smaller than 64 bytes will be in one block of memory, all smaller than 256 will be in one, etc.

[jade glacier]

Do you pool the buffers themselves then?

rather than have each entity own a byte[] of its own?

[graceful zephyr]

the ones that get sent to the background thread? yes

it's a few massive buffers which are split up between entities of the same size

or similar sizes

rather

[jade glacier]

interesting

[graceful zephyr]

so i'll have maybe like 5-6 different size buffers

of like 128 bytes, 512 bytes, 1kb and 4kb maybe

or w/e sizes the entities and up as

[jade glacier]

makes me want to rethink the way I am doing it, which is each entity has its own byte[]... that really should be pooled or should write into a larger buffer and use pointers

[graceful zephyr]

thing is the entities dont write into these buffers, the entities themself LIVE in this buffer

[jade glacier]

Doesn't that make your unserialized/uncompressed fields intermingle with the serialized data though?

and get in the way of your memcopy?

[graceful zephyr]

no, the serialized data is kept in other buffers

[jade glacier]

ah ok, was mostly talking abou those buffers

and what they are, how they are allocated, and if/how they get aligned if they even do

[graceful zephyr]

they are byte* buffers allocated from malloc they can also be splited/chunked like i talked about

but thats less of an issue generally

since they will only live on the background thread anyway, it can pool those itself, etc.

[jade glacier]

Leads to more questions, but they aren't important 😃

Overall, super intelligent design you have going there

[graceful zephyr]

yeah the serialization buffers are less important

they can be handled in a myriad of different ways, that's not complicated, can just do regular ulong[] for those allocated separately

it doesnt affect performance much

[jade glacier]

At this point I am just implementing the entire concept of pre serialization and unique sends to each connection... NST didn't have that. So just trying to make it so at some point I can modify it to throw the serialization process onto another jobs thread.

[graceful zephyr]

i dont see how thats possible when you depend on unitys transforms, etc

as you cant access any unity stuff

from a background thread

at all

[jade glacier]

I would have to snapshot it first yeah

[graceful zephyr]

well, some static methods in math and such but thats about it

ye

[jade glacier]

Not tackling that yet, or maybe ever

just wanted to make it not a massive refactor for myself if I try, so just planning ahead a little

My assets only ever will be for networking newbs who want drag and drop. That kind of performance is probably not in the cards

At this point, I just want to add per connection unique sends

[graceful zephyr]

yeah, i dont see you needing that type of performance

[jade glacier]

and will be done with that

#1 goal right now though is just making a solid tick engine

[graceful zephyr]

sounds like a good goal

[atomic cliff]

Hey, is photon pun a good choice for mp in the vein of clash royale?

Oh well, ping me if anyone knows. It'll be super helpful to me thanks.

[split crescent]

clash royale is implemented using a lockstep model

you can see it in how they bake the command delay into unit spawning

[atomic cliff]

Nah, i just mean the matchmaking itself is similar to the one in clash royale

a public room with two player, the game itself is a fast paced arcade-y kinda thing

I doubt lockstep would work for that

Sorry for not being exactly clear on that

But what i need is basically just easy skill based matchmaking with real-time input, and it seems photon pun fits the bill, but i just need to know if there are better alternatives

[split crescent]

you should generally treat matchmaking as a separate concept/appliance from your core networking implementation

[atomic cliff]

Okay, noted. I am new at this networking stuff

So, is photon pun latency good enough for real time matches with only two players?

I read that it doesn't scale very well, but i only need 2 players in any room at a time

But that may change

[warm jungle]

So, I'm having confusion on how to setup something to sync positions between an obj on each client. There are two issues I'm running into.
For one, normally if I was to spawn something in, like the players, or a bullet or something, I can send a message to the server to spawn that for all clients and give it the same unique id on each client.

However, if its just a box that exists in the world by default, I cant do that? I can have the boxes request a unique id, but then theyll just be different unique ids cause theyre considered different objs from each client. Any idea on the best way to deal with this? Manually assigning ids doesnt seem very sustainable or good practice.

Aside from that this is how I was currently trying to sync the position https://hastebin.com/ajofuminiq.cs and it doesnt work very well. its laggy and glitchy, and I'm not sure the best way to go about doing this.

[weary walrus]

@warm jungle frame interpolation for the latter

not sure what you mean about the box that exists in the world by default. If it's part of the scene, then it should have an ID that can sync when clients connect. it's server owned.

[warm jungle]

I talked with someone on GDL about it and got an idea of what to do for it

for the id

Im not sure what you mean for the frame interpolation - i was under the impression I'm already doing that? I'm doing a MoveTowards on the syncing

[weary walrus]

http://www.gamasutra.com/blogs/MarkMennell/20140929/226628/Making_FastPaced_Multiplayer_Networked_Games_is_Hard.php

essentially instead of rendering each new update. you replay the game a frame behind, and interpolate from the last position to the next one. this way you can still be interpolating while waiting for the next frame to come from the server

[warm jungle]

I.. don't really understand and don't get why this would really help? If youre just playing the frames but just a frame behind im not sure how youd exactly set that up, or how it would help

[weary walrus]

because it wouldnt be jittering, you're smoothing between two server approved frames

[warm jungle]

But am I not already doing this? The client moves, sends an update to the server, the server sends that new position to the client, the client interpolates towards that new point

[weary walrus]

is the server sending the position back to the client?

and your just setting that?

frame interpolation is for rendering other clients (proxies) smoothly

if you want to respond to updates from the server for your own client position, that is client side prediction

when the client gets an update from the server of his position, that's from some time behind. so you need to accept that, and then replay batched events

[warm jungle]

When the client moves, it sends its position to the server.
The server then sends that position out to all the clients.
All of those clients set those values to another variable, and then in update i have a transform.position = MoveTowards(transform.position, thatNewPos, Time.deltaTime * speed)

[weary walrus]

so when you say "sends that position out to all clients", is that including the client that sent it?

[warm jungle]

yes

[weary walrus]

so the controller will constantly snap back?

that's where your jittering..

[warm jungle]

Well it shouldnt tho, cause the original client that sent it would already be in that position right? so it would interpolate.. toward where its already at, aka go nowhere

[weary walrus]

no, cause lag exists

by the time i get the message back from the server, my local instance has moved forward

this is where client side prediction comes into play

http://www.gabrielgambetta.com/client-side-prediction-server-reconciliation.html

[warm jungle]

Okay, so. Like in that it refers to input right? Player presses a button, and you gotta get all the buttons in the right order to be in the right position.
But in this case I'm not dealing with button presses, but only position updates, so I dont even really care about the entire back catalog of where ive been, just where i should be now

[weary walrus]

the concept is the same

either by syncing inputs, or by syncing position

by the time the server responds to the originating client with the 'approved / updated' position. the client has already moved past that server tick

you could have the controller client entirely ignore server updates for his entity

[warm jungle]

Yeah, I thought about that, but I'm not sure how I can detect if this specific client is the one who sent the message
and yeah, so if the server is always late sending back the new position, how can that be fixed or helped? If its always behind it never helps unless i just ignore the client who sent

Also I assume I also want to have objs that are moving due to the server update be ignored, but idk how to detect if its moving based on the server updating it, or if its from something else moving it

[weary walrus]

you normally would keep a buffer of past positions, and if they send a state update at a certain server frame that doesnt align with your local clients registry, then you know you have diverged from the server and need to reconcile

and you should definitely be able to tell which client is sending you position updates

[glass bobcat]

Also security wise trusting a client is bad unless you don't care about that in your case

The client could just move however it wants or even teleport

[weary walrus]

yep, never trust the client

[graceful zephyr]

@warm jungle how's it going lol

Just read all the stuff in here

[warm jungle]

uhhh baaad

and im about to head to bed

but confusingly

[graceful zephyr]

@warm jungle what's the issues you're having

[jade glacier]

@warm jungle look at litenetlib Manager, some guy made and shared it. Does a lot of the stuff you are trying to figure out. Can see how he did it.

https://github.com/insthync/LiteNetLibManager

[weak plinth]

https://www.pcgamesn.com/steam-dedicated-servers
Highlights:

Relay servers provided by Valve, free of charge for any game released on Steam (opt-in, price is essentially included in the 30% tax). Developers can either proxy to their own game servers or use Valve-provided dedicated ones (only for Valve-approved developers at the moment).

Servers are "hidden" from the outside world, players can no longer know their IPs and connect directly

Free of charge DDoS protection included as well

All connections are routed through Valve's backbone, which provides lower pings and higher speeds and throughput

Platform-independent account identity support (Xbox Live, Steam, PSN, etc.)

Beta only at the moment. Developers should not use this in the production environment until all the functionality is shipped

[vital hawk]

with that, this is my main concern Clearly, as with all Steam updates, it comes on Valve time.

still haven't seen Source 2 SDK they promoted years ago being free for developers who at least release on steam

so, without any library/engine SDK on my hands, they can talk as much as they want but it will not change anything 😄

@weak plinth where did you get those highlights from? the article you linked didn't contain all that info

[weak plinth]

reddit ny bad

[vital hawk]

I found the reddit thread, no source links there either

[weak plinth]

https://www.reddit.com/r/gamedev/comments/am1stl/looks_like_steams_getting_dedicated_servers_for/

ohh source

[vital hawk]

yes, that's the one I'm reading, and that's commenting the same link

but the post that put those there doesn't tell where the info is from, so it doesn't look credible at all

like, the part where relay servers provided by valve free of charge part, or the platform independent thing

the article itself links this as the only source https://www.resetera.com/threads/valves-new-multiplayer-api-in-public-beta-vac-for-everyone-dedicated-servers-ip-hiding.96853/

[weak plinth]

the the free of charge is doubtful

[vital hawk]

but that's not really going into business details

that person have multiple threads for it even 😄

but yeah, I call BS

[warm jungle]

I mean, if valve is looking to compete without lowering it's percentage, this is a way to do it

Doesn't seem that far our to me

[weak plinth]

are the others like discord & epic offering much outside of a place to stick your game? i figured steam had them beat in services pretty badly out of the gate

[vital hawk]

sure, but hosting dedicated itself couldn't still be totally free or included on that 30%, it would get crazy expensive on some games for them

@weak plinth bit offtopic here but sure, there are other stores too, gog, humblebundle, itch all accept indies

[weak plinth]

yeah, was just moreso commenting on the fact i don't think they need this to stay competitive even and keep their % the same

[vital hawk]

atm, there's no real threat

[weak plinth]

any suggestions on an approach to out of game notifications? initially i was going to do long-polling, but i'm thinking of using websockets for friend invites, party updates, etc. any good websocket libs i could use that you know of? (looking at this: http://sta.github.io/websocket-sharp/)

[graceful zephyr]

@weak plinth for which platform?

There's notification APIs for all of them now adays

[weak plinth]

android / ios is what i'd like to build for if that's what you mean

[graceful zephyr]

But it's usually platform specific

Yes, they both have notification APis don't they?

Use those

[weak plinth]

oh sorry, by out of game i meant like the menu scene, social scene, stuff like that

basically when they're not connected to an online game

[jade glacier]

http://www.hatebin.com/zftrbecinf Nothing I would rather be doing on a Saturday morning than be adding a zillion callbacks that should have already been there to a dead networking library.

[compact bramble]

I find pretty much all networking libraries die because it's a fun problem to start but one that never runs out of problems.

Those that survive must be backed by cash or open source.

I guess a solution like Quantum is to be both the problem and the answer. The internet matters less in this case 😄

[jade glacier]

UNet didn't though fail for much more of a reason than Unity failed to finish and continue to support its growth. I'm glad to see it die though because all of these HLAPIs are based on around some really bad networking concepts (RPCs and Syncvars) and really get developers going down bad paths.

Quantum and Bolt (both being fholms work) are the only two that actually ground up respect AAA methods for networking. They are closed, but there is nothing stopping these open libraries from embracing the same principles... they just don't for whatever reason.

All of the HLAPIs for networking seem to go straight for abstractions with RPCs and syncvars with little or no consideration for the actual important stuff, which is the tick-based syncing up of all of the clients and server, and the bitpacking that you generally do on top of those tics.

[compact bramble]

UNet was a result of the company having no funding for it and no management.

Now I'm pretty sure there's a decent amount of support behind it, still it's left me burned once so I'll be approaching with caution this time round.

It's strange really that a big company like unity wouldn't put maximum resources behind something, but I guess finding the right staff is also a hard job even with endless cash.

shrugs - it is all guesswork we shouldn't have to even discuss but a failure makes that get discussed because unity's failure is a customer failing too

[jade glacier]

If Unity tried to maintain an internal game like Epic does with their library, I suspect the engine would have a lot less of this headscratchers built into it.

[compact bramble]

They are though

for years, Unity's made small internal games to test stuff, and more recently it's been big tech demos and more recently still an entire FPS networked game that you can get from github

[graceful zephyr]

@jade glacier the open unet derivatives don't use those techniques because they all want to have this mega generic library which can do everything but also sucks at everything except for the simplest games

Like the way you network different genres of games is massively different

Not just deterministic vs not, but the requirements are just different within one genre like fps/tps where there's widely different implementations used depending on the specific game, etc

Like the way fortnite and cs are networked is just different at it's core even

@compact bramble big difference between small internal games and public facing massive hits like fortnite, unreal tournament etc

[jade glacier]

I get that in the big scheme of things, but they didn't even make an effort to ground up build layers that branch into ANY trees that are useful.

[graceful zephyr]

Nope, because those things are really hard to get right

[jade glacier]

@graceful zephyr That fact that I and every other dude in these net channels are making tick engine and input syncing says something about that layer ... just missing....

That layer alone branches into 90% of the next layers

[graceful zephyr]

You can easily build an architecture which you realize half way through just doesn't work

[jade glacier]

yeah of course

I think they have been slow to realize that though, because they werent making ANY actual game with UNET

it would have reversed course early on had anyone building it been making it for any type of game

It seems like someone got really excited about weaving all kinds of cool stuff like RPCs and Syncvars...

without asking if they were even the right path

They are 2 layers above what needed to be made first

[graceful zephyr]

Anyone with half decent experience in networking could've looked at unet hlapi and said it was a dead end

Literally first thing I said when I looked into the dlls

[jade glacier]

The weaving was cool, but it was icing on bad cake

[graceful zephyr]

Meh, weaving stuff is at best... Almost useful

[jade glacier]

I wasn't even experienced and 2 weeks in I was trying to get out from under RPCs and Syncvars... not sure how that got designed by someone obviously competent as they were

What is scary is PUN and Forge seem to repeat the same thing

[graceful zephyr]

PUN is just a different target tho

[jade glacier]

Lots of rpc and syncing... but still no foundation in syncing up clients into a common tick

Yeah, but PUN can easily escape this mess. I suspect in fact it could even do a really great job of it

that relay server could be a beast, rather than a dumb mirror

I know why it is what it is

And am not saying anyone did anything wrong

Just that Unity networking is in a sad state because everyone building these things but you seems to understand the fundamental backbone of networking

Quantum and Bolt are still the only two engines I can name that treat the tick like a AAA title would and should

And that bitpack like they should

I feel like a broken record with this stuff, but its a wall that is just right there in the middle of the networking room

[vital hawk]

but AAA would also never work on closed source libraries where you can't even fix things when they break :p

[jade glacier]

My issue with UNET HLAPI isn't even that its closed, its just that it skips right over packing states and inputs into ticks... and goes right to messaging

Its like making games using email as the transport LOL

HLAPI though was actually open, though lot of good that really did.

Anyway, to put a pin in my rant... anyone making MLAPI or HLAPI level networking stuff, really need to spend a little time coming up with some good generic tick engines and tick based input/state syncs... or they are just making garbage... imo

[graceful zephyr]

@jade glacier quantum doesn't do a lot of actually networking tho

It just sends input but sure

[jade glacier]

It does a different thing with the ticks

[graceful zephyr]

yes sure

[jade glacier]

but it still must have a rock solid means of getting all of the clients on the same clock and lining things up

[graceful zephyr]

Yes

[jade glacier]

Anyway, not looking to poop on the channel with my ranting over and over about that

[graceful zephyr]

Gotta put shitpants to bed

[jade glacier]

Youre going to need a new name for him once he gets potty trained

[graceful zephyr]

Yes I know

The name is starting to stick too much

[jade glacier]

I assume you don't say it in English

[graceful zephyr]

Oldest is still called nugget because that's the nickname I gave him

[jade glacier]

Pooperhossen or something I assume 😛

[graceful zephyr]

Can't have this one being called shitpants when he's 5

We do say it in English

Me and the wife talk English

With each other

[jade glacier]

Ah yeah, I forget that swedish wasn't her primary language

[graceful zephyr]

Ye, so no he's shitpants, verbatim

[jade glacier]

Its bizarre being in Portugal, hearing all kinds of accents but most speaking English, since any mixed group that is the default.

I'm always like... why are you speaking English??

Because that one friend.

[graceful zephyr]

I lost my swedish accent

As I talked so much English day to day the past decade

[jade glacier]

Kind of a shame English became the dominant language... its not the prettiest or easiest

Yeah, your typing is better than most Americans

[graceful zephyr]

I prefer it over swedish

[jade glacier]

I'm working on Spanish, that seems like the ideal second language after English

[graceful zephyr]

It's gotten to the point where my English is better than my swedish and I find myself thinking in English and using English grammar rules when speaking swedish

[jade glacier]

Not a bad problem to have when you work internationally like you do

World Citizen.

[graceful zephyr]

Talk English at home, work from home and all work is in English, all tv shows are in English, type in English all day etc

[jade glacier]

Ah ;yeah, you are definitely becoming native English

[graceful zephyr]

Maybe 10% of my speaking and typing is done in swedish

Tops

[jade glacier]

Anyway, didnt mean to keep you from shitpants

[graceful zephyr]

Oh I put him to bed while talking to you

He's out now

[jade glacier]

Quick

[graceful zephyr]

Yes it is 9 PM here, way past his bedtime

So he was out in a few minutes

Now he is sleeping on my chest

[jade glacier]

So you put him to bed... on your chest?

[graceful zephyr]

By put to bed I meant put to sleep

He's going in his crib in a few minutes

[jade glacier]

😛

[graceful zephyr]

Should've been more clear ;p

Just it takes me tops 10 mins to do it if he gets to fall asleep on me, takes like an hour if it's in the crib by himself

[jade glacier]

baby science

[graceful zephyr]

So I walk around in a circle with him and talk crap on discord on my phone

I started on my new idea we talked about a few days back

Just prototyping some stuff, trying to figure out the implementation details

[jade glacier]

How many days to get a basic enough test happening to see how its working? Though seems like you can predict most of the results.

I'm still trying to make sense of the memory footprint, seems like with a large number of netobjects and a high player count, that would really start to add up?

[graceful zephyr]

It's not more than any other solution, it's just that you don't think about the memory when dealing with classes and monobehaviour

The memory footprint is an absolute non issue

[jade glacier]

True, its already a bloated giant with those anyway

[graceful zephyr]

It'll take me.. IDK two weeks maybe for a first prototype

Maybe less since I already figured out a lot of the details in my earlier prototyping, but maybe more if I run into some big issue I have not forseen

[jade glacier]

Looking forward to hearing

[graceful zephyr]

Going to hammer away at it tonight

Soon as I can get out from under mr shitpants

[jade glacier]

It may happen really quickly

[graceful zephyr]

Mostly just dealing with memory layout stuff atm

Like how to structure the data in a way that gives me the least overhead when passing it on to the background thread

And restructuring of the socket hookup as I need to move some stuff around

Due to how the new background serializer will work

[jade glacier]

all a paygrade above me for sure

[graceful zephyr]

We'll see how it goes, still a few question marks to figure out

[jade glacier]

Your transport is going to be a beast with this many use cases having been applied to it

[graceful zephyr]

But nothing too big I'm worried about

The transport layer is as simple as possible

No reliability, only ack/nack

[jade glacier]

What's your next layer above that called then? I assume that manages the order and reliability?

[graceful zephyr]

It does yes

The layer above is the networking library itself

[jade glacier]

No abstraction of it, interesting

[graceful zephyr]

That's split into two main pieces

The background thread and the unity main thread stuff

[jade glacier]

conceptually tidy

[graceful zephyr]

The socket library itself has an abstraction for plugging in different subsystems like steam, .net sockets, photon cloud, etc

[jade glacier]

I'm fuzzy on the distinction between the transport and the sockets

[graceful zephyr]

Game => net lib main thread => net lib bg thread => transport => socket adapter

[jade glacier]

Ahh, ok

I never touch the transport and socket - so they were just the same thing to me

"The mailbox" 😃

[graceful zephyr]

Difference between transport and socket is a hairline one, but for me the socket is the raw thing I call send on to put stuff on the wire, the transport is the connection abstraction, ack/nack ontop

[jade glacier]

I see the value of keeping them separate if libraries like Steam and Photon don't make any transport level choices for you

I just assumed their api was that entire pipe

[graceful zephyr]

the only thing my transport layer asks of the socket abstraction is two things:

1. provide me with some way to identify who im sending/receiving stuff to (ip address, steam id, etc.)
2. provide me a way to send unreliable data to them

and basically every socket api in existance does those two things, .net sockets, steam, xbox, psn, switch, etc

they all allow you to do those two things

and then i build the rest of it myself on top

so it can run on any socket layer

if they provide more features, fine... but i dont use those

and yes im on my laptop now as you can tell by my typing speed 😛

[jade glacier]

I'm jealous of that nice clean abstraction... as I sit here modifying my adapters from UNET to work with PUN.... not at all clean or tidy

You are unleashed

[graceful zephyr]

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace NetCode.Sockets {
  public abstract class SocketImplementation {
    public Peer Peer { get; internal set; }

    public abstract SocketEndPoint EndPoint { get; }
    public abstract Boolean Bound { get; }
    public abstract Boolean SupportsPollTimeout { get; }

    public abstract Boolean IsEndPointCompatible(SocketEndPoint endpoint);

    public abstract void Bind(SocketEndPoint endpoint);
    public abstract Boolean Poll(Int32 timeout);
    public abstract Int32 Send(Byte[] buffer, Int32 length, SocketEndPoint endpoint);
    public abstract Int32 Recv(Byte[] buffer, Int32 length, out SocketEndPoint endpoint);
  }
}

that is my base class for the socket abstraction

[jade glacier]

Nice and short

[graceful zephyr]

its obviously modelled on classic raw sockets

but its similar to the systems i've built in the past and it works well, it can be adapted to any networking api ive come across

atm SocketEndPoint is a class, which i want to turn into a struct instead

but thats fine for now doesnt matter

[jade glacier]

not a pooled class candidate?

[graceful zephyr]

nope, and it doenst need to be a class either

i was just lazy as its simpler to do it with a class

[jade glacier]

ah

[graceful zephyr]

its basically

i can either just implement the core idea of how it should work and use plain .net sockets

or i could build the real real thing with needs me to do a whole native plugin and interface with sockets in C

just to get rid of one allocation point during development

which is just... meh, doesnt matter for now

3 hours vs 3 days of work sort of thing

[jade glacier]

//TODO:

[graceful zephyr]

ye

[jade glacier]

Do you recall if PUN has delegate callbacks for network events, or just the interface methods? I need to make use of callbacks in a static class, but not sure thats doable @graceful zephyr

[graceful zephyr]

there is a acallback yes

[jade glacier]

hell if I can seem to locate them

[graceful zephyr]

its not 'in' pun tho

its in realtime

which is one step down

[jade glacier]

The derived base class methods I know, and the interface I know... but not finding delegates

[graceful zephyr]

its on PhotonNetwork.peer or something

PhotonNetwork.OnEventCall

at least in pun 1.x

[jade glacier]

digging through the PUN2 migration notes now

PhotonNetwork.OnEventCall is gone. You have two options: use PhotonNetwork.NetworkingClient.EventReceived(EventData) instead. use IOnEventCallback.OnEvent(EventData).

[graceful zephyr]

okok

😦

😃 i mean

[jade glacier]

{
    PhotonNetwork.NetworkingClient.EventReceived += OnEvent;
}
        
private static void OnEvent(EventData data)
{

}``` 
Not puking red at me, so looks like the right track. Thanks @graceful zephyr

[graceful zephyr]

np

[jade glacier]

The docs seem to indicate its a callback like the interface ones, but it only seems to fire for network messages, and not events like OnConnect... oh well. Going to have to change my static to a singleton, which is annoying.

[graceful zephyr]

morning o/

[jade glacier]

Werd

[glass bobcat]

hey

[graceful zephyr]

morning o/

god this channel is low traffic 😄

[jade glacier]

Bad networking

Wasnt able to find a way to get system callbacks with PUN, so just resorted to making my static adapter class into not static, instead a non-MB Singleton... Just so I could put their callback interfaces on it. @graceful zephyr

Annoying, but not the end of the world

And good morning

[graceful zephyr]

what do you mean system callbacks?

you mean for connected/diconnected/etc ?

good morning to you too o/

[cosmic atlas]

mornin

[jade glacier]

@graceful zephyr joined room, the whole list of pun events

[graceful zephyr]

@jade glacier ah yeah thats only doable with the regular callback interface

@cosmic atlas morning o/

[cosmic atlas]

adding a "Refresh Games" button while contemplating if I should just implement some sort of auto matchmaking system ;_;

[jade glacier]

I do have my tick engine and input syncs working with PUN now so the surgery was a success

[graceful zephyr]

nice

@jade glacier i wanna discuss your speed up/slow down tick system for the clients at some point, did you test it with different ping conditions/simulated lag, etc ?

on that topic: you know photon has a builtin lag simulator right?

`Photon Lag Simulation Gui' script

[jade glacier]

I have to choose how to deal with BIG CHANGES in latency. Right now it triggers a hard realignment

It's not a realistic scenerio...

[graceful zephyr]

i would have it use the same system for everything, it gets too messy otherwise

[jade glacier]

But I want it to be able to survive me turning on 300ms of delay.. and removing it

Right now that cause packets to get fully discarded... Since a rapid catch-up with physx isn't reasonable

[graceful zephyr]

yeah it is

[jade glacier]

I didn't, I use a third party net sister

[graceful zephyr]

https://docs.unity3d.com/ScriptReference/Physics.Simulate.html

[jade glacier]

Distressor

[graceful zephyr]

you can do rapid catchup with physx

using that

[jade glacier]

I have to think about how playing with uses time to aggressively will break other things

Though realistically that should never happen

On the flip side... It will extrapolate a bunch... But that's easy

[graceful zephyr]

what i would do is that if you want to use the tick system and all of that with your system, basically it's implicit that they need to hand over control of fixed timesteps to your library

[jade glacier]

It's just on the .. what if I get a crap ton of updates at once

[graceful zephyr]

@jade glacier have you seen the overwatch videos on how they deal with this?

[jade glacier]

Have not, but should so I'm not reinventing this wheel

The system isn't deterministic, so I can toss excessive updates flooding in.. but as long as I am spending the time on this, figure I may as well try to do it so it doesn't do that, and keep it future deterministic friendly

Ok, mobile typing is failing me here

Laptop time

[graceful zephyr]

If you want to make it deterministic then running several physics steps to catch up is required

[jade glacier]

I think you are right about taking over the timestep completely, and having devs make any changes to it through my property

so I can just manhandle the fixedTimeStep duration without fighting them

They really being me when I finally make something with my own networking code

[graceful zephyr]

bolt takes full control over the fixed timesteps

[jade glacier]

I have no idea how often anyone even would want to tinker with the fixedupdate rate in a networked game, but as long as I am writing this I should write in handling for it

[graceful zephyr]

and dont evne let the user fuck with it

and it works fin

it will literally override it every frame to make sure its the right value

[jade glacier]

Right now they could try to change it, but every tick I would just be overwriting their changes

[graceful zephyr]

ye

[jade glacier]

So I could make a property that changes the target rate

[graceful zephyr]

if they want to change the timestep

[jade glacier]

The main question right now is how and if to really make this make sense in PUN

[graceful zephyr]

that has to be done from the server

what we do in quantum is that with every update we send out the current timescale of the server for each tick essentially

[jade glacier]

server auth in PUN isn't really a great idea. I have it doing it... but not sure there is a real point to it

[stuck void]

hey guys 🐸

[graceful zephyr]

o/ yo

[jade glacier]

All of the players can and generally do talk directly to one another, so nudging to keep locked up with the server does nothing for client to client

morning

or eve

[stuck void]

yeah im having an issue with bad latency testing as well

[graceful zephyr]

@jade glacier its morning (timezonesarealie)

[jade glacier]

If its morning in Britain... its morning everywhere.

[stuck void]

hosting both my master and game server on my poop adsl wifi at home and testing client thru my laptop on mobile data hotspot for a realistically horrendous connection xD

[graceful zephyr]

😄

[stuck void]

its 7:50 pm here

[cosmic atlas]

haha 😃

[jade glacier]

That was how I originally tested my net code for mobile.... I would host on my wifi laptop on home cable... and connect my phones over mobile carriers @stuck void Gave a pretty good realistic case of lag and jitter

[stuck void]

seems I can have my attacks miss-but-not-miss with a bad connection, so I'm gonna have to build a system to sync em up at the very least so it doesnt look so out of sync visually

[jade glacier]

What kind of game is it, and what are you syncing stuff up with?

[stuck void]

cant do much about actually missing or not missing since its fully server auth

but I'll need something to at least let the client visually know so I dont have a projectile whoosing past but they still take damage

[jade glacier]

ahh, server auth with user inputs... so some inputs are getting lost?

[stuck void]

not quite, I've got an almost-reliable system on top of udp, but the delay in the client receiving the attack message coupled with the input delay can result in close calls that arent in sync

[jade glacier]

That's pretty common without going deterministic, since the player in his world is actually in a future position relative to the network objects being shown to him

So rockets flying by but actually hitting you in the server report is a pretty common artifact

[graceful zephyr]

i spent the weekend designing the memory model for my new networking system to shuffle all the serialization to a background thread, it's looking like it will work out

[stuck void]

yeah, I did delay the server attack slightly based on the lowest average latency across all players in the area which helped a fair bit, but its still possible to consistently make them go out of sync with close dodges

[graceful zephyr]

well i spent the weekends with the kids, spent the evenings working on the netcode :p

[jade glacier]

@graceful zephyr Are/Have you considered proposing this entire system to the guys at Exit? Might get backing to finish it out as a bolt replacement?

[graceful zephyr]

just want to do it as a hobby thing for now, no idea how to monetize it later

i hate having the pressure of having to deliver something when im inventing new stuff

[stuck void]

im thinking I'll just keep track of abilities and projectiles on a list like I do with enemies and players, and let the server tell the client whether or not it hit, so at worst even if it slips by a little bit it'll still play its impact animation and disappear on the client

[jade glacier]

It will be a quick turnaround if you decide to once you have all of these parts made

[graceful zephyr]

like this is probably the 10th iteration of my new system, and i think i finally got it in place

started in late 2017 so :p

[jade glacier]

Its going to be beastly when you finish it, though these things are never really done... there is always some new tech idea that could be added

You can also consider letting some owner stuff get passed through the server before it becomes server authority states

[graceful zephyr]

me or ada?

[jade glacier]

like enemy weapon fire you might be able to have passed along immediately @stuck void

for the cosmetics of it

[graceful zephyr]

ah

[jade glacier]

Some risk there, but you can shave the server buffer out of the picture with a little client to client trust on the cosmetics