Sfs | Garry's Mod | Page 1

flat wedge Mar 24, 2024, 4:53 AM

#

If these benchmarks hold true, I'll add this to Express asap

dusk fossil Mar 24, 2024, 6:33 AM

#

👀

west lintel Mar 24, 2024, 11:14 AM

#

Might sound stupid but is it only able to decode on the same realm as it was encoded? Or am I missing something here?

#

-- CL code snippet
-- tSend is a sequential table
local sEncoded, err, err2 = HG.Core.Serializer.encode(tSend)
if (err) then
  HG.Core.LogError("Serializer", "Error encoding Bodygroups data:", err, err2)
  return
end

net.Start("HG.Network.Bodygroups.Save")
  net.WriteString(sEncoded)
  net.WriteUInt(iSkin, 4)
net.SendToServer()
print("CL:", sEncoded, HG.Core.Serializer.decode(sEncoded))

-- SV code snippet
local sBodygroups = net.ReadString()
local xData, err, err2 = HG.Core.Serializer.decode(sBodygroups)

if (err) then
  HG.Core.LogError("Serializer", "Error decoding data:", err, err2)
  print("SV:", sBodygroups, xData)
  return
end

#

Also, HG.Core.Serializer = include("modules/bin/sh_serializer.lua") is just a shared global for using its methods

old hearth Mar 24, 2024, 2:53 PM

#

Ur supposed to use writedata not writestring

#

as ur sending binary data

old hearth Mar 24, 2024, 3:15 PM

#

@flat wedge show ur results too!

old hearth Mar 24, 2024, 3:19 PM

#

flat wedge If these benchmarks hold true, I'll add this to Express asap

blazing fast netstream alternative is next plan :3

west lintel Mar 24, 2024, 3:42 PM

#

yh figured it out, my mind wasn't fully woke up this morning 😭

#

got really neat results w/ big inventory transfers, congrats!

old hearth Mar 24, 2024, 4:12 PM

#

can u share them

daring cobalt Mar 24, 2024, 5:22 PM

#

old hearth blazing fast netstream alternative is next plan :3

Add AI on it and be ready to open a LLC

old hearth Mar 24, 2024, 5:37 PM

#

lol

#

its just a simple idea

old hearth Mar 25, 2024, 1:35 AM

#

kinda sus after adding garrysmod types to the encoder

#

oopsie that was because i had this

#

(still sus)

viral crypt Mar 25, 2024, 6:09 PM

#

im benchmarking it right now against PON in ULib as ULib sends a lot of small/large tables over

#

📎 message.txt

#

seems like pon is smaller for larger files loads

#

pretty significiant too

#

added systime tracking to it

📎 message.txt

#

sfs is pretty much faster

#

for anyone wondering im doing this in ULib.clientRPC

old hearth Mar 25, 2024, 6:23 PM

#

u shouldnt include compress within the test

#

also dont forget to check decode as pon will be way slower

viral crypt Mar 25, 2024, 6:32 PM

#

i care about the size most tbh

old hearth Mar 25, 2024, 6:34 PM

#

im kinda curious about what data ur encoding, are they mostly strings

viral crypt Mar 25, 2024, 6:36 PM

#

i think so yeah, not entirely sure though

#

ill print the table rq

old hearth Mar 25, 2024, 6:37 PM

#

as im testing rn and for most types except strings sfs is way smaller

#

eg. encoding "true"

#

1 byte in sfs and 4 bytes in pon

#

as pon has to take a table

#

local value_to_encode = {"xd", 3, "play_time", 24759}

sfs 22 pon 30

#

unless ur encoding arrays then ur using a wrong function to do it, and i bet its the case?

viral crypt Mar 25, 2024, 6:40 PM

#

💀

📎 message.txt

#

yeah thats a lot of strings

#

ill dump them to a file

old hearth Mar 25, 2024, 6:41 PM

#

pon will be smaller in some cases

#

with strings

viral crypt Mar 25, 2024, 6:43 PM

#

local tbl = {}

for i = 1, 10000 do
    table.insert( tbl, tostring( {} ) )
end

local pc = util.Compress( pon.encode( tbl ) )
print( #pc )

local sc = util.Compress( sfs.encode( tbl ) )
print( #sc )

14883
24153

#

yeah strings are a big difference

#

without the compression

189402
209621

old hearth Mar 25, 2024, 6:44 PM

#

no the diff here is because of arrays

#

u should use encode_array for arrays

#

and give it the size

viral crypt Mar 25, 2024, 6:46 PM

#

local tbl = {}

for i = 1, 10000 do
    table.insert( tbl, tostring( {} ) )
end

local p = pon.encode( tbl )
local pc = util.Compress( p )
print( #p, #pc )

local s = sfs.encode_array( tbl, #tbl )
local sc = util.Compress( s )
print( #s, #sc )

188980    24503
180003    22644

old hearth Mar 25, 2024, 6:46 PM

#

wtf

#

why am i getting diff results

#

are u using a modified pon version

#

wtf why is pon with me giving diff result every time

viral crypt Mar 25, 2024, 6:47 PM

#

im using 1.2.2

#

📎 message.txt

#

tostring will be different in size

#

for me its the same if i use the same table

old hearth Mar 25, 2024, 6:49 PM

#

thats weird sfs giving me same size?

#

`
sfs 180003
pon 141591

sfs 180003
pon 141744

sfs 180003
pon 139068

`

#

or is it because im testing with luajit directly

#

lemme test with gmod lol

#

bruh why am i getting diff results using luajit directly with pon

#

and why is pon giving smaller/bigger size with same string size

old hearth Mar 25, 2024, 6:56 PM

#

viral crypt 💀

could u send this so i can test it

viral crypt Mar 25, 2024, 7:02 PM

#

https://github.com/npc-servers/ulib

GitHub

GitHub - npc-servers/ulib: ULib: A Lua library for more rapid devel...

ULib: A Lua library for more rapid development on Garry's Mod servers - npc-servers/ulib

#

probably needs ulx too

old hearth Mar 25, 2024, 7:02 PM

#

i meant the big table, dump it as json

viral crypt Mar 25, 2024, 7:03 PM

#

👍

flat wedge Mar 25, 2024, 7:04 PM

#

Gangbox has functions to generate random data that might be useful for your testing:
https://github.com/CFC-Servers/gm_gangbox?tab=readme-ov-file#generating-random-data

viral crypt Mar 25, 2024, 7:04 PM

#

gang gang

old hearth Mar 25, 2024, 7:05 PM

#

god i need a way to figure out if table is an array or not

#

the problem with doing that check is holes

flat wedge Mar 25, 2024, 7:06 PM

#

if it has holes then it's not an array right

viral crypt Mar 25, 2024, 7:06 PM

#

yeah

old hearth Mar 25, 2024, 7:06 PM

#

ye but nil and null

#

undefined

viral crypt Mar 25, 2024, 7:06 PM

#

gotta loop over the entire table harold

old hearth Mar 25, 2024, 7:07 PM

#

WHY CANT WE GET ACCESS TO THE ARRAY PART

viral crypt Mar 25, 2024, 7:07 PM

#

📎 ulib_rpc_xgui.getchunk8374.txt 📎 ulib_rpc_ulib.ucl.initclientucl41510.txt 📎 ulib_rpc_ulib.ucl.initclientucl82123.txt 📎 ulib_rpc_xgui.getchunk3078.txt

#

here's the largest chunks

#

and no, i didnt accidentally send that twice ulib seems to do that tf

#

wtf

flat wedge Mar 25, 2024, 7:08 PM

#

viral crypt Mar 25, 2024, 7:08 PM

#

does that have sub tables

flat wedge Mar 25, 2024, 7:08 PM

#

yes

viral crypt Mar 25, 2024, 7:08 PM

#

oh i see yeah

flat wedge Mar 25, 2024, 7:09 PM

#

it doesnt do nonsequential though is the only downside

old hearth Mar 25, 2024, 7:10 PM

#

bruh sfs will prob win with a really big margin

viral crypt Mar 25, 2024, 7:11 PM

#

god i hate ulib

old hearth Mar 25, 2024, 7:13 PM

#

bruh found a bug thanks to gb.randomtable lmfao

viral crypt Mar 25, 2024, 7:14 PM

#

👀

old hearth Mar 25, 2024, 8:00 PM

#

ok fixed now back to the testing:
pon 215194 compressed: 97651 sfs 134524 compressed: 90841

#

local tbl = gb.RandomTable(10000)

local p = pon.encode( tbl )
local pc = util.Compress( p )
print( "pon", #p, "compressed:", #pc )

local s = sfs.encode( tbl )
local sc = util.Compress( s )
print( "sfs", #s, "compressed:", #sc )

#

ok figured out a way to have support for nested arrays, not 100% accurate for tables with holes but better than having a nested array with no holes get encoded as a table, i noticed that pairs and next always go through the array part then the hash part, so doing next(tbl, #tbl) will tell us if it has a hash value or not (but if it's an array with holes it will be encoded as a hashtable) and thats better than pon's method which prob loops twice and hurts performance for big tables

#

@flat wedge what do u think

old hearth Mar 25, 2024, 11:11 PM

#

ok tables now get encoded as arrays if they are (not consistent if has holes)

old hearth Mar 25, 2024, 11:14 PM

#

viral crypt

ok i tested all of them and i dont think it's that big difference considering they are all strings, it's the only position where will pon win in size as it uses two bytes to identify it as a string (but it wont win with all strings, some strings will be smaller in size using sfs)

#

but is it worth it if ur just going to use strings? u decide, using ur first big table u sent running 1k times (encode + decode):

#

and dont forget max size int for pon is 32 bit

#

testing using gb.randomtable with 1k fields (encode + decode):

#

prob u wont have to think about it too much lol

viral crypt Mar 25, 2024, 11:24 PM

#

the speed increase is definitely a big bonus

old hearth Mar 25, 2024, 11:26 PM

#

u can also increase it a bit if you have a custom type function https://gist.github.com/CapsAdmin/0d9c1e77d0fc22d910e182bfeb9812e5

#

ok using a custom type function can make a big perf diff for big tables/arrays

dim storm Jul 16, 2024, 6:38 PM

#

@old hearth Issue in sfs.table (2 pic)
Result(1 pic) true true
=> test key is missing (4 pic, code 3 pic)

PoC

local tbl = {}
tbl.test1 = "test1"
tbl[#tbl + 1] = {1, 2, 3}
tbl.test2 = "test2"
tbl[#tbl + 1] = {4, 5, 6}
tbl.test3 = "test3"
tbl[#tbl + 1] = {7, 8, 9}

local tbl_len = #tbl
print(tbl_len > 0, next(tbl, tbl_len) == nil)
PrintTable(sfs.decode(sfs.encode(tbl)))

#

The fact check is false-positive or not reliable

flat wedge Jul 16, 2024, 7:23 PM

#

dim storm <@223217429684289536> Issue in ``sfs.table`` (2 pic) Result(1 pic) `true true` =...

Report on Github?

dim storm Jul 16, 2024, 7:45 PM

#

flat wedge Report on Github?

For first here

#

Tomorrow I'll publish it on Git, but that's tomorrow

flat wedge Jul 16, 2024, 8:53 PM

#

Okay

old hearth Jul 19, 2024, 6:35 AM

#

dim storm <@223217429684289536> Issue in ``sfs.table`` (2 pic) Result(1 pic) `true true` =...

I don't have my PC rn, but that's not possible to achieve unless something is fucked up with your lua env
https://github.com/rangercyh/lua5.1/blob/bd830426035f6db4d02bff3088f35299637b0ef5/ltable.c#L162

GitHub

lua5.1/ltable.c at bd830426035f6db4d02bff3088f35299637b0ef5 · range...

lua 5.1 source code. Contribute to rangercyh/lua5.1 development by creating an account on GitHub.

dim storm Jul 19, 2024, 6:51 AM

#

old hearth I don't have my PC rn, but that's not possible to achieve unless something is fu...

I'll test it later on a clean one

dim storm Jul 19, 2024, 2:48 PM

#

old hearth I don't have my PC rn, but that's not possible to achieve unless something is fu...

Pure install gmod dedicated server without any external, same code, give same unreliable results. As a fact that the problem is not ours. In addition, disabling JIT did not make any changes to the reliability of the check

#

Just another sample

#

More for fun, not serious. So that it would not be possible to say that this is something we have on pure gmod. This check is just as unreliable when running in online Lua interpreters.
https://onecompiler.com/lua/

Lua Online Compiler & Interpreter

OneCompiler's Lua online editor helps you to write, compile, debug and run Lua code online.

old hearth Jul 19, 2024, 3:07 PM

#

yes I can replicate on it, please submit an issue on github so I can check when I have access to my pc

dim storm Jul 19, 2024, 3:13 PM

#

old hearth yes I can replicate on it, please submit an issue on github so I can check when ...

Okay

old hearth Jul 19, 2024, 9:03 PM

#

@dim storm could you test with this check added to the existing checks
next(tbl, tbl_len -1) == tbl_len

dim storm Jul 20, 2024, 6:04 PM

#

old hearth <@219709046863691776> could you test with this check added to the existing check...

Better, but also unrealible in some cases, mostly on big tables i think

old hearth Jul 20, 2024, 6:06 PM

#

dim storm Better, but also unrealible in some cases, mostly on big tables i think

I ran phatso's random table library for half an hour when I pushed that array update and caught nothing, if you have any cases for big ones would be nice

dim storm Jul 20, 2024, 6:19 PM

#

old hearth I ran phatso's random table library for half an hour when I pushed that array up...

We managed to catch one, but unfortunately it was lost. This was an example of a 6~ row table

#

We'll try to catch it, but it's really much better

old hearth Jul 20, 2024, 7:17 PM

#

ok

#Sfs

` sfs 180003 pon 141591

sfs 180003 pon 141744

sfs 180003 pon 139068

`
sfs 180003
pon 141591

sfs 180003
pon 141744

sfs 180003
pon 139068