ToolKitty

[dark bay]

Interesting it looks like the network can be in any shape we want.

E.g.

Alice <---> Bob <---> Fred.

If Fred changes his doc, it will sync to Bob, which in turn will sync to Alice.

With web rtc, each RTCPeerConnection hooks up just two ppl. To hook up more than 2, you need to make many RTCPeerConnection.
Anyone in the graph can make the extra connections. So the network graph can be in any shape.

We might be able to build something more high level to handle the possibility of multiple connections instead of just one.

[zenith void]

Would be really cool!

Kind of surprising to me that automerge-repo does not have dependency-free adapter for webrtc

[ocean notch]

AI have failed me

[dark bay]

Just copy mine into a playground?... it's confirmed working 😉

But I split the call and ice into two separate QRs which might not be necessary now that the qr scanner is working better.

(Many clicks)

Transfer call, Transfer back answer, Transfer ice.

[ocean notch]

it was only for the compression
it failed to turn it into a playground
also it had a flaw, that it used a dynamic dictionary instead of the static I defined

can't trust AI to write the code for you

did you end up needing all the candidates or you could omit all but the first?

[dark bay]

I didn't try to eliminate. I miss used the qr-scanner library, after I realised my error it scanned all of them with no problem.

Was starting to think about multiple connections and making it more user-friendly.

Maybe it should flow like so:

• invite button shows offer qr.
• join button brings up scanner.
• after joiner scans, it will show accept qr for the invitor.
• the invitor presses scan, and scans the accept qr.

And allow multiple invites to join multiple ppl.

And for PC... copy paste codes, not as friendly.

[dark bay]

... unfortunately I'm gonna have to write a signalling server in php to keep this other guy happy ($)

[ocean notch]

always take the path of least resistance, another nature lesson

[zenith void]

Came across this article on bluesky: https://tailscale.com/blog/how-nat-traversal-works

I didn't understood what that stun/turn/ice stuff in webrtc was for and got explained in it well

Kind of funny and sad all the hoops we have to go through to do the thing the web is all about: connecting computers

[dark bay]

while ur here... is solid-meta the way we are meant to control the title of a web app?

and is meta production ready? 😛 .... sorry, just piecing something together quickly.

[zenith void]

yes, especially if ur using ssr

[zenith void]

for sure

[dark bay]

Cheers

[zenith void]

if ur just csr'ing u can just yolo it and either just change the title in the index.html or document.title = ... if u don't care about seo

[dark bay]

i am just csr'ing... just wanted to do things the solid way.

and there won't be any seo.... its just a monday . com like app

so maybe document.title = ... is enough

actually "Job Management System" is what this guy is calling it.

[zenith void]

lol didn't know about this site

that's so funny to call it monday

[dark bay]

this one? https://monday.com/

[zenith void]

ye

it's funny to claim that day

such a cursed day

[dark bay]

everyone loves "Mondays" xD

[zenith void]

exactly

really funny to tap into that energy for a brand

[dark bay]

thank god for phpstan .... (type system for php). Keeps the dinosaur alive.

[dark bay]

css modules can just have ".css" extension instead of ".module.css" ? (The guy I'm doing this for wants to do his own styling, but I don't wanna confuse him.)
suppose I can just test and see for that one.

[zenith void]

no, it needs the .module.css

so vite knows which css-files to mangle

[dark bay]

right... thanks!

now.... onto php.... pain xD

painful hyper protocol

actually phpstan will lessen my pain (trying it for the 1st time)

and it has vs code plugins

and mysql database to setup rooms for the signaling server

[dark bay]

now that I'm in php.... 6 letter invite codes for connecting ppl over web rtc makes more sense and is easier to do.

[zenith void]

u want to have a php server that manages the webrtc connections for toolkitty?

[dark bay]

just for this monday . com ... this other guy will pay for his own hosting

$4 / month shared hosting, doesn't get much cheaper than that

[dark bay]

Not for tool kitty, I'm a bit cheap 😛

[zenith void]

can't u have a node server for the same price?

or is a requirement?

[dark bay]

its pretty straight forward.... offer data and invite code held in mysql database

[dark bay]

maybe.... its a pre-existing server he already has.

[zenith void]

gotcha

is a long time i wrote php

i actually didn't hate it back then

[dark bay]

there is so many ways to do things, its hard to know which path to take

[zenith void]

if the server is already there it's a good choice i think

[zenith void]

choices are hard, so when they are made for it's a good thing 🙂

[dark bay]

starting very basic:

php shared hosting does not support persistent connections, so I need to poll

I make an invite, and pool for rtc_answer to not be NULL to know when someone joins.

I'll throw in an expiry date time column too

And just poll once every 10 seconds or so.

Will see how it goes, if not stable enough --> switch to node

by persistent connections, i mean php shared hosting does not support "web sockets"

there is a web socket library for php, but it won't work on php shared host

[zenith void]

gotcha

[dark bay]

got phpstan working... easier to setup than I expected

I probably will need to also store the automerge documents in mysql via php, just incase the users browser clears out the indexeddb

[zenith void]

at that point, isn't it easier to go the websocket way?

[dark bay]

.......siiiigh..... wife kicking me off soon to go down town lol

[zenith void]

i m gonna crash soon too 🙂

[dark bay]

I think it will be fine.... Just gotta implement my own StorageAdapter and tunnel it through PHP REST API calls.

I've done way more complicated stuff before.

[dark bay]

catch ya later... and thanks again for ur help

[zenith void]

always a pleasure!

see u tomorrow!

[ocean notch]

the path of least resistance

[dark bay]

codeium (AI) + phpstan, helped me smash out that php signalling server like it was nothing... AI is helpful sometimes.

[ocean notch]

which model?

[dark bay]

GPT4o (preview), then falls back to GPT3 after going over the limit

I forgot how handy it is to have AI

I don't use AI on the TV or on Termux on mobile.

its even auto-generating the client side typescript code to interface with the php rest apis for me

that's what I'm using: https://codeium.com/

has an "e" sorry :P.... there is another AI without the "e"

just the free tier

the type signatures in the php from php stan probably helped the AI a lot.

[ocean notch]

got it, I use gemini and gpt
both failed for example to give me regexp that match ipv6

but it nice to see that it can help with PHP

[dark bay]

That's strange. I fought it would of handled that.

Gemini fails for me to on ipv6

I guess AI is good for stuff u already know the answer to, in order to save on typing.

[ocean notch]

yeah basically all the regexp that I found on a search failed simple cases.
you are right it is probably just GIGO

The ipv6 is not too trivial
you have

::abab
abab::
abab::abab
abab:abab::abab
aaaa:aaaa:aa:aaaa:aaaa:aaaa:aaaa:aaaa

:: can only show once 
can only be at trailing or heading if less than 8 parts
should not match if there are less than 8 parts without :: in any of the separator (including head/trail)

so looks like we are still safe from AI

[dark bay]

That's hard for a regex. U'd probably need a parser for it.

[dark bay]

I'll share some of the code back to tool kitty from "monday . com replica" for the multiple rtc peers (2 or more)

[dark bay]

the network adapter for an existing web rtc connection implemented (without need for signal server). I'll definately copy her back to tool kitty.

and if i just use the indexeddb storage adapter for now, that will be really good for demos. And when I get more time, I can implement another storage adapter for the php backend.

good old php:

cool.... u can add additional network adapters at any time during runtime.

Just gotta work out how to remove them at runtime too:

    let connectionsUi = createConnectionsUi({ server, });
    let networkAdapters = createMemo(mapArray(
        connectionsUi.connections,
        (connection) => new WebRtcAutomergeNetworkAdapter({ connection, }),
    ));
    createComputed(mapArray(
        networkAdapters,
        (networkAdapter) => {
            automergeRepo.networkSubsystem.addNetworkAdapter(networkAdapter);
            onCleanup(() => {
                // How to remove network adapter?
                //automergeRepo.networkSubsystem.
            });
        },
    ));

but thats fine... still good for giving initial demo. I think I just need to implement something inside the network adapter to make/allow them to be removed.

[dark bay]

bit of debugging and I should be all good.

RangeError: could not decode sync message: not enough input
    at __wbg_new_dd6a5dd7b538af21 (@automerge_automerge-repo.js:2134:15)
    at automerge_wasm.wasm.js_sys::RangeError::new::h031611c1a19a934c (wasm://wasm/automerge_wasm.wasm-006ab882)
    at automerge_wasm.wasm.automerge_wasm::error::<impl 

all the code is there, just seeking the bugs

data is definately flowing over webrtc

I suppose I'll try the broadcast network to see if the issue is with my WebRtcAutomergeNetworkAdapter.ts

... ahh its a version conflict in my dependencies

so close to working I can taste it xD

ahhhhh right.... can't use JSON.stringify directly, because it contains a Uint8Array

thats why

gotta incode that Uint8Array to go through RTC

bcuz RTC only accepts strings for sending

Hell Yeah!!! It Works!!!

bet I am the first one to automerge through PHP (though technically through web rtc with a signalling server written in PHP)

now I am stuggling to remember the screen recording software I use in windows to demo it xD... its installed though

ahhh.... Loom

will open 1 window in chrome, the other window in firefox to prove that it is not Broadcast channel.

Here she be, Automerge over a PHP server (shared hosting):
https://youtu.be/0GM33YU1umA

the initial ui is very rough 😄

[ocean notch]

what is this a chat based on tasks?

[dark bay]

Very early prototype of a task list synced over the network via automerge.

Though a PHP server 😆

The kind of server with no persistent connections (Web sockets)

[ocean notch]

nice, what is the php server role?

handling the share codes?

[ocean notch]

you can add auto copy to clipboard when the share code is clicked
add some indication that it was copied to clipboard

[dark bay]

They'll shout across the room.

[dark bay]

Yep.

[ocean notch]

what happen if say connection is lost, can that happen?

[dark bay]

Yep... they just work offline and reconnect and automerge gets it all back in sync.

Both offline / online

[ocean notch]

so automerge always need the entire data?

[dark bay]

Yep... full history.

[ocean notch]

is that for data that is also on the database?

[dark bay]

Will be. I'll implement a storage adapter for it.

Got enough for a demo.

[ocean notch]

ok so maybe I get it, the syncing is only for the data in the browser not the data on the database

in other words, this is not a local first application

[dark bay]

It's using IndexedDB for its storage at the moment. And syncs over web rtc with a php signalling server.

[ocean notch]

yeah I get it now, looks good

[dark bay]

It's works offline too.

So I would consider it local first.

[ocean notch]

yeah I suppose it it

they said it could not be done,
https://regex101.com/r/gN7dgK/1

but with ~3000 char regexp I can finally match ipv6 🤣

[dark bay]

Oh wow 😮

3000 chars lol

Lots of combos

[ocean notch]

might be the first working regexp ever
maybe the ai will train on that one day

I think there might be smaller regexp for this I am just not an expert
with regexp

[dark bay]

An actual parser would be more suitable.

[ocean notch]

I will try to make a parser, I am not a parser expert but I will try

[dark bay]

U can find a library for 1 too.

[ocean notch]

no fun in that

but if I fail, I will let you know, so you can help me find one 😀

[dark bay]

Recursive decent parsers are the easiest to write. I'd research those.

LL(1) parsers are the hardest to write, but are O(1) memory and O(1) time.

Unless ur using a parser generator, LL(1) becomes easier. (Generates parser code from grammar)

@scarlet belfry thank you for your createDocumentProjection<T>(...). It works well.

[dark bay]

I wanna do more. But my wife will get up me 😆.... so many possibilities have opened up.

I think when I get a chance to get back to ToolKitty I'll make every file in the reactive virtual file system a automerge document.

Just gotta work out a story for the pixel editor. If it is fine for the data to be in compressed png file or if it should be a raw uncompressed file.

Automerge will probably remember a full history of every pixel change.

Granted they are gonna be low-res.

Everything else besides the pixel editor will be fine. Because everything else is just simple json data.

[zenith void]

i would look into having a separate automerge-document for each of the textures

and then having a separate automerge document that hooks everything up

[dark bay]

Doc IDs in a Doc sort of thing.

[zenith void]

exactly

it's common practice in automerge

[dark bay]

Sounds like a plan

I keep filenames too for the sake of zip import/export maybe

I wanna also allow the user to use their own 3rd party tools on the data too.

Which should be fine.

files: { filename: string, docId: string, }[]

It may need to diff on import to discover changes.

[scarlet belfry]

haha, classic

this is the experience everyone i know has had after Automerge clicks

[dark bay]

A<->B<->C chain works (3 web rtc connections in a linear chain). I update C, which causes B to update, which causes A to update. Unlimited number of Web rtc connections in a chain:

Web rtc mesh doesn't scale. But u can trade off latency for performance, by placing them in a linear chain instead. As a linear chain, it will scale. But of course, it can open the door for bad actors.

[zenith void]

What would bad actors look like?

[zenith void]

ye also if u ever want to integrate the repl-toolkit it's kind of handy to have some sort of filesystem-like reference for if u want to do imports in ur scripts.

[dark bay]

Actually maybe not. Bcuz A,B and C would only be ppl we know. I was thinking about B lying to A about changes made by C.

[dark bay]

Like implementing ur own git blame... and B is blaming C for something B did 😈

[dark bay]

Then u need a large network of nodes to vote on what really happened (consensus). And before u know it, u have implemented a blockchain.

[ocean notch]

you probably just need to sign/verify changes

which mean your signaling server will also need to take the rule of the key manager

but if every user can make any changes, then they can just make the change, the only difference is if there is importance to who make the change such as in audit, you can then track the authenticity of the changes.

[ocean notch]

grok actually gave me a much smaller expression
https://regex101.com/r/Wf25pU/1

got to 177 bytes down from 2962

[dark bay]

Well done grok!

[dark bay]

"monday . com" didn't work on the that guys network. But worked on the home network. Demo failed 😆

Will need to investigate some more. It could be their VoIP phones their network connection goes though. Who knows.

[dark bay]

I might need to add a TURN server apparently.

And stop using Firefox :p

[zenith void]

that bloody NAT traversal shakes fist to the cloud

[dark bay]

I think I'll give in to peerjs just for demonstrations purposes.

I reckon my php is not waiting long enough for all the ICE candidates.

I can return to php if peerjs works (confirming the location of the problem)

I read some ppl have to wait upto 60 seconds for all the ice to trickle through.

In peerjs, you can supply your peer id, and I think that is what the automerge peer js network adapter does, just uses the document ID as the peer id.

Ohh... it's 3rd party. Not a part of automerge repo packages.

[zenith void]

[it seems like pvh (one of the core members of automerge) left the route of P2P in their research](#1200006941586509876 message)

[zenith void]

fucking hate it so much that it's so painful to connect two computers 😕 makes my blood boil

[dark bay]

It's fine 🙂... it used to be near impossible.

At least these days we have web rtc.

I'll take this thing for a whirl:
https://github.com/automerge/automerge-repo-network-peerjs

[zenith void]

ye we have the primitives at least on the web, but the infrastructure is fighing against us

[dark bay]

And me sticking to old tech :p

Php is not meant for persistent stuff.

[zenith void]

Definitely added layers of friction

[zenith void]

I love how in this article it's like ICE is just throwing everything at the wall until the first thing sticks

[dark bay]

Pretty much. U keep getting candidates until one of them work.

When I get back to the php. I think I just need to implement trickling of ice, instead of trying to grab all the ice at once.

Which would require both sides to continuously poll the php server as they send stuff to the php server... (faking a persistent connection). Just until the web rtc peer connection is complete.

And worse case scenario, I'd just implement a php polling network adapter. It will have high latency, but it will work.

As for ToolKitty.... trickling qr codes.... 😜... I might need a better plan.

[ocean notch]

why not, does it not have GC?

[dark bay]

I mean persistent connections (web sockets)

[ocean notch]

http://socketo.me/

lol http

I am not sure if this is standalone server, but I got that impression that it was

[dark bay]

Won't work on shared hosting 😉 ... I've tried that one.

But can't remember what the issue was. It was a long time ago.

Something to do with one thread per connection, the thread is not freed until the connection ends. And their are only finite threads.

And ratchet needs to keep running in the background which u couldn't do either.

[ocean notch]

sound like a shared hosting problem less of a php problem

[dark bay]

True... I'd probably switch to node than remaining on php if had the choice.

[ocean notch]

or long polling worst case, if you can't switch hosts

[dark bay]

They have their emails.... and WordPress business site on it.

And they want to manage their own server.

WordPress runs on php. And requires no programming ability to update it.

Yep, u can run WordPress on node too :p, just a bit more complicated.

And they'll want to keep using like cPanel.

If I make them switch, and everything is different, they'll freak out.

... recreate cpanel in solid... they'll never know 😆

[ocean notch]

lol, how

yeah, I wouldn't switch

[dark bay]

php is available as a node package. Throw in mysql, then u have a platform that can run WordPress :p... but I won't do that.

https://www.npmjs.com/package/node-php

A vps would be ideal. Their just so expensive though.

But then u loose cpanel (closed source) and have to do everything urself in a Linux terminal.

[ocean notch]

yeah

not

[dark bay]

Old version of php then.

[ocean notch]

what the price for vps compared to shared

[dark bay]

Modern php has opt in type checking.

[dark bay]

$4 / month vs $40 / month for low end 1 CPU I think

[ocean notch]

how much memory you get in shared?

[dark bay]

Shared hosting is cheaper bcuz u have many ppl on thenone host sharing resources of the 1 machine.

Will check

Host papa has a lot of bad reviews, but that's who we are with 😆:
https://www.hostpapa.com.au/buy/aussie-hosting-sale-2025/?gad_source=1&gclid=Cj0KCQjws-S-BhD2ARIsALssG0bBIGdzkSrI5Q1N6iGLjPnw0eSyZpnptE-vl0M3XQQ0-zdR_0dygUIaAtLAEALw_wcB

Ohhh tax deductible. I didn't think of that.

Doesn't tell me much about the actual hardware in shared hosting.

Just storage limits.

Hostinger is much better than host papa. In my experience. Avoid host papa like the plague.

[ocean notch]

yeah, pretty cheap though

[dark bay]

Shared resources shared costed

[ocean notch]

does it mean you can take all the ram

[dark bay]

Until u read the fine print. They triple the price after the first year :p

[ocean notch]

was looking for the fine print, to see limits and such

[dark bay]

U get throttled I think.

[ocean notch]

but it was too fine, can't see it

[dark bay]

Renewing prices... da da daaa:
https://www.hostpapa.co.uk/renewal-pricing/?_gl=1*9qlseh*_gcl_aw*R0NMLjE3NDIyODU3NjcuQ2owS0NRandzLVMtQmhEMkFSSXNBTHNzRzBiQklHZHprU3JJNVExTjZpR0xqUG53MGVTeVpwbnB0RS12bDBNM1hRUTAtemRSXzBkeWdVSWFBdExBRUFMd193Y0I.*_gcl_au*MTcxNjI0NTQ2Mi4xNzQyMjg1NDQx*_ga*MTI2MzA4MDU3OS4xNzQyMjg1NDQx*_ga_0DC3VEFEN7*MTc0MjI4NTQ0MC4xLjEuMTc0MjI4NTg2NC42MC4wLjA.

Gotta convert from brittish pounds to aud for comparison.

Very deceptive pricing.

[ocean notch]

some usage https://www.hostpapa.com/knowledgebase/server-resource-usage-specification-chart/

[dark bay]

1 core, 1gb ram on cheapest shared.

[ocean notch]

and one cpu core, but I wonder if you run a process 100% is that going to flag you?

[ocean notch]

how heavy of usage you have, how much bandwidth for example

[dark bay]

I think it is all load balanced.

I never do or have done anything heavy on the server

I always get the clients to do the heavy work.

Many PCs more powerful than one PC.

Most consumer PCs these days are 16Gb or more ram... and ur using a shared host where everyone (maybe 50 ppl per shared) is sharing 1gb ram :p

[dark bay]

For my bandwidth I am just logging job details so we know how much to charge our clients for software usage.

[ocean notch]

all this runs on one papa shared hosting account?

[dark bay]

We normally charge them around either $3 per square metre or 3% on buildings our clients sell using our software.

[dark bay]

Yep

Only issue we had with them is the sudden price rise after renewal.

Would of been cheaper to go with hostinger. Atleast hostinger are not as deceptive with their pricing.

[ocean notch]

you need 40square meter a year to cover hosting

[dark bay]

Pretty much :p

[ocean notch]

digital ocean seem expensive for 2gb ram
https://www.digitalocean.com/pricing/droplets

[dark bay]

We got a meeting 2mrw with a engineer/software dev to help optimise our engineering.

[ocean notch]

what is wrong with what you have now?

[dark bay]

We just have tabled engineering. And for anything non standard (outside of tables), the engineer takes too long to get back to us.

If we can do the engineering ourselves (the calcs), it would be much better.

I'm no engineer :p

[ocean notch]

tabled engineering ? where is that terminology originated from?

[ocean notch]

you are just the brain behind the operation

[dark bay]

My boss says our cad software kicks sketchups ass, and says he can do non standards in it really quick... but, still need the engineering to know the member sizes / spacing to use.

We've just started working on the non standard cad software for the last 3 months.

[ocean notch]

what is non standards?

[dark bay]

Anything u can get information for from an engineering table or anything that is a bit weird. E.g. non rectangular regions.

Or the customer wanting their building made with non-standard materials not covered by the lookup tables.

One cool thing ToolKitty is pretty much powered by the same reactive entity component system used in the CAD software at work.

I know we can do big things with ToolKitty, the reactive entity component system is a very powerful way to model things.

[ocean notch]

so tool kitty going to be the next sketchup?

[dark bay]

More like a light weight retro game creation platform, I guess is the goal.

I get bored of CAD... wanna make games 😆

Sketch up capabilities can help with level creation... there is overlap.

[ocean notch]

how is the game creation platform going anyway?

[dark bay]

At a stand still. Keep having to go to work :p

I get a chance for a little hack each weekend.

[ocean notch]

is the monday app part of work or extra

[dark bay]

Little bit by little bit

[dark bay]

That is an extra work on the side.

Consumes some weekend time.

[ocean notch]

so kitty is going to stand still for a bit

[dark bay]

Just for a little while.

Monday is teaching me automerge, that I plan to use in kitty.

Still some gain.

[ocean notch]

what is the use case for automerge in hello kitty?

[dark bay]

Fun 😁

U get to co-op design levels.

[ocean notch]

so live collab?

[dark bay]

But also works single person level design

[dark bay]

Yep

[ocean notch]

could be fun

[dark bay]

Maybe multiplayer level testing too

But different technologies will be needed for multplayer in game

[ocean notch]

multiplier in games can be interesting.

[dark bay]

There is a good chance ToolKitty will keep being worked on. Otherwise boredom will come around and kick my ass.

Even if I have to do it all in termux while laying in bed at night.

[dark bay]

How about we have peerjs as an option in ToolKitty. The user can choose to connect via qr codes, or the shared free signalling server provided by peerjs.

If they have the option, and the peerjs signalling server is down for whatever reason, then they can connect via qr codes.

[dark bay]

Writing peerjs code for Monday now, I can reuse it :p

[dark bay]

Will re-demo Monday 2mrw sometime. It is working on my end with peerjs, but will see what happens on their network.

The proper (non php) signalling server will probably help. Then I can fix the php signalling server if that works.

[zenith void]

sounds like a plan!

[dark bay]

Looky looky.... some cPanel shared hosting providers support node applications:
https://docs.cpanel.net/knowledge-base/web-services/how-to-install-a-node.js-application/

Just gotta find a cheap one. (That's just cpanel docs, not a hosting provider.)

Actually, that is probably not shared hosting, will need to research some more.

Shared = cheap 😀

But this guy, apparently it's shared hosting with nodejs:
https://youtu.be/emDCYo5Rz0E?si=GevpGP6kYofQXRfU

It's in his cpanel, so only if ur host supports it ig.

[ocean notch]

if they give everyone the same IP
then you can't really listen to port 3000 or any other port for that matter.

[dark bay]

That's a good point.

[dark bay]

There might be some trickery where the request subdomain proxies the port to the correct running node instance.

Like a proxy listening on port 3000 that redirects based on subdomain.

[ocean notch]

oh yes, if you have multiple processes of nodes, you can reverse proxy based on the domain

[dark bay]

Ohh... the Monday demo went good today (on Wednesday 😅).
I guess Monday is just not a good day of the week.

So peerjs was perfect. It was just the php not collecting enough ice.

I told them, it's automerge, I didn't take credit for the syncing.

They were impressed.

All they want to begin with is the task list turned into sticky notes they can drag around the screen, and have the ability to assign them to certain users. (Tags and filters basically)

Also automerge has something call et-something updates that don't get stored in the doc. (Good for drag animation, then perform the actual doc update on drop)

Forgot the name, but starts with e.

Emtherical or something

Can't find it in the docs for the life of me 😆, but I swear it started with e.

Found it... Ephemeral

[ocean notch]

#1344274340484616244 message

[dark bay]

Thought it seemed familiar... cheers 🍻

How do I even pronounce that word.

Played it google... doesn't even sound English 😆

[ocean notch]

it is an odd word

[zenith void]

do you have a link to the docs?

[ocean notch]

https://automerge.org/docs/repositories/ephemeral/

[zenith void]

a ye ok

that i know

but it's like: here is a broadcastchannel and do with it what you want

so it's still up to the coder to make that into something meaningful

[ocean notch]

it does look like it

[zenith void]

it would be nice if they had something like handle.ephemeral((doc) => ... )

[ocean notch]

can you not create a companion document for "ephemeral" data
and discard when the "owner" is done with it

[zenith void]

probably

played around with a similar idea for a bit

[dark bay]

That would be nice if they had that.

I'm sure I can work around it.

Don't wanna persist every pixel location for a drag animation.

Just the finish location

I'll have a two worlds attack. A reactive ECS that is kept in sync with an automerge doc... that way I can update the ECS we without touching the doc for drag animations.

A double project that u can touch the middle layer if need be.

[dark bay]

Maybe ur solution is easier, a companion doc.

Automerge can also do all the undo/redo for us... no need to implement that instruction based undo/redo manually anymore.

And the undo/redo will persist between sessions.

[ocean notch]

sound nice, seem like it can take on many responsibilities

[dark bay]

If I get a chance, I'll have a crack at it 2nyt in ToolKitty.

Will then try to also knock down atleast 1 task on github.

I notice in pixel studio, their undo/redo persists between save and reload. Which is unusual. I wonder if they are using some sort of sync technology too.

[zenith void]

eh, they have a history, but not really a proper undo/redo system iirc

[dark bay]

Either that or they are putting in the extra effort to make their undo/redo stack a serializable.

[dark bay]

I see.

I suppose a history is an undo/redo system of sorts. Just not a memory efficient one.

Unless it's immutable data with structure sharing... that was the Haskell way :p

[zenith void]

no, sorry i wasn't being clear, i meant that you can view snapshots history, but you can not edit them

there is https://github.com/onsetsoftware/automerge-repo-undo-redo but when i used it in combination with chee's automerge-solid-store i couldn't get it to work

it might have something to do with the store-abstraction, because back then it couldn't handle conflict patches

i think we landed since then on the idea to simply fetch the document again once the automerge-solid-store came across a conflict patch (but not sure if that's already implemented in it)

https://www.youtube.com/watch?v=uP7AKExkMGU is an interesting talk from a year ago on research for undo/redo w automerge

explains some of the complexities around undo/redo in a crdt application

https://dl.acm.org/doi/10.1145/3642976.3653029 the actual paper

Automerge v1 used to have undo/redo but they didn't port that to v2 for some reason

[dark bay]

Ahh right.... I'll stick to our instruction based undo/redo then 😉 ... it's not hard to keep track of.

I've hook up that 6 digit alphanumeric numeric invite/join via peerjs as an option:
https://clinuxrulz.github.io/kitty/#/automerge-webrtc-test

Much easier to connect.

I'll probably just turn that page into a connection management page.

Then get cracking with hooking automerge into kitty.

But not 2nyt... today has been big enough already :p

Work makes ya tired, not much energy left in the afternoon.

[dark bay]

I see, it's not as straightforward in a collaborative setting.

[zenith void]

ye, it's a Hard Problem

especially with json, where anything can be mutated

[dark bay]

"move" can sort of be fixed, by a flat data structure of items that hold IDs of their parents I think.

And undo/redo don't exist if everything is a do from automerge perspective, and u manage undo redo in an instruction based was on ur side.

Granted their could be some undo/redos rendered invalid, bcuz of a pateners edit.

I think from a end user point of view, a user should only be able to undo/redo things they have done and not what their partner has done. Otherwise it gets too confusing anyway.

[dark bay]

Actually that approach can still end up with invalid data, even though the json would avoid cycles.

[zenith void]

what about nested stuff?

// 0. empty toDos
const toDo = []
// 1. user A adds toDo 
const toDo = [{ title: 'buy eggs', description: 'buy 20 eggs' }]
// 2. user B changes toDo description
const toDo = [{ title: 'buy eggs', description: 'buy 40 eggs' }]

should user A still be able to undo action 1.?

in this trivial case it feels like yes to me, but if you generalize it, it could be a whole book that user B wrote inside the object that user A added.

[ocean notch]

free storage
https://turso.tech/pricing

wonder if we can make a very unsafe signaling for the webrtc
or I am sure you will find something to use it for.

[dark bay]

Yes feel correct to be too.

[dark bay]

Also, if A deletes something that B has edits in, B should be able to undo an edit in what A deleted, and if A redos then it should come back with Bs undo applied to it.

That can be achieved by having a isDeleted instead of actually deleting. Just to hide it from the UI.

[dark bay]

It would be confusing for the end users, but I think it is correct.

[dark bay]

5Gb is very generous for a free tier. We have peerjs's free online signalling server hooked up as an option. (6 digit alphanumeric invite codes)

[ocean notch]

oh they provide the codes?

so that is good then for the signaling

[dark bay]

U get to make ur own peer id 😉

let id = 85875338_Kitty_${code_here}

And just use the same prefix for both peers.

(Just incase too many ppl doing 6 digit code thing on the free signalling server)

[ocean notch]

smart

[dark bay]

I was surprised automerge let's you provide only 1 storage adapter. But I guess that makes sense since a server could have its storage adapter and be connected via a network adapter.

[ocean notch]

just finished a proof of concept seem like you can use node as a passthrough proxy
similar conceptually to virtual hosts the shared hosting use.

each client can start its own instance, and the server only exposes a single port

the tricky part is to get each app its own ssl termination, without the proxy needing to know of each app certificates
and just let the app handle the handshake, a typical reverse proxy also terminate the tls.

[dark bay]

Very cool. The ssl (https) stuff is way over my head, I just let the hosting provider take care of that.

[dark bay]

When u think about, shared hosting is already doing this magic. Everyone connects to ur site through port 443 (https) yet u sharing the same ip with a bunch of other ppl. The only thing that distinguishes u is the url domain name.

[dark bay]

Probably time to do a proper connection management page, so I can sign off web rtc. The other pages can make use of the connections setup in the connection management page through their network adapters for automerge.

To keep things simple the connection management page can provide a signal of network adapters for the other pages to use.

[dark bay]

If I am tired of dealing with CSS styling but still want something that looks OK, is daisey UI + tailwind the best way to go?

I literally have no patience for style 😛, and don't really care about bundle size at this stage.

[dark bay]

... i'm using it, the CSS reset has me annoyed, everything just looks like normal text unless u go and write CSS for it otherwise, and my artistic skills are nil.

[ocean notch]

stuff is way over my head
whoops, know your audience

yes, shared hosting use the same idea to support virtual hosts ( specifically multiple domains on the same ip )

the concept is simple the server need to know the hostname then from there it can just server that client file

with ssl, there is one problem is that the request is encrypted so you can't check the header
instead the ssl handshake itself, the client sends the domain ( they call it SNI - Server Name Indication )

and from there it is the same story of how they used to do that with http://

so my proxy just needs to read the SNI from the initial handshake
and then just proxy the connection to the ip:port of that domain

nothing too magical.

[ocean notch]

styling, programmers worst enemy

and we are not yet even talking responsive design

[dark bay]

Ur surprising me with ur knowledge. Are u planning on selling nodejs shared hosting?

[ocean notch]

haha I don't think there is much demand for that.

also I think node per client might have more resource requirements then
a say nginx or apache where you can have 10,000 clients
and resources are only used when requests are made
over holding 10k node processes for each client ( 1 server vs 10k+1)

I also didn't think shared hosting was still a thing now that most people go the vps
as it could be as cheap as shared hosting for some use cases.
and provide better isolation and security

[dark bay]

True... me and my workplace are a little behind in that area.

[ocean notch]

its ok, you don't fix what ain't broken

and if what you have works for you then why not.
the alternative do have the potential to be more expensive

it only when what you need can not be provided. is when you need to look into alternatives

[dark bay]

Yeah... if we do switch, it will need WordPress support.

[ocean notch]

maybe if budget allows, you keep the wordpress where it is
seem to do fine where it is.

then for the advance stuff you look into the alternatives

[dark bay]

very simple connection manager

generate several invites, shoot them to your friends (1 invite per friend), and they can all connect at the same time.

and u can kick them if they are naughty.

nicknames are auto-generated so users don't need to think of names.

there will be a table of connections/invites below in the black area 😛

[ocean notch]

how does kicking works?

[dark bay]

Just disconnects them for now. If u don't send them another invite code, then they can't join.

Haven't thought it through 100% :p

[ocean notch]

so in your case the person who invites is the bridge between all the invitees
or they all open a connection to each other?

[dark bay]

If they delete all ur levels while offline, then join you. All ur levels get wiped out.

[dark bay]

Ppl can invite each other. And ppl can kick/disconnect any connected peer.

But not sure if kick makes sense

There is no restricted access in automerge repo by default.

I guess the inviter is whose level (document id) u will be working on.

Not really sure yet.

[ocean notch]

I guess if you invite someone you are inviting them to share the whole project?

[dark bay]

Feels unsafe :p

[ocean notch]

so you want to invite per part of the app?

[dark bay]

Maybe it should only establish a connection, and u have to shoot individual document invites after connected.

[dark bay]

Yeah, that way might be safer.

All the document IDs are UUIDs that would be impossible to guess.

It's possible just to connect them all to automerge and ur documents will still be safe.

[ocean notch]

yeah ok

so connection is step one

step 2 the inviter decide what he wants to collab on

but this might be complex UI wise

how should collab work?

do I open a view, say share, then I go to another view
is the collaborator going to see the switch?

[dark bay]

And only share documents IDs for documents u wish to share.

[ocean notch]

how does it even work with the automerge

how do you decide who going to get what document ?

[dark bay]

I think the remote network adapters register interest in certain documents.

If the remote peers don't have a document ID, then they can not access that document.

The document ID in a way is the password to access he document it reference (its a UUID, impossible to guess one)

[ocean notch]

once you give the id, there is no way to revoke access?

[dark bay]

Ahh true... u can fork ur document id into a new document ID to keep it safe.
There is no revoke.

They sync automatically. Unless we implement our own automerge repo with access control. And use the bare bones automerge.

[dark bay]

U can share a fork, and merge it back manually and only share a forked document too.

It's tricky

Maybe export before invite for a backup.

Or.... get rid of the kick button, and only invite ppl u trust :p

[ocean notch]

backup sound good

can you fork the whole thing

and let the collab be on the fork

when the session is done, if user is happy. he commits the fork

[dark bay]

Rename kick to disconnect so it seems more friendly.... (I'm just signing out)

[dark bay]

The problem is, I think forks have a document ID commit history that can be used to backtrack to the original document id (unsure)

Sort of like git log

So a bad actor can still get at the original document.

An export and import to remote works around that though.

[ocean notch]

what if you fork the entire automerge

like pretend when you collab you split your computer to 2
the collab is on computer 2 and the collab

if all goes well
you switch to use computer2 or actually allow the sync back from 2 to 1

basically same as a backup / snapshot so maybe simpler

[dark bay]

Messy... but that might do the trick.

[ocean notch]

I am just hypothesizing without really knowing how automerge work

[dark bay]

automerge documents have a merge function, what u suggested will work.

And exported automerge documents generate binary data with no document id attached. (Safe)

But less fun :p

It's more fun to see stuff change on ur screen while they do stuff at the same time.

Maybe clone, then share

Get the end user to keep their own backups

As long as they have a backup they are safe

[ocean notch]

yes the fork is for the real time sharing so you should still be able to see it

at least that is what I had in mind

backups sound like good idea anyway

[dark bay]

Then u can also merge back into ur original if ur happy with the changes.

Sort of like github fork and pull requests

[ocean notch]

is automerge become your source of truth for the data?

[dark bay]

Yes I think so

Atleast for this side project.

[ocean notch]

how does it work with the reactivity part?

[dark bay]

I can embed an automerge doc deep inside, and it all works the same as before on the surface.

The reactive entity component system will wrap an automerge doc.

There is a projectDocument(...) to react to changes made by peers

It basically turns a automerge doc into something like a solidjs store

[ocean notch]

so if i understand this correctly

you take doc A
fork it you have A1 A2
you start a collab transaction on A1
if all goes well you commit() - which mean you just discard A2
if there is a problem you rollback() - which mean you merge A2 back to A1

does this technically work?

[dark bay]

Correct... except u have A and A2 after the fork.

[ocean notch]

true

you mean A1 is just A

[dark bay]

Also automerge documents are immutable, they develop a long connecting history chains (there is always a backup)

[ocean notch]

ok this seem like easy to solve

the more challenging thing will probably be how you actually share
in the UI

it will probably be nice to have share whole project for people you trust
and share specific documents for people you don't

then I guess you can treat each share as a "session" and you can jump to different sessions

[ocean notch]

does that mean you can just take a commit id and revert to that on a document level?

[dark bay]

Pretty much.

So it's always safe ig

[ocean notch]

ok so I should just think of it like git

[dark bay]

It's like git but more fine grain.

[ocean notch]

basically each document is a git repo?

[dark bay]

Yes and no, they may be connected by commit chains or not.

Like 1 repo with many branches ig

Each doc is like a branch

If it's an orphan branch, then it's like another repo

[ocean notch]

oh interesting

when you sync changes from the other person

do you have something like a middleware where you can block some changes
for example write to a document the user does not have permissions ?

[dark bay]

Nope... just hidden documents IDs is my only defence.

[ocean notch]

so that mean we have to work on A2 ?

[dark bay]

Yep

And each file in the reactive virtual file system will be an individual document

And the file system itself will be a document that references other documents by document id I guess (still working that part out)

A little tricky, bcuz u gotta share multiple documents to even use the level editor.

Image files, tileset meta data files, level file.

[ocean notch]

yeah, so when you share do you have to fork all dependencies to protect them all?

[dark bay]

Yep

And change the references in the file data to reflect the new IDs of the forked documents.

Not just a copy of the data.

Unless I have secondary IDs so the file data doesn't need to change.

[ocean notch]

so it is either this, or invite only people you trust

[dark bay]

Trust is sounding easier

We will need a virtual file system manager. So we can have multiple named virtual file systems too.

Like an isolated environment for files like native mobile apps use for their app data.

We can select a file system we wanna work on and have backups

Fork the whole filesystem and collab

Which is kinda the same as export/import zip ig

[ocean notch]

this can get complicated, what if you have 2 tabs
and you share each with different users ?

[dark bay]

Should be fine... each browser tab is a separate process with its own memory.

[ocean notch]

you will need full clone for each collab

and your persistence layer will need to support all

[dark bay]

Yep, and a 500mb limit.

Ig when u fork it doesn't consume more memory unless the file data changes

And it only stores the diffs in memory (what has changed)

[ocean notch]

and we need to anonymize the ids and de anonymize when commiting

[dark bay]

Maybe trust + manual backups

Manual backups are export/import.

It's can be like a primarily single user app, but u can collab with friends if u wish too (as an extra feature)

[ocean notch]

well some good thinking today.

some interesting edge cases.

and good conclusions for a starting point

[dark bay]

hmm... might be more suitable for generating a peer ID from a 6 digit invite code:

export function inviteCodeToPeerId(inviteCode: string): string {
    return md5(SALT + inviteCode);
}

at least it will look like a proper peer id

[dark bay]

Neither here nor there. The end user never sees the peer id. :p

[zenith void]

Have you looked at the bee/keyhive stuff? Automerge+auth layer. They talked about it in their latest community meeting:
https://us02web.zoom.us/rec/share/1D-BF8Zr7p5hQ7AV9jmz7yBC-2skAk7csE2I5w3wjdrofTu0-9laN1BfArmFm64.Jzt7kaWVyC9t7yFM?startTime=1740071291000

[dark bay]

oh.... I haven't seen that yet. I'll watch it tonight in bed. (like a bedtime story)

[zenith void]

It's a good watch! You will see me and chee fanboying in the chat too 😁 + they have been keeping notes of their research too. decentralized auth is quite the mindfuck 🙃

[dark bay]

A very scary bedtime story then :p

Hopefully their library can abstract away all the complication so we don't have to think about it.

[zenith void]

That's the goal!

Also ToolKitty crossed 1500 messages. When we get at 2000 we should throw a party 🥳

[dark bay]

Hahaha.... we shouldn't celebrate my spammyness

[zenith void]

Our spammyness 🤝

[dark bay]

Good thing it's in here and not on off topic

[zenith void]

I m very pleased with this #1343933024223363143 😄

Was a good addition

[dark bay]

Here is ur web rtc connection manager:
https://youtube.com/shorts/Ejy1oqCWG7k?si=XfLorPRXqcTJh0aN

[zenith void]

Lesgooo

Looks clean too 👌

[dark bay]

All thanks to daiseyui... my css sucks 😆

[zenith void]

That's what component libs are for!

I can do a design pass for toolkitty if u want at some point. I m not too terrible with it.

[dark bay]

That would be neat. Just beware, it's missing a lot of functionality. It is kinda mashed together like Frankenstein.

[zenith void]

Ye, that's why rn is mb not the best time to do it.

But u do a have a bunch of cool pieces

[dark bay]

Getting there slowly 🐌

[zenith void]

Once u wanna bring them together in something more coherent I can do a pass

[dark bay]

Cheers 🍻

U know one thought I had was. Instead of doing many mini projects. I would put anything I was currently interested in one monolithic project (Bundled together). That way the project will always be worked on.

[zenith void]

I thought u were already doing a mono repo?

[dark bay]

Yep... that's the thought that lead to ToolKitty.

I mean, that's was the thought that inspired it.

Oh lol... what on earth... the joined connections in the video have a different nicknames from the joiners 😆... I think I messed up somewhere, how is that even possible?

I must of generated the nickname twice per client by accident.

Time to watch that long automerge auth layer video.

[dark bay]

And I feel a sleep 10 minutes will try to re-watch it later.

Even if I can read the api docs, I'll get a feel for it.

[dark bay]

Is there a link to the github repo for keyhive?

I've come up with a cleaner strategy for hooking up automerge into the ECS.
All I need to do is make a ECS system that syncs the scene to an automerge doc.

That will allow me to switch between different sync engines if need be, or even switch sync off and on.

[dark bay]

And anything using ecs, will instantly be able to use automerge as an opt in.

Interestingly though, a regular ECS (non reactive impl) handles reactive to some extend as the pattern does not impose any limitations of what components contain (e.g. POJO not required). It is perfectly fine for components to contain signals.

A projection over a regular ECS can make it a reactive ECS. Allowing u to use any existing regular ECS libraries in the wild as a reactive ECS.

[zenith void]

In case u hadn't found it yet: https://github.com/inkandswitch/keyhive

[zenith void]

That does sound interesting. Then you don't have to buy in 100% into automerge. Maybe it could also help with the ephemeral state if you are being more explicit with the syncing.

The con is that you will have duplication of state ig

[dark bay]

Yep. That's no biggy. Computers and phone have a lot of ram these days.

It's common in ECS to take the synchronised duplicate state approach when mixing with physics engines.

As the physics engines will have their own representation of bodies to make their updates more efficient.

[zenith void]

Makes sense

[dark bay]

They call that approach, the two worlds approach.

[dark bay]

Next goal I guess is that automerge ecs system for syncing a scene against an automerge document.

that createDocumentProjection thing is so clever. It's basically just powered by solid's produce on a solid store, and applies automerge patches through the mutable object provided by produce.

[dark bay]

made a start, she is a tricky beast:
https://pastebin.com/raw/HJtuj3BP

once I reach the component level, everything is just a solid store. So hopefully I can reuse createDocumentProjection at that point.

If I would of just used a Store for the ECS World, it would be a lot easier too. I'll make that a last resort if I get stuck.

[dark bay]

Actually, I can use the TypeSchemas to generate sync code for the components.

[dark bay]

OK... the 1st half is done for observing an automerge document to update a ECS scene.... now to observe the ECS scene to update the automerge document (the second half)

all untested at this stage :P... will be a fun debug session.

I need to keep track of if the transaction was caused by automerge or by the user in a flag to prevent an infinite loop.

[dark bay]

not sure if this pattern can work, will see:

function syncWorldToAutomergeDoc(world: EcsWorld, docHandle: DocHandle<any>, isTransactionAutomerge: () => boolean) {
    createComputed(on(
        world.entities,
        mapArray(
            world.entities,
            (entity) => {
                if (isTransactionAutomerge()) {
                    return;
                }
                // handle entity added by user here
                onCleanup(() => {
                    // handle entity removed by user here
                });      
            },
        ),
        { defer: true, },
    ));
}

wanna observe the world, and only do something if it was the "user" that modified the world, and not "automerge"

and also wanna skip the first load

a little bit sketchy

two way data binding in the goal there, and prevent infinite loop

I think I'll have to think a bit more about that one

maybe I need to wire it up normally, and use the isTransactionAutomerge() flag callback only surrounding the code that is about to mutate the automerge document.

otherwise I'll loose sync connections

[dark bay]

debugging time 😅 (the two way binding automerge ecs system is complete and untested)

[zenith void]

lesgooooo

[dark bay]

Needed to walk away from the code before testing, feels kinda sketchy:

https://github.com/clinuxrulz/solid-kitty/blob/main/src/ecs/systems/AutomergeEcsSyncSystem.ts

Trying to convince myself that there will not be an infinite loop.

And I was a bit lazy at the component level.

Will test it in a little while. Just thinking about where my document IDs will go.

At the moment I have a filesystem with a folder called levels containing like: level1.json, level2.json, etc.
Each containing the level data (as a serialised ecs scene)

I could have a file like: level1.id, and if it ends in the extension id, then it contains just a document ID.

Will think about it a bit.

[dark bay]

OK... I'll use automerge documents for all the files in the filesystem :p, gotta get some consistency going.

[dark bay]

Deciding if I should make virtual folders separate automerge documents too. Because then I can expose the same api currently used for the virtual file system.

The folder id becomes the automerge document id

If I can keep the same api, I can save on a lot of hook up code.

Current plan:

type VfsFile = {
    type: "File",
    docUrl: string,
};

type VfsFolder = {
    type: "Folder",
    docUrl: string,
};

type VfsFolderContents = {
    [name: string]: VfsFileOrFolder,
};

type VfsFileOrFolder = VfsFile | VfsFolder;

export type AutomergeVirtualFileSystemState = {
    root: {
        docUrl: string,
    },
};

[dark bay]

Actually won't be exactly the same api exposed, bcuz I'll be pulling automerge docs out instead of text/binary data for the fine grain updates through automerge.

[dark bay]

Getting through the automerge virtual file system slowly... another (maybe smarter) thing I could of done was an abstract interface over the ECS so I could use physical use a real automerge doc under the hood, instead of two way data binding (to avoid extra complexity). But I will try the two way data binding 1st, since I already put the effort in.

[vocal saffron]

Haha

[dark bay]

hahaha

I've almost done enough for testing... give me a chance xD

but yeah, I got last backup options just in case.

I'm hooking up the automerge virtual file system as we speak

while baby siting, divided attention

[ocean notch]

haha all I saw was

(to avoid extra complexity). But I will try the two way data binding 1st,

just a heads up that you pick the simpler solution, I am at the point that I no longer understand what you are doing
so I am just waiting to see the outcome

[dark bay]

there is a trade off.... if I go two way data binding, then I am not dependent on automerge (not a 100% vendor lock in)... If I deeply embed automerge, then it is a 100% vendor lock in.

[ocean notch]

ok , so I prefer the no lock in

[dark bay]

I'm trying no lock-in first 😄 ... will see how it goes... but the easier path is vendor lock in

I'm pretty sure automerge is awesome and won't disappear overnight, so vendor lock in is OK too

[ocean notch]

it is not about disappear but more about unnecessary
complexity

also possible bloat and possible overhead

each feature should be like an onion layer
and the system should not be like cutting onions that makes you cry.

[dark bay]

Yeah.... usage wise though, the two way data binding is just adding an ecs system to do the binding (very isolated)

Just the code for the two way binding itself with automerge patches is a head spin.

[ocean notch]

yeah, might not be possible?

[dark bay]

The system can be switched on and off

[dark bay]

I've done the code for it already. Just not tested yet.

Need a few more parts b4 I can test it

Just need to get our virtual file system to pass around automerge document references instead of blobs

[ocean notch]

so the automerge document is what you actually save in the virtual file system?

[dark bay]

Yes

And the file system itself will be an automerge document too

So we can share file systems

[ocean notch]

how do you turn off automerge in this story?

[dark bay]

By using the file/folder id to either be what it is at the moment or a automerge url

Having the same id take on 2 forms

Both type string

Even making an abstract interface for the vfs to switch between the two

With that said though, automerge will work on a single PC via indexeddb

[ocean notch]

so you have the VFS, you have automerge and you have solid store

[dark bay]

Our old vfs + automerge vfs

Not 100% confident I'll avoid vendor lock in :p

But will try

[ocean notch]

when you say vendor lock in ,you mean a different sync engine ? or no sync engine

[dark bay]

Yeah... different or none

I suppose that can be worried about later with a refactor

[ocean notch]

that will not age well

[dark bay]

Say automerge is no longer open source one day, and they wanna charge for usage, u never know.

[ocean notch]

you can still use whatever version before they change license

[dark bay]

Sync engines will all have a similar interface in case of a need for a switch I'm sure.

[dark bay]

True

[ocean notch]

what is the preformance and cost of automerge?

I hear it keeps all history?

[dark bay]

We can always abstract the sync engines later on for easy switching too

[dark bay]

It keeps all its history, but u can manually/programmicly cull it.

Like if u know all ur peers are in sync, u can do an orphan branch with no history and start from there.

[ocean notch]

ok that good, so no infinite bloat

[dark bay]

Bloat is an opt in 😉

Or opt out

[ocean notch]

are you still using solid store for reactivity?

[dark bay]

Performance wise automerge is getting faster everyday. But need to test it some more.

[dark bay]

ReactiveMap mainly from solid primitives

Solid stores leak when u delete keys

[ocean notch]

how does it leak?

[dark bay]

Solid stores are not made to be used like a map

The keys never truly get deleted when u set them to undefined

Just incase the key is being observed and will be set in the future (is the argument)

ReactiveMap doesn't have that problem

[ocean notch]

oh I see, so the signal for the key is never freed?

[dark bay]

But if u are using them not as a map, then it is fine.

Like a fixed type signature

[ocean notch]

it is not bug but a feature

[dark bay]

Fixed number of keys

[dark bay]

Hard to say... I'm not convinced :p

Bcuz u can reference count ur observers to know when a key can be truly deleted

(onCleanup to decrement ref count)

[ocean notch]

well in a way, they can free the signal when there are no observers

but the chance of that happening when you call the delete is hard to enforce as the user

yeah, you will need a GC trackign which cost, and solid mostly choose lighter paths

[dark bay]

Yeah

It might of been left out for performance reasons.

[ocean notch]

it will be nice if there was an api freeSignal() which can remove the signal from all its observers

[dark bay]

Solid 2.0 might have something comming for it

[ocean notch]

then
delete(store,key)

can safely remove the signal for the key

[dark bay]

Atleast we have ReactiveMap we can use from solid primitives

[ocean notch]

so the ReactiveMap can not handle the createComputed(()=>{m.has(key)})?

[dark bay]

It's if u want a map and not a struct then use a map sort of thing :p

[dark bay]

It can handle that

Bcuz it does reference counting

[ocean notch]

ohhhhh

I see

[dark bay]

It's very smart under the hood that one

The has will even create a signal if one does not exist, and reference count it.

It's what I wish store was :p

[ocean notch]

needs to see the implementation, it sounds good

[dark bay]

It's based on another solid primitives called trigger I think

The trigger does the reference counting.

[dark bay]

guess its time to push the automerge file system into the texture atlas / level editor and try her out. I'll do it in a separate branch for a play, and to see how well its gonna work.

I don't have the exact same interface exposed on the automerge virtual file system vs our reactive virtual file system, so I think I'll just go for it and plug her in with changes

I can abstract away later to supply a common interface between both later on

basically I can not do sync by using just blobs for data :P.... so the interface had to change anyway

[dark bay]

The exercise will let me understand the shape of the common interface

[dark bay]

Usability is not too bad, here is how you sync an ECS world against a automerge document:

let textureAtlasWorld = new EcsWorld();
let syncSystem = createAutomergeEcsSyncSystem({
    registry,
    world: textureAtlasWorld,
    docHandle: textureAtlasData3,
});
if (syncSystem.type == "Err") {
    return asyncFailed(syncSystem.message);
}
let syncSystem2 = syncSystem.value;
onCleanup(() => syncSystem2.dispose());

[dark bay]

Let's of debugging when get to the end of this.... everywhere the code touched the old reactive virtual file system is getting changed 😜

Which is fine, bcuz the old rvfs didn't support fine grain sync. So it's shape had to change anyway.

[dark bay]

...oh autosave doesn't make sense anymore... code I can delete, yay!

Technically all the texture atlas code is hooked up to automerge now (untested)
Onto hooking up the level editor

[dark bay]

1 last file to sort out, but ran out of time 2nyt. After that I can start debugging.

[dark bay]

Down to no type errors (on branch "automerge"), debugging starts tomorrow.

Opened in Termux while laying in bed:

[plugin:vite:import-analysis] Failed to resolve import "solid-js/types/server/reactive.js"

😆

[ocean notch]

when did you add a server to the story?

[dark bay]

Wdym?

[ocean notch]

solid-js/types/server/reactive.js

isn't the app CSR

[dark bay]

Ahh... that was an autocomplete accident.

[ocean notch]

I was scared

[dark bay]

It's peer to peer still

Just gotta find time to debug the "automerge" branch before I merge it into "main"

From termux mobile, it seems buggy still.

[ocean notch]

need some pc time 🙂

[dark bay]

Debugging on a mobile phone sucks :p

[ocean notch]

even on PC it is bad, so I can imagine how bad it can be on mobile

[dark bay]

Kiwi browser gives u dev tools atleast:

[ocean notch]

oh that is nice

[dark bay]

It's a shame kiwi browser is being discontinued.

It's the only android browser with dev tools in existence.

[ocean notch]

yeah very sad.

no idea why browsers don't ship with it, where there is more mobile users than pc

[dark bay]

99.9% of ppl are not developers I guess.

[ocean notch]

perhaps that is why, but developers that develop for mobile need better tools to create stuff for mobile users

[dark bay]

I'm debugging for desktop from a mobile 😆

But also for mobile too ig

U can remote debug ur mobile from desktop

[ocean notch]

yes, at least that, but do you need to be usb connected or wifi works?

[dark bay]

Used to be usb only, but wifi works now too.

Kiwi browser is just a fork of chromium with dev tools enabled.

Would be nice if there was just a flag in chrome for android, to just switch it on.

It works fine. (In kiwi)

[ocean notch]

If it easy I hope someone will just release chromium at least with the dev tools enabled

[dark bay]

Sorted a couple of more bugs... then got kick off the pc :p

It's gonna be sick when it's done.

Fine grain collab

[dark bay]

ok... two way databinding against automerge doc is a no go, because automerge doc diff event handlers are deferred

I'll have to deeply embed an automerge doc inside EcsWorld. no choice

I'll make an interface against the EcsWorld for easy switching called IEcsWorld the "I" for interface

[ocean notch]

so for 2 way data binding you needed sync code?

sync as in not async

[dark bay]

I got myself in an infinite loop, and the change events from automerge were delayed, so I couldn't just use a flag to know what transaction I was in.

so I'd update, world, that would update doc, that would update world, ...

what I really gotta do is make a proper projection:
I should only ever write to the automerge doc
And only read from the EcsWorld
And have something in between to update the world from the doc

that way I won't have an infinite loop

I'm gonna use IEcsWorld to abstract that projection away

export interface IEcsWorld {
    entities(): string[];
    entitiesWithComponentType(componentType: IsEcsComponentType): string[];
    createEntityWithId(entityId: string, components: IsEcsComponent[]): void;
    createEntity(components: IsEcsComponent[]): void;
    destroyEntity(entityId: string): void;
    getComponent<A extends object>(
        entityId: string,
        componentType: EcsComponentType<A>
    ): EcsComponent<A> | undefined;
    getComponents(entityId: string): IsEcsComponent[];
    unsetComponent(entityId: string, componentType: IsEcsComponentType): void;
    unsetComponents(entityId: string, componentTypes: IsEcsComponentType[]): void;
}

the interface will allow me to write to automerge doc, which updates the EcsWorld, and I only ready from the EcsWorld

[ocean notch]

yeah, I think async code is the enemy of this scenario

so you are planning on making the data flow in one direction?

[dark bay]

yeap, back to 1 directional data flow

[ocean notch]

how is EcsWorld get updated when automerge is updated?

[dark bay]

automerge doc has event handlers to know what parts get updated

it gives you patch information, (a path which a change description)

its tricky but worth it

[ocean notch]

ok so automerge doc emits patching messages

you listen to those changes, and you can know when a document changes
so you can update the IECsWorld

what is the IECsWOrld?

[dark bay]

IEcsWorld is the shape of the EcsWorld class to hide implementation details from the rest of the code.

IEcsWorld will pretend automerge docs don't exist

Local_User_Change -> Update's autmerge doc -> remote user gets it -> remote users EcsWorld updated

and vice verse (swapping local and remote)

it will allow 2 or more ppl to work on the same level in real time without saving. It will all just be live and in-sync.

the old-two-way-data-binding was observing ecsworld and updating automerge document (causing infinite loop)

[ocean notch]

so this will just be in memory? at what point is that commited to storage?

[dark bay]

automerge automatically commits to storage as well

through its storage adapter

I think it has like a 3 second or so throttle

maybe shorter than that

and you and ur partner can also continue working on the level offline, and when a connection comes back all the changes catch up and sync together

[ocean notch]

old-two-way-data-binding was observing ecsworld and updating automerge document (causing infinite loop)

what if escworld was another automerge?
wouldn't it eventually realize that there is nothing to update

[dark bay]

not sure 😛 ... I have abandoned two way now, too much of a head spin.

deferred update events sort of kill it

I'll keep pushing, pretty sure I am on the right track now

IEcsWorld will allow me to switch to another sync engine down the track if I need to

or use no sync engine at all too

EcsWorld is basically the scene for the entity component system

[ocean notch]

yeah , I am just thinking out loud

unidirectional works for UI frameworks I guess it might work for this too

[dark bay]

I was foolish to go two way 😛

unidirectional to the rescue

gotta learn somehow I guess

I should of listened when u showed me the sinking ship xD

will this is the start of her:
https://github.com/clinuxrulz/solid-kitty/blob/automerge/src/ecs/EcsWorldAutomergeProjection.ts

All the read operations go through world
And all the write operations will go through docHandle
And I will add event listeners to update world from changes that happen to docHandle

[ocean notch]

so world is reader and docHandle is writter

[dark bay]

docHandle gets updated both locally and remotely via the magic of automerge

[dark bay]

Yes, they call that a projection

[ocean notch]

got it

[dark bay]

so the rest of the code will work with IEcsWorld and not even know it is talking to automerge

it will be like magic

[ocean notch]

that we call abstraction?

[dark bay]

yep 😛

[ocean notch]

how far is this from a working poc

[dark bay]

got cards with the visitors soon, so may be next weekend

don't think I can do it quick enough for today

then back to work for the week

[ocean notch]

sounds goods,
you playing cards?

[dark bay]

very soon... got a couple of minutes, will see if I can do some magic

I've got the two way code that I can copy into the one way code

[ocean notch]

last minutes changes

[dark bay]

just need to structure it

[ocean notch]

me talking slows you down 😄

[dark bay]

its fine... I won't finish it today

I've hit another complication

IEcsWorld.getComponent is both read and write

because the returned component can be mutated

so I need to abstract the component as well like I did with the world

IEcsComponent wrapped around EcsComponent so I can capture the writes and redirect them through the docHandle

a component is a part of an entity in a scene... like a frame for a tile for example

I could change it so components are immutable and send new components to update... but wanna keep same interface so I don't have to change the code that touches it.

quite tricky 😛

I'll do as much as I can for today

I might be able to cut a EcsComponent is half, a read half and a write half, and stitch them back together.

I think carving my EcsComponents in half is gonna work

[ocean notch]

so you need to wire the mutation of the component to the docHandle

[dark bay]

TADA:

    createEntityWithId(entityId: string, components: IsEcsComponent[]): void {
        this.docHandle.change((doc) => {
            let components2: { [t: string]: any } = {};
            for (let component of components) {
                let component2 = component as EcsComponent<object>;
                let component3 =
                    saveToJsonViaTypeSchema(
                        component2.type.typeSchema,
                        component2.state
                    );
                components2[component2.type.typeName] = component3;
            }
            doc[entityId] = components2;
        });
        for (let component of components) {
            let component2 = component as EcsComponent<object>;
            let key = entityId + "_" + component.type.typeName;
            let dispose = createRoot((dispose) => {
                let componentJson = createMemo(() =>
                    saveToJsonViaTypeSchema(
                        component2.type.typeSchema,
                        component2.state,
                    )
                );
                createComputed(on(
                    componentJson,
                    (componentJson2) => {
                        this.docHandle.change((doc) => {
                            doc[entityId][component2.type.typeName] = componentJson2;
                        });
                    },
                    { defer: true, },
                ));
                return dispose;
            });
            this.keepAliveMap.set(key, dispose);
        }
    }

will debug it later

tricky as hell

gotta start cards... catch ya later

[zenith void]

ugh hate those auto-imports

u can exclude them from typescript if u wanna: https://www.typescriptlang.org/docs/handbook/release-notes/typescript-4-8.html#exclude-specific-files-from-auto-imports

[dark bay]

That last TADA trick above is not gonna work. But I have another solution.

It won't catch changes from peers

I've gotta do a proper projection through EcsComponent like I did to EcsWorld. Not out of the woods yet, but very close.

[dark bay]

Cheers, so u can exclude certain patterns of imports.

[zenith void]

exactly

[dark bay]

I believe this will work for creating a projection for EcsComponent:

    createJsonProjection(initS: S, json: Accessor<any>, setJson: (x: any) => void) {
        let [state1, setState1] = createStore(initS);
        let [state2, setState2] = createStore(initS);
        createComputed(on(
            json,
            (json2) => {
                let s = loadFromJsonViaTypeSchema<S>(this.typeSchema, json2);
                if (s.type == "Err") {
                    return;
                }
                setState1(s.value);
            },
            { defer: true },
        ));
        let state2Json = createMemo(() => saveToJsonViaTypeSchema(this.typeSchema, state2));
        createComputed(on(
            state2Json,
            (state2Json2) => setJson(state2Json2),
            { defer: true, }
        ))
        return new EcsComponent({
            type: this,
            state: state1,
            setState: setState2,
        });
    }

the json and setJson route through the automerge doc

its like a 2 way data bind, but only going in 1 direction

the downside is the duplicate state

Usage:

let component6 = componentType2.createJsonProjection(
    untrack(component5),
    createMemo(() => doc[entity][componentTypeName]),
    (x: any) => docHandle.change((doc2) => doc2[entity][componentTypeName] = x),
);

where doc is a makeDocumentProjection of the docHandle

sort of like a swap the wires approach

still an infinite loop xD ... will keep trying

[dark bay]

Cross wires did the trick!

[dark bay]

@ocean notch got ur poc floating in the gif animation above this message. Still bugs for me to sort out, I'll think it will be the comming weekend job.

[dark bay]

Actually I can probably improve on this. Don't think it is required to make any stores here. I should just be able to tunnel it though.

[zenith void]

lesgoooooo

man i love these super low pixel marios so much

[dark bay]

Some ppl have gone lower too:
https://www.deviantart.com/eriklectric/art/Nanosprites-21-nano-Mario-sprite-sheet-916645944

[zenith void]

pretty! but not 100% sure i would read them as mario if i would see them in the wild

that moustache

https://www.deviantart.com/eriklectric/art/Nanosprites-13-Mario-and-Yoshi-animated-899065419 looks cool tho in motion

[dark bay]

Yoshi 😆.... took a long time to figure out what's going on. (Is he getting eaten by a plant?)

[zenith void]

lol not really sure 🤣

i thought he was just squatting in them

🤣

the power of ✨ imagination ✨

[dark bay]

The artist limited himself to exactly 10 pixels per character as a challenge.

[dark bay]

I think I'll just merge into the main branch in case I delete my automerge branch by accident

I've introduced more bugs than it had before, but it is more or less working well

turns out it is possible to keep that reactive ECS api as in and plug it into an automerge doc

saving on rewriting a lot of code

I'll make a demo video of it working, then get back to working on features over the coming weekend

and I'm pretty hellbent on including automerge, bcuz its awesome!

[dark bay]

Arrg.... should of implemented share document id via qr code. I ran out of time. Painful to test between phone and tablet.

Maybe I can squeeze it in via Termux 2nyt.

Or maybe it is more tidy to shoot the virtual file system document id over when establishing a peer connection via the connection manager.

Ohh.... the mobile web browser (chrome) can generate qr codes for sharing links.... all good, don't need to add anything.

[zenith void]

woot wooot

[zenith void]

o rly they have built in api for that?

[dark bay]

Hit a little glitch... my granularity is down to the component level, and my level grid is a single component.

So each time I insert a level cell, it updates all level cells over the network.

I might have to work on the projection a bit more to get granularity down a bit deeper.

Otherwise works will.

Each ecs component is not really a json object, but the TypeSchema allows it to be serialised/deserialised to/from json.

The automerge doc is holding the components in json format and being projected to the reactive ecs via the serialisation/deserialisation of the json at the component level. Limiting granularity down to the component level.

If I had only limited it to json data types in the ecs components, then I could go deeper easier. (E.g. removing invariant map from type schema)

[dark bay]

U goto the share button in the browser, then there is a share by qr button. My url contains the document id.

I could make each level cell a separate entity as a work around too. By my entity IDs are UUIDs, and would be wasteful on storage space.

I should be able to do something with the TypeSchema to generate something that achieves a lower level granularity.

Will try to do a demo vid on the phone of it working.

[dark bay]

ToolKitty + Automerge:
https://youtube.com/shorts/4WvyUR9kGhg?si=KCqXLGFYd_tKcX61

[dark bay]

Ohh.... I could cheat a lower level granularity by diffing the components against the automerge doc before setting it ig.

[zenith void]

kinda like reconcile?

not 100% following why the patching technique doesn't work

[dark bay]

Bcuz my components are not compatible with an automerge doc, they contain classes like Vec2. So it needs to pass through conversion to/from json. It can be fixed, I just didn't preplan enough.

[elfin lintel]

ngl this post is a world record with the most conversation

1907 messages is nuts

thats triple and close to quadruple the amount of messages of https://discord.com/channels/722131463138705510/1346367267990147154 lol

[zenith void]

party at 2000 messages!

[elfin lintel]

yeh

im in

[zenith void]

the party will consist of me spamming party 1000 times

[elfin lintel]

thats kinda big brain idea

ngl

imagine tho if it hits 2k msgs

[dark bay]

Better in here than off topic :p

[elfin lintel]

we will be like those scientists working in nasa and finally seeing the rocket launch successfully xD

[dark bay]

In here is like [off topic]²

[elfin lintel]

nah

offtopic ↑↑↑ 2

[dark bay]

That operator looks familiar, not sure where from.

[elfin lintel]

pentation

its applying tetration

repeatedly

a niche thing in math

[zenith void]

i think this place is more [on topic]

[elfin lintel]

yeh

[zenith void]

*most of the time

[elfin lintel]

haven't read all messages

so dunno

[dark bay]

It's mostly just me thinking out load for 1000 messages.

Actually there is a lot of zulus and bigmistqkes in there too when I scroll up.

[zenith void]

For sure, we r along for the ride

[dark bay]

Creating a full on projection down to the field level via TypeSchema is still possible to. Just haven't had time to do it.

Not a very clean projection at the component level yet:
https://github.com/clinuxrulz/solid-kitty/blob/1ae20ad03a496fa6c602a7ca105dbf6fb3c5c601/src/ecs/EcsComponent.ts#L32

[dark bay]

Every person out of my 1.6K views are the exact same age bracket... what are the chances?

It must just be me rewatching my videos 1.6k times 😆

[dark bay]

Coolioo:

[dark bay]

Gotta fix some of the math in there, seems to only work properly with squares, not rectangles.

[zenith void]

lol, i m also in that bracket 🤣 the other viewer

[dark bay]

The stats don't lie :p

I'm doing a tidy up 🧹 of the automerge virtual file system to make it more ergonomic. If I can, I'll try to expose the same interface of the virtual file system in solid primitives.

[dark bay]

I believe automerge can obey the SyncFileSystem interface.

Which is strange, bcuz it can talk to indexeddb which is async.

But I guess it's done in memory, then written to indexeddb once in a while via the storage adapter.

[dark bay]

ahhh.... sorry, I'm not sure if it makes sense to have the same API for an automerge vfs as the virtual file system from solid primitives.

for reading a file from the vfs of solid primitives always returns a string for the file contents, but I want an automerge doc to come out.

and If i could, it would definitely be the async not the sync interface due to repo.find()

in this automerge file system I have made every folder and file a separate automerge document

the reason being, to allow for loading just a part of the data in a large file system into memory, Just in case it is really big.

and my time is up.... Termux xD

[elfin lintel]

i mean we do probably need a graph for how many people of the specific age exist

assuming the max age a person can live is n

and the chances of it will be

   44
   ∑ f(x)
   35
-----------
   n
   ∑ f(x)
   1

i think

where f(x) gives the population of the age

wait

im wrong

[dark bay]

I'm too tired to analyse that :p... good nyt all... @elfin lintel thanks for visiting.

[elfin lintel]

np

tbh im also lazy to work it out

[zenith void]

a ye, be aware: if you are ever thinking of doing a flat filesystem in automerge like { 'some/path/index.': ... } you will have to escape and unescape the /. it's a bit annoying, but issa bug.

[dark bay]

Ahh... true 👍... was thinking also, only .json files need special treatment as automerge docs, the rest can more or less obey the virtual file system API from solid primitives.
However I would like readFile to return a Blob instead of a string. The string says text only.

[dark bay]

Ohh... I didn't interpret that correctly. Forward slashes don't work as keys in automerge json objects?
That's bizarre.

Under the hood the automerge file system I've done is tree-like. So shouldn't be a problem in theory.

[zenith void]

Ye I think it was something with the patch generation mb 🤔

[dark bay]

Did u submit an issue/bug in their github? Surely they would fix that straight away. Version 3 is just days away. Orion is working on it as we speak.

I suppose they said JSON-Like data structure.

All good they know:
https://github.com/automerge/automerge/issues/825

Problem only occurs in the keys, not the values. Thankfully.

[ocean notch]

so no flat paths?

[dark bay]

Not as a Map basically. U can still do flat paths as an array.

There will be a party soon 🎂

[dark bay]

Instead of creating a dependency on solid primitives filesystem, I will create an identical interface as a proof of concept I can pull out as a separate project.... that readFile returning a string instead of a Blob is bothering me.

same for writeFile

even a Uint8Array for targets that do not store mime types

maybe there should be readFile(path: string, mode: "Text" | "Binary"): ... and the return type depends on the mode

the mime type is optional for a Blob object, returning blob is probably fine for binary

and Blob has an interface to easily extract text from it... so return blob always maybe.

[dark bay]

Line 335 downwards, I have an improved reactive automerge vfs ready to hookup/swap-over:

https://pastebin.com/3YMdqD9f

Files and folders are by ID, so you can rename them without the viewers refreshing on their content.

The improved version should fix up the few bugs that got introduced when I swapped over to automerge.

Ohhh.... the delete keyword can be used as a method name too ryt? I probably should call it something else, just in case.

[dark bay]

Almost the automerge file system adapter. I might break it off as a separate project. Other ppl might find it useful.

[dark bay]

Here we go, a reactive automerge virtual file system implementing the AsyncFileSystemAdapter but with Blobs instead of text for data, completely untested:

https://github.com/clinuxrulz/solid-kitty/blob/d64a425ebf6172e600861a5db9cf73912c20c3d3/src/AutomergeVirtualFileSystem.ts#L44

Enjoy 😄

oh... with a bug... I forgot onCleanup(() => { ...refCount--; ... }) in 1 spot. (it's leaking)

I haven't hookup up the improved automerge filesystem yet... Just thought I'd have a crack at that interface as a proof of concept.

12 messages away from cake time 😛

[dark bay]

Almost finished hooking up the cleaner rework of the automerge virtual file system... hopefully will be done by 2mrw. Then back to features (github issues pretty much)

[dark bay]

Feels bug free again after the hookup, will merge to main branch and build an update:

Back to program features

[dark bay]

Actually just found a few minor timing bugs where it tries to read/load virtual files before they are finished being written/created. Will debug that later on PC.

[dark bay]

Next focus, will be auto grow level when adding tiles out of range.

Basic meta data for tile properties (its a ground, its a spike, etc.)

And then a level runner, so we can actually try/play the level.