Core Haxe | Haxe | Page 6

velvet lodge Jun 13, 2025, 8:29 AM

#

but other times (not this one) it acts odd

#

they are private, its just you are in the class itself i suppose, that said, cant hurt to make them @:noCompletion

toxic stratus Jun 13, 2025, 8:30 AM

#

velvet lodge they are private, its just you are in the class itself i suppose, that said, can...

i'm not 🤔

#

Its weird that i'm getting a _lastOrder as well because that's meant to be a getter and a non entity function

#

    public var lastOrder(get, never):Order;
    function get_lastOrder() {
        return this.orders[orders.length - 1];
    }

no idea why that's there

#

maybe a nightly quirk?

velvet lodge Jun 13, 2025, 8:32 AM

#

will have to check, i know it does some things with getters / setters

velvet lodge Jun 13, 2025, 8:33 AM

#

toxic stratus maybe a nightly quirk?

hard to tell

toxic stratus Jun 13, 2025, 8:33 AM

#

yeah, its weird i don't think i saw this in the client side of the application

#

this is the server side

#

and actually, going back to my old question about the timestamp setter

#

if you're making them all have getters and setters anyway, i wonder if i can just override it 🤔

#

oh... probably not. It's gonna be in the same class lol

velvet lodge Jun 13, 2025, 8:36 AM

#

toxic stratus i'm not 🤔

so actually, they are set public, feels like a bug / oversight

#

should be better now

toxic stratus Jun 13, 2025, 8:38 AM

#

nice!

#

some old code i no longer need x)

toxic stratus Jun 13, 2025, 9:01 PM

#

class EntityTools {
    public static function insertOrUpdate<T:IEntity>(entity:T, table:String, ?succ:T->Void, ?err:Dynamic->Void) {
        trace("test");
        var e:Dynamic = cast entity;
        e.add().then(successItem.bind(succ, true), (err) -> trace(err));
    }

    static function successItem<T>(?callback:T->Void, log:Bool, entity:T) {
        if (callback != null) {
            callback(entity);
        }
        trace('here');
    }
}

Can confirm that this works 😅

toxic stratus Jun 15, 2025, 9:32 AM

#

public var lastOrder(get, never):Order;
function get_lastOrder() {
  return this.orders[orders.length - 1];
}

This adds a column lastOrder into the table

#

not sure if intentional, the behaviour seems inconsistent

toxic stratus Jun 15, 2025, 7:46 PM

#

@:exposeId
@:fieldset(NoRelations, [!orders, !account])
class Position implements IEntity {
    @:ignore public var client:Bool = #if client true #else false #end;
    @:ignore public var rest:RestExchangeBase;

    public var account:Account;
    @:size(20) public var id:String;
    public var symbol:String;
    public var side:Side = None;
    public var price:Float = 0;
    public var quantity:Float = 0;
    public var liq:Float;
    public var open:Int = 0;
    public var startTime:Float = 0;
    public var endTime:Float = 0;
    public var stopOrder:Order;
    public var takeOrder:Order;
    public var leverage:Float = 1.0;
    @:cascade public var orders:Array<Order> = [];
    public var updated:Float = 0;
}```
with this structure setting `stopOrder` and `takeOrder` to `null` doesn't remove the reference, in this particular case i'd expect the "reference" to be zero'd in some way

#

either by setting to actual 0 or something like -1

toxic stratus Jun 15, 2025, 9:58 PM

#

If i want to rename a column, how do I go about it without data loss? will e2 pick it up or should i rename in the db manually, rename the class locally and then everything is good?

velvet lodge Jun 16, 2025, 5:20 AM

#

toxic stratus ```hx public var lastOrder(get, never):Order; function get_lastOrder() { retur...

looks like a bug

velvet lodge Jun 16, 2025, 5:21 AM

#

toxic stratus ```hx @:exposeId @:fieldset(NoRelations, [!orders, !account]) class Position imp...

you mean, you do:

myPosition.stopOrder = null;
myPosition.update().then(...);

and stopOrder isnt null in the db?

velvet lodge Jun 16, 2025, 5:22 AM

#

toxic stratus If i want to rename a column, how do I go about it without data loss? will e2 pi...

you will defo lose data currently if you rename a field in the haxe classes, so you have to do it manually, eventually i want to have a migration system, but it doesnt exist yet, and im not 100% sure what it would look like

toxic stratus Jun 16, 2025, 7:06 AM

#

velvet lodge you mean, you do: ```js myPosition.stopOrder = null; myPosition.update().then(.....

correct

velvet lodge Jun 16, 2025, 7:07 AM

#

toxic stratus correct

defo feels like a bug, but one i thought i would have encountered by now... which seems odd

#

crazy, yeah:

    function testBasic_NullRef(async:Async) {
        var mainEntity = createEntity("this is a string value for entity #1", 123, 456.789, true, new Date(2000, 2, 3, 4, 5, 6));
        mainEntity.entity1 = createEntity("sub entity", 456, 789.123, false, new Date(2001, 3, 4, 5, 6, 7));

        profileStart("testBasic_Primitives");
        measureStart("add()");
        mainEntity.add().then(entity -> {
            measureEnd("add()");

            Assert.equals(2, entity.basicEntityId);
            Assert.equals("this is a string value for entity #1", entity.stringValue);
            Assert.equals(123, entity.intValue);
            Assert.equals(456.789, entity.floatValue);
            Assert.equals(true, entity.boolValue);
            Assert.equals(new Date(2000, 2, 3, 4, 5, 6).toString(), entity.dateValue.toString());

            Assert.equals(1, entity.entity1.basicEntityId);
            Assert.equals("sub entity", entity.entity1.stringValue);
            Assert.equals(456, entity.entity1.intValue);
            Assert.equals(789.123, entity.entity1.floatValue);
            Assert.equals(false, entity.entity1.boolValue);
            Assert.equals(new Date(2001, 3, 4, 5, 6, 7).toString(), entity.entity1.dateValue.toString());

            measureStart("refresh()");
            return entity.refresh();
        }).then(entity -> {
            measureEnd("refresh()");

            Assert.notNull(entity.entity1);

            entity.entity1 = null;

            return entity.update();
        }).then(entity -> {
            Assert.isNull(entity.entity1);

            return entity.refresh();
        }).then(entity -> {
            Assert.isNull(entity.entity1);

            profileEnd();
            async.done();
        }, error -> {
            trace("ERROR", error);
        });
    }

#

the last Assert.isNull(entity.entity1); fails... 😮

toxic stratus Jun 16, 2025, 7:26 AM

#

velvet lodge the last `Assert.isNull(entity.entity1);` fails... 😮

I think it's because its an Int column and you can't null it by default so I think it makes sense in this case it should be something like 0 or -1

velvet lodge Jun 16, 2025, 7:26 AM

#

i think you can null an int column in the db

toxic stratus Jun 16, 2025, 7:27 AM

#

you have to set it to be nullable tho

velvet lodge Jun 16, 2025, 7:27 AM

#

that shouldnt be an issue, the schema is created by entities, im not even sure thats the case though... i think its just not nulling things, it might even be a db-core issue

toxic stratus Jun 16, 2025, 7:30 AM

#

yeah nvm ignore me

#

it is nullable in the db

velvet lodge Jun 16, 2025, 7:31 AM

#

so thats a bit crazy, so when it builds up the recordset to add into the db in the update it creates:

#

{
                        basicEntityId : 2,
                        stringValue : this is a string value for entity #1,
                        intValue : 123,
                        floatValue : 456.789,
                        boolValue : 1,
                        dateValue : 2000-03-03 04:05:06
                }

#

notice how there is no entity1 field, which means the update sql simple wont include it

toxic stratus Jun 16, 2025, 7:32 AM

#

ahhh interesting

#

you're using null to exclude things

velvet lodge Jun 16, 2025, 7:32 AM

#

it means, in the "toRecord" code in entities2 there must be a != null

toxic stratus Jun 16, 2025, 7:32 AM

#

yeah, i defo relied on that behaviour a lot in direct use

velvet lodge Jun 16, 2025, 7:33 AM

#

toRecordFn.code += macro @:privateAccess {
    var record = new db.Record();
    $b{[for (entityField in entityDefinition.primitiveFields()) {
        macro {
            record.field($v{entityField.name}, entities.EntityManager.instance.convertPrimitiveToDB($i{entityField.name}, $v{entityField.type}, $v{entityField.options}));
        }
    }]}
    $b{[for (entityField in entityDefinition.entityFields_OneToOne()) {
        var foreignKey = entityField.foreignKey();
        macro {
            if ($i{entityField.name} != null) {
                record.field($v{entityField.name}, $i{entityField.name}.$foreignKey);
            }
        }
    }]}
    return record;
}

yup

#

i need to think about this, seems intentional

toxic stratus Jun 16, 2025, 7:33 AM

#

yeah, there are some cases where you don't want to submit things that are null

velvet lodge Jun 16, 2025, 7:35 AM

#

exactly, like if you are just updating the, say, name of an entity, it could nuke your relationships

#

i mean, if you loaded it from the db, then you would be fine... or if you loaded it from network (serializers) and called .refresh, then you would be fine also

#

but if you forget to .refresh, then you could nuke your 1 -> 1's :/

toxic stratus Jun 16, 2025, 7:38 AM

#

would making a field be marked as nullable make this work?

velvet lodge Jun 16, 2025, 7:38 AM

#

i dont think so...

toxic stratus Jun 16, 2025, 7:38 AM

#

that way, you can expect null and "whatever happens is okay"

#

weird problem

velvet lodge Jun 16, 2025, 7:39 AM

#

yeah, both "paths" are valid, thats the issue

#

i almost think you want a sort of "explicity nulled" or something, but i think entities internally might also do ".entity1 = null"

toxic stratus Jun 16, 2025, 7:40 AM

#

then another issue occurs there, how do you make a compatible syntax for that

#

for haxe there's 0 = null or 1 = object

#

you would need a second thing to flip, no?

velvet lodge Jun 16, 2025, 7:41 AM

#

null would be null in the db

#

stopOrder is an object, so it would be 0, it would be null

#

though i suspect the same thing happens for primitives also

#

actually, no, it doesnt

toxic stratus Jun 16, 2025, 7:42 AM

#

yeah, i've never had this occur outside of entities

velvet lodge Jun 16, 2025, 7:42 AM

#

$b{[for (entityField in entityDefinition.primitiveFields()) {
    macro {
        record.field($v{entityField.name}, entities.EntityManager.instance.convertPrimitiveToDB($i{entityField.name}, $v{entityField.type}, $v{entityField.options}));
    }
}]}

velvet lodge Jun 16, 2025, 7:47 AM

#

toxic stratus yeah, i've never had this occur outside of entities

im not sure what you mean here, its clearly an e2 issue, its not nulling the db entry that links stopOrder... and its because its skipping "things that are null", which is reasonable in one use case, and not in another (even using the same entity you can hit both use cases)

toxic stratus Jun 16, 2025, 7:48 AM

#

velvet lodge im not sure what you mean here, its clearly an e2 issue, its not nulling the db ...

a response to though i suspect the same thing happens for primitives also but I also never used anything but primitives outside of e2

#

lol

velvet lodge Jun 16, 2025, 7:48 AM

#

yeah, primitives are fine, there isnt the "if != null"

#

for relationships there is, and its reasonable, but also not reasonable depending on use case... :/

var entity = new Entity();
entity.entityId = 1;
entity.name = "some name";
entity.update(); // <--- just nuked all relationships if i implement a fix (ie, remove the != null check)

#

it seems like a usecase that wouldnt happen, BUT, in my apps, entities often come down the wire to and from the client (serialized) and are partial

toxic stratus Jun 16, 2025, 8:22 AM

#

seems like it would be a common thing to do with api's as well, only send out what's changed

velvet lodge Jun 16, 2025, 8:43 AM

#

OK, i think i have a solution, just need to write a bunch of UTs to test it

#

    public function field(name:String, value:Any = null):Any { // if value is non null, this is effectively a setter
        if (value != null) {
            data.set(name, value);
            return value;
        }

        return data.get(name);
    }

this shitty pattern in db-core just bit me too

#

well, it works:

#

just dunno if ive broken a bunch of other stuff in the process

velvet lodge Jun 16, 2025, 9:13 AM

#

toxic stratus ```hx @:exposeId @:fieldset(NoRelations, [!orders, !account]) class Position imp...

this should be sorted now

toxic stratus Jun 16, 2025, 9:18 AM

#

velvet lodge this should be sorted now

nice! thanks

#

hopefully nothing is broken

#

likely there is tho

#

i vaguely remember trying to do something like this in my app side and ran into some complication

#

just can't recall what it was

velvet lodge Jun 16, 2025, 9:20 AM

#

i dont think there is tbh... were going to give other projects the once over, but the UTs show that none of the "normal" behaviour has changed since they all pass - and they are quite comprehensive

toxic stratus Jun 16, 2025, 9:20 AM

#

nice!

#

where would I go to start implementing the currency/json types

#

i'm guessing its going to be a simple mapping of types to dbcore

#

cause you did add it to core already

velvet lodge Jun 16, 2025, 9:23 AM

#

yeah, so, just so you know... the change was fairly simple: entities2 already creates setters for 1 -> 1 entities, this is because it doesnt like updated refs, and prefers to copy over the values if you do someEntity.subEntity = anotherEntity, in that scenario, the subEntity wont actually be a ref to anotherEntity but its data will be copied over (this is also so you dont nuke data, a nice feature in itself i forgot about)

this is extended now to "explicitlyNulled", so internally, e2 will bypass the setter, and prefer to directly set the variable (_subEntity), but in user code, if you explicitly set an entity to null, it will hold a flag for that that entity was nulled, and it will then write an actual null to the db

velvet lodge Jun 16, 2025, 9:24 AM

#

toxic stratus i'm guessing its going to be a simple mapping of types to dbcore

i dont think its going to be quite that straight forward tbh... it doesnt exist in db-core either

toxic stratus Jun 16, 2025, 9:25 AM

#

velvet lodge yeah, so, just so you know... the change was fairly simple: entities2 already cr...

oh cool, why copy rather than use the existing object? Isn't that more work?

velvet lodge Jun 16, 2025, 9:26 AM

#

because you can nuke data

toxic stratus Jun 16, 2025, 9:26 AM

#

velvet lodge i dont think its going to be quite that straight forward tbh... it doesnt exist ...

I'm sure you added decimal

velvet lodge Jun 16, 2025, 9:26 AM

#

https://github.com/core-haxe/db-core/blob/main/src/db/ColumnType.hx

#

decimal exists, but thats what Float maps too

toxic stratus Jun 16, 2025, 9:27 AM

#

Oh, i'm confused I defo remember asking you for the decimal type and i'm sure you added it

#

maybe you just parsed it on the incoming

#

decimal is stored as string in the db but i think you had it get parsed into float, so probably a db-mysql thing instead then

velvet lodge Jun 16, 2025, 9:28 AM

#

probably, yeah

#

i think a new type needs to be added to db-core, that maps to "that other one" i cant remember what it is now, and then e2 needs that mapping too

#

maybe Currency on both sides is "fine"

toxic stratus Jun 16, 2025, 9:29 AM

#

I think so

velvet lodge Jun 16, 2025, 9:30 AM

#

but it still needs plenty of mapping, i also suspect that the Currency abstract will come through in e2 as a Float (since the macro it uses followWithAbstracts) - so that will need to handled specially (i think)

toxic stratus Jun 16, 2025, 9:30 AM

#

Its clear

#

i tried doing it the other day with json but couldn't get it to work so likely more complicated than i thought

#

just have a getter that parses it locally instead

#

or i just missed a spot

#

but i tried to follow the Date type around

#

also learnt that you could have enum's named the exact same name as a haxe type from that lol

toxic stratus Jun 16, 2025, 12:14 PM

#

think i'm done with 99% of e2 porting

#

these kinds of things are a pita, but also a blessing. Found lots of things that could be done better along the way

velvet lodge Jun 16, 2025, 1:44 PM

#

e2 convert? blobcheer

toxic stratus Jun 16, 2025, 1:46 PM

#

I think so, now its time to figure out the workflows 😄

glacial relic Jun 16, 2025, 1:47 PM

#

excuse the interjection - what's e2 ?

toxic stratus Jun 16, 2025, 1:49 PM

#

glacial relic excuse the interjection - what's e2 ?

https://github.com/core-haxe/entities2

#

just a shorthand way of saying it x)

#

Basically, a class represents a database Table

glacial relic Jun 16, 2025, 1:49 PM

#

ORM mapping then

toxic stratus Jun 16, 2025, 1:49 PM

#

correct

glacial relic Jun 16, 2025, 1:50 PM

#

ok - thanx - just curious core haxe looks like it's headed to become Haxe Django

#

cool

toxic stratus Jun 17, 2025, 9:37 AM

#

I'm having issues with serialising IEntity's to send out over a websocket

#

properties not existing etc

#

are there any solutions

#

I'm assuming the issue is something like, none of the serializers know that var account:Account will turn into a property and may even have a different name, so on unserialising its referencing the wrong things?

toxic stratus Jun 17, 2025, 10:24 AM

#

all the data is there, its just not accessible 🤔

velvet lodge Jun 17, 2025, 12:19 PM

#

what does your serialization look like?

toxic stratus Jun 17, 2025, 7:54 PM

#

velvet lodge what does your serialization look like?

I think I may have figured it out

#

I didn't realise how deep the serialisation was going so the things it was serialising wasn't really things i wanted to serialise

#

might be a coincidence that it only started occuring post e2 integration

velvet lodge Jun 17, 2025, 8:08 PM

#

what is the serialization lib?

toxic stratus Jun 17, 2025, 8:09 PM

#

velvet lodge what is the serialization lib?

std haxe

velvet lodge Jun 17, 2025, 8:09 PM

#

haxe. Serializer?

toxic stratus Jun 17, 2025, 8:09 PM

#

yeah

#

filtrek told me a cool feature of it and I think it actually does work fine, was just an error that could come from e2 related things

velvet lodge Jun 17, 2025, 8:11 PM

#

might wanna consider using this: https://github.com/core-haxe/serializers

you just do implements IHaxeSerializable on the entities and you will get to and from string functions

#

i use them alot with entities and they work fine

#

you will get to and from string functions
sorry, i mean function serialize():String; and function unserialize(data:Any):Void; functions

toxic stratus Jun 17, 2025, 8:12 PM

#

lol ofc there's a serialise lib as well

#

what do I have to do in those functions

velvet lodge Jun 17, 2025, 8:13 PM

#

nothing

#

its all handled for you

toxic stratus Jun 17, 2025, 8:13 PM

#

can you ignore properties?

velvet lodge Jun 17, 2025, 8:13 PM

#

yeah, @:ignore iirc - same as entities

toxic stratus Jun 17, 2025, 8:14 PM

#

I'm guessing you can't serialise an enum if it's the 'root' has to be a class?

velvet lodge Jun 17, 2025, 8:15 PM

#

why would you serialize an enum? You would just serialize enum values, no?

#

maybe im not understanding

toxic stratus Jun 17, 2025, 8:16 PM

#

enum foo { A(value:obj)}

#

and serialise A

velvet lodge Jun 17, 2025, 8:16 PM

#

but you would only serialize uses of that, not the enum itself, like var myFoo:Foo = A("bob")

toxic stratus Jun 17, 2025, 8:16 PM

#

yeah

#

some serialisers don't support that

velvet lodge Jun 17, 2025, 8:17 PM

#

i think it would work, it uses haxe.Serializer under the hood, and i think that works with things like that

toxic stratus Jun 17, 2025, 8:17 PM

#

ahhh

#

cool, will defo use it if current setup breaks

toxic stratus Jun 18, 2025, 11:27 AM

#

velvet lodge yeah, @:ignore iirc - same as entities

passing thought but is there a way to override that

#

I have something in an entity that I don't want to go to the db, so its marked as @:ignore but I would want it to be serialised

velvet lodge Jun 18, 2025, 11:34 AM

#

i suppose a "@:force" could be added, but it all feels very "not the way entities is supposed to be used", the whole point (really) is that your entity is a data unit, not a dumping ground for "random" shit

toxic stratus Jun 18, 2025, 11:35 AM

#

It certainly isn't random shit

velvet lodge Jun 18, 2025, 11:35 AM

#

well, buts its not "entity shit" either

toxic stratus Jun 18, 2025, 11:36 AM

#

it's a data unit, so it's the place to put it

velvet lodge Jun 18, 2025, 11:36 AM

#

but a data unit that shouldnt be serialized to the db, but should be serialized to and from the client? Doesnt sound right

#

lemme guess, that cache thing again on date tools?

toxic stratus Jun 18, 2025, 11:37 AM

#

class Position implements IEntity {
    @:ignore public var client:Bool = #if client true #else false #end;
    public var account:Account;
    @:size(20) public var id:String;
    public var symbol:String;
}

I have a client property that tells the client that the object is created on the client or server

#

if the object is sent from the server, it should take priority over the client created version

#

this flag has no real business being in the database

#

client can't update the db

velvet lodge Jun 18, 2025, 11:40 AM

#

this flag has no real business being in the database

then i would argue it has no place being in the entity, not sure what alt system you would use, but i think if you have this sort of "well, this is in the entity, but its not really in the entity" then its sort of a misuse of the entity itself (imo)

#

function createOrder(position:Position, source:ClientOrServer)... or something

toxic stratus Jun 18, 2025, 11:41 AM

#

its sent over a websocket

#

possible to work around for sure

velvet lodge Jun 18, 2025, 11:41 AM

#

toxic stratus its sent over a websocket

and?

#

class OrderRequest implements IHaxeSerializable {
    var position:Position;
    var source:ClientOrServer;
    var someOtherParam:Int
}

#

sending entites "as is" is fine i suppose, you will, imo, quickly get into a situation where you'll want additional (non-entity) data passed through also, so you'll end up shoe-horning it into the entity "because its easier"

#

eg, this is my UserApi interface... which is used to build an RPC interface (transport doesnt matter, http, rpc, etc)

📎 UsersApi.hx

toxic stratus Jun 18, 2025, 11:45 AM

#

i'm defo putting some "non entity" stuff in there, and not treating it as a data-only thing

#

i'm treating it more like a "HandlePositionTableData" zone

velvet lodge Jun 18, 2025, 11:46 AM

#

toxic stratus i'm defo putting some "non entity" stuff in there, and not treating it as a data...

exactly, its a misuse... i mean, its "fine", but i think you'll either need to "deal" with it being in the db, or ill add a "force" to serializers that opens it up for abuse (though thats not really my problem)

#

i think eventually, you are going to end up wanting an RPC type interface anyway rather than just passing a single entity back and forth

toxic stratus Jun 18, 2025, 11:47 AM

#

velvet lodge i think eventually, you are going to end up wanting an RPC type interface _anywa...

another thing to learn... ?!

#

i really don't even know what the difference is between rpc and what i'm doing currently

velvet lodge Jun 18, 2025, 11:48 AM

#

well, its not that much more, i dont mean a PROPER rpc system, i just mean that instead of passing and entity back and forth you pass a request / response that wraps and entity (or 100)

toxic stratus Jun 18, 2025, 11:49 AM

#

the main issue i'm solving with the current setup is basically making the mysql db "event-like"

velvet lodge Jun 18, 2025, 11:49 AM

#

class OrderRequest implements IHaxeSerializable {
    var position:Position;
    var requetor:User;
    var someOtherEntity:Entity;
    var source:ClientOrServer;
    var someOtherParam:Int
}

#

if you are using the "single entity" approach, then you would end up creating a new entity call "OrderRequest" and that is defo not correct (unless you wanted OrderRequest in the db ofc, which is perfectly reasonable in itself)

toxic stratus Jun 18, 2025, 11:51 AM

#

your projects sounds very tidy

#

😅

#

you would probably hate to see some of my entities x)

#

It just seems like the right place for it, but maybe that's just where my ignorance level is at

#

most are pretty much "just entities" though

#

just 1 or 2 aren't

#

i'll probably revisit this at some point when i get to that stage of the project

toxic stratus Jun 19, 2025, 11:05 AM

#

#

I've managed to some how create an infinite loop of entity creations

#

#

Any idea what could cause this? The way I triggered it was on the client side I pressed an action button repetitively a few times, the writes are all completed but now it seems to be stuck in a perpetual read cycle

#

another action just occurred and took it out of it the cycle

toxic stratus Jun 20, 2025, 6:31 AM

#

class Main {
    static function main() {
        trace("Hello, world!");
        new Main();
    }

    var db:IDatabase;

    public static var universe:Universe;

    function new() {
        db = DatabaseFactory.instance.createDatabase(DatabaseFactory.MYSQL, {
            database: 'test',
            host: 'localhost',
            user: 'root',
            pass: 'password'
        });

        EntityManager.instance.database = db;
        connected();
        var timer = new Timer(200);
        timer.run = this.loop;
    }

    function connected() {
        var user = new User();
        user.name = "Test Main";

        var pos = new Position();
        pos.symbol = 'BTCUSDT';
        pos.orders = [];
        pos.add().then((position) -> {
            for (i in 0...2) {
                var order = new Order();
                order.exchange = 'Bybit';
                order.symbol = 'BTCUSDT';
                order.side = 'Buy';
                order.price = 10;
                order.quantity = 0.5;
                order.add().then((response) -> {
                    pos.orders.push(response);
                    pos.update().then(null, (err) -> trace(err));
                }, (err) -> trace(err));
            }
        }, (err) -> trace(err));
    }
}

Was trying to isolate the loop bug and came across another bug

#

In this example, i'd expect there to be 2 orders in the pos.orders array

#

#

but I'm getting 4 in the database (2x duplicates)

#

maybe just a side effect of the loop/async nature here, but thought i'd mention just incase

#

looking at my db I do think reference duplication does seem to occur in general

#

i've got 13 orders linked to a position in my main db and there's 144 references :o

#

I think what's happening there is every table.update() a new reference is being added to the database

velvet lodge Jun 20, 2025, 6:50 AM

#

    function connected() {
        var user = new User();
        user.name = "Test Main";

        var pos = new Position();
        pos.symbol = 'BTCUSDT';
        pos.orders = [];
        pos.add().then((position) -> {
            for (i in 0...2) {
                var order = new Order();
                order.exchange = 'Bybit';
                order.symbol = 'BTCUSDT';
                order.side = 'Buy';
                order.price = 10;
                order.quantity = 0.5;
                order.add().then((response) -> {
                    pos.orders.push(response);
                    pos.update().then(null, (err) -> trace(err));
                }, (err) -> trace(err));
            }
        }, (err) -> trace(err));
    }

this seems a very weird way to use entities

toxic stratus Jun 20, 2025, 6:51 AM

#

I don't actually use it like that, that's just the repro

velvet lodge Jun 20, 2025, 6:51 AM

#

var pos = new Position();
pos.symbol = 'BTCUSDT';
pos.orders = [];
for (i in 0...2) {
    var order = new Order();
    order.exchange = 'Bybit';
    order.symbol = 'BTCUSDT';
    order.side = 'Buy';
    order.price = 10;
    order.quantity = 0.5;
    pos.orders.push(order);
}
pos.add()...

#

are you still adding the order though...

#

if you do want to manually add orders, and later associate them with the position, i think there is meta for that... "@:mustExist" or something

toxic stratus Jun 20, 2025, 6:53 AM

#

in my app there is no for loop adding orders

velvet lodge Jun 20, 2025, 6:55 AM

#

that all said and done, even though its not a pattern i use myself, i would expect something like this to be fine, so if it doesnt work, its a bug:

position.add();
order1.add();
order2.add():
position.orders = [order1, order2]
position.update();

#

(obviously promise based in reality, code above is just psuedo)

toxic stratus Jun 20, 2025, 7:00 AM

#

in my app it works like this

#

each press of the green button is 1 more order added to the position

#

4 clicks of the button is 4 orders associated to the position

#

#

but here's what the reference table looks like

velvet lodge Jun 20, 2025, 7:18 AM

#

so yeah, you are adding new orders that you are add()'ing to an existing positions, normally i would just add them to the position.order array and call update... but, i still think your pattern is perfectly reasonable

#

i think what is happening is you are add()'ing the order manually, then associating it with the position, then updating, and that, internally, is add()'ing it again

#

if you annotate the order array with @:mustExist, does that change / help?

toxic stratus Jun 20, 2025, 7:56 AM

#

velvet lodge so yeah, you are adding new orders that you are add()'ing to an existing positio...

This is defo something that's occuring, does that mean i'm also duplicating in my orders table? hmm.. will check

#

will check the mustexist thing now

velvet lodge Jun 20, 2025, 7:57 AM

#

keep in mind that mustexist also changes the behaviour, it means "my" pattern of "just add the orders to the array and call update" wont work at all anymore... the entities "must exist" in the db already

toxic stratus Jun 20, 2025, 7:59 AM

#

hmmmmmmmmmmm, i think I coded things with that idea in mind but will confirm

toxic stratus Jun 20, 2025, 8:00 AM

#

velvet lodge keep in mind that mustexist also changes the behaviour, it means "my" pattern of...

same behaviour

#

maybe not the right flag?

#

it is the right flag

#

but yeah, an order shouldn't implicitly be added to the orders array on a position

#

an order can reject, not be filled yet etc etc

velvet lodge Jun 20, 2025, 8:04 AM

#

toxic stratus an order can reject, not be filled yet etc etc

so the order is an entity in its own right, and then later, based on some status, you associate it with a position?

toxic stratus Jun 20, 2025, 8:17 AM

#

velvet lodge so the order is an entity in its own right, and then later, based on some status...

correct

velvet lodge Jun 20, 2025, 8:19 AM

#

so yeah, it makes perfect sense, and feels like a bug... as i mentioned, the pattern you are using feels perfectly valid, even if i would maybe approach it somewhat differently, so ill create some UTs and see if i can repro (not going to happen now though, too much on)

toxic stratus Jun 20, 2025, 8:19 AM

#

no worries

#

might be related to another thing, i was noticing that my entities were getting created a ton of times, so i suspect what's happening is

#

they are getting created in proportion to the number of references around

#

but the arrays themselves seem accurate

#

like if there's 10 additional records in the db, i don't think i'm actually getting 10 duplicates locally. Confirmed, no additional records locally, so maybe you're handling the duplicates?

toxic stratus Jun 20, 2025, 8:53 AM

#

Curious, you said you would handle it differently - how would you handle it?

toxic stratus Jun 21, 2025, 12:05 PM

#

If I have a trigger on a table will e2 remove that?

#

I can do the same thing in code, just wondering if I should forget about them as an option

velvet lodge Jun 22, 2025, 1:15 PM

#

i doubt it... but defo test that, dont take my word for it 🙂

toxic stratus Jun 29, 2025, 11:40 AM

#

it seems like many-to-many relationships works, but how would I construct a find query on them?

#

like, find all items with tags x, y and z in entity MyTable

velvet lodge Jul 1, 2025, 6:29 AM

#

i think you'll have to use a raw query for now, with inner joins, its not ideal because it means "knowing" about the table structure, but glint has had to do it for an "advanced" find

velvet lodge Jul 2, 2025, 4:59 PM

#

toxic stratus

think i can repro this

toxic stratus Jul 2, 2025, 5:29 PM

#

velvet lodge think i can repro this

there's a good chance your existing projects just have phantom data in regards to this issue

velvet lodge Jul 2, 2025, 5:29 PM

#

i think thats exactly what i just found

toxic stratus Jul 2, 2025, 5:30 PM

#

its unlikely to be that heavy

#

just something to be aware of

velvet lodge Jul 2, 2025, 5:30 PM

#

cleaning up is easy, making sure it doesnt happen again is something ill need to repro in a UT

toxic stratus Jul 2, 2025, 5:30 PM

#

The repro i gave should be easy to UT it right?

toxic stratus Jul 15, 2025, 12:51 PM

#

i just tried this with a few thousand orders but I got tons of duplicates in the db, admittedly it is a system i'm only just working on now, but want to know how this behaviour is meant to work

toxic stratus Jul 15, 2025, 2:25 PM

#

I think I understand what's going on, when I add the entity that references an order that isn't added to the database, it will add that order to the database. But I think what's happening is, every time that parent entity gets updated, it readds the order to the database because maybe it didn't get marked as being added?

Haven't confirmed, mostly just spilling thoughts as i wait for db to delete 60k records x)

velvet lodge Jul 15, 2025, 2:32 PM

#

ive been working towards repro'ing it in a unit test (that was last week though), for simple cases (to my surprise) i didnt see the issue - but there certainly is one, i just need to hone in on what the sequence is

#

.update could be a good hint

#

ok, actually, i did update, albeit a simple case

📎 TestBasic.hx

#

class Object1 implements IEntity {
    public var name:String;
    public var array_1:Array<Object2> = [];
    @:cascade public var array_2:Array<Object2> = [];
}

class Object2 implements IEntity {
    public var name:String;
}

toxic stratus Jul 15, 2025, 2:42 PM

#

velvet lodge ive been working towards repro'ing it in a unit test (that was last week though)...

class Object1 implements IEntity {
    public var name:String;
    public var counter:Int = 0;
    public var array:Array<Object2> = [];
}

class Object2 implements IEntity {
    public var name:String;
}

var foo = new Object1();
var bar = new Object2();
bar.name = "Test";
foo.array.push(bar);

for (i in 0...10) {
  foo.counter = i;  
  foo.update();
}

I think this might be a simplified version of what's going on in my app

#

it doesn't happen on all of them but some of them so it might be a case of increasing the loop counter. Maybe it's an async related issue?

velvet lodge Jul 15, 2025, 2:53 PM

#

you arent using promises properly

#

you are like "fire and forgetting" them

#

you are expecting them to be sync here and they arent (at all)... especially since your db is remote

#

but there is an issue anyway, i can see it in my dbs, so for sure, there is something going on

toxic stratus Jul 15, 2025, 8:00 PM

#

It was pseudo code for the app flow, i'm not actually using it like that

#

it might still repro the issue

#

because i'm not doing that in my app and still get the dupes

#

it's important to note this though, because, the thing that's having issues is Object2 is the one getting duplicated, not Object1

#

I'm interfacing with Object2 so maybe Object1 isn't handling the async properly

#

maybe object2 is returning the done callback before array has finished doing the insert ops (and toggling whatever flags it needs)

#

Yea, if I preinsert all the orders, there's 0 duplicates

velvet lodge Jul 15, 2025, 8:48 PM

#

if you are doing something like this though:

for (i in 0...10) {
  foo.counter = i;  
  foo.update();
}

Then odd things are going to happen... you need to wait for the update to finish before updating again, because things could be in a weird state

#

im starting to consider that this is whats happening in my apps somehow, i defo have orphans, but im just not sure where they are coming from because in the unit tests, which is a somewhat controlled environment, it doesnt happen

velvet lodge Jul 15, 2025, 8:52 PM

#

toxic stratus Yea, if I preinsert all the orders, there's 0 duplicates

do you have a somewhat minimal repro of when it doesnt work (and the preinsert for good measure), im not surprised (i dont think) that if you preinsert its all fine, because then you are just updating an association

#

i actually think i might need to look into a locking mechanism, but these can be messy and i wondering if then entities is trying to do "too much"

toxic stratus Jul 15, 2025, 9:04 PM

#

velvet lodge if you are doing something like this though: ```js for (i in 0...10) { foo.cou...

var i = 0;
var waiting = false;
while (i < 10) {
  if (waiting) {
    continue;
  }
  waiting = true;
  foo.counter = i;  
  foo.update().then(function(_) { i++; waiting = false}, null);
}```
Defos, I was adding a system and locking until a response doesn't affect this particular issue, it results in the same

toxic stratus Jul 15, 2025, 9:05 PM

#

velvet lodge i actually think i might need to look into a locking mechanism, but these can be...

it will heavily add to latency issues as well, preadding the orders to the db (with no lock required here) causes the app to lock up a bit altho i don't actually understand why this occurs

#

Just an additional note, i'm not sure if the above will repro, its just the kind of flow going on here, maybe try with 100 loops

#

I haven't really nailed down when its occuring, but my suspicion is it occurs when you have to reupdate the same Object2

velvet lodge Jul 16, 2025, 6:04 AM

#

toxic stratus ```hx var i = 0; var waiting = false; while (i < 10) { if (waiting) { cont...

this will likely skip a bunch of items... its not really a lock, more of a "skip"...

velvet lodge Jul 16, 2025, 6:04 AM

#

toxic stratus it will heavily add to latency issues as well, preadding the orders to the db (w...

yeah, the latency is what im worried about tbh....

velvet lodge Jul 16, 2025, 6:09 AM

#

toxic stratus I haven't really nailed down when its occuring, but my suspicion is it occurs wh...

is it only ever one call coming in that is adding the objects? Because if it is, then it blows my idea out the water - any actual code for me to see? (or is it top secret kinda stuff? )

toxic stratus Jul 16, 2025, 7:44 AM

#

velvet lodge is it only ever one call coming in that is adding the objects? Because if it is,...

@:exposeId
class Position implements IEntity {
  public var symbol:String;
  public var account:Account;
  @:cascade public var orders:Array<Order> = [];
}

@:exposeId
class Order implements IEntity {
  public var symbol:String;
  public var price:Float;
  public var account:Account;
}

orders[aid] = getAllOrdersFromBybitCryptoExchange();
for (order in orders[aid]) {
  EcsTools.set(aid, new PositionOrder(order));
}```

We start in `AccountSync.hx` to get all the orders from exchange API then push them out to another system
Then we go to `PositionSystem.hx` which receives the above orders
```hx
override function update(_) {
  iterate(orders, (entity) -> {
    if (handleOrder(account, order)) {
      universe.deleteEntity(entity);
    }
  });

  for (p in _positions) {
    //only attempt to update a position when we have a new version and not in the process of updating
    if (!updatingPosition.exists(p.pid) || (!updatingPosition[p.pid] && p.updated > lastUpdated[p.pid])) {
      trace('here');
      updatingPosition[p.pid] = true;
      lastUpdated[p.pid] = p.updated;
      p.update().then(function(_) {
        emitPositionUpdate(p);
    updatingPosition[p.pid] = false;
      }, (err) -> {
    updatingPosition[p.pid] = false;
    trace(err);
    });
  }
}

function handleOrder(account:AccountId, order:Order) {
  if (!_positions.exists(pid)) {
    var state = settingUpPosition.get(pid);
    if (settingUpPosition.exists(pid) && state) {
      return false;
    }
    trace('setting up position');
    settingUpPosition.set(pid, true);
    EPosition.find(query($account == account && $symbol == order.symbol)).then((position) -> {
      //create a position if null and add it the db, then remove the lock otherwise add the position to a local map and continue on
    });
    return false;
  }

  switch (order.status) {
    case FILLED | PARTIALLY_FILLED:
      p.addOrder(order);
      //triggers `p.updated` timestamp to get raised
    default:
  }
}

#

That's not a repro or anything, but that's an exact cut down version of what's going on in my app

#

  iterate(orders, (entity) -> {
    if (handleOrder(account, order)) {
      universe.deleteEntity(entity);
    }
  });
``` receives the updates from `AccountSync.hx`'s EcsTools.set....

#

I couldn't seem to repro it, I did give it a few tries :/

#

not everything in my app is getting duplicated tho, there's like 1000 demo orders i'm syncing for the test and there will be a bunch that don't duplicate then all of a sudden one will get duplicated like 8 times

#

and it repeats like that sporadically

#

hopefully there's some hint or correlation to your project in there

velvet lodge Jul 16, 2025, 7:57 AM

#

toxic stratus not everything in my app is getting duplicated tho, there's like 1000 demo order...

yeah, the orphans in my db are defo not "all the time"... so it feels like a race condition of sorts, which is hard to repro locally since its "too fast"... i think ill plug the UTs into a remote db at some point and see if that works better (as in worse 😄 )

toxic stratus Jul 16, 2025, 8:01 AM

#

I did my tests on a remote db

#

the only thing i didn't try is doing a loop of 1000 for some reason lol

#

just tried, made no difference test is probably too simple

#

seperate question, on an entity if I do orders = []; would it delete all the relationships from the respective table?

velvet lodge Jul 16, 2025, 8:07 AM

#

toxic stratus seperate question, on an entity if I do `orders = [];` would it delete all the r...

this is exactly the path me and @old juniper are starting to go down... im not sure it explains everything, but it does explain some things... the question is how to fix

toxic stratus Jul 16, 2025, 8:08 AM

#

velvet lodge this is exactly the path me and <@633942991660384276> are starting to go down......

yeah, i just randomly walked into this issue just now

#

from what I expect, these relationships should be gone but they just appeared randomly

velvet lodge Jul 16, 2025, 8:09 AM

#

yeah, there is defo a bug there, but im not sure if its the "whole picture" or part of it

toxic stratus Jul 16, 2025, 8:10 AM

#

it's certainly curious how we've all ended up in a similar place though, must be something simple..... back to the repro 😄

#

repro attempt worked perfectly :/

toxic stratus Jul 16, 2025, 8:45 AM

#

well that's interesting 😄

#

I don't have a clue what's going on here my app isn't connected 🤣

toxic stratus Jul 16, 2025, 9:07 AM

#

cursed as hell, i've restarted literally every part of the chain and these things won't delete

#

right click -> empty table did the job

toxic stratus Jul 17, 2025, 10:39 AM

#

Is it possible to limit how deep the cascade goes?

#

I only want to remove the relationships but i suspect its removing the og data as well

#

maybe not.... something else weird is going on

#

I'm inserting 68 records but db always seems to have an arbitrary amount less than 68

toxic stratus Jul 17, 2025, 11:10 AM

#

exchange.data.Order.count(query($account == 2)).then((count) -> {
    trace('$count order records');
    trace('${orders.length} orders');
}, (err) -> trace(err));```

#

#

but i guess cascade does remove the original order records, which isn't really what I want 🤔

#

it's odd tho, because if that was what was going on I should end up with 0 orders rather than > 0

#

(don't mind me, just rubber ducking a lil bit now lol)

toxic stratus Jul 18, 2025, 8:35 AM

#

class Foo extends IEntity {
  public var data:Int = 0;
}```
1 benefit of this applying a default value is that all prior entries that existed prior to the point of column get auto filled in

velvet lodge Jul 22, 2025, 9:17 AM

#

@toxic stratus - at some point "soon", i have some pretty huge updates for e2... would be interested if you could plug it into your app and see if anything breaks (it will be a branch)... its not totally ready yet, but so far all the apps ive tested work (including UTs) - but would defo be interested in feedback

toxic stratus Jul 22, 2025, 10:37 AM

#

yeah that's fine with me

velvet lodge Jul 23, 2025, 4:56 AM

#

toxic stratus yeah that's fine with me

branch experimental/better_query_cache in e2
(need latest db-core)

toxic stratus Jul 23, 2025, 10:02 AM

#

Anything i should be weary of?

velvet lodge Jul 23, 2025, 10:02 AM

#

nope, nothing should be different functionally, but you should see significant perf improvements, especially if your db is remote (which it is for you)

#

(doesnt fix the orphans, just significant improvments in the query cache - which was totally broken 😮 )

toxic stratus Jul 23, 2025, 10:04 AM

#

how does query cache work?

#

if its the same query, take from the cache?

velvet lodge Jul 23, 2025, 10:05 AM

#

it looks to build up a list of ids "it will need" and then caches the results, it also (now) does some clever things to reduce queries that can be reconstructed from the cache

velvet lodge Jul 23, 2025, 10:05 AM

#

toxic stratus if its the same query, take from the cache?

yes, but only per call

#

so you will still always get "live" data, its just if a call wants "id 1" three times in the same call stack, it will get the other 2 from the cache

toxic stratus Jul 23, 2025, 10:06 AM

#

ohhh, that is a nice addition

#

should I try things without the fieldsets**?

velvet lodge Jul 23, 2025, 10:08 AM

#

the cache has always been there - it was just broken, but now its a) not broken and b) much smarter (it splits queries and allows cached results to be "reconstructued" from previously cached results with a different query... also, another big feature is it will "pre cache" queries... so it will first (again, per call) look to collate ids, and make a single query with all the ids (and cache the result, as well as the "split" results)

velvet lodge Jul 23, 2025, 10:08 AM

#

toxic stratus should I try things without the fieldsets**?

shouldnt make a difference, but mainly just make sure you app works the same

toxic stratus Jul 23, 2025, 10:09 AM

#

velvet lodge the cache has always been there - it was just broken, but now its a) not broken ...

oh that's another cool thing, i recently learnt about that when performing some db backups and was curious about how it structured the data

#

will git everything up and switch the branch

velvet lodge Jul 23, 2025, 10:11 AM

#

the main think im interested in is that everything works exactly the same, id also be interested in if there is a perf increase in your app since you talk remote to the db, so latency "per query" is serious

toxic stratus Jul 23, 2025, 10:13 AM

#

would be quite curious if it did help, i think i've managed to make it relatively "nice" for the most part

#

Will see if I can get some timings there

velvet lodge Jul 23, 2025, 10:14 AM

#

yeah, fieldset will still help ofc by totally just not making queries, but e2 makes SIGNIFICANTLY less queries now

#

i had a totally non trivial test (lots of relationships, very deep structure) and it went from 181 queries to 10 (granted the query cache was totally broken with the 181 version and i hadnt realised)

#

so if you assume 10ms latency per query (which isnt unreasonable) thats 1.8 seconds in just network comms, not even running the queries

#

realistically in my apps, i never talk remote to the db - its via an api, so no network lag - but thats not the point... 181 queries is INSANE

toxic stratus Jul 23, 2025, 10:23 AM

#

Fieldset:
Account(main) Loaded: 0.174 seconds
Account(Test1) Loaded: 0.135 seconds
Account(Test2) Loaded: 0.158 seconds
Account(Test3) Loaded: 0.141 seconds
Account(Test4) Loaded: 0.14 seconds
Account(Test5) Loaded: 0.139 seconds

No Fieldset:
Account(main) Loaded: 0.904 seconds
Account(Test1) Loaded: 2.343 seconds
Account(Test2) Loaded: 0.581 seconds
Account(Test3) Loaded: 1.625 seconds
Account(Test4) Loaded: 0.151 seconds
Account(Test5) Loaded: 0.893 seconds

So here's some befores

velvet lodge Jul 23, 2025, 10:23 AM

#

oh, nice one... will defo be interesting to see the "afters"

toxic stratus Jul 23, 2025, 10:28 AM

#

velvet lodge oh, nice one... will defo be interesting to see the "afters"

Fieldset:
Account(main) Loaded: 0.259 seconds
Account(Test1) Loaded: 0.241 seconds
Account(Test2) Loaded: 0.223 seconds
Account(Test3) Loaded: 0.25 seconds
Account(Test4) Loaded: 0.2 seconds
Account(Test5) Loaded: 0.225 seconds

No Fieldset:
Account(main) Loaded: 0.288 seconds
Account(Test1) Loaded: 0.296 seconds
Account(Test2) Loaded: 0.235 seconds
Account(Test3) Loaded: 0.321 seconds
Account(Test4) Loaded: 0.193 seconds
Account(Test5) Loaded: 0.254 seconds

#

Pretty good 😄

velvet lodge Jul 23, 2025, 10:29 AM

#

slower with fieldsets though, which seems odd

toxic stratus Jul 23, 2025, 10:29 AM

#

only just clocked that

#

was comparing to no fieldset

#

maybe fieldset is ignored in places with latest version?

velvet lodge Jul 23, 2025, 10:30 AM

#

i dont think so... but ill double check... maybe i missed something on fieldset... dont think i did though

toxic stratus Jul 23, 2025, 10:30 AM

#

timings are a lot more consistent overall though

velvet lodge Jul 23, 2025, 10:30 AM

#

yeah, its defo better... and more importantly, app works exactly the same?

toxic stratus Jul 23, 2025, 10:30 AM

#

app seems to be working fine at first glance

velvet lodge Jul 23, 2025, 10:30 AM

#

perfect

toxic stratus Jul 23, 2025, 10:30 AM

#

no errors or anything obviously wrong

#

not sure what would happen if something updated something in the cache and i read from the same query or cases like that yet though

velvet lodge Jul 23, 2025, 10:31 AM

#

what cache? e2 query cache?

toxic stratus Jul 23, 2025, 10:31 AM

#

yea, like backend updates a value in the db

velvet lodge Jul 23, 2025, 10:32 AM

#

it wont matter... its restricted per call, the cache isnt global or anything, its destroyed after use (ie, after the call "find" for example, is complete)

toxic stratus Jul 23, 2025, 10:32 AM

#

ohhhh, that's neat

velvet lodge Jul 23, 2025, 10:36 AM

#

eg:

findByIdFn.code += macro @:privateAccess {
    var queryCacheId = entities.EntityManager.instance.generateQueryCachedId();
    if (fieldSet == null) fieldSet = new entities.EntityFieldSet();
    return new promises.Promise((resolve, reject) -> {
        init().then(_ -> {
            return findInternal(query, queryCacheId, fieldSet, 1);
        }).then(entitiesList -> {
            entities.EntityManager.instance.clearQueryCache(queryCacheId);
            resolve(entitiesList[0]);
        }, error -> {
            entities.EntityManager.instance.clearQueryCache(queryCacheId);
            reject(error);
        });
    });
}

#

queryCacheId stuff - it gets passed around and the cache only applies to that call ("find" in this case)

toxic stratus Jul 23, 2025, 10:38 AM

#

that's really nice

#

no need to get the same account on all results

#

basically makes fieldset a bit useless 😄

velvet lodge Jul 23, 2025, 10:41 AM

#

toxic stratus that's really nice

exactly, and the precache is smarter now... so it will build up a query like "select * from t where x in (1, 2, 5, 7, 8)"... then it will cache this result, but the best part is that its all the query ast in db-core, so i can then split that into individual cache entries for 1, 2, 5, 7, 8... and then later if another query comes through with "select * from t where x in (1, 2, 8)" (which is also a query expr) then i can say "oh, i have all those individual ids cached already, and can "reconstruct" a recordset from those ids

velvet lodge Jul 23, 2025, 10:44 AM

#

toxic stratus basically makes fieldset a bit useless 😄

not really, feildsets were never supposed to be a perf enhancement, they are there so you can stop certain things from being populated (like sensitive db data)... previously in entities1, you would have to do something like user.passwordHash = null (if you were sending the entity to the client via serialization)... even worse in e1 if it was a sub object... eg: foo.bar.user.passwordHash = null... in e2 you use fieldsets like User.NoSensitiveInfo

velvet lodge Jul 24, 2025, 6:41 AM

#

toxic stratus Fieldset: Account(main) Loaded: 0.259 seconds Account(Test1) Loaded: 0.2...

when you get a moment, can you pull e2 (branch) and retry? I wasnt honouring fieldsets in the query precache, so it was building up ids for things it would never query (because of fieldsets)

toxic stratus Jul 25, 2025, 10:06 AM

#

velvet lodge when you get a moment, can you pull e2 (branch) and retry? I wasnt honouring fie...

Account(main) Loaded: 0.222 seconds
Account(Test1) Loaded: 0.183 seconds
Account(Test2) Loaded: 0.179 seconds
Account(Test3) Loaded: 0.2 seconds
Account(Test4) Loaded: 0.168 seconds
Account(Test5) Loaded: 0.179 seconds

#

that's with fieldsets now

velvet lodge Jul 25, 2025, 10:27 AM

#

same ish...

#

dont really understand how it can be slower than the original version though... you sure you pulled latest... maybe network variance

toxic stratus Jul 25, 2025, 11:33 AM

#

#

no more things to pull

#

but i'll do another test in a bit, need to do some stuff first tho

toxic stratus Jul 25, 2025, 1:45 PM

#

Account(main) Loaded: 0.133 seconds
Account(Test1) Loaded: 0.203 seconds
Account(Test2) Loaded: 0.176 seconds
Account(Test3) Loaded: 0.193 seconds
Account(Test4) Loaded: 0.128 seconds
Account(Test5) Loaded: 0.149 seconds

#

maybe recompile didn't occur from the last one because these are better on average

#

2nd one being higher is odd

#

ohh... i did a mistake 🤦‍♂️

#

need to reprocess the data

velvet lodge Jul 25, 2025, 1:55 PM

#

OK, thats seems better though

#

little bit of variance isnt super surprising, but they are all less than the "non optimized" version

#

what sort of load is this out of curiosity? Like, how many records (ish) - including relationships etc

toxic stratus Jul 25, 2025, 1:57 PM

#

it's quite a good stress test all around, even for my account sync setup

#

relationships vary from position to position

#

will give some details after reprocessing the data

toxic stratus Jul 25, 2025, 2:06 PM

#

velvet lodge what sort of load is this out of curiosity? Like, how many records (ish) - inclu...

** No Fieldset**
Account(main) Total Records=35 Total Orders=78 Time: 0.312 seconds
Account(Test1) Total Records=112 Total Orders=225 Time: 0.385 seconds
Account(Test2) Total Records=21 Total Orders=44 Time: 0.237 seconds
Account(Test3) Total Records=77 Total Orders=155 Time: 0.312 seconds
Account(Test4) Total Records=1 Total Orders=2 Time: 0.257 seconds
Account(Test5) Total Records=37 Total Orders=74 Time: 0.259 seconds

Fieldset
Account(main) Total Records=35 Total Orders=0 Time: 0.219 seconds
Account(Test1) Total Records=112 Total Orders=0 Time: 0.188 seconds
Account(Test2) Total Records=21 Total Orders=0 Time: 0.184 seconds
Account(Test3) Total Records=77 Total Orders=0 Time: 0.203 seconds
Account(Test4) Total Records=1 Total Orders=0 Time: 0.179 seconds
Account(Test5) Total Records=37 Total Orders=0 Time: 0.206 seconds

#

each order/record will have a reference to an account as well

#

and account will reference a user

velvet lodge Jul 25, 2025, 3:33 PM

#

toxic stratus > ** No Fieldset** > Account(main) Total Records=35 Total Orders=78 Time: 0.312 ...

cool - so i think im going to merge this into main pretty soon... all my tests are good, timings dont improve drastically (because its all localhost effectively) but query count defo does

toxic stratus Jul 25, 2025, 3:41 PM

#

yeah man it's super cool how much it's reduced

#

it would basically deadlock the page

#

for seconds in some cases

#

now it's basically nothing

velvet lodge Jul 25, 2025, 3:59 PM

#

i would still advise that dont call directly into the db (ever! nothing to do with e2)... but thats not the point... 181 queries vs 4-5 is crazy... granted not trivial relationship wise... but thats defo NOT the point

toxic stratus Jul 25, 2025, 4:02 PM

#

velvet lodge i would still advise that dont call directly into the db (ever! nothing to do wi...

wdym? (want to make sure I understand the advice)

velvet lodge Jul 25, 2025, 10:16 PM

#

toxic stratus wdym? (want to make sure I understand the advice)

i mean that the db access should be behind an api layer... if you had a "findUsers" kinda thing, you wouldnt want that talking to the db directly, you would want that exposed as an api function, and that api function would talk to the db... that api layer would include the security, etc

#

atm, when your app wants "orders" its talking direct to the db asking for them - the reason why i never noticed the latency is because the client talks to the api and that talks to the db which is "0" network latency, so it just never came up the huge amount of queries it was creating

toxic stratus Jul 25, 2025, 10:18 PM

#

velvet lodge i mean that the db access should be _behind_ an api layer... if you had a "findU...

that's the ultimate plan

velvet lodge Jul 25, 2025, 10:18 PM

#

whats "crazy" is once i plugged my local app into a remote db (which is bad mojo) is saw stupid stupid times... ...

#

so for sure, e2 (well, the query cache) was bugged to hell

#

not the sort of thing i can reliably unit test either... well, not true-ish, i have a few unit tests now for the query count and the query cache hit count... so if something gets better or worse, these UTs will fail

velvet lodge Jul 25, 2025, 10:22 PM

#

toxic stratus that's the ultimate plan

i think you defo want that pattern / abstraction... direct db access is bad (unless its a local app, but even then i would argue it cant scale)... but, in this specific case it led me down a path of bugs and "i dont like this, thats crazy, that many queries?!?!" 😄

toxic stratus Jul 25, 2025, 10:24 PM

#

with the way I've developed my app switching out to an api isn't that big of an issue

velvet lodge Jul 25, 2025, 10:24 PM

#

nice, sounds like you already abstracted out the data layer on the client 👌

toxic stratus Jul 25, 2025, 10:25 PM

#

so I'm happy to dev with direct access, and I'm gonna be the primary user and tester anyway for the foreseeable future so no need to worry about the api abstraction for now

#

I'm sure some will bundle sql with their apps tho so still useful

velvet lodge Jul 25, 2025, 10:26 PM

#

i use a pattern like: var client = RpcClient<ISomeApi>()... then the macro builds "all the parts"... best part is that macros also build the (dynamic) parts of the server also

toxic stratus Jul 25, 2025, 10:26 PM

#

lol

velvet lodge Jul 25, 2025, 10:26 PM

#

so if you change the api, then server or client will fail to build

toxic stratus Jul 25, 2025, 10:26 PM

#

magic 😄

velvet lodge Jul 25, 2025, 10:28 PM

#

yeah, its actually really nice, i want to extract it at some point and make it a little more generic... its fairly generic, but i feel like it probably has some shortcuts in it for glint specifically (i dont remember any, but im assuming)... but the client -> server comms is a dream (over any transport, http, websockets, files if you wanted, its all pluggable)

#

(trying to find an example to gush about 😄 )

#

@:expose @:keep
@:operation(glint.api.users.UsersApi.findUsers)
class FindUsers implements IRpcServerOperation {
    private static var log = new Logger("glint.api.users.server.operations.FindUsers");

    public function execute(criteria:FindUsersCriteria, requestor:UserData):Promise<Array<UserData>> {
        return new Promise((resolve, reject) -> {
            log.info('finding users, criteria: ${criteria} (requestor: ${requestor != null ? requestor.username : null})');

            var queryParts:Array<QueryExpr> = [];
            if (criteria.name != null) {
                queryParts.push(Query.query($username = criteria.name));
            }

            if (criteria.searchTerm != null) {
                queryParts.push(Query.query($username =~ criteria.searchTerm));
            }

            var query = Query.joinQueryParts(queryParts, QBinop.QOpBoolAnd);

            UserData.findAll(query, criteria.pageSize, UserData.NoSensitiveInformation).then(users -> {
                resolve(users);
            }, error -> {
                reject(error);
            });
        });
    }
}

thats the server operation, that gets built into a js (basically) and gets dynamically loaded (based on some config), all the endpoints are created, etc...

toxic stratus Jul 25, 2025, 10:31 PM

#

ohh it plugs with entities

velvet lodge Jul 25, 2025, 10:31 PM

#

entities, serializers and rpc (which isnt released)

#

call is just:

ApiManager.instance.users.findUsers({searchTerm: "*" + searchTermTextField.text + "*", pageSize:ITEMS_LIMIT_PER_TYPE}, null).then

#

quite alot of "magic" going on client side to be fair

#

i feel like ive always been searching for the "holy grail" of client / server comms... and, maybe it will change, but this pattern is insanely productive... you write a server side function, and it all kinda just gets "linked up"

uncut phoenix Jul 25, 2025, 10:34 PM

#

i feel like at this point core-haxe is an entire web framework lol

velvet lodge Jul 25, 2025, 10:36 PM

#

uncut phoenix i feel like at this point core-haxe is an entire web framework lol

to be fair, its just how you put it all together... like, this "RpcOperation" is using core-haxe/modular... didnt need to, but its perfect... the operations are loaded by the server, it creates the endpoint (well, the rpc system does) and it all just slots in, under the hood its using http (or in the future websockets, and could flip between them)... but the libs themselves are totally standalone

#

no part of entities, http, rpc, logging, etc is required for the other

#

actually, maybe logging and promises... thats probably it though

#

also... ... ... ... not a web framework 😉 (could fairly easily create a similar thing with hxwidgets and the same system - modular would be an issue there, but when i have more time, i know it would all work via cppia)

knotty vortex Jul 26, 2025, 4:56 PM

#

@velvet lodge what server backend are you building towards?

velvet lodge Jul 28, 2025, 7:25 AM

#

knotty vortex <@456226577798135808> what server backend are you building towards?

currently im targeting node, but i think all the libs (with the exception of modular) should work as is on most haxe targets

toxic stratus Jul 29, 2025, 8:02 PM

#

loaded up a page on linux and things were taking a long time, thought i had broke the app

#

turns out i just forgot to sync the e2 branch change x)

toxic stratus Aug 4, 2025, 11:45 AM

#

#

(╯°□°)╯︵ ┻━┻

#

Are you good if I make encoding optional?

velvet lodge Aug 4, 2025, 11:48 AM

#

thats in haxe.Http... though its shadowed in core-haxe/http... why wouldnt you have things encoded anyway?

toxic stratus Aug 4, 2025, 11:50 AM

#

velvet lodge thats in haxe.Http... though its shadowed in core-haxe/http... why wouldnt you h...

i'm signing something and its interfering with the exchange validating the signed request due to mismatching url

velvet lodge Aug 4, 2025, 11:52 AM

#

looks like it does similar in "non-shadowed" haxe: https://github.com/HaxeFoundation/haxe/blob/development/std/haxe/http/HttpJs.hx#L127

toxic stratus Aug 4, 2025, 11:52 AM

#

yeah, its defo a std thing

velvet lodge Aug 4, 2025, 11:52 AM

#

its pretty standard / important i think

#

why dont you urlEncode your request url so they match?

toxic stratus Aug 4, 2025, 11:53 AM

#

its typically an optional thing

velvet lodge Aug 4, 2025, 11:53 AM

#

ive never heard of it being optional personally... but maybe i just havent come across it

toxic stratus Aug 4, 2025, 11:54 AM

#

i assumed exchange would expect its page token in original form but I guess I can verify

#

yeah encoding the string causes other issues

#

realistically it makes sense

#

if i'm encrypting a string that needs to be in a certain format, and the validation string in the url is different then its just rejecting the string

toxic stratus Aug 6, 2025, 8:48 AM

#

@velvet lodge I did manage to solve it by making encoding optional, you cool with a pr, i think its done in a non breaking way

toxic stratus Aug 6, 2025, 9:05 AM

#

https://github.com/core-haxe/http/pull/16

toxic stratus Aug 6, 2025, 9:28 AM

#

oh no...

#

You don't shadow httpbase 🤔

#

adjusted the pr

#

I have an alternative change i'd like to propose as well for httpresponse

#

class HttpResponse<T:Dynamic = Dynamic> {
    public var bodyAsJson(get, null):T;
    private function get_bodyAsJson():T {
        if (body == null) {
            return null;
        }

        return Json.parse(body.toString());
    }
}

It is haxe > 4.3 syntax i believe so can conditionally wrap it but it's a nice change none the less I think

#

if no type parameter is passed it just defaults to Dynamic

#

yeah fails on haxe < 4.3

velvet lodge Aug 6, 2025, 10:31 AM

#

toxic stratus <https://github.com/core-haxe/http/pull/16>

would it not be better as a #if?

velvet lodge Aug 6, 2025, 10:32 AM

#

toxic stratus ```hx class HttpResponse<T:Dynamic = Dynamic> { public var bodyAsJson(get, n...

how would you use this though? I mean, how would supply the type param?

toxic stratus Aug 6, 2025, 11:12 AM

#

velvet lodge would it not be better as a #if?

Not all cases require one or the other, node has it as an optional parameter

toxic stratus Aug 6, 2025, 11:13 AM

#

velvet lodge how would you use this though? I mean, how would supply the type param?

client.post().then(function(resp:HttpResponse<MyType>){})

#

on phone atm so may not be exactly correct but I'm using it like that locally and it works fine

velvet lodge Aug 17, 2025, 12:54 PM

#

new addition: https://github.com/core-haxe/observable

GitHub

GitHub - core-haxe/observable: observable

observable. Contribute to core-haxe/observable development by creating an account on GitHub.

#

still wip, need more test coverage, but got bored, working as expected with manual tests

#

basic usage:

class ObservableObject1 implements IObservable {
    public var intVar:Int = 10;
    public var objectArray:Array<ObservableObject1>;
    public var subObject:ObservableObject1;
    public var normalArray:Array<Int>;
    public var objectMap:Map<String, ObservableObject1>;
    public var normalMap:Map<String, Int>;
}

ObservableDefaults.GroupChanges = true; // defaults to true
ObservableDefaults.EliminateDuplicates = true; // defaults to true

var o1 = new ObservableObject1();
o1.registerChangeListener((changes) -> {
    ...
});

o1.intVar = 101;
o1.intVar = 101;
o1.intVar = 102;
o1.subObject = new ObservableObject1();
o1.subObject.intVar = 201;
o1.objectArray = [];
o1.objectArray.push(new ObservableObject1());
o1.objectArray[0].intVar = 505;
o1.objectArray[0].subObject = new ObservableObject1();
o1.objectArray[0].subObject.intVar = 606;
o1.objectArray[0].subObject.intVar = 607;
o1.objectArray[0].subObject.intVar = 608;
o1.objectArray[0].subObject.intVar = 609;
o1.objectArray[0].subObject.objectArray = [new ObservableObject1()];
o1.objectArray[0].subObject.objectArray[0].intVar = 707;
//o1.objectMap = []; // wont work, need to init as map, or use ObservableMap explicitly 
o1.objectMap = ["one" => new ObservableObject1()];
o1.objectMap.get("one").intVar = 808;
o1.objectMap.get("one").subObject = new ObservableObject1();
o1.objectMap.get("one").subObject.intVar = 909;
o1.objectMap.get("one").subObject.objectArray = [new ObservableObject1(), new ObservableObject1(), new ObservableObject1()];
o1.objectMap.get("one").subObject.objectArray[1].intVar = 333;
o1.normalArray = [];
o1.normalArray.push(1111);
o1.normalMap = ["tim" => 2222];
o1.normalMap.set("tim", 3333);

#

( ☝️ that example there leads to a single invocation of the registerChangeListener callback with a list of 24 changes)

toxic stratus Sep 15, 2025, 11:35 AM

#

Does entities/dbcore support haxe enums as a type?

#

enum Foo {
  Color(r:Int, g:Int, b:Int);
}```
Kind of deal

velvet lodge Sep 16, 2025, 8:44 AM

#

toxic stratus ```hx enum Foo { Color(r:Int, g:Int, b:Int); }``` Kind of deal

it doesnt, not currently - i would like to add them, but it wouldnt be trivial

toxic stratus Sep 16, 2025, 8:49 AM

#

yeah seems like a pretty powerful addition

velvet lodge Sep 16, 2025, 8:50 AM

#

i think you could / would just tread them like any other entity (ish)..

#

so you would have a "color" table, with id, r, g, b... and that would link to the instance var (by id)... which is how 1->1 entities work

#

so i think the general system is already there... its "just" doing it, and all that entails

toxic stratus Sep 16, 2025, 8:53 AM

#

Oh interesting

#

that's a cool way to do it

#

i thought it would just get serialised/jsonified 😂

toxic stratus Sep 19, 2025, 12:27 PM

#

https://github.com/HaxeFoundation/haxe/issues/12369#issuecomment-3311588934 Might be useful for you to read

velvet lodge Sep 19, 2025, 1:57 PM

#

OK, intersting about the rejection, so i guess i can do a "if error is MySqlError"... doesnt fix the underlying problem though, correct? Just better error handling

toxic stratus Sep 19, 2025, 2:04 PM

#

yeah, underlying problem is still in haxe

toxic stratus Feb 13, 2026, 1:47 PM

#

import shared.dbtypes.User;
import entities.IEntity;

class SessionToken implements IEntity {
    public var user:User;
    public var token:String;
    public var created:Float;
    public var expires:Float;
    public var lastActive:Float;
    public var ip:String;
    public var name:String;
}

class User implements IEntity {
    public var username:String;
    public var password:String;
    public var passwordSalt:String;
    public var created:Float;
    public var lastLoggedIn:Float;
}

#

Am I doing something wrong here? this seems correct, but I'm getting:

mysql/impl/cpp/DatabaseConnection.hx:163: UKNONWN:,TClass(shared.dbtypes.User)

#

@smoky crane (pinging cause you probably aren't in the thread anymore 😄)

smoky crane Feb 13, 2026, 1:58 PM

#

valid, defo wasnt 🙂

toxic stratus Feb 13, 2026, 3:01 PM

#

#

mysql/impl/cpp/DatabaseConnection.hx:163: UKNONWN:,TClass(shared.dbtypes.User)

#

what in the world is this sillyness

#Core Haxe