Core Haxe

[velvet lodge]

my personal guess is that the query expr isnt explicitly noticing its bool and doing something

[toxic stratus]

but true is probably getting parsed to a json string "true"

[velvet lodge]

oh yeah? Ok, but still, i dont like that you have to use string "true" in haxe... doesnt feel right

[toxic stratus]

i think i'm doing something a bit off the road here anyways 😄

what i want here is a ```json
true

and what haxe `true` is parsing as is a json:
```json
"true"

[velvet lodge]

still, i think ill UT it and see what happens, feels like it should notice its true (actual haxe bool) and either turn it into "true" (mysql string) or 1 (boolean bit - dunno if thats how mysql stores bools, but feels sensible)

[toxic stratus]

i'm only guessing at what i think is going on here anyways

[velvet lodge]

yeah, same, unit test will make it clear

[toxic stratus]

mysql doesn't have a bool field does it?

[velvet lodge]

MySQL treats it as a synonym for TINYINT(1)

ok, and its "TRUE" or "FALSE" - not string literals - no quotes

[toxic stratus]

oh huh

[velvet lodge]

wonder if that applies to json fields though... probably not

[toxic stratus]

so maybe add tiny int here and parse it as a haxe bool? 🤔

[velvet lodge]

the json part i think will be different, i mean, im assuming

[toxic stratus]

for sure, but the json was trying to make up for a lack of a bool type

[velvet lodge]

WHERE jsonfield.someBoolField = TRUE - if that works, its defo ideal

[toxic stratus]

ah i see

i didn't even consider that

[velvet lodge]

yeah, no, its not the same

This is different from a BOOL column, which is really just a TINYINT(1) (numeric).

llm, so could be wrong, but i doubt it - will UT it tomorrow with the variations to confirm

still, i get the feeling that, nothing to do with json columns, core-db isnt handling true (haxe type) properly in the query expr... might be totally wrong about that, but will check it out

i tend to just use 1 and 0 for bool types in dbs... and, i dont think, ever use TRUE or FALSE... i also dont ever remember writing a query expr in core-db like $someField = true... so could just be an oversight (likely)

[toxic stratus]

yeah, i was doing 1/0 but when i saw the json field i thought I could just get it to work

i like being able to do if (obj.isTrue)

[velvet lodge]

yeah, thats the part i feel wouldnt work currently in core db (not a json field)... i know i handle that specifically in entities2 (in entities2 code there is macros like this.$field = record.get($v{field}) == 1)

i think it would be if (obj.isTrue == 1)

(im going to start referring to entities2 as entities, since i archived entities1 - that was news to me... "Past Ian" is more on the ball than i thought 😄 )

[toxic stratus]

lol

just call it e

[velvet lodge]

e^2

[toxic stratus]

😂

[toxic stratus]

trace(Type.typeof(settings.isdefault)); evaluates to a String on js

[velvet lodge]

isdefault is a json field that just contains "true", right?

https://try.haxe.org/#f0b1b994

i think i prefer the hxcpp version here, though i dont like the mismatch

[toxic stratus]

correct

same, on cpp it evaluates to bool

i have no idea why it doesn't here if it parses everything else as a json

[velvet lodge]

thats because the db-core/libmysqlclient parses json fields

[toxic stratus]

is it just a coincidence that js works with json fields then? lol

[velvet lodge]

who knows, it probably parses it, but maybe there is some "if startswith { or [" or something

[toxic stratus]

ahhhh

[velvet lodge]

i cant control that npm lib

[toxic stratus]

well either way its fine... i think

[velvet lodge]

well, its defo not ideal, its a mismatch in behaviour, which i dont like - that said, it defo feels like an edge case

[toxic stratus]

C:\Users\j\Documents\projects\ceramic\ProjectStudy\src\systems\StyleSystem.hx:30: TClass({
        __name__ : String
    }), false

yeah okay, not fine

[velvet lodge]

its arguable that you are abusing the json field?

[toxic stratus]

thought that would still eval to true

[toxic stratus]

absolutely, but, it is valid json

[velvet lodge]

in fact, hang on, why are you using a json field to have "true", why not just a bool?

[toxic stratus]

because i only found out that it existed when you mentioned it not too long ago 😄

[velvet lodge]

agreed

[velvet lodge]

right, and maybe its not nicely supported in core-db... (that part is easily fixable)

[toxic stratus]

i might be able to get that to work tho

wonder how a tinyint field gets eval'd on js

[velvet lodge]

i defo feel like if you just want a bool, then you should use a bool type, if core-db isnt handling it properly, its better to fix that than shoehorn a json type

[velvet lodge]

yeah, defo need unit tests here

[toxic stratus]

testing tinyint now

[velvet lodge]

i think in js "if (someVar)" means "not null", i dont know what happens in js if "someVar === 0"

[toxic stratus]

StyleSystem.hx:30: TInt, false

int gets parsed as an int

[velvet lodge]

some freaky js shit no doubt

[toxic stratus]

so it isn't a real bool

[velvet lodge]

but its not in the db either

[toxic stratus]

no no, it is now

i changed it to tinyint

[velvet lodge]

tinyint isnt a bool... i mean, it is, but its just a 1/0

mysql has no concept of "bool"... as far as i understand

[toxic stratus]

oh, yeah, that's what i thought

that's why i was using the json field lol

[velvet lodge]

mysql being the same as C i presume

i think i just need to write some UTs to properly understand it... but i defo get that "if (record.isDefault)" is better than "if (record.isDefault == 1)", and im almost sure core-db should be able to handle that correctly (since i do in entities)

as far as i remember, in C / C++ its basically:

#define BOOL int
#define TRUE 1
#define FALSE 0

BOOL b = TRUE; // int b = 1;

there isnt some intrinsic "bool" type in computers

[toxic stratus]

yeah i think its typically ints

it's a "small issue" either way

[velvet lodge]

exactly, so its just conversion...

[velvet lodge]

true, but "if (true)" is nicer than "if (1)"

[toxic stratus]

js craziness

[velvet lodge]

... yeah

x = 0 means it wont trigger if (x)

so x == 0 is the same, in js as x == null

so 1 and 0 is true and false in js? I mean, thats "ok" i think, the "0 is null" is odd though

yeah, crazy

x = 0
if (x) console.log("x"); else console.log("not x");

not x

😬

i thought "if x" meant "if its null or undefined", but nope, it also means "if its not 0", it seems, thank god i dont have to deal with vanilla js

[toxic stratus]

yeah, haxe makes things quite nice

i treat js as if its native

and everything makes sense

[velvet lodge]

amen

you cant even do "if x" in haxe (which makes perfect sense ofc)

if you are going to create that construct though, it should at least make sense... i dont even get the "or if its zero" thing tbh

"if x is null, undefined or zero"... that "zero" seems very out of place 😄

[toxic stratus]

if you think about it like its a "speaking language" it makes a bit more sense

nothing == nada == none

[velvet lodge]

dunno... im not on board... 1234/ 0 = undefined (undefinable)... they mean different things

[toxic stratus]

agree

[velvet lodge]

im probably over thinking it no, now one writes anything serious in vanilla js, but its crazy dangerous

[toxic stratus]

well given all these crazy paths

you really would think true =="true" would pass 😂

[velvet lodge]

if (!someVar) {
   throw "some crazy exception that takes the whole system down because null is bad"
}

... ... ... someVar = 0... 😱

mental

note to self: if ever dealing with vanilla js, use if (someVar === null)

[old juniper]

Wouldn't it be more "if x is null, undefined, false or zero" ? that false == 0 would kind of make sense

[velvet lodge]

zero (or false) != undefined

(or null)

undefined is "without a definition", null, is "no value"... false and zero are both values (and definitions)

[old juniper]

Don't worry I agree with you.. But I think there's some logic behind.
if (x) -> if (x==true) -> if (x!=0)

[velvet lodge]

im sure there is logic, but its faulty, zero / false (or 1 / true) shoudnt be lumped in with null / undefined

[old juniper]

Yeah it feels kind of "old" in a way... wouldn't have expected this in js.

[velvet lodge]

thats not accurate though, it means if (x != null && x != undefined && x != 0) (for example)... the negative case is much worse: if (x == null || x == undefined || x == 0)

... ... that 0 just slips through 😮

[velvet lodge]

wouldn't have expected this in js.
really? js is known for this insantity... there is this "whhhhhat" thing? Ill see if i can find it

https://www.youtube.com/shorts/-a6JpgtTRyw

... that was easy 😄

oh, its not the one i meant though

https://www.destroyallsoftware.com/talks/wat

[old juniper]

haha indeed... I knew php was kind of like this .... never knew js was this insane. I should thank haxe too ( for sanitizing it)!

[uncut phoenix]

seen this before... i feel like the audience is on something

[deep ore]

this is unironically correct and sane

[deep ore]

if (x) {} wont run if x is a falsy value in js
as in, null, undefined, NaN, 0, -0 and the empty string IIRC

[old juniper]

I do understand the logic... pretty simple when we think about it...
If you use "+", it tries special concatenator for string.
If it's "-", it considers both a number except that one is not a number so it's Nana

The big video though... there were too many for me.

[deep ore]

ye
it's just type coercion doing its thing

[toxic stratus]

Hey Ian, https://github.com/core-haxe/db-core/pull/9 in reference to this issue

underscore is failing

pls merge the flag

[toxic stratus]

hmm, issue looks similar but may be a case of bad code

got 3 libs, don't want each lib to depend on each other, but at the same time they're all used together

so got duplicated code and mismatching type paths :/

[toxic stratus]

hmm... what's the solution to the problem...

another library! 😄

lets see how it pans out 🙈

[velvet lodge]

i dunno, im a little averse to just adding a flag "because"... so, in all the haxeui-* libs, i need to do the same? I doubt it... i think there is something weird going wrong somewhere else...

-lib db-core

...
...
...

#if db_core
trace("got it");
#end

should work

also, that error doesnt seem to be related? Or, well, i guess, i dont know what Db.insert is doing... looks like a good 'ol type mismatch though

[toxic stratus]

src/systems/PositionSys.hx:79: {
message : Invalid JSON text: "not a JSON text, may need CAST" at position 0 in value for column 'positions.open'.,
call : update
}
Yea, maybe auto conversion to json on nodejs at least isn't a good idea

[velvet lodge]

what was the field value?

[toxic stratus]

bool

true

[velvet lodge]

i dont think this is coming from db-core though... that error looks like its coming from mysql server

but maybe db core is creating a crappy query though

[toxic stratus]

i don't think its core, i think this is the nodejs externs

I think I understand the problem here

i just don't know how or where to solve it

[toxic stratus]

i think i'll give up on json for bool column

a quest for another time x)

[velvet lodge]

cant you use an actual BOOL type?

[toxic stratus]

wdym

mysql doesn't have a bool type

[velvet lodge]

it does apparently

[toxic stratus]

waah i've looked at that list multiple times

i just switched to tinyint and you're telling me this now

i did see a unknown type for the column that i was tempted to mess around with

but it seemed a bit too experimental for my mood currently x)

[toxic stratus]

What is the bool type lol

[velvet lodge]

huh, your are right, i can only find this now: https://github.com/core-haxe/libmysqlclient/blob/main/lib/include/field_types.h#L77

and the comment doesnt fill me with confidence

[toxic stratus]

its alright, getter life is fine

i did think that dbcore could auto convert true to 1 and false to 0 if it was passed as a value but i think that's an added assumption

like a nosql db supports actual bools so that wouldn't be good

[velvet lodge]

i think it could / should if you use "true" in a query

(it might already do that tbh)

but getting it out of the db, it will always be 1 / 0... db core isnt going to know (i dont think) if its a bool or an actual int

(entities2 handles all this ofc 🙂 - but thats because the entity objects are the schemas, so it knows its a boolean from the haxe source)

[toxic stratus]

is that smart tho? postgres supports bool types as well

maybe it can auto convert true/false for the mysql backend?

[velvet lodge]

well, the query just turns into a QueryExpr (so just a sort of small AST for a representation of a query), its then the db plugins job to turn that into something useful... so sql for mysql / sqlite

so if there was another db plugin then it would have to turn the QueryExpr into something that that db platform would understand

eg: eventually ill create a IndexedDB plugin (the database thingy that lives in the browser), that doesnt use sql, so all query exprs have to work with that system (not hard i dont think, maybe even easier than sql)

so, TLDR: i dont think there should be any problem with a QueryExpr, when turning into SQL, will recognise "its a bool" and turn it into "WHERE FIELD = 1"

[toxic stratus]

Ohhh cool i getchu now

lots of layers to abstractions

missed where query expressions are in the chain

[velvet lodge]

yeah, its a super important part of the abstraction tbh... because queries are pretty much everything, but once you say "has to be sql" you are now totally limited to what dbs you can support

another eg: eventually, id like to use this with lucene / solr and that has a totally different query system... so, if queries were "only sql" then its just a non starter

also, something like entities2 living in the browser (as IndexedDB) is a pretty cool idea

[toxic stratus]

question about e2, if I have a record mapped out, and i update the field does that automatically update the database?

[velvet lodge]

no, you need to call .update()

https://github.com/core-haxe/entities2/blob/main/tests/cases/basic/TestBasic.hx#L40-L70

[toxic stratus]

interesting either way

question is, is it worth moving from

Db.update('table', data, query($aid == aid));
😄

[velvet lodge]

the issue here, to me, would be "what is 'data'"... Are you turning it to / from actual objects / typedefs at some point?

[toxic stratus]

Nope

it's a db.Record that's generated by a macro from the class fields

[velvet lodge]

so yeah, its just a collection...

entities creates actual objects, that can link to each other, etc

[toxic stratus]

the relationship stuff sounds interesting for sure

[velvet lodge]

you cant use arrays in your setup... and you certainly cant use an object that has a bunch of arrays contained in it

(not trying convince you, just highlighting the differences)

[toxic stratus]

doesn't matter either way

i'm certainly curious about e2

[velvet lodge]

i would also say, this isnt a million miles away from e2, its just that e2 does it all for you generically... so you dont have think about it anymore... looking at this it seems like a "personal ORM" and entities is a "generic ORM"

[toxic stratus]

yeah, that's kind of why i'm in the middle here

I think I basically have a form of orm with my ecs and the above setup

you've just gone way deeper with the details than i have

mines at version - "it works"

[velvet lodge]

yeah, thats pretty much exactly why e2 exists, i ended up doing it over and over again...

if it works, and you dont need 1 -> 1 or 1 -> many relationships (which i think?) your ORM doesnt handle, then it doesnt matter so much

[toxic stratus]

i think i will and i also want many to many at some point

[velvet lodge]

how would you represent many -> many in haxe code? (the haxe code is the schema in e2)

[toxic stratus]

i have no idea

i have no use for it currently

but i will have things like custom tagging for data

i figured many to many is the way to go about that

[velvet lodge]

in entities it would probably just be an var tags:Array<Tag>

as a member var for an entity (Tag also being an entity ofc)

[toxic stratus]

i think i'm nearing the point where i'll probably get to justify playing with it

couple more backend things to do on the list

then I can run away from the messy code base into e2 for a lil bit

and at least then, I can experiment on a new branch

[velvet lodge]

cool - i would have a play with it in isolation first, you already have an ORM, so its a change

[toxic stratus]

yes, i'll defo do that

i've peaked at it a few times and i don't immediately get it

don't want my project context to mix it up even more

[toxic stratus]

nooo

just hit a moment where e2 would be useful 😅

i have an array of order ids, that i'd really like to be an array of order objects and i suspect e2 would have that setup

[velvet lodge]

yessir

i wanna add maps next, but the linking tables arent quite so simple

[toxic stratus]

aaaand because i don't have access to the order objects some of my tables have additional columns that I probably wouldn't require if the relationships were handled

okay, i think i've got enough convincing to actually look into it 😄

its funny how we can work around problems and not even notice them x)

[toxic stratus]

so the tricky part of figuring out e2 is gonna be all these meta's

[toxic stratus]

yeah, e2 seems confusing as heck

where do you even load data from the db

all the cases seem to just create things locally

am i thinking "too db-like"?

[velvet lodge]

wdym?

[velvet lodge]

also, wdym?

[toxic stratus]

i see no get queries or how to "load" data from the db

everything seems to just magically exist

at least in the test cases

there's no relatable app flow that I can pick up lol

[velvet lodge]

everything does magically exist, yeah:

class MyEntity implements IEntity {
    public var someString:String;
}

EntitiesManager.instance.database = ... // any db-core db

var myEntity = new MyEntity();
myEntity.someString = "bob";
myEntity.add().then(...);

[toxic stratus]

or table definitions

[velvet lodge]

e2 will build all the tables for you, will update them where needed also (schema diffing)

[toxic stratus]

in the ref examples there's var arrayobj:Array<X>

but i have no idea what the type of arrayobj is

[toxic stratus]

its fair, but does that mean i should start a completely new db for e2?

[velvet lodge]

ideally

if the tables already exist, it will modify them

[toxic stratus]

okay, that's good to know, i'll dupe rather than test on my existing db

[velvet lodge]

not sure what you mean here

[toxic stratus]

https://github.com/core-haxe/entities2/blob/main/tests/cases/refs/entities/MainObject.hx

but realistically it doesn't actually matter anymore because of the answer above

[velvet lodge]

you should probably play with it outside of anything, just standalone if you are having trouble grepping it... @uncut phoenix i think has some code that uses it

[toxic stratus]

i am playing with it alone 😂

[velvet lodge]

public var arrayObjectA1:Array<SubObjectA>;

[toxic stratus]

if there's a mini project out there it might be easier to understand than the test cases

for now, i'll see what happens if i just start a new mock db with e2

[velvet lodge]

yeah, the UTs are probably a little over complex and badly named for it to be really useful (since for the most part they specifically testing something), like the ref one you are looking about is all about circular references and refs to refs to refs, etc

[toxic stratus]

    var db:IDatabase;
    function new() {
        db = DatabaseFactory.instance.createDatabase(DatabaseFactory.MYSQL, {
            database: 'entities',
            host: 'db',
            user: '',
            pass: ''
        });

        db.setProperty('autoReconnect', true);
        db.setProperty('autoReconnectInterval', 5000);
        db.setProperty('replayQueriesOnReconnection', true);

        db.connect().then((res) -> {
            if (res.data) {
                trace('connected');
                this.connected();
            }
        }, (err) -> trace(err));

        var timer = new Timer(500);
        timer.run = this.loop;
    }

    function connected() {
        var order = new Order();
        order.oid = '1';
        order.exchange = 'Bybit';
        order.symbol = 'BTCUSDT';
        order.side = 'Buy';
        order.price = 10;
        order.quantity = 0.5;
        order.add();
    }

I have this little bit of code, but i'm getting a stack trace

called from order.add()

anything obviously silly here?

[velvet lodge]

couple of things:

1. i dont think you need to connect
2. you have to actually specify the database on the entity manager

[toxic stratus]

ahh okay

[velvet lodge]

EntitiesManager.instance.database = ... // any db-core db

var myEntity = new MyEntity();
myEntity.someString = "bob";
myEntity.add().then(...);

[toxic stratus]

nice

[velvet lodge]

hmmmmm

im just about to head into a meeting, can you paste the entities (as text) and ill see if i get the same

[toxic stratus]

i kind of understand how you can map up the oids now though

pretty cool

import entities.IEntity;

class Order implements IEntity {
    public var oid:String;
    public var exchange:String;
    public var symbol:String;
    public var side:String;
    public var price:Float;
    public var quantity:Float;
}

class Position implements IEntity {
    public var aid:String;
    public var pid:String;
    public var exchange:String;
    public var symbol:String;
    public var side:String;
    public var price:Float;
    public var quantity:Float;
    public var liq:Float;
    public var open:Bool;
    public var stopId:Order;
    public var takeId:Order;
    public var oids:Array<Order>;
}

Will it be possible to specify a field type? like, i'm assuming Float will map to Double but I want it to be a Decimal in mysql

[velvet lodge]

nope, and it wouldnt make sense to... its not guaranteed to be a mysql (or whatever) db

something like @:size could be reused for it... but currently that only applies to strings

[toxic stratus]

awkward, can i prevent e2 from overwriting the field type then?

[velvet lodge]

also no, you dont really deal with the db anymore... i mean, it might not change the type of the field if its already there, but if that works, its more a bug than a feature

whats the diff between double and decimal? Precision?

[toxic stratus]

decimal is used for currency values

double introduces float errors

[velvet lodge]

i feel like correct "entities" way to handle that would be to introduce a "Currency" type, like there is a "Timestamp" type

[toxic stratus]

yeah, that does seem more correct

[velvet lodge]

i can repro this... odd, will be something simple

"order" needs to be escaped 👌 - interesting its never come up before

[toxic stratus]

the table name? lol

[velvet lodge]

yup

[toxic stratus]

that seems like something the tests should pick up

do they not make any tables?

[velvet lodge]

they do, but obviously never used reserved keywords like "order"

[toxic stratus]

ah, it's a reserved word?

never knew that

[velvet lodge]

ORDER BY

[toxic stratus]

aaaahhh

[velvet lodge]

fixed

[toxic stratus]

wow the issue was in db-mysql

yeah so weird that never came up

actually... i never created a table with dbcore so i guess that makes sense

cool lil find

okay now i'm having a compile time error

    function connected() {
        var order = new Order();
        order.oid = '1';
        order.exchange = 'Bybit';
        //order.symbol = 'BTCUSDT';
        order.side = 'Buy';
        order.price = 10;
        order.quantity = 0.5;
        var pos = new Position();
        pos.oids.push(order);
        pos.add().then(null, (err) -> trace(err));

        order.add().then((_) -> {

        }, (err) -> trace(err));
    }

ERROR C:/HaxeToolkit/haxe/lib/entities2/git/src/entities/macros/EntityBuilder.hx:1103: characters 64-90

1103 | promiseList.push($p{classDef}.deleteById.bind(id));
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
| (?fieldSet : Null<entities.EntityFieldSet>) -> promises.Promise<Bool> should be () -> promises.Promise<Any>
| For function argument 'x'

        | For function argument 'onFulfilled'

position is:

class Position implements IEntity {
    public var aid:String;
    public var pid:String;
    public var exchange:String;
    public var symbol:String;
    public var side:String;
    public var price:Float;
    public var quantity:Float;
    public var liq:Float;
    public var open:Int;
    public var stopId:Order;
    public var takeId:Order;
    public var oids:Array<Order>;
}

[velvet lodge]

runtime error you mean? You dont need to add() pos, pos.oids.push(...) is enough, e2 will take care of adding... also, you are using promises badly

sorry, wrong way round, you dont need to add() order

[toxic stratus]

no compile time error

[velvet lodge]

i dont think null is valid for a promise callback... ... ?

error is super odd though

this works fine for me, no compile / runtime errors:

        var order = new Order();
        order.oid = '1';
        order.exchange = 'Bybit';
        //order.symbol = 'BTCUSDT';
        order.side = 'Buy';
        order.price = 10;
        order.quantity = 0.5;
        var pos = new Position();
        pos.oids = [];
        pos.oids.push(order);
        pos.add().then((_) -> {

        }, (err) -> trace(err));

[toxic stratus]

ahh is the error I forgot to instantiate the array?

[velvet lodge]

doubt it

[toxic stratus]

copy pasted your exact code

maybe this is a haxe 5 error....

yeah, that's it

seems like there's probably a type hint missing somewhere in the chain and it trips up on hx5

4.3.7 works fine

ahh yes it is a problem in e2 i think

just haxe 4 doesn't notice it

var promiseList:Array<() -> promises.Promise<Any>> = [];
that's how promiseList is defined

        var deleteJoinDataFn = entityClass.addStaticFunction("deleteById", [
            {name: "id", type: macro: Null<Int>},
            {name: "fieldSet", type: macro: entities.EntityFieldSet, value: macro null}
        ], macro: promises.Promise<Bool>, [APrivate]);

that's the type of deleteById

is the issue here that Promise<Bool> isn't unifying with Promise<Any>? 🤔

[toxic stratus]

okay, i've got it somewhat working

its very magical how all of a sudden i have tables

makes me cautious 😄

ahhhahh

an entity is a table so queries are all done via Table.find() etc

so as long as i mock up the entity properly, everything should work fine. Wonder how it will play with 2 unique columns

[velvet lodge]

You mean Entity.find?

like Order.find(...)

[toxic stratus]

yyeah

i was trying to figure out how do i run a query lol

[velvet lodge]

yeah, so you can use Entity.find / Entity.findAll which take queryexpr params

but instead of returning records, they'll return populated entities (including the sub entities, arrays, etc)

[toxic stratus]

that's quite nice, does remove some boilerplate that i worked around

[velvet lodge]

what i tend to do also, is create static functions on the entity class like findBySomeField(someField:String) that just calls the find / findAll

eventually these static functions will also be used to create indexes on the db, which should make them super fast...

im just not sure how, yet, to create indexes in db-core in a universal way... im thinking as simple as "createIndex(fields)" is enough for any most db engines

[toxic stratus]

there's no @:unique meta is there?

or a way to mark a field as unique

[velvet lodge]

nope, meta is pretty slim on the ground currently

i dont think thats possible in core db even

yeah

enum ColumnOptions {
    PrimaryKey;
    NotNull;
    AutoIncrement;
}

[toxic stratus]

wonder if i mark it as unique in gui if it'll get overwritten 😄

i'm not sure i like how many assumptions there are for table creations

but maybe that's just beginner pains

like all my fields are marked as nullable

but i don't want it to be possible to send a null value for a lot of these

nice, if i flip a text field to varchar add unique and a length it doesn't get overwritten

[velvet lodge]

@:size(100) will change text to a varchar(100)

[toxic stratus]

i did try that but it didn't auto update the field

[velvet lodge]

yeah, its a known bug, one that ill fix soon(ish)

which also means it will nuke your manual changes

[toxic stratus]

as long as i can reapply them i dont particularly mind

[velvet lodge]

you wont be able to, fixing that bug will mean the entity schema (the classes) will always win

[toxic stratus]

then I won't be using e2 unfortunately 😄

[velvet lodge]

cool - your call - @:unique wouldnt be hard to add, but its not some thing im going to be adding "now"

[toxic stratus]

yeah, its cool

if i can't work around what's "not there yet" I can't really use it either way

[velvet lodge]

well, at the moment, it will preserve any manual modifications you make to the table... but its a bug, not a feature

[toxic stratus]

true, guess i could just not update 😄

Also, any particular reason why you default to nullable?

Wouldn't it be better to mark a field as nullable if I have var field:Null<Float>?

[velvet lodge]

you mean turning var x:Float in var x:Null<Float>?

[toxic stratus]

I mean, all of these allow NULL

but i haven't specified that anywhere in the schema

[velvet lodge]

just the default i presume, its not explicit in the table creation i dont think

surely it makes sense anyway? I mean, no value = null

[toxic stratus]

I want it to error if there's no value in some cases

like if oid is null, i want to know when its null, it shouldn't be

[velvet lodge]

i mean, @:notNull is one of the other metadatas i want to add, but me thinks you rely on db validation too much 🙂

[toxic stratus]

is that a bad thing?

[velvet lodge]

well, i defo personally prefer to not have to make a round trip to the server to find out if something is null or not

[toxic stratus]

Its fair, but, i guess its just a secondary stop point

I can review how i write things and see if there's issues there

[velvet lodge]

barring any fuck ups, imo, the apps job is to have the data is as ready a state as possible before it delivers it to the backend, and the backend validation is more of a backup

[toxic stratus]

that's how i think i'm doing it

but in this case there's no backup

[velvet lodge]

sure, but you said you want it to error in some cases, which indicated to me that you were relying on the db to tell you?

[toxic stratus]

fair, maybe i am

[velvet lodge]

dont get me wrong, im not saying @:notNull isnt useful / important to catch bad data santization on the front end...

but it doesnt, imo, remove the need for sending "good data" to the backend

[toxic stratus]

for sure

also why not just check if the type is null wrapped? wouldn't that be a more "haxey" way to handle it?

rather than have the meta

[velvet lodge]

could be an option, yeah... just have to think what that means for non nullable types... i have a feeling i turn primitives into null wrapped types anyway

[toxic stratus]

you'd have to if you want something to be nulled

[velvet lodge]

no, i mean, i think i turn Float into Null<Float> via macros

[toxic stratus]

yeah

it works with abstracts as well which is nice

doesn't seem to trip up on that

i really like how the relationships are handled

i don't actually have to have an aid column in every table (was avoiding relationships)

in code I just make a simple account:Account field instead

e2 does actually reduce a good chunk of code that i've already written, just as i work through things

the question becomes, is there an e2 migration service if I wish to use it? 😄

[toxic stratus]

    message : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'order
ADD `account` INT' at line 1, 
    call : addColumn
}```

probably another escape issue i guess

https://github.com/core-haxe/db-mysql/pull/6

probably another silly question

var user = new User();
user.name = "Test Main";

var account = new Account();
account.user = user;
account.name = "Test";
account.add().then(null, (err) -> trace(err));
//--------------
class Account implements IEntity {
    public var user:User;
    public var name:String;

    public var altname:String;
    public var exchange:Exchange;
    public var key:String;
    public var secret:String;
    public var passphrase:String;
}

class User implements IEntity {
    public var name:String;
}

No user table is created and no account table is created, and no errors

...maybe it's the change i just did 🤔

unlikely, just double checked it

[velvet lodge]

so, entities tables are created on demand as they are used, if you dont use an entity, it wont create tables, data, etc. However, if you want to force creation of all entities you can use EntitiesManager.instance.initAllEntities (its promise based, ofc, so you'll need to make sure you use it properly)

[toxic stratus]

aren't i using an entity in the first few lines though?

[velvet lodge]

oh yeah, you are, missed that

not sure then, null as a callback still looks really weird to me

[toxic stratus]

will try with a value

[velvet lodge]

maybe that is screwing with something?

[toxic stratus]

same deal

[velvet lodge]

not sure, defo does work, so something weird you must be doing... is your app exiting before its had a chance to add them or something? Sure you are looking in the right db, etc

[toxic stratus]

app is still running and that's all the code besides db creation

weird....

i defo refreshed it 10 times

but just refreshed it again and it's there now :/

[velvet lodge]

its not a local db, correct? Maybe the provider had a slow down or something

[toxic stratus]

not local, yeah

well, my bad!

the relationship stuff is so damn cool man

i don't know how you figure this stuff out

i think i want to use it, but its a completely different workflow to what i have

its gonna be some work switching it out

[velvet lodge]

yeah, i quite like it all 😊

it certainly stops me from writing the same ORM for each project over and over again, and invariably, every project needs some type of ORM

[toxic stratus]

i'm just mentally, and roughly going through it

and i think i can drop like 2 or 3 tables, a bunch of code querying everything because I assume with this

if i do Orders.find(query($account == 6)) everything that returns is gonna be auto mapped

i don't need to run any additional queries looking for related accounts etc

still not completely clear tbh, but, i'm sure its gonna work well

[velvet lodge]

exactly, yeah, you dont need to think about tables or anything anymore, the entities are the schema

[toxic stratus]

gotta let go of that, my brain is still in query mode

[velvet lodge]

yeah, i guess it takes some time to get out of "SQL Mode", but on the plus side it means you dont have sql queries littered around your app, which is always a pain to manage

[toxic stratus]

i'm actually really looking forward to that part

it was defo on my list to go on a big query hunt and start refactoring stuff

granted my setup isn't too bad for that because of the ecs nature

[velvet lodge]

like, you change a field name... oh, now i have to go through a bunch of SQL queries and update... that kinda thing just evaporates

[toxic stratus]

I don't think the way that I interface with things is going to change that much, the thing that's gonna change the most is data is going to be more accessible with less work

[velvet lodge]

i suppose you dont need your DB.query class (or whatever it was)

[toxic stratus]

will have to see, first step first is start to port some data and get a more applicable glimpse of how things look

I have logging stuff in there as well

was about to add a column in heidi.... nope, i'm meant to add it to the class

[velvet lodge]

heh heh 😄

[toxic stratus]

public var active:Int = 0; syntax idea... registers as a default value for the field

[velvet lodge]

... it would happen anyway though wouldnt it?

why would it need to be a default on the db?

[toxic stratus]

i don't know man, i've been running two typers

maybe completely unnecessary work

yep, just works

[velvet lodge]

yeah, as i mentioned, the set of haxe classes is effectively the schema, so it would be perfectly doable to say "if its 1, put it as the default value in the db table creation", and it would be fine, but in practice, you are already doing that when you set it to a default value in the haxe classes, so really the default value on the db column would be kinda meaningless

with the added pain of, if you changed it to "2", db-core / entities would have to notice that and update the db schema

[toxic stratus]

yep, slow abandoning sql thinking

oh, is there a way to add a field to an entity but not have it db related?

in the case of a getter, a local state var etc

[velvet lodge]

@:ignore

[toxic stratus]

perfect 😄

expected that a setter on a field doesn't compile?

    @:ignore public var date:String;
    @:ignore public var time:String;

    public var timestamp(default, set):Timestamp;

    function set_timestamp(value) {
         date = DateTools.format(Date.fromTime(value), '%d/%m/%Y');
         time = DateTools.format(Date.fromTime(value), '%H:%M:%S');

        return timestamp = value;
    }

[velvet lodge]

whats the error?

[toxic stratus]

 ERROR  C:\HaxeToolkit\haxe\std/haxe/macro/Printer.hx:130: characters 18-20

 130 |   return switch (ct) {
     |                  ^^
     | Uncaught exception Null Access

      ->  C:\HaxeToolkit\haxe\std/haxe/macro/ComplexTypeTools.hx:40: characters 10-43

      40 |   return new Printer().printComplexType(c);
         |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
         | Called from here

      ->  C:/HaxeToolkit/haxe/lib/entities2/git/src/entities/macros/ClassBuilderTools.hx:51: characters 34-69

      51 |                 var typeString = ComplexTypeTools.toString(arg.type);
         |                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
         | Called from here

      ->  C:/HaxeToolkit/haxe/lib/entities2/git/src/entities/macros/EntityBuilder.hx:67: characters 13-60

      67 |             entityClass.substitutePrimaryKeysInQueryCalls();
         |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
         | Called from here

      ->  src/Order.hx:3: character 1

       3 | class Order implements IEntity {
         | ^
         | Called from here

 ERROR  src/Main.hx:43: characters 16-27

  43 |    var order = new Order();
     |                ^^^^^^^^^^^
     | Order does not have a constructor

         | For function argument 'onFulfilled'

[velvet lodge]

probably not handling something, like the default... isnt that a kinda weird way round to handle that? Wouldnt it be more correct to have getters that return the date / time parts of the timestamp

[toxic stratus]

want to cache the values

[velvet lodge]

why? Because of cost?

[toxic stratus]

it'll be used in haxeui tables

cost, just doesn't seem like a good idea to have as a getter when the value only needs to be set once

[velvet lodge]

cache it in the getter then, but seems like a micro optimization if you ask me

[toxic stratus]

i went through my whole codebase hitting these things because they were significant in tracy 😄

[velvet lodge]

@:ignore public var _date:String;
public var date(get, never):String;
private function get_date():String {
    if (_date != null) return _date;
    _date = DateTools.format(Date.fromTime(timestamp), '%d/%m/%Y');
    return _date
}

feels more correct and more localised than doing it in a setter to me

although, if timestamp changes then its a problem actually

[toxic stratus]

indeed

[velvet lodge]

... fixed

[toxic stratus]

lol thanks

[toxic stratus]

huh, i'm confused

Am i reading this right, it seems like you're converting floats to decimal's, but they're coming through as doubles on the db side

maybe decimal here isn't sql decimal, but a number with a decimal place

[velvet lodge]

there is another set of mapping also, which is entity field type => core db field type

although, im guessing Decimal = Decimal there

    public function haxeTypeToDatabaseType(haxeType:ColumnType):String {
        return switch (haxeType) {
            case Number:        'INT';
            case Decimal:       'DOUBLE';
            case Boolean:       'INT';
            case Text(n):       'VARCHAR($n)';
            case Memo:          'TEXT';
            case Binary:        'BLOB';
            case Unknown:       'TEXT';
        }
    }

i still think double is the right choice here, but i think a new Currency that maps to DECIMAL might be wise

(i dont want to add a Double to the enum because its too specific)

[toxic stratus]

cool makes sense

[toxic stratus]

uurrr, i think i've ported all the data over to an entities like structure

i guess now I try to branch my code and see if I can implement it all in the actual project 😅

[toxic stratus]

Am i correct that @:cascade will basically "run down the list"

so if i delete something in my Position table, it will delete all the relationships linked to it as well?

[velvet lodge]

yeah, entity typed variables marked @:cascade (one-to-one or one-to-many) will be deleted when the containing entity gets deleted

[toxic stratus]

nice

how do you do ordered/limit queries with entities?

like a Table.GetFirst('column')

[velvet lodge]

order doesnt exist in db-core yet, so you cant... limit i think there is a function for it, but i forget... two secs...

[velvet lodge]

i have no idea what this is supposed to be doing

[toxic stratus]

ah yes, i do a raw query in my previous setup

[velvet lodge]

raw still works... you can use Query.raw(...) in entities

[toxic stratus]

oh interesting

[velvet lodge]

findByPage is i think how you would create a LIMIT query

[toxic stratus]

I might see if i can reconfigure my current setup to see if I can keep it a bit more localised

[velvet lodge]

i would just sort on the client side with array.sort until i add proper db ORDER BY

[toxic stratus]

seems like a decent option as well, don't know why i didn't

wait, i know why

[velvet lodge]

order by + limit

[toxic stratus]

when you want 1 of something

there's nothing to sort on the client :D

[velvet lodge]

yeah, order by + limit, you'll have to use raw for now

[toxic stratus]

no worries, i'm happy with that

[velvet lodge]

im not 😄

[toxic stratus]

😄

[velvet lodge]

theres a couple of place in some apps ive written where ive used it, but its "ok" (useful to have the option) but its certainly not ideal

means you have to use / know the table structure, which is NOT ideal at all...

[toxic stratus]

i just like we're not blocked

and figuring out a clean api for adding order somewhere seems a bit of a nuisance in itself 😄

[velvet lodge]

defo

[toxic stratus]

enums are nice

[velvet lodge]

its more about where to put it

would it be a parameter on every function call in db-core?

[toxic stratus]

I just do GetFirstRecord(table, where, value, orderby) and no need to think about it

but it's a private api

[toxic stratus]

yeah.... seems awkward

firestore did have a query chaining kind of setup

maybe applicable here

query($id == 4).order(ASC).then(...)

[velvet lodge]

a dsl would be built ontop of this, you'd still need to be able to use the lib "as is" (especically in e2 macros)

query($id == 4) is a queryexpr though... i think, if anything, it would be query($id == 4, [fields], DESC).. but even that doesnt really fit into the query expr (at all)

[toxic stratus]

then it's not a param and an easy/intuitive opt in

yeah, that looks odd

[velvet lodge]

its even about the look of it, its that query(...) builds a query ast

so not sure how that would be represented in the ast... metadata maybe, or even a sort of QueryExprSort or something, either option is messy

[toxic stratus]

ah yes, we're back in the abstract complication land

[velvet lodge]

yeah

[toxic stratus]

maybe a different backend would help make things a bit clearer

with the figuring out what's the ideal abstraction

and now that i say that, it probably explains why there's so many haxeui backends lol

[toxic stratus]

I'm getting a bunch of these errors

exchange.Exchange is:

package exchange;

enum abstract Exchange(String) from String to String {
    var Binance;
    var Bybit;
    var Coinbase;
}```

and they're all coming through on the class

any ideas on where to look?

I think there's an error forwarding issue in the entity macro somewhere

[toxic stratus]

it's weird because it did work in an isolated project so I don't really think the issue is with the abstract

[toxic stratus]

yeah, issue is somewhere else. Changing it to a string gets me Order.hx:8: character 1 : String has no field data

[velvet lodge]

changing what to a string?

[toxic stratus]

sorry realised I never sent the code

the line it's complaining about is var exchange:Exchange;

which is the abstract above, but I'm not sure it's actually relevant

[velvet lodge]

can you paste (as files) the Position.hx, Order.hx, Balance.hx and PositionHistory.hx?

could be some other macro in your project (ceramic / state thingy) interfering perhaps?

[toxic stratus]

Could be, but i've never gotten the error prior to e2

Okay, so I just went for a walk and got back and I think i've got a pretty big hint

it is something to do with the variable name

var exchange:Exchange

If i change the var name to test the error goes away

[velvet lodge]

feels very odd, exchange isnt a reserved word in anything afaik

[toxic stratus]

maybe there's an incomplete type path somewhere?

the error exchange.Exchange has no field data

looks like the package, package exchange.Exchange

[velvet lodge]

you couldnt repro this outside your project though, correct?

[toxic stratus]

i haven't tried, i've tried to figure out what's the link between everything

the package in those files is exchange.data

[velvet lodge]

be interesting to see if it happens in your "e2 only" test app

[toxic stratus]

I can try, i think it's a decent hint

[velvet lodge]

ah, ok, i think i can understand it though... the variable name is exchange, the package is exchange

[toxic stratus]

yeah

[velvet lodge]

sounds like enough to try and repo this end

yeah, i can repro

[toxic stratus]

nice, just reprod it as well

[velvet lodge]

... gonna be fun to track down, i get a no stack trace

[toxic stratus]

yeah, tried to get the debugger to trip up in the macro but no dice

i have no idea on where it does things 😂

[velvet lodge]

im not sure there is a way round it

it basically creates code like:

exchange.data.Something.DoSomething

but you have a member var called exchange

i think you would have a similar issue if you tried to it in user code

i think i can minimize the collision, but i dont think its fully removable

                        var joinTableName = entityField.joinTableName();
                        var joinForeignKey = entityField.joinForeignKey();
                        var classDef = entityField.toClassDefExpr();
                        trace(classDef);
                        trace(entityDefinition.className.split("."));
                        var temp1 = classDef.copy();
                        var temp2 = entityDefinition.className.split(".");
                        temp1.pop();
                        temp2.pop();
                        if (temp1.join(".") == temp2.join(".")) {
                            trace(">>>>>>>>>>>>>>>>>>> they are the same");
                            classDef = [classDef.pop()];
                        }

this can minimize it, but you'll still get the issue of you try to use an "exchange" var in an entity that isnt in the same package

problem is that you cant create imports in macros, so you have to use fully qualified names

[toxic stratus]

what's going on?

it thinks the package exchange is the same as the variable exchange?

[velvet lodge]

the issue is that, in macros you have to use fully qualified paths (you cant add imports), so, the macro is creating code like:

exchange.data.Order.findById(...)

but exhcnage is a membmer var, so it thinks you mean the member var

[toxic stratus]

ohhh

[velvet lodge]

i can minimize it by checking the packages, and if they are they same, doing:

Order.findById(...)

which would work for your case, but the issue is technically still there, and im not sure its solvable

i think the saving grace is that usually you have a single package of entities, so it should be fine for like 99.9% of cases

[toxic stratus]

yeah, seems like it's a shadow collision. Maybe rudy would know a way around it

[velvet lodge]

i dont think there is a way round it, its a language "thing" (that makes perfect sense really)

you just could work around it in user code by doing an "import exchange.data.Order"

[toxic stratus]

isn't that just normal?

[velvet lodge]

wdym?

you cant add imports in macros

[toxic stratus]

!!

oh, nvm, yeah i don't understand the solution for my side

[velvet lodge]

i think the solution is the one im proposing, if the packages are the same, then just use "Order" not "package.Order"

it minimizes the issue, but i suspect, 99% of the time, entities are all in one package anyway

so, to trigger the issue you'd have to a) have entities in two different packakges AND b) use a variable name in one of them that is the first part of the package of another

[toxic stratus]

there are 2 packages here that trip up and knowing the issue i'm confused by it

[velvet lodge]

which seems pretty minimal

[toxic stratus]

my exchange library entities and the projects db.HyroAccount

[velvet lodge]

do you understand the issue? Im not sure you are following what i mean

[toxic stratus]

yeah, i do now

was just a bit slow

[velvet lodge]

i think ill add in the "mitigation"... and i think it will be "fine" for most use cases, you'll have to kinda try to break it (or have a bit of a weird setup) to trigger it

[toxic stratus]

sounds good

much thanks!

[velvet lodge]

fixed

[toxic stratus]

😅

just this one left for some reason

[velvet lodge]

ok, so yeah, this is the edge case... totally different package

[toxic stratus]

what's the way to work around it at a user level

[velvet lodge]

dont call the var exchange, or move the HydroAccount into the exchange package

[toxic stratus]

i can live with platform

and theoretically e2 will just handle the refactor for me 😄

[velvet lodge]

until you use platform as a package also 😦

[toxic stratus]

😅

If it makes you feel any better I've just realised that because e2 handles relationships at the point of the query

I actually don't need every table to have the exchange column 🤣

I only need the column on the account and the rest can just become getters

although, i find it curious why the ignore isn't working here

[toxic stratus]

Account.findAll(query($user == 5)).then((accounts) -> {});
also another weird thing. This query is returning 0 results

but there are defo matches

[velvet lodge]

hmmm

[velvet lodge]

its still a varaible in the class... so it will still mess things up

[toxic stratus]

ahhh I thought it was just because of the macro parsing it

[velvet lodge]

nope, the issue is the variable is called exchange and there is code that is generated like exchange.data.Something.Else(...)

@:ignoring the var doesnt change that conflict

[velvet lodge]

this im not sure about, but for sure this works as its the most basic thing

can you trying something really quick though, in EntityBuilder.hx there is entityClass.substitutePrimaryKeysInQueryCalls()... can you comment that out just to sanity check something

[toxic stratus]

still 0 results

hmm... actually maybe initialisation order has changed due to using entities

Db.getRecords('account', query($user == 5), (records:RecordSet) -> {
    trace('Direct: ' + records.length);
});
Account.findAll(query($user == '5')).then((accounts) -> {
    trace('Entities: ' + accounts.length);
});

It gets weirder... Both of these return 0 results

db is defo connected

!!

[velvet lodge]

im guessing the second shouldnt be a string?

[toxic stratus]

was just testing

figured out the issue

dbs.init("tracker", "", "", "");

was still using old db name x)

[velvet lodge]

right

that'll explain it

[toxic stratus]

entities created the table on the og db

so it never actually errored

silly oversight x)

[toxic stratus]

public var entryTimestamp(default, set):Float; {}

this field is being ignored

probably due to the setter

[velvet lodge]

yeah, i think fields that are getters / setters are explicitly ignored because they arent really "schema like"

like, there is behaviour associated with them, so the assumption is made to ignore them

[toxic stratus]

okay then

i've been marking all my getters/setters with ignore 😂

[velvet lodge]

it might not be the case, but i think so

[toxic stratus]

i swear they were going through at some point

[velvet lodge]

i could be totally wrong

it might be that im not handling (default, set)

[toxic stratus]

so could i

yeah, that was a recent addition so maybe just an oversight

you know, i think the auto overwrite tables thing is quite dangerous

[velvet lodge]

defo can be

can lose data, for sure if you change a field name

[toxic stratus]

I just got a little scare because I just realised that the misnamed db issue from earlier could have erased data in my original db by sheer coincidence

[velvet lodge]

... with great power... ... .... ....

[toxic stratus]

😄

[noble scaffold]

comes greater power

[velvet lodge]

realisitically, your local app should in NO WAY be connected to your prod server, and you should be using a local version, so on your head be it 😄

[noble scaffold]

Oh by the way Ian totally related to programming.

I am doing a playthrough of factorio and trying to launch the rocket.

[toxic stratus]

for sure it would have been my mistake but what a massive mistake that would have been 😅

[noble scaffold]

I got the space age dlc, I wanna see how crazy it gets :,)

[toxic stratus]

I am going to immediately backup my og db now that i've been reminded how important the data is x)

[velvet lodge]

ive also got the dlc, tried to play a few times, but never gotten even close to getting off the planet... keep giving up after a few hours

[noble scaffold]

What roadblock are you hitting?

[velvet lodge]

i mean, it would be like running a bunch of sql commands on your prod server, regardless of entities, its not a good move

[velvet lodge]

time

[noble scaffold]

Are you starting fresh each time?

[velvet lodge]

im going to start again though, this time going to go huge

[velvet lodge]

yeah

i get to my normal "right, trains trains everywhere" and then get all "fuck this, i cant be botherered"

[noble scaffold]

Well that's the problem, the starting phase is so tedious and slow

[noble scaffold]

nah that's where you have to double down

you gotta make ugly complex systems it's part of the game

[velvet lodge]

defo, thats the part you have to push through

i just got totally bored of laying tracks for 8 hours when i defo should be doing other things

[noble scaffold]

What I've been doing to get past it, is I've been building factories in horozontal focusing on something I want

so I don't have one mega complicated spagetti monster

I only lay tracks to move resources

[velvet lodge]

i defo have it all planned out

[noble scaffold]

damn 😅 that's so overkill

[velvet lodge]

the distances are HUGE 😄

[noble scaffold]

Yeah but still you have 4 drop off points in the middle of nowhere

[velvet lodge]

well, thats the begginning of the main factory the other one above is just my starter factory

[noble scaffold]

I get what you're going for, but to me it seems you are scaling way to fast.

Did you have blue science by that point?

[velvet lodge]

naw, i dont think so, ill get blue conveyers in my starter soon

[noble scaffold]

It's so overkill 😂

[velvet lodge]

i dont start building the "main factory" until i have blue conveyors because its soo much effort to replace red -> blue

the point, to me at least, is defo overkill... even worse is how i build all my production lines as loops as to not waste ANY resources (ie, they feed back into themselves)... totally over complicates everything for little gain

and every time i restart im like "right, no loops this time"... and then i start building the loops

one of the simpler examples, couldnt find an older save game

also, you say overkill, but have you seen the insane amount of resources needed for the late game stuff, its crazy

[noble scaffold]

yeah but why not scale after you hit the point when you actually need it

this is just insanity

[velvet lodge]

i can have a bank of 64 factories working flat out producing copper wire and it still not enough

[velvet lodge]

well, i dont build it all at once, its modular for sure

[noble scaffold]

what are you producing o.O

[velvet lodge]

blue conveyors require 15 blue circuits

[velvet lodge]

awful, i hate it 😆

[noble scaffold]

But but but WHY I mean I know it's faster and better, but the resource sink is just insane.

Were you doing oil yet?

[velvet lodge]

yeah, i get oil as fast as possible

not sure about this play through, but generally as soon as i can i do

even if its a little hodge podge to start with

[noble scaffold]

The research labs setup is smart

I had no idea you could do that 😅

[velvet lodge]

trust me (bro), your setup will bite you in the end... everything will crawl to a halt 😉

[noble scaffold]

I'll just make another one! 😅

[velvet lodge]

yeah, thats what im doing... well, was doing... big(ish) start base, but its all just there for the "final base" which is the one that will just max out everything so i dont have spend years removing shit at upgrading it...

[toxic stratus]

is that ingame years lol

[velvet lodge]

defo works, and eventually its 100% needed

[toxic stratus]

(no idea on factorio)

[velvet lodge]

feels like literal years when you are expanding things and go "oh, shit, this wont fit" and then have to spagettify everything

px doesnt seem to care though looking at his train setup, so more power to him 😆

[noble scaffold]

Yeah I gave up caring about grand setups

I am using the train there to just move around

My current save there I want to build a nuclear reactor

[velvet lodge]

well, you've inspired me to start again, check back in a few months and lets compare notes and see your frustration and my inner zen / peace 😄

[noble scaffold]

so I laid a bunch of tracks to find a uranium deposit

[noble scaffold]

Okay deal

I would be suprised if you hit inner zen, but I'm also more curious if one or both of us will launch the rocket.

[velvet lodge]

we'll both launch one (probably), but for you it will be like pulling teeth, for me it will just be "yeah, its all there, throughput is x 1000 of items per second"... 😄

you'll be gray, and ill still be my youthful self 🤣

[noble scaffold]

I don't KNOW, I don't have the same impulse of needing to go through the early stages and grinding again.

I think that's where the youthful energy goes to die!

[velvet lodge]

everything. must. be. perfect.

but also, it really helps the late game imo

[noble scaffold]

Also this is where I will build my end base because it's so pretty

[velvet lodge]

i hate you, im considering discord blocking you 😆

[noble scaffold]

I have struck a cord I didn't even know existed, 😅

[velvet lodge]

yeah, its new to me too, im going to lose sleep over your factory setup 😄

[noble scaffold]

I think you will lose your mind if I build a giant spaghetti monster and launch the rocket and I show the picture of the end base and it's so incredibly hideous and awful. You ask me oh how many teeth did you pull and I just reply, oh it was a breeze, super simple

I think there's a high probability especially given the drones you get at end game

I could build total nonsense and still get it to work.

Oh no did I get blocked... 😅

[toxic stratus]

Ian has probably started a game

@:isVar public var testSetter(get, set):Timestamp;

function set_testSetter(value) {
    date = DateTools.format(Date.fromTime(value), '%d/%m/%Y');
    time = DateTools.format(Date.fromTime(value), '%H:%M:%S');

    return testSetter = value;
}

function get_testSetter() {
    return testSetter;
}

Just checked this out and its also ignored so I guess we can't have setters on fields

[velvet lodge]

haha, no, was eating 😄

[velvet lodge]

i think properties in general

[toxic stratus]

was browsing through the code and found some thoughts

not sure if its about this kind of thing though

just seems related to properties

[velvet lodge]

i think its better, in a way, to think of entities as typedefs...

i mean, from a data pov

i wouldnt want getters / setters to be in the db, nor would i want to decorate them all with @:ignore... they are behaviour type things, not "just data"

[toxic stratus]

just seemed like a good place to init some variables

it isn't actually affecting the db data, but its preparing the data for the client

[velvet lodge]

maybe the isVar could be a modifier there, like, if its isVar it will end up in the db

[toxic stratus]

the alternative here is I basically have to add getters/setters to each of these props

@:ignore public var entryObj:Date;
@:ignore public var exitObj:Date;
@:ignore public var entryDate:String;
@:ignore public var entryTime:String;
@:ignore public var exitDate:String;
@:ignore public var exitTime:String;

which is equally meh

[velvet lodge]

thats because you are insisting on caching all this stuff...

one thing i think i am going to add is some type of overridable function: onBeforeLoad and onAfterLoad (or something)

then the entryObj could be cached in a getter (which feels right to me) and the onAfterLoad would null it (or reset the cache basically)

eg:

[toxic stratus]

bro it was necessary

[velvet lodge]

58 million iterations?

[toxic stratus]

yaaa

33seconds running time

[velvet lodge]

58 million iterations

[toxic stratus]

😂

you think i was joking

[velvet lodge]

no, i dont think you are joking, im just saying that 58 million iterations isnt realistic, if it were in a hot loop, maybe, but it shouldnt be

anyway, i have no fundemental issue with the cache, i just think its in the wrong place

it should be in the getter, the problem there is if timestamp changes, your cache is wrong / old value

[toxic stratus]

there's a lot of places where i use date stuff in my app so it all compounded

[velvet lodge]

58 million times? 👀

[toxic stratus]

😂

[velvet lodge]

5800 times would be excessive

[toxic stratus]

surprised me as well

i was just doing Date.now() etc every where

i saw that figure and was like NOPE

tracy very good x)

[velvet lodge]

i still dont really get how you can do a date.now 58 million times in 33s tbh... but, fine

[toxic stratus]

it defo aint that high now, i should probably check at some point tho

still in e2 migration mode

[velvet lodge]

yeah, but it sounds like you are still calling that function 58 million times in 33s, it just is returning the cached value...

and i dont really understand why its being called that much in the first place... indicates your app is sitting there idle calling the function in a hot loop

[toxic stratus]

hmmmm, i'm not sure if that's the case post review

i don't think the 58m figure moved to cache vars

but that was like 2 months ago now so i have little memory of what i did now x)

all i know is app ran pretty good after addressing it

it is possible

just something for down the road

[toxic stratus]

nudge: https://github.com/core-haxe/db-mysql/pull/6

Oh man, you really do see the difference in data response time with entities, will need to handle that at some point

how do you handle that in your apps?

queries are typically instant

[toxic stratus]

oh you know what I think is happening, at least on this page

For this particular example i'm querying for 30 records and its taking about 900ms to load in

Is it pulling all of the associated relationships in this query?

Is there a way to say "no need to get the relationships for this query"?

for cases like this that might be a decent solution

1 trade can consist of at least 2 orders, so that's a minimum of 90 queries going on here and there's probably more queries to look up the relationship id which if i understand right, would add another 30

[toxic stratus]

where can i trace the final built query? want to see if i can figure out what's going on

i've put traces in all of the functions in databaseconnection for nodejs but none of them have it

found it

very confused

        var now = Date.now().getTime();
        Account.find(query($accountId == 1)).then((resp) -> {
            trace('entity elapsed: ${Date.now().getTime() - now}');
        });

        var now = Date.now().getTime();
        Db.get('account', query($accountId == 1), (resp) -> {
            trace('dbcore elapsed: ${Date.now().getTime() - now}');
        });

This returns around 300ms each, so something else is adding latency, but heidi completes the query immediately so it isn't a network issue

so weird

if I switch to my old database the same dbcore query takes 36ms

hmm no further leads, but just left with more weirdness unfortunately

[velvet lodge]

i think its sub queries as you say, although 300ms seems excessive

what does the Account entity look like

in fact, can you dump all your entities in a zip?

[velvet lodge]

you can use fieldsets

[velvet lodge]

30 distinct queries?

[toxic stratus]

Nope, a singe query

        var _loadStart = Date.now().getTime();
        PositionHistory.findAll(query($entryTimestamp >= startTime && $entryTimestamp <= endTime && $account == aid)).then(function(records) {
            var _loadEnd = Date.now().getTime();
            trace('Elapsed: ${_loadEnd - _loadStart} length: ${records.length}');

[velvet lodge]

yeah, ive been running some tests locally, and it seems network latency and query count is problematic

[toxic stratus]

it can't be latency though because heidi returns instantly and the old db returns instantly

[velvet lodge]

still involves a network though, its not local connections... and even if it runs "instantly" it still has to deliver the result to you... even 10ms adds up here

[toxic stratus]

i did try to compare both dbs last night and couldn't find any differences

[velvet lodge]

under the hood though, entities is doing more querying

[toxic stratus]

I'd like to run the entire query stack for the page but i'm getting new weird and fun error this morning lol

[velvet lodge]

in my example, from one of the unit tests, i have an entity, granted its super nested and has a BUNCH of sub entities and sub-sub entities, etc (so lots of queries)... when i run the test against a local db, it takes 11ms... ... when i run it against a remote db its a whopping 2000ms

[toxic stratus]

it seems really odd tho, no? my old db was running on the exact same server and 2seconds for a small example seems crazy

[velvet lodge]

again, its network latency, it adds up fast...

each query is a round trip to the server

even 5-10ms here is very significant

[toxic stratus]

yeah, 30*10*relationships

SELECT * FROM positionhistory_orders
WHERE (positionhistory_orders.positionHistoryId = ?);`
Is it possible to output the queries with the values?

[toxic stratus]

i noticed the option but didn't know how to use it

[velvet lodge]

they can / will help in the short term

at the top of the entity, do something like:

@:fieldset(NoSomeField, [!someField])

and then when you call find, use that fieldset

SomeEntity.find(..., SomeEntity.NoSomeField).then(...)

they were originally intended to be used to not query certain data for security reasons, but they "accidently" double as a way of reducing queries

but the fundemental issue remains... ive never hit this because i wouldnt in a million years allow my client apps to make direct queries to a db, and everything goes through an api, which lives on the same network (effectively) as the db... but regardless, its defo an issue, and the diff between 11ms and 2000ms is... ... ... shocking

[toxic stratus]

yeah, that's the plan, it just seemed odd. Like poeple actually do ship mysql in their clients and even a 100+ queries seems like it should be pretty damn quick

especially with a small dataset

my db only has no more than around 3-4000 records

[velvet lodge]

its not the data size, its the general network latency... if you had a latency of 5ms (which is tiny, its going to be bigger), then 100 queries is 500ms

eg, a single query on my non-local db is about 45ms

that drops to about 0-1 locally (and the 1s are 0.999999999999s so rounding)

[toxic stratus]

yeah, added the fieldset and now my response is 30-100ms

have noticed something else weird going on

all my haxeui page changes are occuring 3 times

[velvet lodge]

exactly, less queries = less latency, its an "ok" solution, but im a little surprised just how high it was without query tuning (especially for me locally)

[toxic stratus]

yeah the example seems quite simple, like at first glance it seems like "how does anyone else handle this"

[velvet lodge]

less queries

[toxic stratus]

i noticed no joins in the queries

maybe that would be faster?

[velvet lodge]

entities1 was join based, and it can get completely out of hand fast (query wise)

its one of the reasons e2 exists actually

and in my initial tests things looked great... i didnt (obviously) factor in network...

so i wonder if some type of hybrid approach can be used

adds a HUGE amount of complexity to already complex code though

[toxic stratus]

yeah the joins aren't easy with tons of relationships

and the coding nature of entities really encourages relationships lol

[velvet lodge]

coding nature of entities really encourages relationships
exactly

i cant remember the details exactly, but with a deeply nested entity in v1, i was getting like 30 joins and a result set of 1 million rows or something crazy

granted it was an extreme example, but still

you'll have to live with fieldsets for the time being until i can digest it properly... suffice to say, i can repro it (and worse) locally

[toxic stratus]

jesus

fieldsets would help there as well

[velvet lodge]

yeah, fieldsets didnt exist in v1 - they are pretty powerful

[toxic stratus]

no probs, fieldsets solve the bulk of the delay just an interesting problem in general to poke at

[velvet lodge]

you saying interesting, i say fucking annoying 😄

[toxic stratus]

could probably have some preset fieldsets as well

Layer 0 - No Relationships
Layer 1 - 1 Level
Layer 2 - 2 Levels
etc

[toxic stratus]

its interesting because of how seemingly simple but damaging it is 😄

[velvet lodge]

its not how fieldsets work, they exclude fields by name, not by type, so "no relationships" would have to be handled explicitly, and i dont really like / want to do that

also, you shouldnt have to use it... fieldsets were never supposed to be a "oh, queries are taking too long? Use a fieldset"

it was supposed to be "oh, leaking sensitive info to the client after serilaization? Use a fieldset"

[toxic stratus]

😆

you can use a hammer for lots of things!

[velvet lodge]

true enough...

there is valid in using them to exclude entire subobjects... but perf should have to be one of them

if you want to use it for perf, cool, it should make a fast thing faster, but 2000ms vs 11ms... is crazy

[toxic stratus]

fieldsets likely don't have the flexibility for the problem as well

like, i might want my orders but don't need the relationships on the orders

[velvet lodge]

should be able to do that with field sets

[toxic stratus]

oh really?

[velvet lodge]

iirc you can pass down fieldsets from other entities and exclude them

[toxic stratus]

Oh cool!

[velvet lodge]

so

Account.findByBy(..., Account.NoSomeField | Order.NoSomeOtherField)

something like that anyway

[toxic stratus]

neat

[velvet lodge]

again, the idea being that you wouldnt want, for example, User.passwordHash coming through, but User could be "anywhere"

[toxic stratus]

Does that actually make things more secure?

If the request is on the clientside, couldn't the client just modify the request to re-inlude the hash?

[velvet lodge]

possibly, though no easily... and anyway, it should be an api function anyway, its for things like a "findUsers" api call not returning things you dont want it to

rather than having to null results so they dont end up coming down the wire

[toxic stratus]

class UISettings<T> implements IEntity {}

Can we get type params on entities 😄

useful for general purpose + json fields

probably a complicated ask in general actually

[velvet lodge]

yeah, i think that would be very complex... and would probably only really work with @:generic... defo not something im thinking about addding. I even dropped inheritance because, in practice it felt like it was "cool" but not really very useful and added a bunch of complexity

[toxic stratus]

A simple workaround could just be that it maps an unknown type to Dynamic and just doesn't error

because project does compile but its giving a macro error

[toxic stratus]

oh nvm, anything marked as Dynamic is ignored as well 🤔

[toxic stratus]

I was browsing the internals of e2 and saw a Date type

what does that become? mysql date or timestamp?

[toxic stratus]

Would be cool if entity.table was publicly exposed

Would allow for making some static extensions that can be universal to entities but not specific to any one in particular

without requiring a table name

there goes that idea 🥲

[velvet lodge]

why would insertOrUpdate be useful, thats what add does

or, well, update

if it doesnt exist it will add it

[toxic stratus]

didn't know that, but also, it's a small example there's other extensions that will be useful

    public static function getFirst(account:AccountId) {
        return find(Query.raw('SELECT * FROM `order` WHERE `account` = \'$account\' ORDER BY `created` ASC LIMIT 1'));
    }

I'm duplicating this code in multiple places as another example

I don't necessarily need to set succ/err on promises everywhere and can just handle that in a single place

[velvet lodge]

so where is this getFirst function? I would expect it to be on the Order class

[toxic stratus]

Yeah, it's in the respective entities

[velvet lodge]

and the query is the same?

[toxic stratus]

besides the tablename yeah

[velvet lodge]

feels very "meh" to want to put that single (tiny) query in a generically typed class

[toxic stratus]

even if not a sort by field option isn't an awful idea

[toxic stratus]

it's not just about the query, it's also about other things, like this

i have to do the whole promise boilerplate with callbacks everywhere

around 40% of these, i don't need specific actions

[velvet lodge]

cant you handle the error in the getFirst function? Its like an additional 3-4 lines

[toxic stratus]

then i lose the flexibility of being able to manage the error in a place i might want to

[velvet lodge]

you mean on the thing that calls getFirst?

[toxic stratus]

yeah

i haven't gotten this part of the project to compile yet though, and i'm gonna try casting to dynamic and see if that lets me through

[velvet lodge]

then a moving it all to another class aint gonna help you, you are still going to need handle the promise error handler in the code that calls getFirst, regardless where the error originates

[toxic stratus]

It will

entity.add().then(null, (err) -> trace(err));
entity.insert(); // static extension

static function insert(entity, ?succ, ?err) {
  entity.add().then(successHandler.bind(succ), errorHandler.bind(err));
}

the promise is already handled and its handled a lot better than a lazy trace

[velvet lodge]

this seems immensibly over complicated, and you run the risk of using entities before they are "done", ie, you arent really using promises correct here at all

[toxic stratus]

how so

i've never ran into an issue with this pattern on top of promises

[velvet lodge]

entity.insert(); // static extension
trace(entity.primaryKey); // may have written to the db, may not have, insert should be promise based and you should use then, its an async function

[toxic stratus]

but if i want entity.primaryKey I pass a succ function to insert

entity.insert((succ) -> {
  //normal callabck
});

[velvet lodge]

ok, then why not just uses promises? You are creating (potential) callback hell over the top of a promises??

[toxic stratus]

because I can handle a bulk of my use cases in a better way

Proper logging without the additional work

[velvet lodge]

seems completely mental, but, ok 🙂

[toxic stratus]

its defo easier

maybe i didn't make it clear

[velvet lodge]

entity.insert((succ) -> {
});

vs

entity.insert().then((succ) -> {
});

and, now do it where you want to add five entities one after another

entity.insert((succ) -> {
  entity.insert((succ) -> {
    entity.insert((succ) -> {
      entity.insert((succ) -> {
        entity.insert((succ) -> {
        });
      });
    });
  });
});

?

[toxic stratus]

it's not necessarily for all circumstances

yes in that situation its stupid

[velvet lodge]

just feels like a bad pattern in general, and i dont even see the benefit, except having to type "then", but now you cant chain promises, or use PromiseUtils.runAll / runSequential...

[toxic stratus]

but there's a lot of situations where you just want to do entity.add()

and because you're in a flow you end up just doing entity.add.then(null, (err) -> trace(err)) which is a pretty lazy error handler

[velvet lodge]

fire and fogetting something like a database add feels dangerous, unless its like logging or something, and even then, id like to know it was written (or not)

which is a pretty lazy error handler
better than no error handler?

[toxic stratus]

mine has error handling by default

[velvet lodge]

ok, so you are handling promise errors globaly

[toxic stratus]

and success handling by default

so you can get a nice log (If you want) that something completed or errored without manually adding it everywhere

it does lead to a better experience in general

code is less messy and more accurate

[velvet lodge]

still feels like a total anti pattern and misuse of promises, but well, its not my app 🙂

[toxic stratus]

hmm, i haven't had a problem with it tho

the worst problem i had when i first used it was line numbers/call location

but that's solvable with PosInfos

[velvet lodge]

i think you are sidestepping the async nature of promises for the sort term benefit of not explicitly handling errors

entity.add.then(null, GlobalErrorHandler.instance)

is how i would do it by NOT using callbacks and having this little function that is basically telling me "use promises like they were sync"

there is a tempation to do something like:

entity.field = 0;
entity.add();
entity.field = 100;
entity.update();

and this is totally wrong... and the result would be "who knows", depends on a few things

[toxic stratus]

I can see that occurring for a public lib but i've never personally done that

[velvet lodge]

you never need to add an entity and then update it?

or worse, have a process that is, at different point in time, updating some entity, like a timer or something?

[toxic stratus]

well previously i had my insertOrUpdate command 😆

and you said that update does that

[velvet lodge]

it does, i think add might also update if it already exists all, cant remember

actually, thats not true, add will always add a new entity

but update will add if it doesnt have a PK, update if it does

[toxic stratus]

this is probable but how does using a promise the normal way avoid this? they would both be subject to the same issue here

updating, adding is defo cool to know

[velvet lodge]

yeah, not a great example tbh, but i more meant that the nature of the promises "instructs" you to think of it as an async operation, imo, your helper hides that (with the added "benefit" of putting you in callback hell if you want to make, say 3, updates in a row)

but as i said, its not my app, so you do you 🙂

[toxic stratus]

it would be used in context specific situations, not all

there are a lot of times where all you want to do is a simple instruction without the nesting

its just cutting out some boilerplate

[velvet lodge]

makes it worse, again, imo: "heres a util function that helps, dont use it all the time though, use proper promises the rest of the time"

where as im of the opinion: "use proper promises, all the time, its consistent"

something that has always annoyed me about haxe, just tried to add this function:

    function doSomething<T:IEntity>(instance:IEntity) {
        return instance.add();
    }

ofc, .add doesnt exist on the IEntity (since im just using it for macro building), so i thought, ok, cool, ill add it (probably not a bad idea anyway), so:

@:autoBuild(entities.macros.EntityBuilder.build())
interface IEntity {
    public function add(fieldSet:EntityFieldSet):Promise<IEntity>;
}

and now:

./cases/basic/entities/BasicEntity.hx:11: character 1 : ... error: cases.basic.entities.BasicEntity should be entities.IEntity
./cases/basic/entities/BasicEntity.hx:11: character 1 : ... have: (...) -> promises.Promise<cases.basic.entities.BasicEntity>
./cases/basic/entities/BasicEntity.hx:11: character 1 : ... want: (...) -> promises.Promise<entities.IEntity>

even though class BasicEntity implements IEntity

[toxic stratus]

consistency is a fair point for sure

[toxic stratus]

i've never understood that either

[velvet lodge]

i can cast it i think in the macro function...

and probably no negative side effects... unless i lose the typing of BasicEntity in the promise result

[toxic stratus]

i think you would

[velvet lodge]

prolly :/

[toxic stratus]

so T basically can't be forwarded

[velvet lodge]

actually, i dont even think i can cast it

i would have to change the function signature of the add to return an IEntity, then i would defo lose type info

i could do "implements IEntity<TheType>" to be fair

so actually, i think the error is "alright" now im thinking a little better about

[toxic stratus]

is it possible to make the _ private?