Core Haxe

[toxic stratus]

just use the new json field type in your mysql db 😂

[velvet lodge]

heh, yeah

[toxic stratus]

test has begun now 😄

oh no

how much of entities requires metadata usage? 🤔

do you recall what was the issue with the meta.json thing i submitted a while back?

[toxic stratus]

okay, so i guess entities doesn't like constructors

    - type 'Exchange' not supported for field 'exchange', skipping
    - type 'OrderStatus' not supported for field 'status', skipping
    - type 'Side' not supported for field 'side', skipping
    - type 'OrderType' not supported for field 'type', skipping

enum abstract's aren't supported

lol sorry 😂

[velvet lodge]

yeah, enums not supported yet

are those four field types enums?

[toxic stratus]

they are

[velvet lodge]

so yeah, not supported yet im afraid 🙂

[toxic stratus]

hmm, okay. I shall branch it off and just do some passive port work and test it when they get added 😄

[velvet lodge]

i think i might store them as strings in the DB... oh actually, how complex are these enums?

can you paste the defintions of the four of them?

Not sure how you could store "MyEnum.Foo(x, y)" in the db... 🤔

[toxic stratus]

nah, they are pretty basic

i just want consistency between various potential exchange implementations

enum abstract Side(String) {
  var None;
  var Buy = 'buy';
  var Sell = 'sell';
}

enum abstract OrderStatus(String) {
    // fell through to default
    var NONE; 
    // order has been sent to server
    var PENDING;
    // order is active, waiting to be filled
    var OPEN;
    // order has been partially filled
    var PARTIALLY_FILLED;
    // order has been changed but not clear on why
    var CHANGED;
    // order is filled
    var FILLED;
    // order was rejected by server
    var REJECTED;
    // order has been cancelled by user
    var CANCELED;
}

enum abstract OrderType(String) {
    var MARKET; 
    var LIMIT;
    var STOP_LIMIT;
    var TAKE_PROFIT;
    var ERROR;
}

enum abstract Exchange(String) to String {
    var Apex;
    var Binance;
    var Coinbase;
    var KucoinFuture;
}

[velvet lodge]

so there are no params? Its literally just enum X { A, B, C}

[toxic stratus]

But it would be nice if all basic types were supported, so an float enum abstract did map to an actual float in the db

[velvet lodge]

i think this could be supported fairly easily... will have to have a think / play

[velvet lodge]

yeah, defo think that is possible... my concern is when the value changes

[toxic stratus]

but I guess that actually doesn't matter if the entity lib will just convert everything to expected working types 🤔

[velvet lodge]

yeah, thats kinda what im thinking... if i store the actual enum name, then it doesnt matter... ... i think

[toxic stratus]

oh, yeah, that's a weird macro timing question

var Sell = 'sell' something like this may not actually be valid

[velvet lodge]

like say you have:

enum abstract Side(Float) {
  var None = 0;
  var Buy = 123;
  var Sell = 456;
}

And is store "Sell" as 456 in the db... sounds like a good idea, until you change your mind and make "Sell" 789, then things go weird... if i just store "Sell" and make sure it assigns the right one, i think you avoid all that... ... i think

[toxic stratus]

yeah that makes sense

[toxic stratus]

how do we use limit?

I tried adding it to queryexpr but that doesn't seem like its correct (doesn't work)

oh wait... vsc didn't show me errors, may work

[velvet lodge]

queryexpr takes valid haxe synax, not sure how that would work with "LIMIT"... maybe ".page" will work?

https://github.com/core-haxe/db-core/blob/main/src/db/ITable.hx#L15 ?

though it might be nice for ".all" to have a "max" param... 🤔

[toxic stratus]

oh interesting, never knew of page

i mean... really... all you gotta do is just find some weird haxe quirk to make it make sense 😄

query((example == 'stuff') ("LIMIT" = 5))

behold! valid haxe syntax! 😄

[velvet lodge]

LMIIT feels very "sql"...

i feel like find can / should have a maxRecords:Null<Int> = null, or something

[toxic stratus]

ahhhh something like find(query, 10)

that makes a lot more sense, yeah

[toxic stratus]

10/10 Ian 😄

[velvet lodge]

noice

[toxic stratus]

It has been over 24 hours now and db still works

[velvet lodge]

and presumably didnt drop any queries either?

[toxic stratus]

there were no queries over the night, but, I just ran one now and its entered into the db fine, there does seem to be some missing data but I need to figure out where that's from - it could still be my app

[velvet lodge]

hmmm, right... be good to know that nothing was lost... i dont think its db-core if the data was missing and nothing was run... but... well, always good to know that its not the case

[toxic stratus]

the timing of the missing data makes it seem like it's not a db-core issue to me

I just performed an action, and it was inserted, but the data that is missing did not come in and I see no fail responses from the db side

I recently removed all traces to do with that aspect assuming it was all fine and good - dang 😄

[velvet lodge]

yeah, i really need to put logging into these libs... it becomes a pain to debug...

[toxic stratus]

speaking of which it seems like I need to start doing some logging things for my app

server/remote based debugging is a pain in general

[velvet lodge]

i end up reverting to "trace" also, which could be pretty problematic for a running system

[velvet lodge]

https://github.com/core-haxe/logging ?

[toxic stratus]

The theoretical approach I have to logging will be centered around trace

i want to do something like

trace("data", warning, obj)

[velvet lodge]

kinda similar interface: https://github.com/core-haxe/logging/blob/main/src/logging/ILogger.hx

[toxic stratus]

I need to figure out what I want just yet

because what i want will likely be my forever logging setup at this point 😂

[velvet lodge]

yeah, at some point i need to go through the core haxe libs and litter logging all over the them

some use it, some dont... all should really, especially the db type ones

[toxic stratus]

i want to be able to remotely configure logs as well

so i can just toggle some setting in a db somewhere and see traces live for that espect

pushing changes just so I can get a new trace output and then removing it is not a great workflow 😄

[velvet lodge]

yeah, i want live reload of log params... so the system can be running and i can change a file or something and the logging will reinit, ive done that in a live system im using, but its not built into the logging lib, and it should be

[toxic stratus]

I want to be able to just be a bit carefree with my traces, have them anywhere and just adjust what is useful later on

[velvet lodge]

i would personally advise against getting too fancy with log config (in general), like i wouldnt make it come from a db... file, or env var or something makes more sense

[toxic stratus]

how come?

[velvet lodge]

because, imo, you are adding overhead to what should be a fairly simple system...

[toxic stratus]

ah, i have gone overboard with this idea in the past

it can be a rabbithead

i think I have a general idea of where to cap it out

[velvet lodge]

yeah, its easy to get distracted and go "but look how cool the logs are" and then realise the system just works 100 times better with everything off 😄

[toxic stratus]

just having gotten into the weeds on it yet as i want some core features working first

[toxic stratus]

i've had this before as well 😂

[velvet lodge]

yeah, i might give the logger the once over this eve, make sure its fit for purpose

[velvet lodge]

same 😄

[toxic stratus]

but there are times where I want to have logs of stuff but don't want it traced into the console

so i want some level of flexibility

[velvet lodge]

you can use "adaptors" in core-haxe/logger... not trying to sell you it, just saying the adaptors can do "anything"

LogManager.instance.addAdaptor(new ConsoleLogAdaptor({
    levels: [LogLevel.Info, LogLevel.Error], // will show all logs by default
    formatter: new MySuperFormatter(), // defaults "DefaultFormatter"
    packages: ["foo.bar"] // will show all packages by default
}));

(and you can use any number of adapators ofc)

[toxic stratus]

yeah, but, my main issue at this point is i haven't gone down to identify what i need at a "core" level yet

so I'm not sure how to use a logmanager at all with limited context

I just mostly have some idea/picture in my head

[velvet lodge]

im not 100% sure i follow... you mean you dont know what you want to capture data-wise?

[toxic stratus]

I know I want "things", but I haven't isolated what "things" are actually useful, what things are just "fun ideas" etc

I have this idea in my head that I want to just use haxe's default trace for everything (easier api to quickly get into the habit to using)

one issue i've had in the past is i'll make some log manager

and then never use it

and default to trace lol

[velvet lodge]

yeah, i tend to do that also, but its just because im lazy / dont give proper logging its due power until something goes wrong

[toxic stratus]

but if I make my logging system all work from haxe's default trace then change becomes easy

[velvet lodge]

the other important thing i need to double check with core-haxe/logging is if i can filter out noise to just one function... i think i can

[velvet lodge]

not sure how that would look... you would just override trace to do "things"?

[toxic stratus]

at least that's how I think it'll work in my head

[toxic stratus]

yeah, you can pass custom parameters to haxe's trace and parse them out however you wish

[velvet lodge]

yeah, it sounds neat in theory, i just wonder if its a little hacky in practice... like maybe you want an actual trace, not "write to a file" or... something... dunno

also, at least semantically, "trace" is different to "log"... Not that that matters much i guess

[toxic stratus]

no custom params means no fancy trace

[velvet lodge]

right, that would be a nice feature... until it bites you 😄

[toxic stratus]

yeah, at least at the moment I think it'll work I currently have 0 habit of using more than 1 parameter with a trace

so I'm not entirely sure where the issues may arise just yet

[velvet lodge]

oh yeah? im always doing: trace(">>>>>>>>>>>>>>>>>>>>>> here", a, b, c, d, e, f)

[toxic stratus]

lol I just do trace('foo' + a)

or

trace('here') ;
trace(obj);

[velvet lodge]

yeah, i used to the same, and in java you have to, which catches me out least 83 times a week

[toxic stratus]

lol

the benefits of committing to a single language 😂

[velvet lodge]

yeah, sitting in intellij (in a java project) staring at the screen not being able to work out why "trace" isnt recognized is another favourite passtime of mine 😄

[toxic stratus]

lmao

[uncut phoenix]

i try to use vim keybinds in eclipse

works greddat

[merry sinew]

TIL adaptor can be spelled with an "o"

[velvet lodge]

american / british english maybe / probably?

yeah

Since the two words share the same meaning, it's best to use adapter in the U.S. and adaptor in the U.K.

[cinder summit]

Still can't get core's sqlite to work right

||@velvet lodge shadow ping||

Full main class

[velvet lodge]

whats the error?

[cinder summit]

it doesn't run any of the sqlite

just prints my first trace and exits

[velvet lodge]

so the "dbc.open.then" never triggers (if you put a trace before "return dbc.exec")

[cinder summit]

Let me check

...the trace runs but the table isn't created

[velvet lodge]

hmmmm, and no error?

[cinder summit]

Nothing caught by the reject and nothing in stderr

I get this

[velvet lodge]

seems very odd... the usagle looks pretty much exactly the same as the unit tests and they are running fine 🤔

[cinder summit]

goddammit is this a crinfarr error

[velvet lodge]

haha

lemme try your exact sql this end... see if that also doesnt work

[cinder summit]

I would send you the exact text file but it's insanely large

[velvet lodge]

            final dbc = new Database('ips.sqlite.db', ReadWrite);
            dbc.open().then(_ -> {
                return dbc.exec("CREATE TABLE servers (
                    ip text not null primary key,
                    ports text not null,
                    timestamp float not null
                );"); 
            }).then(_ -> {
                trace("done");
            }, error -> {
                trace(error);
                trace(haxe.Json.stringify(error));
            });

[cinder summit]

actively losing my mind

[velvet lodge]

i can see what it is... you should have gotten an error though

{"name":"Error","message":"SQLITE_ERROR: Some of statement ignored: \"\r\n \""}

im gonna get the lib to trim() sql... avoid these types of things

[cinder summit]

I honestly didn't know sqlite cared about trailing newlines

[velvet lodge]

alright, if you pull latest sqlite3 it should be fine now

you might want to review your error handling though as i certainly got a (useful) error

[cinder summit]

wait this is actually hilarious

I didn't stringify the error, I just traced the message

I think the \r\n erased the actual error

so it was throwing fine but the message deleted itself

[velvet lodge]

should have still got an error i think.. lemme verify

            final dbc = new Database('ips.sqlite.db', ReadWrite);
            dbc.open().then(_ -> {
                return dbc.exec("
                CREATE TABLE servers (
                    ip text not null primary key,
                    ports text not null,
                    timestamp float not null
                );
                "); 
            }).then(_ -> {
                trace("done");
            }, (error:SqliteError) -> {
                trace(error.message);
            });

./cases/util/DBCreator.hx:45: SQLITE_ERROR: Some of statement ignored: "
                "

[cinder summit]

weird

maybe its the fact that I'm using the vscode internal terminal

[velvet lodge]

so yeah, you might want to just check to make sure you get an error out before you update

otherwise you are going to have pains chasing your tail without errors

very briefly looking at your code, it looks fine though... so defo odd

although, hang on, you are capuring the error and just rethrowing it, right?

[cinder summit]

old version, I swapped it to a trace

[velvet lodge]

does it hit the promise error handler? (like with a trace)

[cinder summit]

but throw was doing nothing

[cinder summit]

that might actually be it

[velvet lodge]

and it doesnt trace?

[cinder summit]

trace works now, I'm calling schrodinger's error

[velvet lodge]

well, i think you were just rethrowing the error as an exception, the app was then bombing out and you were none the wiser

[cinder summit]

does sqlite have WAL

[velvet lodge]

yeah

[cinder summit]

nice

[velvet lodge]

cant remember how you set it now... two secs

from db-core/db-sqlite:

                if (_properties.exists("journalMode")) {
                    _db.run("PRAGMA journal_mode=" + _properties.get("journalMode") + ";").then(_ -> {
                        resolve(new DatabaseResult(this, response.data));
                    });
                }

[cinder summit]

Trying to see how many concurrent threads I can use to count the lines for an accurate progress bar

looks like 2

...looks like FileInput has an internal mutex

damn

[velvet lodge]

how many lines in the file? Didnt i have a play with this and got it down to like sub second? I cant remember now, i might be thinking of something else

[cinder summit]

Smaller file but similar use case

I think it's like 40 million

[velvet lodge]

i cant remember what i tested with, but i remember i managed to get it silly fast... maybe it wasnt 40m records though... dont remember

somewhere in haxe chat, but ages ago

[cinder summit]

#haxe message my search filter mastery comes in handy

[velvet lodge]

oh, much slower than i thought then

it was taking hours though iirc... so maybe it was "sub minute" i remember, not "sub second" 🙂

[cinder summit]

yea but my solution at the time involved directly calling a child process that ran sqlite

[toxic stratus]

btw ian

server has been running for 2 days

no issues with db connections x)

[velvet lodge]

noice

[deep ore]

how is the websocket support in Core Haxe? does the JVM target work nicely? 👁️
I'm aware of this other WS library https://github.com/colyseus/colyseus-websocket-hx

[velvet lodge]

web socket support is still being mirgrated over from hxWebSockets

i think client stuff is done(ish), but i need more unit tests and more targets... maybe ill have some time this weekend, not sure

suffice to say, the intention is 100% to move things over to core-haxe/websockets (https://github.com/core-haxe/websockets) and have as many targets supported (and unit tested) as possible

[deep ore]

I see, thanks for the heads up 🙂

[toxic stratus]

any thought to supporting enum abstracts in db stuff?

Wait, i'm confused

I have another database operation that works with enum abstracts

[toxic stratus]

lol nvm, i misremembered that the abstract issue was on entities 😂

[velvet lodge]

haha, yeah

[velvet lodge]

and i still havent looked at it, im not even sure what info i can get in the macro, presumably "everything"

[toxic stratus]

are there any utilities in db core for handling timestamp column's in db-core?

[velvet lodge]

nothing in built i dont think... what are you trying to do? Will haxe date time hold timestamps? How are they stored in the db... longs?

[toxic stratus]

the timestamp column type specifically

just parse it "easily"

it doesn't come in, in unix timestamp format

[velvet lodge]

ill check it out... defo sounds like a nice thing that db-core (well, the plugins really) can do

[toxic stratus]

yeah, it allows you to do more with date/time related queries than just "simply" sort by asc/desc

[toxic stratus]

is there a way to handle sorting results? 🤔

I want to get the most recent record inserted

typically i think you'd just sort by desc and grab the first one

[velvet lodge]

not possible yet, but defo needs to be added... and soon

[toxic stratus]

okay that's cool

but thanks to this being missing, i found a cool use for triggers 😄

I have a table_status table that tracks when another table is last changed

I added a data column that holds the last bit of data that was inserted 😂

so i can just use that for now

triggers are cool yo!

OH MY, I can build json in here as well 😄

[velvet lodge]

jvm support (client only currently) added:

[toxic stratus]

it's nice seeing so many greens 😄

[velvet lodge]

yeah, the unit testing was sorely missing from hxWebSockets... its one of the reasons i never really want to make any changes

[toxic stratus]

how do you unit test these things? something like http/sockets seems awkward to unit test cause you'd need some kind of service to connect/post/delete etc

[velvet lodge]

https://github.com/core-haxe/http/tree/main/tests/cases

plenty of dummy services about

i could also mock them, but id actually prefer real tests (even though that makes them more integration / functional tests rather than unit tests)

[toxic stratus]

i think for stuff like this, functional tests matter the most

[velvet lodge]

agreed

websockets is more involved to test... i spin up a dummy echo server (native in nodejs) using new Process() and use that to test things

ill do the same with a server written in websockets once i have that working... but having clients tested against a (non-haxe, non-this-lib) nodejs ws server is defo useful also

[toxic stratus]

for client use, would you say core-haxe/websockets is usable now?

also is there an internal type for db-core promise errors?

i get no completion whenever I work with the promises and wanna see if I can get that inferred

seems like ops (insert/update/etc) have the following error structure```hx
typedef Error = {
message:Stirng;
call:String;
}

[velvet lodge]

https://github.com/core-haxe/db-core/blob/main/src/db/DatabaseError.hx ?

[velvet lodge]

... maybe? Probably?

[toxic stratus]

i had a browse through the api and i guess it isn't possible with the current promise structure

there's no way to define an expected type for an error promise

[velvet lodge]

sure there is: ..., (error:DatabaseError) -> { ... }

[toxic stratus]

yes, i meant in an inferred way

[velvet lodge]

ah right, no, currenly its a dynamic i think, i wonder if you could make a generic version? Not sure

[toxic stratus]

yeah, not sure myself either

and i think it would be somewhat of a big change

[velvet lodge]

i think the main issue would be that technically, a promise can reject anything... like one branch might to "reject(1)" another "reject('a string')", etc

[toxic stratus]

it's similar with the http lib where i keep forgetting api structure so have to keep appending HttpError/HttpResult to the function params

[deep ore]

neat!

[toxic stratus]

this is defo not a super important thing but it looks like you may have some changes to perform when haxe 5 comes out

https://github.com/core-haxe/queues-core/blob/main/src/queues/macros/QueueTypes.hx#L16

https://github.com/core-haxe/db-core/blob/main/src/db/macros/DatabaseTypes.hx#L15

i was just a lil tempted to see how things were in vscode x)

[velvet lodge]

ill have to check it out, it seems onAfterInitMacros is available in 4.3.3 so i might just be as simple as changing things to use that

[toxic stratus]

just to let you know

server has been running for a week

absolutely no issues with the connection

and i discovered that the missing data issue was a problem with my own code

[velvet lodge]

great! 👌

[velvet lodge]

also great! 😄

[toxic stratus]

I'm getting a weird error, or maybe i'm just doing something stupid and it is late

result.table.update(query($pid = position.pid), record);

is resulting in the following error:

{
  message : Unknown column 'RxxdtVUyOd' in 'where clause', 
  call : update
}

where RxxdtVUyOd is not the column, it's a column value

the column should be $pid 🤔

the extra weird part is, I use this exact same query prior with a findOne which successfully works

[velvet lodge]

hmmm, odd... so fineOne works, but then the same query in update doesnt?

[toxic stratus]

correct

[toxic stratus]

case SearchAndUpdate(table, key, query, value, callback):
    var record = RRecord.fromDynamic(value);
    db.table(table).then((result) -> {
        return result.table.findOne(query);
    }).then((result) -> {
        trace(result);
        if (result.data == null && !this.insert_cache.exists(key)) {
            this.insert_cache.set(key, true);
            trace('adding');
            return result.table.add(record);
        } else {
            trace('updating');
            return result.table.update(query, record);
        }
    }).then(function(result) {
        if (this.insert_cache.exists(key)) {
            this.insert_cache.remove(key);
        }
        if (callback != null) {
            callback(Success('Successfully updated record', result.data));
        }
    }, function(err:Dynamic) {
        trace(value);
        trace(err);
        trace(err.message);
        // if (err.message.contains('Duplicate entry')) {
        //     EcsTools.addComponents(event); //recycle for an update instead
        // }
        if (callback != null) {
            callback(Error("Failed", err));
        }
    });

here's the full code block for context

to even get to the update condition findOne has to work

[velvet lodge]

what does trace(Query.queryExprToSql(query)) result in?

[toxic stratus]

DatabaseSystem.hx:137: pid = RxxdtVUyOd

[velvet lodge]

and is that right? I mean, does that make sense?

[toxic stratus]

yeah it makes total sense

the column name is pid and the value i want to match against is RxxdtVUyOd

[velvet lodge]

cool - you might want to trace out the sql the update produces

see whats going on there

[toxic stratus]

How do I do that?

[velvet lodge]

https://github.com/core-haxe/db-mysql/blob/main/src/db/mysql/MySqlTable.hx#L251

trace that

[toxic stratus]

UPDATE position_status SET pid = ?, exchange = ?, symbol = ?, price = ?, quantity = ?, id = ?, pnl = ?, order_ids = ?, status = ?, created = ?, closed = ? WHERE (pid = RxxdtVUyOd);

[velvet lodge]

looks alright?

[toxic stratus]

yeah it seems fine

[velvet lodge]

whats the error again?

[toxic stratus]

  message : Unknown column 'RxxdtVUyOd' in 'where clause', 
  call : update
}

it defo exists

[velvet lodge]

can you trace out the findOne sql also?

[toxic stratus]

SELECT * FROM position_status WHERE (position_status.exchange = ? AND position_status.symbol = ?) LIMIT 1;

oh, this is an earlier find

2 secs

SELECT * FROM position_status
WHERE (position_status.pid = ?) LIMIT 1;

oh the update seems to omit position_status from the where clause

[velvet lodge]

hmmm, interesting, defo a difference... ill check it out when i get a minute (prolly not till the weekend, swamped atm)

[toxic stratus]

i think i fixed it

does the change make sense to you?

[velvet lodge]

make the PR, easier to see in context, we can discuss there

[toxic stratus]

okii

https://github.com/core-haxe/db-core/pull/3

also it appears like my change is causing issues with db-core as well? :/ locally it doesn't damn haxe 4.2.5

[velvet lodge]

change looks good to me... nice catch, one thing, can you also make sure other where queries are supplying all params... i can see how that isnt supplying table.name

[velvet lodge]

you mean the GH actions?

[toxic stratus]

yeahe

CI breaks on 4.2.5 with my recent pr for db-core

[velvet lodge]

hmmmm, right

[toxic stratus]

found 2 more places

[velvet lodge]

thanks!

[toxic stratus]

thanks right back for the walkthrough on where to look 😄

[toxic stratus]

@velvet lodge

core-haxe is failing with my fix for the macro error

it seems like they're all failing on node due to this

[velvet lodge]

guess ill revert that also?

[toxic stratus]

are you cool if i force the type to array<dynamic>?

[velvet lodge]

or is this a non haxe5 fix?

[toxic stratus]

it's a hxb haxe fix

but it works for 4.3+, if you want to revert that's fine as well

[velvet lodge]

looking at the line, 2 secs

[toxic stratus]

ahh was just about to link you to them

[velvet lodge]

its in the logging?

[toxic stratus]

what is?

[velvet lodge]

the error

[toxic stratus]

ah, i'm not sure lemme check

nah, seems fine

i think it was db-core and queues-core i patched

[velvet lodge]

is it not this line: https://github.com/core-haxe/db-mysql/blob/main/src/db/mysql/MySqlDatabase.hx#L30 ?

[toxic stratus]

it is yeah

[velvet lodge]

so thats the logging, no?

though i cant repro it on try.haxe

[toxic stratus]

ahhh i wasn't following, i didn't actually get the line before you 😂

https://github.com/core-haxe/logging/blob/main/src/logging/Logger.hx#L29

[velvet lodge]

yeah

https://try.haxe.org/#ae191D16

works here though

[toxic stratus]

hop back to haxe 4.2.5

[velvet lodge]

ah, so its a 4.2.5 issue?

[toxic stratus]

yea

[velvet lodge]

well, im defo happy to remove 4.2 for core haxe

[toxic stratus]

makes sense

coolsies

[velvet lodge]

is that the only issue? Why did this just start happening?

oh, because i only just added the logging maybe

[toxic stratus]

I think it was because of this patch? https://github.com/core-haxe/db-core/commit/9a8201dedc1436a3e02d9963e409a39954f132bc

[velvet lodge]

defo looks like thats the only issue... ill drop 4.2 support in the actions

[toxic stratus]

but i never actually checked to see if it was my fault

[velvet lodge]

i dont think so... i think its when i added all the logging

[toxic stratus]

ahhh i just assumed it was me 😅

[velvet lodge]

in facr, its only using 4.2...

alright, updated them all to 4.3.3, see what happens

[toxic stratus]

@candid zealot What's your issue with db-core? Is it all setup correctly?

[candid zealot]

I don't believe it is, I think I forgot to compile it

Will double check this theory later

[candid zealot]

Yeah no I don't think I have it compiled

But idk if thats nescessary

I downloaded it into my project, added -cp db-core/src, but I cannot import it into my main file or anything

[toxic stratus]

why use a cp and not --lib db-core?

[candid zealot]

Ohhh, I tried -lib db-core not --lib db-core

[toxic stratus]

how did you install it?

[candid zealot]

Had to download it, there's not a haxelib thing for it

At least the command said as much

[toxic stratus]

haxelib git db-core https://github.com/core-haxe/db-core

it's worth learning a bit about haxe :p

[candid zealot]

I was trying haxelib install db-core

[toxic stratus]

yeah

to install from git you need to provide the source

[candid zealot]

There isn't an awful lot of resources on haxe, I've been trying to learn

So what does that do different than manually? Does it compile it? Integrate it into the project?

[toxic stratus]

just ask here for general questions like that, this kind of thing is pretty common in haxe and afaik there is docs on this

[toxic stratus]

it installs it like haxelib install db-core

so any kind of config that the library may have, may have been missed with your approach

[candid zealot]

Ah, noted

[toxic stratus]

on that note, did you install db-sqlite?

[candid zealot]

But yeah, searching for anything haxe related is a pain, I feel so many resources are tied up in the API or 3 hour long lectures

I believe I did, was trying everything

It's under the dependencies, right?

Ran into issues using raw sqlite because it said the example I was using was making numerous requests at once but it wasn't 😭

But I look forward to getting home and trying out db-core then lol

[toxic stratus]

it isn't, db-core is an abstraction layer, you need to specify what abstractions you want

it allows you to switch from sqllite/mysql/another db protocol without the need for changing code

but yeah, as ian said you can use sqlite directly if preferred

[candid zealot]

Ah, all I need is sqlite and castledb I think.

So maybe it's not even worth it lol

[toxic stratus]

up to you, if you don't plan on doing anything too complex db-core probably makes the api easier

[candid zealot]

Ah, true enough. Less things to go wrong too

I just want a basic relational db, just want to be able to traverse a list, hop between items, and tweak some variables, nothing crazy

Just using it as a ledger basically

[deep ore]

you could use the js target with Bun which has built-in sqlite support

oh nvm, i didn't realize what channel i was in, disregard

[toxic stratus]

yeah, it works well for me

i'm using it to track orders and assigned data

a simple db.table.findOne(query($id == order_id)) picks up what i want

[candid zealot]

I've been trying this command and one for sqlite3. they both flag errors, for possibly dead dependencies. db-core for one called "promises" and sqlite3 for "libsqlite3" any idea what im doing wrong @toxic stratus
haxelib git db-core https://github.com/core-haxe/db-core

[velvet lodge]

they arent dead deps... they are just deps... core-haxe has a bunch of deps that are interused

https://github.com/core-haxe

also, on that page there is a section "update/install scripts" that you should be able to just copy and paste into a .bat or .sh...

it wil install all of core haxe, but its not huge (size wise)

( @candid zealot )

[candid zealot]

finally got it working, thank you all so much! I dont think i would have figured it out without your help

Theres not enough resources online for even chatgpt with a web api like bing's is able to help all too much lol

[velvet lodge]

tbf, its all just kinda "haxey" stuff (like knowing how to use haxelib git or haxelib dev)... if you are new to haxe (which im assuming you are?) then yeah, can not be obvious at first glance

[candid zealot]

newish to haxe but not programing. Using it for a little more than a month, I've read through the website a number of times, watched most the videos, but it feels like the only way to learn a lot of the stuff is to dev deep into the API or through the community, unless im missing a major source or smthn and just being dumb

[velvet lodge]

newish to haxe but not programing
yeah, i meant haxe specifically

unless im missing a major source or smthn and just being dumb
i think what you listed it pretty much it... the discord here is super helpful usually, and a quick way to get answer for things

[deep ore]

you are spot on. Haxe doesn't have a lot of up to date tutorials for a lot of things. For lots of libraries, you'll have to look at the source code to figure out how to use them

[candid zealot]

Trying to figure out how the sqlite code works, but it seems a lot of the functionality is in the externs. Would really appreciate if someone just confirmed/denied my understanding but basically

import sqlite.Database;
import sqlite.SqliteError;

class Main {
    static function main() {
        var db = new Database("somedb.db");
        db.open().then(result -> {
            return db.exec("CREATE TABLE Persons (PersonID int, LastName varchar(50), FirstName varchar(50));");
        }).then(result -> {
            return db.exec("INSERT INTO Persons (PersonID, LastName, FirstName) VALUES (1, 'Ian', 'Harrigan');");
        }).then(result -> {
            return db.all("SELECT * FROM Persons;");
        }).then(result -> {
            for (person in result.data) {
                trace(person.FirstName, person.LastName);
            }
            return db.get("SELECT * FROM Persons WHERE PersonID = ?", [1]); // use prepared statement
        }).then(result -> {
            trace(result.data.FirstName, result.data.LastName);
        }, (error:SqliteError) -> {
            // error
        });
    }
}```

example code ^

so what appears to be happening is we make a new db and store a reference to it as a variable, this makes sense. From there we use .open() to enable us to insert commands into it, .then() lets us insert commands, not entirely sure what result -> {return...} means, db.exec() appears to be a function to insert SQL commands as strings in order to make changes and db.all() appears to be a way to insert SQL strings in this case a query but i assume its just for generally access groups of data

[velvet lodge]

so all the libs in core-haxe are promise based, this makes them asynchronous ... is that what you are not used to here?

the ".then" just means that its callback (function(result) { ... }) will be called when the function completes "at some point in the future"

result will be the result of the call

as for db.all, db.exec, they are just things exposed from the mysql interface... "exec" in mysql is where you dont expect a result, "all" is when you do... technically i think you could use "all" for everything here, but semantically, "exec" makes more sense for things like "create"

[toxic stratus]

db.get("statement") is where you specify your sql actions
a promise is just an asynchronous callback, it's the same in javascript. It just returns the response from your actions

db.get('').then((result) -> {}, (error) -> {});
//is the same as
db.get('').then(function(result) {}, function(error) {});

in haxe the top one is just a shorthand function

what ian is doing in the example is just promise chaining, using the response from one promise to pass it to the next

otherwise you'd just continuously nest the promises

[candid zealot]

makes sense

Just trying to make sense of the syntax really

So in this case db.open() creates the original result thats put into the first then() function

[toxic stratus]

yeah altho you aren't using the result there. The main reason you're doing it in the callback is to know that the db connection is successfully openend

[candid zealot]

ah, as to throw an error in the case its not?

makes sense

thanks for the help! I'll get some sleep pretty soon, might write some basic documentation for at least myself so I can remember whats going on lol

[toxic stratus]

if you have docs, just add them to the repo. i think the docs on the repo make a lot of assumptions

like you're familiar with dbs and promises

if you're not, I guess it can be hard

[candid zealot]

definitely. I havent made many contributions to any repos but my own (its a bit intimidating to say "hey my code is good enough for everyone to use", even when invited lmao) but if its good enough and just docs I could probably

got invited to retool and submit something on beziers to a bigger project, too scary jk

[toxic stratus]

if its wrong or unsuitable the repo owner will just say no thx or offer corrections, saying "hey i think this may help" isn't gonna immediately affect the repo

and most often these things aren't done because the repo owner doesn't need them, not because they don't want it

[velvet lodge]

two things are constant: ill never said no to a beer, and ill never say no to doc PRs... 😆

[candid zealot]

the project was bevy, i dont think I'd have the mental fortitude to handle the inquisition around the PR lmao

just scrolling github drama is fun sometimes

[toxic stratus]

i avoid all drama 😂

waste of time 😅

[candid zealot]

is there any reason why this code only prints the output when there is no existing "somedb.db" file?

class Main {
    static function main() {
        var db = new Database("somedb.db");
        db.open().then(result -> {
            return db.exec("CREATE TABLE Persons (PersonID int, LastName varchar(50), FirstName varchar(50));");
        }).then(result -> {
            return db.exec("INSERT INTO Persons (PersonID, LastName, FirstName) VALUES (1, 'Ian', 'Harrigan');");
        }).then(result -> {
            return db.all("SELECT * FROM Persons;");
        }).then(result -> {
            for (person in result.data) {
                trace(person.FirstName, person.LastName);
            }
            return db.get("SELECT * FROM Persons WHERE PersonID = ?", [1]); // use prepared statement
        }).then(result -> {
            trace(result.data.FirstName, result.data.LastName);
        }, (error:SqliteError) -> {
            // error
        });
    }
}```

[toxic stratus]

I would imagine because the first two conditions likely return an error, probably like 'table Persons' already exists

put a trace in the error callback and it will let you know what's going on

it's the last one in the chain

[velvet lodge]

yeah, that sounds like exactly the issue

[candid zealot]

sorry, which condiitons?

[toxic stratus]

if the table already exists, it can't create another table with the same name

[candid zealot]

how would i just access the table then?

and yeah that seems to be the issue in this case "table Persons already exists"

what little resources I could find plus the robot says this is just how you access a table

[toxic stratus]

a select statement

[candid zealot]

OH WAIT

[toxic stratus]

there's a ton of resources for sqllite

[candid zealot]

so the issue is CREATE TABLE Persons, not the new Database

for the haxe implementation, i assumed the issue was the new Database("somedb.db") not the SQL

[toxic stratus]

lol

[candid zealot]

I am the wisest of men jk

[velvet lodge]

you might want to follow NotBilly's advice and put a trace in the error handler, would likely give you good info

[candid zealot]

I did indeed, currently trying to decipher a new error i just made. This ones just syntax

Its sql so like billy says, lots of resources

so i can figure this new one out : P

[toxic stratus]

with the promise chain, errors 'fall through' to that one callback - maybe that's why you thought it was db 🤔

[candid zealot]

It was my illiteracy with the overall code, and some bad assumptions on my part in this case. Up until you suggested it I wasnt using the errors, as going off a bad hunch

I do need to read more about promise, because without it the source code may as well be Greek to me

[velvet lodge]

this is js: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise but the idea is the same (and the syntax is almost identical)

[toxic stratus]

Can i do queries like field > number

(in db-core)

[velvet lodge]

sure, why are you asking, does it not work?

[toxic stratus]

i checked the readme and it only seemed like = was a thing

my bad!

[velvet lodge]

https://github.com/core-haxe/db-core/blob/main/src/Query.hx#L9-L21

||, &&, =, !=, <, >, <=, >=, in

[toxic stratus]

just a fun little fact

db has been running for 2 months and no issues x)

[velvet lodge]

nice, good to know 🥳

[toxic stratus]

Is there a convenient way to exclude a field from being added when posting data to the DB?

I could make a field in the class where I just construct a record and send that, i guess....

(just thought about that now)

i think just making an abstract class with a record field may be better than what i'm doing now

that'll be a refactor in quite a lot of places

[velvet lodge]

i dont think so, i mean, it will take the record and use that to update the record... saying "heres the record to update, but not all of it" seems weird

[weak talon]

Doesn't seem weird to me. Eg if the user wants to update their name, it seems logical to only update the Name entry in their record instead of fetching the entire record, updating one field, and then rewriting the entire record.

[toxic stratus]

yeh, right now i'm just doing Record.fromDynamic(data) which is great for convenience, but means anything thats in the object that isn't in the db means the whole entry won't get inserted/updated 😂

[velvet lodge]

ok, hang on, there could be an angle here, two secs, lemme check how fromDynamic works

ok, so there could be an option param on fromDynamic which would be "exceptions"

like: Record.fromDynamic(data, ["nope", "no_way", "heeeeeelll_no"])

[toxic stratus]

it could work, but, essentially would mean i'd have to mock that up everywhere i wanna send the record

it may be useful in some cases

but i think just doing the refactor may be best

[velvet lodge]

i feel like an exclusion list is sensible anyway... can defo think of occasions when you are like "heres the data, but oh, its junky, so forget about these fields"

[toxic stratus]

i was hoping for a @notForDB kinda meta 😆

[velvet lodge]

but where would you put that meta?

[toxic stratus]

ya, when i asked the question i didn't think that would be incompatible with fromdynamic 😂

but the meta would be in the data object for the db

class Foo {
  public var timestamp:Float;
  @exclude public var time(get, never):Float; //format timestamp into readable time for UI
}

[velvet lodge]

but the meta would be in the data object for the db
im not sure what means, core-db is just dealing with records, not anything you turn them into

core-db doesnt have a clue what "Foo" is... entities would, but thats a different story

[toxic stratus]

yeah, i figured this may be something in the scope for entities

[weak talon]

That kind of meta seems like something for a macro-powered library built on top of db-core.

[velvet lodge]

yeah, which is exactly what entites is

[toxic stratus]

maybe i'll have a play with making my own mini macro for something like this

[velvet lodge]

maybe, and im not sure of the full scope of your stuff, you could just extend Record and use that?

[toxic stratus]

i haven't messed around with macros in quite a while

[velvet lodge]

like MyRecord extends Record then override the fromDynamic and toDynamic (or the fields, or whatever)

[toxic stratus]

how would i achieve the above result using that?

[velvet lodge]

i guess you could override fromDynamic, have a "black list" and not allow those fields through... dunno

although, then you would need to use MyRecord.fromDynamic, so its not an override at all - my bad

[toxic stratus]

class Position {
 public static function fromRecord(record:Record) {
  var id = record.field('id');
  var pid = record.field('pid');
  var symbol = record.field('symbol');
  var side = record.field('side');
  var price = record.field('price');
  var quantity = record.field('quantity');
  var timestamp = record.field('timestamp');
  var open = record.field('open');
  var exchange = record.field('exchange');
  var updated = record.field('updated');
    
  var p = new Position(exchange, symbol, side, price, quantity, timestamp);
  p.id = id;
  p.pid = pid;
  p.open = open;
  p.updated = updated;

  return p;
  }
}

right now, a lot of the db types i use have this kind of static in them

[weak talon]

Oh that is definitely material for macro-powered automation.

[toxic stratus]

making some macro to just handle something like this with a "what goes into the record" would be nice

[velvet lodge]

almost certainly outside the scope of db-core though

[toxic stratus]

completely fair

i guess i'll get to have a play with macros again soon x)

would be nice to go @record var pid:String tbh

[toxic stratus]

just to confirm, the kind of macro i want to do here is an autobuild macro?

[velvet lodge]

depends, where are thinking of putting it? @:autoBuild macros will run on "classes that extend this class", @:build macros will run on "this class"

[toxic stratus]

oh that's useful info, autobuild sounds like the one then. I think I'd be able to do something like

class MyData extends MyRecord {
  @record var pid:String;
}

@:autoBuild(Macro.build())
class MyRecord {}

is it better to just "skip" the sub type?

ahh, nvm actually, i think i may keep it. Then i can just be typesafe everywhere in my code

enum DBEvent {
  Insert(table, data:Any, callback);
}

I'm doing this atm, i could change data to data:MyRecord which would be a lot better

[toxic stratus]

a bit off topic

but i've never done db polling before

it works quite well

i'm only checking once a second

definitely probably resource intensive, but, i think its fine for my needs and i'm hosting my own mysql server on a cheap vps

i'm not really going to be limited by read/write costs

[velvet lodge]

there are probably other ways (like reading log files "binlog" seems to be one option), as for your app getting the notif, you could always have websocket open, that would be realtime

but a 1s poll is probably also fine... even less, .1 if you wanted "near realtime"

[toxic stratus]

I read about those 2 options

but no idea how to implement either of them

[velvet lodge]

basically, you would have a seperate process, a websocket server in this case, that would be reading binlog in real time and working out events like "data added", "data removed", etc... it would then dispatch those events to any clients that are listening through the websocket channel...

[toxic stratus]

var http = new HttpClient();
var url = new HttpRequest('https://api.deepcoin.com/deepcoin/market/candles?instId=BTC-USDT&bar=1m');
url.method = HttpMethod.Get;
http.makeRequest(url).then((res) -> trace(res), (err) -> trace(err));

I have no idea why

but this request and only this request fails

if you go to the url directly in the browser, works completely fine https://api.deepcoin.com/deepcoin/market/candles?instId=BTC-USDT&bar=1m

[weak talon]

Define "fails". Is there any kind of error message?

[toxic stratus]

it seems to not be able to connect at all

[Error: Unable to create XMLHttpRequest object.] {
  __previousException: undefined,
  __nativeException: [Circular *1],
  value: 'Unable to create XMLHttpRequest object.',
  retryCount: 0
}

fails on interp as well

[weak talon]

(What error does it give there?)

[toxic stratus]

2 secs

{
    "bodyAsString": null,
    "retryCount": 0,
    "headers": null,
    "message":"A connection attempt failed 
because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.\r\n(connect, )","body":{},"httpStatus":null,"bodyAsJson":null
}

[old juniper]

is it the only one that has a "hyphen" ?

[toxic stratus]

i think so, but, i also think i have used hyphans in the url previously

https://api.deepcoin.com/deepcoin/market/instruments?instType=SPOT no dash url also fails in the same way

i suspect it's with the host specifically but I have no idea why

[velvet lodge]

weird... does the api need some headers that are missing?

[toxic stratus]

nothing according to the api

[velvet lodge]

you sure, not even an agent or something? Does it work in postman?

it does

[toxic stratus]

I send an agent

what's the headers that postman sends?

[velvet lodge]

sending "Host" also? i get a 400 without Host in postman

[toxic stratus]

does http change headers in anyway?

'Host' => 'api.deepcoin.com' I send this

i'm getting an error saying it's an array for some reason?

[weak talon]

Host is a required header per the HTTP specification.

[toxic stratus]

yeah, that seemed to be the connection issue. Just no-one else seems to care for that lol

[weak talon]

And per the spec:

All Internet-based HTTP/1.1 servers MUST respond with a 400 (Bad Request) status code to any HTTP/1.1 request message which lacks a Host header field.

[velvet lodge]

seems odd... smells like a bug... a strange one though

[weak talon]

It might be a good idea to make HtppRequest set the Host header per default, since it knows the URL (and thus the domain) already.

[velvet lodge]

yeah, feels sensible

[toxic stratus]

Not really sure what's going on

class Main {
    static function main() {
        trace("Hello, world!");
        var http = new HttpClient();
        var url = new HttpRequest('https://api.deepcoin.com/deepcoin/market/instruments?instType=SPOT');
        url.headers = ['Host' => 'api.deepcoin.com', 'User-Agent' => 'CustomApp'];
        url.method = HttpMethod.Get;
        http.makeRequest(url).then((res) -> trace(res.response.bodyAsString), (err) -> trace(Json.stringify(err)));
    }
}

this still exhibits the connection issue on interp

[velvet lodge]

ill take a look in a bit, later this eve

[weak talon]

That might just be interp being broken, it has had issues with HTTP(S) in the past iirc.

[velvet lodge]

would be good to confirm if thats the case or not

[toxic stratus]

fair, just trying to see why host becomes an array atm

seems to make no sense

internally i see nothing that's causing this

[toxic stratus]

could using hx4compat be causing the issue?

ahh nvm

ohhh!!

I have commented out a lot of things and no dice

I just commented out the http library and things immediately started working

okay... I don't think it's the http library

but i do think it is something to do with one of Ian's libs

[toxic stratus]

ohh, i've been sending messages in the wrong channel lol

well, for context #1199984627020992553

[velvet lodge]

something todo with hxb?

[toxic stratus]

yeah

[velvet lodge]

well, im happy to make / accept code changes that fix it (so long as they dont break anything else ofc), but i cant really support some experimental feature in a future version of haxe that is till WiP... that said, if there is something one of the libs is doing wrong that is triggering this behaviour, obviously, im happy to review / change that

[toxic stratus]

we're unaware of what is going on atm, I don't think the issue is necessarily with the libraries

we're trying to find out why it causes a compiler error

[toxic stratus]

Are you good with this change? The issue is absolutely to do with a compiler bug, rudy was able to repro it

but this type hint works around the issue, so if anyone else wants to use latest haxe won't get the compiler error

[velvet lodge]

does it work "everywhere"?

or, well, looks like js only?

[toxic stratus]

it works for me on 4.3.3 and nightly

But there isn't a CI for promises

I can't imagine it breaking as Promise.reject is "meant" to return this type anyway

buuuut you never know

[velvet lodge]

whack it in... there are unit tests in all other libs and they all use promises, lets see what happens

[toxic stratus]

i posted a PR but it didn't trigger any rebuilds

[velvet lodge]

it wont... it probably should, but it wont

[toxic stratus]

ahhh fair

[toxic stratus]

@velvet lodge did you happen to have a look at this?

[velvet lodge]

i havent, is this interp only?

[toxic stratus]

no, has issues with js as well

[velvet lodge]

gotcha, will check it out over lunch

[velvet lodge]

    function testBasic(async:Async) {
        var http = new HttpClient();
        var url = new HttpRequest('https://api.deepcoin.com/deepcoin/market/instruments?instType=SPOT');
        url.method = HttpMethod.Get;
        http.makeRequest(url).then(result -> {
            Assert.equals(1, 1);
            trace("---------------------------------------- here", result.response.bodyAsJson);
            async.done();
        }, err -> {
            async.done();
            trace(err);
        });
    }

this works fine for me, didnt even need the headers

nodejs: fine
hxcpp: fine
hl: fine
neko: fine

[velvet lodge]

i do get this though

[toxic stratus]

with the host it connects but I get the host is an array error

[velvet lodge]

i get that, and i can see why, easy fix, but i didnt even need it

[toxic stratus]

really? weird

[velvet lodge]

nope

[toxic stratus]

it flat out breaks for me

[velvet lodge]

that array thing looks like another bug in haxe.http

         for (i in headers) {
             var arr = Reflect.field(h, i.name);
             if (arr == null) {
                 arr = new Array<String>();
                 Reflect.setField(h, i.name, arr);
             }
 
             arr.push(i.value);
         }

... ... ... lol?

[weak talon]

Huh.

[velvet lodge]

yeah, it always sends headers as arrays... always

{
        Host : [api.deepcoin.com],
        User-Agent : [CustomApp]
}

sometimes arrays of headers is useful, but it shouldnt be always

[weak talon]

Also I just noticed that sys Http actually does set the Host header already: https://github.com/HaxeFoundation/haxe/blob/2aa1b528f9782e506f0e370f3203f1af7460bc11/std/sys/Http.hx#L203

[velvet lodge]

might be why i dont have any issues...

also, thats sys.Http

this is HttpNodeJs

[weak talon]

Hmm, I would expect node's http impl to also set the Host header itself.

[velvet lodge]

probably does, i dont think Host is needed to be sent... i didnt need to send it

but regardless it shouldnt be turning headers in arrays anyway

[velvet lodge]

alright, the array thing is fixed now, but i cant repro your actual issue... ive also added unit tests for that url both with headers and without, both work

( https://github.com/core-haxe/http/commit/5439bf64f8641c33213b0737f785097c78ea8e2c )

[toxic stratus]

what in the world

[velvet lodge]

¯_(ツ)_/¯

[toxic stratus]

I am very confused

this doesn't work for me at all

[velvet lodge]

do you get an error or something?

[velvet lodge]

is this nodejs? Or browser?

[toxic stratus]

electron/browser

[velvet lodge]

have you included hxnodejs?

Although, i would assume it would work from browser/non-node electron too

[toxic stratus]

I have got hxnodejs included, but other domains work

    function testBasic() {
        var http = new HttpClient();
        var url = new HttpRequest('https://api.deepcoin.com/deepcoin/market/instruments?instType=SPOT');
        url.method = HttpMethod.Get;

        url.headers = [];
        
        url.headers.set('Host', url.url.domain);
        url.headers.set('User-Agent', 'Custom Trading App');

        http.makeRequest(url).then(result -> {
            trace("---------------------------------------- here", result.response.bodyAsJson);
        }, err -> {
            trace(err);
        });
    }

this results in the above error as well

[velvet lodge]

whats the difference there anyway?

[toxic stratus]

headers

[velvet lodge]

right

lemme try in haxeui browser and also electron real quick

browser (which make sense)

removed headers:

Access to XMLHttpRequest at 'https://api.deepcoin.com/deepcoin/market/instruments?instType=SPOT' from origin 'null' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

[toxic stratus]

i never thought to check dev tool output

[velvet lodge]

because im running from file system

[velvet lodge]

i think that it seems you are missing some type of lib or something that node is looking for

[toxic stratus]

why does it work fine for everything else? o_o

[velvet lodge]

not for that api though i assume

[toxic stratus]

the api doesn't seem to mention any specific requirements, it's just a rest api

[velvet lodge]

but they have COR access control headers

they must do... other wise browser wouldnt send that error... which you are right though, seems odd for an api... ill check electron too... but i think what is happening after that is your app is erroring and then failing to find some lib

[toxic stratus]

lets see if adding 'Access-Control-Allow-Origin': ['*'] will solve it

[velvet lodge]

i think thats set on their side, not yours

[toxic stratus]

ahh

well, i can see if turning off websecurity in my electron app will solve it lol

[velvet lodge]

works for me on electron

[weak talon]

Why is it trying to load node:internal/errors while trying to send an XMLHttpRequest...
Something is very wrong here.

[toxic stratus]

wth

[velvet lodge]

i think its after it fails... not sure

[toxic stratus]

i don't have a clue what's going on overall

to me it seems odd that a single domain is failing

[velvet lodge]

what webPrefrences do you have in the electron BrowerWindow?

[toxic stratus]

        webPreferences: {
            backgroundThrottling: false,
            contextIsolation: false,
            nodeIntegration: true,
            preload: path.join(__dirname, 'preload.js'),

        }

[velvet lodge]

not really, it obviously has different security than others... it would be weird if two calls from the same api failed

[velvet lodge]

seems similar:

        window = new BrowserWindow({
            show: false,
            webPreferences: {
                nodeIntegration: true,
                contextIsolation: false
            }
        });

this work for you?

[toxic stratus]

how do i run it?

[velvet lodge]

haxe electron.hxml

then in the build/electron folder electron .

[toxic stratus]

huh, i tried that and got electron errors

[velvet lodge]

what errors?

[toxic stratus]

ahhh i didn't cd into electron

just build

[velvet lodge]

that'll do it

[toxic stratus]

it runs

but i don't get anything obvious

[velvet lodge]

dunno what any of those access denied things are

did the app run?

[toxic stratus]

i don't think it's loading the right thing

i got a standard electron init window

rather than a haxeui app with a button

[velvet lodge]

did you put the "."

[velvet lodge]

☝️

[toxic stratus]

yep this is with electron .

[velvet lodge]

no clue, i have no idea what its complaining about...

[toxic stratus]

gonna see what happens if i try to catch the exception in vscode

so it's still a timeout error...

[toxic stratus]

import haxe.Json;
import http.HttpMethod;
import http.HttpRequest;
import http.HttpClient;

class Main {
    static function main() {
        trace("Hello, world!");
        var http = new HttpClient();
        var url = new HttpRequest('https://api.deepcoin.com/deepcoin/market/instruments?instType=SPOT');
        url.headers = ['Host' => 'api.deepcoin.com', 'User-Agent' => 'CustomApp'];
        url.method = HttpMethod.Get;
        http.makeRequest(url, url.headers).then((res) -> trace(res.response.bodyAsString), (err) -> trace(err));
    }
}

-cp src
#-D analyzer-optimize
--library http
-main Main
--js bin/main.js

does this work for you guys?

running node bin/main.js

I get:

src/Main.hx:13: <ref *1> [Error: Unable to create XMLHttpRequest object.] {
  __previousException: undefined,
  __nativeException: [Circular *1],
  value: 'Unable to create XMLHttpRequest object.',
  retryCount: 0
}```

[velvet lodge]

--library hxnodejs

then i get an error though, but a proper http error

and if i remove url.headers = ['Host' => 'api.deepcoin.com', 'User-Agent' => 'CustomApp']; it works fine

[toxic stratus]

adding nodejs i get the same error

oh, my node is outdated lol

okay, i remove that line and I get: ```
node:events:505
throw er; // Unhandled 'error' event
^

Error: connect ETIMEDOUT 90.207.238.183:443
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1158:16)
Emitted 'error' event on ClientRequest instance at:
at TLSSocket.socketErrorListener (node:_http_client:442:9)
at TLSSocket.emit (node:events:527:28)
at emitErrorNT (node:internal/streams/destroy:164:8)
at emitErrorCloseNT (node:internal/streams/destroy:129:3)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
errno: -4039,
code: 'ETIMEDOUT',
syscall: 'connect',
address: '90.207.238.183',
port: 443
}

i keep that line and i get

TypeError [ERR_INVALID_ARG_TYPE]: The "options.headers.host" property must be of type string. Received an instance of Array
    at new NodeError (node:internal/errors:372:5)
    at validateString (node:internal/validators:120:11)
    at calculateServerName (node:_http_agent:349:5)
    at Agent.addRequest (node:_http_agent:259:26)
    at new ClientRequest (node:_http_client:305:16)
    at Object.request (node:https:353:10)
    at haxe_http_HttpNodeJs.request (C:\Users\Jazz\Documents\Projects\haxe\node\test\bin\main.js:575:37)     
    at C:\Users\Jazz\Documents\Projects\haxe\node\test\bin\main.js:1509:10
    at new Promise (<anonymous>)
    at thenshim_js_JSPromiseFactory.make (C:\Users\Jazz\Documents\Projects\haxe\node\test\bin\main.js:2266:10) {
  code: 'ERR_INVALID_ARG_TYPE',
  retryCount: 0
}

but wasn't this solved? hmmm... 🤔

[velvet lodge]

did you update?

[toxic stratus]

apparantly my global http had a .dev file in it for some reason

src/Main.hx:13: {
  body: {
    length: 9,
    b: Uint8Array(9) [
      78,
      111,
      116,
      32,
      70,
      111,
      117,
      110,
      100,
      bufferValue: [ArrayBuffer]
    ]
  },
  retryCount: 0,
  message: 'Http Error #404',
  httpStatus: 404,
  headers: {
    h: [Object: null prototype] {
      'content-type': 'text/plain; charset=utf-8',
      'content-length': '9',
      connection: 'close',
      date: 'Thu, 04 Apr 2024 16:17:14 GMT',
      server: 'prism',
      'trace-id': 'f9c2314ff81fc317e8dd8924502aa73b',
      'x-cache': 'Error from cloudfront',
      via: '1.1 668006c1cb101e4e3461ceae5f2ccbe2.cloudfront.net (CloudFront)',
      'x-amz-cf-pop': 'AMS58-P6',
      'x-amz-cf-id': 'IRCrayZgKXxKU01IcUARgAHXvq-IexUwdJFgbq-GttEgn7NEBoa2Ug=='
    }
  }
}```

now i get that

ohoh

i finally got a response in a isolated environment!

[toxic stratus]

this immediately works as well

i had to add a index.html file

now, for some reason this fails in my actual app

[toxic stratus]

I think i'm getting messed with

I gave up, went and did other things, had a nap and now it is randomly working everywhere

I'm gonna git push everything for a sanity check incase it breaks again

[toxic stratus]

:/

yep, can confirm it is something external

was working before i went to bed

[velvet lodge]

external? I dont really understand why its looking to use XMLHttpRequest if this is nodejs (in electron)

what does your .hxml look like?

[toxic stratus]

I say external because in the space of 5 minutes it went from not working to working to not working

but maybe that's just an invalid conclusion

lemme try the testcase you sent yesterday 🤔

okay, your test case works fine

so something in this setup is causing this weird issue

[velvet lodge]

whats -lib nodejs:git ?

[toxic stratus]

use the git version of the library nodejs

it's akin to -lib haxeui:1.16.0

[velvet lodge]

yeah, i mean what is the lib nodejs?

isnt it hxnodejs?

[toxic stratus]

I just named it nodejs

[velvet lodge]

... ... ... right, sounds odd, but i suppose no issue, i guess you can call them anything locally

[toxic stratus]

haxelib git nodejs https://....

it was just a natural thing

nothing intentional

[toxic stratus]

....and it just randomly starts working again after I fully close and reopen the app

[velvet lodge]

very odd, are you being api throttled or something?

[toxic stratus]

I thought that was the case, but then yesterday whilst it was working and I was 'actually' getting throttled, it told me specifically

[velvet lodge]

no clue then

[toxic stratus]

yeah... it's defo something specific to this project

and something intermittent

it's no longer working now but the test app works fine

oh

you know what

there is a difference

your project i'm running on 4.3.*

this one i'm running on nightly

dang it not the problem, lol, i'll stop rubberducking here

[velvet lodge]

haha, rubber duck away 😄

[toxic stratus]

on native i get {"retryCount":0,"message":"EOF","httpStatus":0,"body":{"length":0,"b":[]},"headers":null,"bodyAsString":null,"bodyAsJson":null}

[toxic stratus]

I have figured out a workaround

package haxe;

#if sys
typedef Http = sys.Http;
// #elseif nodejs
// typedef Http = haxe.http.HttpNodeJs;
#elseif js
typedef Http = haxe.http.HttpJs;
#else
typedef Http = haxe.http.HttpBase;
#end

😅

it still intermittently fails, but it is WAY more consistent if I used the js one

[velvet lodge]

something is buggered in your env tbh...

[toxic stratus]

probably

i entertained the idea of doing a windows reinstall yesterday

but i also don't want to do one

maybe its time, i don't think i've reset it in quite a few years now

[velvet lodge]

i doubt its that honestly, more likely some type of node / haxe + node fuck up

[toxic stratus]

yeah, no idea what it is

i have reinstalled all aspects of the chain

and yesterday i tried doing postman to the API

i intercepted my browser request, loaded it into postman with the same headers and it also failed in postman....

a simple curl request failed as well

[velvet lodge]

oh, so maybe not node then

[toxic stratus]

it's defo some kind of environment problem

[toxic stratus]

i've moved over to core-haxe/websockets

they seem to be working fine for me!

no idea if much has changed with them yet

LOL

some very coincidental comedic timing here

oh that's weird, build passes for hxcpp 🤔

ohh secure server specifically nvm, cool, well was just checking my app anyway

[velvet lodge]

youre using client only right?

[toxic stratus]

yeah

[velvet lodge]

and its "all good"?

[toxic stratus]

for nodejs only

doesn't work for cpp

[velvet lodge]

ah, because of the wss?

[toxic stratus]

yeah

[velvet lodge]

gotcha... ill take a look (maybe over the weekend)

[uncut phoenix]

With hxwebsocket is there a way to send a message every x miliseconds?

haxe.Timer won't work because of threads

(using non core haxe version, this is just the best spot for other ian-ware)

[toxic stratus]

that's odd

I think there's code in ceramic that allows this but I think I've used timers for hxwebsockets

[uncut phoenix]

this isn't a ceramic project

[velvet lodge]

ian-ware
😄

i think i get the issue though... and it should be fixable... i aim to use any "main loop" more, but first i just need to get unit tests fully sorted so changes can be tracked for regressions (which was a serious issue in hxwebsockets - you never knew what you were breaking)

[uncut phoenix]

discord has this thing where it has you send a "heartbeat" based on a particular interval provided to stay connected

[toxic stratus]

that's common websocket behaviour

[uncut phoenix]

i would assume so

[velvet lodge]

yeah, i totally get it... "new Timer(sendMySuperMessage, 1000)" may very well not work "all the time" (on all haxe targets)

[toxic stratus]

do you have an update loop at all?

or is it in the timer

[velvet lodge]

its the timer, well, ish i think - ill have to setup a test project... but im fairly sure a cli app sending a message to a socket server every 1s isnt going to work "out the box"

[uncut phoenix]

i can create an event loop outside of the thread

and do it that way?

kinda hacky, but if it works

[velvet lodge]

ill think there are flags in hxwebsockets (for the server) to use the main loop, these should be extended to the client also, although, is the client threaded?

im not sure it is

doesnt sound like the responisbility of the ws (client) lib

[uncut phoenix]

I think it is?

websoscket

[toxic stratus]

Yo ian

client.post(url).then(
  function(resp:HttpResponse<MyDataType>) {}, 
  (err:HttpError<MyErrorType>) -> trace(err)
);

Thoughts on a change like that?

It would just type the json dynamic

[velvet lodge]

whats the change? the type param?

[toxic stratus]

yeah

[velvet lodge]

i think it should be HttpResponseJson<MyDataType> or something though

assuming json doesnt seem right

[toxic stratus]

oh, make a different type?

[velvet lodge]

well, i just mean that what if the service is returning xml, or something else?

[toxic stratus]

I was thinking of making use of the default type param syntax we have now

so no type paramater specified can just fall back to normal Dynamic

It might lock http lib to 4.3 tho

[velvet lodge]

but what im saying is that, presumably, MyDataType is going to be from json -> MyDataType - in this case

[toxic stratus]

or whenever that syntax was added

[velvet lodge]

which means its going to need to be parsed (somehow), which is fine... but what if i want to use the same syntax for a server that returns xml, or, dunno csv, or yaml, etc, etc

[toxic stratus]

it wouldn't conflict

presumably if you want those types

you would do bodyAsString as opposed to bodyAsJson 😄

[velvet lodge]

its not about conflicting, its about expectations... this is saying "HttpResponse<Bob>" assumes json... and i dont think im happy with that assumption

[toxic stratus]

hmm

[velvet lodge]

HttpJsonResponse<Bob> seems good, just like HttpXmlResponse<Bob> would or HttpYamlResponse<Bob> or HttpMyCustomFormat<Bob>

so it would be something like class HttpJsonResponse<T> extends HttpResponse implements IParsable or something similar (id have to look)

[toxic stratus]

hmm, that's a lot more to work on than expected 😄

[velvet lodge]

its cleaner / more extensible though... assuming Json isnt something i think is "fine"

[toxic stratus]

i understand

was just going off of what's there

[velvet lodge]

ill take a look over the weekend though... i think it should be pretty simple (ish)... i think all the bits are already there

and i do like the idea behind the change, for sure

the rest lib already does it... but i dont like the rest lib anymore... it needs an overview now ive started using it in production apps

so moving this type of stuff to the http lib seems like a really nice idea...

[toxic stratus]

what don't you like about it?

there is something in it that i don't like but i can't put my finger on it

[velvet lodge]

the rest lib?

[toxic stratus]

ya

[velvet lodge]

it makes too many assuptions, its too rigid

[toxic stratus]

I'm not a big fan of the

makeRequest(url).then(function(incoming) {
  incoming.response.body...
}

I should just be able to do incoming.bodyAsJson

[velvet lodge]

in the rest lib, or the http lib?

[toxic stratus]

http lib sorry

[velvet lodge]

https://github.com/core-haxe/http/blob/main/src/http/HttpResult.hx

it looks like i was thinking i might want to expand on that... though i agree i think, the client could happily be in the HttpResponse also - that HttpResult class does smell kinda useless, likely going to break alot of things changing it

(not saying im not going to, just its not a 5 second change, will likely break the rest lib and all my production projects)

[toxic stratus]

yeah, but, it was my first thought. client could just live in response

will break a lot of things for me as well

[velvet lodge]

no pain, no gain i suppose 💪

[toxic stratus]

but i'm ecs so actually, it will be very central 😝

[velvet lodge]

its more the rest lib that it will fuck up totally, and im defo using that (even if its fallen somewhat out of favour with me now)

i dont really ever use http directly in prod projects i dont think

[toxic stratus]

tbh if it's mostly through the rest lib the fix will likely be quite simple

i don't think i need the full fledged rest lib much

[velvet lodge]

not sure, no idea how intertwined things are there - i think quite alot

[toxic stratus]

might be ignorance, I just want to do be able to use rest api's lol

[velvet lodge]

thankfully eveything is UT'd to the hilt, so i can be fairly sure if they pass ill be alright

[toxic stratus]

haxe.http doesn't allow me to do that

[velvet lodge]

well, thats what the rest lib is supposed to be for... but actually, its not too useful unless you can control the client and the server (ie, api building)... which is how i use it... and it its "fine", but still a little esoteric

at somepoint there will be a "rest2" lib, which will look to address all of this stuff... once i get some spare time - at the moment, i defo use it, but its well... ... "alright" (because i do control the full api (ie, client and server)

[toxic stratus]

there's some features that i think would be cool to have but i wonder if you would consider them as part of http over rest

[velvet lodge]

ill take a look at getting rid of HttpResult thought... and also take a look at generic type params for HttpResponse (im quite excited by that one, it looks like a nice change)

[toxic stratus]

something like an auto paginate function would be super cool

[velvet lodge]

i would say that shouldnt be part of either lib... its way too specific... but if it were to go anywhere, it certainly wouldnt be the http lib (but i would be reluctant to add it to the rest lib also tbh)

[toxic stratus]

fair

[velvet lodge]

that said, a "request batcher" could be kinda neat...

[toxic stratus]

it's mostly cause i find paginated results a nuisance

i have a solution that i can do

just never implemented it

[velvet lodge]

yeah, i write alot of threaded request batching - using paging (in python) for one of my clients... it works really nicely, but its not that simple either

[toxic stratus]

something like

oh, trying to jot down a dummy api shows the difficulty in getting that right x)

request.batch.params(['start', 'end']).range(1000, 3000).resultsPerPage(250)

the response would be an array of all the results from each request

but yeah, very specific

[velvet lodge]

very specific

[velvet lodge]

fyi, HttpResponse its completely removable... i dont even every use the .client part of it... anywhere... so i think i was coding defensively

ive removed it locally and i had to make super minimal changes to rest (1 line) and similar changes to my prod apps - all basically the same change, which were these types of things:

.then(result -> {
   var x = result.response.bodyAsJason
})

to

.then(response -> {
   var x = response.bodyAsJason
})

so nothing major at all

i also found a "response.response.bodyAsJson", so yeah, defo happy to remove it

[velvet lodge]

(HttpResult gone)

[toxic stratus]

I do resp.response so many times it bugs me 😂

[velvet lodge]

yeah, im not really sure what the original purpose was, only that i thought i might want "a bunch of stuff" in the HttpResult class... but in reality there has only ever been "response" and "client" and client hasnt been used once anywhere, ever

so anyway, all gone now... defo better

(didnt even move client to httpresponse class, as its just not needed - at least not yet - if it is at some point, i can always whack it in there)

[toxic stratus]

from what I can tell, if you need client you have access to the client anyway from the actual request

[velvet lodge]

yeah, exactly, simply no reason for it to have ever existed, which means that HttpResult would only contain HttpResponse which makes it totally superfluous - hence why its been removed in latest

[toxic stratus]

messing with the json field type

it gets auto parsed into an object!

pretty neat

think i'll probably move hbot away from firebase land

[toxic stratus]

DBEvents.Update('state', record, Query.query($key == field))

You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'key = ?, value = ? WHERE (state.key = ?)' at line 1,

Apparently I have a syntax issue here

what's the way to diagnose this or do you see anything obviously incorrect here?

man discord search in projects is terrible 😭

think i got it, queryexprtosql

nope, that's correct

[velvet lodge]

you shouldnt use queryexprtosql directly - thats what the db plugins use to build the sql

what is "$key" and "field" in your code?

i wonder how Query.query($key == field)) changed into WHERE (state.key = ?) ... ...

like, where did the "state" come from... is that the table name?

[toxic stratus]

key is the column name in the database field is the value

[toxic stratus]

state is the table name

[velvet lodge]

i wonder why it added the table name to the field

[toxic stratus]

didn't I add this a while back?

because it wasn't working for some reason

[velvet lodge]

not sure

what does the actual call to update look like?

(i mean the db-core update call)

[toxic stratus]

https://github.com/core-haxe/db-core/pull/3

[toxic stratus]

                    db.table(table).then((result) -> {
                        
                        trace(value);
                        return result.table.update(query, value);
                    }).then(function(res) {});

[velvet lodge]

can you go into the guys and print out the full sql?

[toxic stratus]

where do i go for that again

[velvet lodge]

somewhere around here: https://github.com/core-haxe/db-mysql/blob/main/src/db/mysql/MySqlTable.hx#L251

[toxic stratus]

right clicking update takes me to an interface 😂

[velvet lodge]

... because its all pluggable 😉

[toxic stratus]

UPDATE state SET key = ?, value = ? WHERE (state.key = ?);

[velvet lodge]

and what does "values" var show trace?

[toxic stratus]

key=next_roundup; value=712;

which is all expected in the db

[velvet lodge]

is value in the db a string... or?

[toxic stratus]

ahh, no

[next_roundup,712,next_roundup]
that's what the values var in mysqltable shows

which appears correct

[velvet lodge]

should be fine i think... but i wonder if 712 is an int in your code and a string (text) in the db?

[toxic stratus]

it is an int

but in the db, it's a json field type

maybe dbcore is incompatible with that?

[velvet lodge]

"711" isnt valid json, dunno if that has any effect?

[toxic stratus]

it is valid!

[velvet lodge]

although its in there so thats seems to be fine

[toxic stratus]

i was pleasantly surprised when i found that out x)

[velvet lodge]

i still think you are passing an int to it, which makes it wrong

[toxic stratus]

note, i can read these values from the table perfectly fine

everything is typed correctly and parses as expected

[velvet lodge]

like, there is a diff between:

WHERE id = 123
and
WHERE id = '123'

[toxic stratus]

oh, i think i have it set to Any

i wonder if it would work better with dynamic

no difference

[toxic stratus]

also, i'm not looking for a where on the number

[velvet lodge]

i think its the type of the value itself... like if you did:

stringField = Std.string(field);
DBEvents.Update('state', record, Query.query($key == stringField ))

[toxic stratus]

i'm "whereing" on the string next_roundup

[velvet lodge]

good point... but the same applies to the "SET"

so where you are setting 712 in the record, i think it should be Std.string(712)

[velvet lodge]

exactly

[toxic stratus]

yeah

[velvet lodge]

'712' would work

[toxic stratus]

it's weird that it let me enter 711 directly in the field

but i also checked a json validator for this as well

[velvet lodge]

but the fact remains, in your code you are passing an int, and it should be a string

[velvet lodge]

but thats in the editor, which presumably is doing "other" stuff

            refreshSchema().then(schemaResult -> {
                var values = [];
                var sql = buildUpdate(this, query, record, values, MySqlDataTypeMapper.get());
                return connection.get(sql, values);
            }).then(response -> {
                log.endMeasure("update");
                resolve(new DatabaseResult(db, this, record));
            }, (error:MySqlError) -> {
                log.error("update", error);
                reject(MySqlError2DatabaseError(error, "update"));
            });

The interesting part here is i do have the schema (refreshSchema), so its possible for it to auto convert, but its not something im going to get round to anytime soon

[toxic stratus]

stringing it also fails

but if i string it in heidi it works

[velvet lodge]

what do you mean "stringing" it? And why does it have quotes around it now in the trace?

[toxic stratus]

turning the int to a string

Std.string

[toxic stratus]

i tried a few ways

[velvet lodge]

im confused as to why it has quotes around it now in the trace

[toxic stratus]

Std.string, "'"+712+"'", '"712"'

all fail

[velvet lodge]

hmmm, i would have hoped / thought std.string would have worked... will have to look deeper into what MySqlDataTypeMapper is doing here

[toxic stratus]

in the db this query goes through successfully so i tried copying the format

[velvet lodge]

... really feels like the std.string should have worked... the typemapper isnt doing much except in the case of bytes

i actually get a constraint failure when trying to add data to json column type

2024-06-19 13:09:42 > ERROR       > db.mysql.MySqlTable > d1101662-2661-a95c-a597-eeff2a023901 > add {
        name : Error,
        message : CONSTRAINT `person.firstName` failed for `persons`.`person`
}

but not when i change the value to "{}" (instead of "Ian")

and when i do that this also work record.field("firstName", 111);

but this doesnt: record.field("firstName", "bob");

... i kinda feel like you are abusing the json column type here... though i dont quite understand why i get wildly different results to you though

[toxic stratus]

how so?

[velvet lodge]

by putting non json things in it

[toxic stratus]

it is json

[velvet lodge]

711 isnt json

[toxic stratus]

it is valid

[velvet lodge]

hmmmmm

[toxic stratus]

it's a property table in mysql

some fields are going to just have a single value

others is going to have a full on object