Rich Text - Uncreatively Named Text Handling Prototype Tool | Graphics Programming | Page 1

molten galleon Sep 7, 2023, 11:12 PM

#

My little prototype project for figuring out the end-all text rendering and input support lib for integrating into my game engine. If you can type it I should be able to render it (and not segfault my engine when pressing backspace on a multi-unit codepoint like I do currently).
https://github.com/forenoonwatch/rich-text

old harbor Sep 7, 2023, 11:13 PM

#

molten galleon My little prototype project for figuring out the end-all text rendering and inpu...

#

i firth

verbal surge Sep 7, 2023, 11:14 PM

#

I see many faces in that zalgo

old harbor Sep 7, 2023, 11:15 PM

#

smiling ones too

molten galleon Sep 7, 2023, 11:15 PM

#

one common side effect of text rendering is hallucinations

old harbor Sep 7, 2023, 11:15 PM

#

heh

molten galleon Sep 7, 2023, 11:16 PM

#

this might also be a good time to test whether my ICU cmake build actually works on msvc

old harbor Sep 7, 2023, 11:17 PM

#

if you need testing on lunix you can hit me up

#

i use arch btw

verbal surge Sep 7, 2023, 11:17 PM

#

When I’m old and I’ll have nothing better to do, I’ll read this:
https://en.wikipedia.org/wiki/Computers_and_Typesetting

Computers and Typesetting

Computers and Typesetting is a 5-volume set of books by Donald Knuth published in 1986 describing the TeX and Metafont systems for digital typography. Knuth's computers and typesetting project was the result of his frustration with the lack of decent software for the typesetting of mathematical and technical documents. The results of this proj...

#

It’s incredible how Knuth got most of it right, but then everyone forgor

molten galleon Sep 7, 2023, 11:17 PM

#

luckily I have my laptop with kubuntu to start with

#

the funny thing is, I'm not even doing it "the hard way" and NIHing everything

#

I'm pretty much at the end of where "all inclusive" stops including all

old harbor Sep 7, 2023, 11:18 PM

#

best way of doing

molten galleon Sep 7, 2023, 11:18 PM

#

you'd think I would just be able to plug a stack of 3 industry standard libs (ICU, harfbuzz, freetype), and go along with my life

#

but it's just the beginning

verbal surge Sep 7, 2023, 11:19 PM

#

Just embed Chromium in your app/game, ez

molten galleon Sep 7, 2023, 11:20 PM

#

that's not good

#

oh, could this be related to the thing sean mentioned the other day about MSVC and utf8

#

verbal surge Sep 7, 2023, 11:23 PM

#

Yeah, UTF-8 and MSVC don’t work that well. I just assume that it can only do ASCII and put UTF-8 in JSON/txt files

#

It’s really weird that they’re doing UTF-8 directly in their source files

molten galleon Sep 7, 2023, 11:27 PM

#

well it is an official unicode org library

#

wow apparently unless you define this, windows.h #defines min and max

#

oh now I'm really in the tough stuff

#

looks like building the ICU data lib gives me error LNK2001: unresolved external symbol _DllMainCRTStartup

#

and all the stuff that interacts with icu-le-hb gives me inconsistent dll linkage, but I'm gonna call that the easy one because I bet I can figure out what define triggers that to line up nicely

molten galleon Sep 8, 2023, 12:16 AM

#

oh boy almost got it building on MSVC

#

in other news, PSA to all you guys, if you don't wanna bloat your repos with raw font data, use file(DOWNLOAD), peep my fonts/CMakeLists.txt for that

#

ok awesome, MSVC build officially works, tomorrow I will fix linux

verbal surge Sep 8, 2023, 6:32 AM

#

molten galleon wow apparently unless you define this, windows.h #defines min and max

It also defines RGB macro 👽

molten galleon Sep 8, 2023, 1:52 PM

#

Alright, I am finally building on Linux, which means I can finally get back into the meat of it

#

So, in order to handle text input properly, the main problem I have to solve is building a TLAS (Text Layout Acceleration Structure), basically something that can let me quickly convert from a pixel pos to a text index, and from a text index back to a pixel pos

#

so far from my investigation into the wacky world of text, it seems like the granularity you need to express in the TLAS is by character break, and if your glyph count is < your character break count you have to just evenly subdivide the glyph bounds to get break positions. The remaining question there is what you'd do otherwise, when your glyph count is > than your character break count

#

#

the two things I'd probably test are the leading edge of the first glyph, and the min(leadingEdge) of all glyphs, and of course I say "leading edge" because it's reversed in RTL text

#

and I'm only thinking horizontal text atm, but for vertical text it should all be the same with the primary and secondary axis flipped

#

#

another interesting thing to note is that the sample algorithm ICU provides for paragraph layout decides that the best way to handle line height is by taking the max(lineHeight) and imposing it on all lines, so here where I render all lines as size 24 and line 2 as size 48, all lines now get way more spacing

#

I'm pretty sure roblox does the same thing from my tests, fancier engines like MS Word don't, I personally don't like it aesthetically but comes with the convenience of being able to very quickly solve for your secondary axis

#

actually it's absolutely not what they do

#

so I won't do that

#

#

and it seems the LineHeight value, which acts as a scalar on top of the computed line height, works relative to whatever the RichText computed for the font

#

so, line height = max(line lineHeight across font runs)

#

meaning I can't use a cheeky uniform (int)(mouseY / lineHeight) vertical lookup

#

minor loss

#

all fine in the name of pretty text

molten galleon Sep 8, 2023, 3:50 PM

#

so, right now, just to be somewhat space efficient I am thinking of making the TLAS (and yes I am gonna continue unashamedly calling it that for lack of a better name) have 2 phases of lookup, a binary search to find the line, and then another binary search to find the X offset

#

when your cursor falls anywhere on the line it counts as being on that line, so you don't need the full text hitbox, but the thing I need to ponder is how to encode the horizontal data

#

in this example here 1 line is pictured as made of 3 text runs, and of course the middle one is arabic script so it's RTL, meaning in terms of string index, the low end of the index is on the right side of the arabic block

#

I guess keeping highlights in byte order is what causes this behavior, and obviously I can technically do better, but I kinda like this little text rendering easter egg that's literally everywhere

#

I guess you start to experience an interesting tradeoff there, and I need to think more to see how their implementation probably handles it

molten galleon Sep 8, 2023, 4:56 PM

#

okay I think it was perhaps obvious, it's because highlight box rendering is completely disjoint from selection, and because your cursor is basically implemented as 2 indices, selectionStart and cursorPosition, so if you have:

AB[CD[EFGHI
  ^  ^

with DEF RTL as above, your byte layout is

AB[CFE]DGHI
  ^   ^

(selectionStart and cursorPosition marked)

#

so now you don't have to depend on render information about visual runs to highlight a contiguous block of text to do a non-visual operation (e.g. ctrl+x, delete, etc)

molten galleon Sep 8, 2023, 9:37 PM

#

hmm been prematurely bikeshedding the TLAS idea, I think I need to store lookups for pixel indices by line -> by run -> by glyph so I can get an O(log n) lookup but then I was thinking... why?

#

I mean you do several O(n) passes on your text anyway as part of rendering it, and I'll deal with Big Text when I get there

#

plus when dealing with BigText you still need to do a layout pass at least once to figure out what portion fits in frame

molten galleon Sep 9, 2023, 4:27 PM

#

ugh, just doing horizontal char advances and the logic is already stupidly convoluted

#

not to mention it doesn't even work right yet

old harbor Sep 9, 2023, 4:34 PM

#

: )

#

famous words

#

"doesnt even work right yet"

molten galleon Sep 10, 2023, 12:45 PM

#

apparently nvim secretly adds a newline when saving a file and doesn't want to show it

#

#

probably great to avoid that whole "file needs to end with a newline" thing with old C compilers

#

wow I messed up my logic so hard because I trusted the line counter in nvim instead of using n++

#

okay at least the basics work like I expect

#

however I still need to fix

many-to-one char:glyph layouts
layout-generated line breaks as opposed to \n (require dealing with the affinity bit like in the article)
word break logic shouldn't actually rely on the ICU wordbreak iterator, but that's unrelated to cursor positioning

molten galleon Sep 10, 2023, 1:37 PM

#

after this little study (n = 3), it seems that affinity is generally a mouse click only thing, and otherwise advancing is doing by the cbrk iterator as normal

#

but you can clearly tell that you're on the same character position with a different bit affinity based on your mouse cursor

#

and I can tell the reason for this choice is entirely because otherwise you would create a dependency between your visual layout info and your text, whereas doing it this way means you only depend on the content of your text to determine the next cursor position, except with your mouse which is inextricably tied to doing a visual query

molten galleon Sep 10, 2023, 5:08 PM

#

and this covers many chars to one glyph, in LTR and RTL, the first code so far that I have to change algorithms explicitly due to direction because of the format icu::LayoutEngine gives glyph data, which is a problem for another day. First 2 are Segoe UI الله, second is ﬃ, which it seems notepad a) ligaturizes and b) splits exactly like I expected

verbal surge Sep 10, 2023, 7:54 PM

#

molten galleon

https://stackoverflow.com/questions/14171254/why-would-vim-add-a-new-line-at-the-end-of-a-file
Hmm, interesting

Stack Overflow

Why would Vim add a new line at the end of a file?

I work with Wordpress a lot, and sometimes I changed Wordpress core files temporarily in order to understand what is going on, especially when debugging. Today I got a little surprise. When I was r...

molten galleon Sep 10, 2023, 9:07 PM

#

interesting

#

study #2: word break iteration, and a closer look at boundaries of bidi text runs. Here's where I'm actually starting to get divergent behavior between my samples:

Word jumps are always to the beginning of a word (in byte order). Forward goes to the beginning of the next word, back goes to the beginning of the previous word. Easy enough and onsistent among all 3.
Words are blocks of non-whitespace characters, word boundaries are just found by checking state transitions of whitespce<->non-whitespace and not using a word brk iterator.
Chromium (#2) is the only engine of the 3 that doesn't respect the non-breaking space in buffi ng. I am currently of the opinion this is incorrect.
Notepad (#1) and Chromium place the cursor at the end of the buffi ng run for the start of the arabic run, while Roblox (#3) places it at the beginning (right side) of the arabic run. I don't think there's a right answer here but it's an interesting divergence.

#

bonus sample, VS code, very much the black sheep, forward jump goes to the end of the next word, backwards jump goes to the beginning of the previous word. NBSP compliant. Interesting behavior, I will have to keep it in mind for when feature creep keeps me working on a text editor for the next 5 years

molten galleon Sep 11, 2023, 12:28 PM

#

study #3: Home/End, Ctrl+Home/Ctrl+End

Ctrl+Home and Ctrl+End are really simple, just jump to char 0 or end of string. Nothing much to say about it
Home/End keep you on the current line and don't always drive you towards the start/end of the string like word jumping. Simple enough
Home/End respects soft line breaks (confirmed in all 3 cases), meaning that despite its simplicitly a Home/End jump is actually a post-layout query

#

https://docs.gtk.org/Pango/pango_rendering.html
Pango has a nice breakdown of the text layout/rendering pipeline, definitely worth citing

#

speaking of which, despite pango seeming like the de facto lib to use for a text stack, it has 0 documentation anywhere on input support, same with cairo or graphite or anything of that ilk

molten galleon Sep 12, 2023, 10:17 AM

#

(Notepad: l-a test)

#

(Notepad: Cluster test)

livid patrol Sep 12, 2023, 10:18 AM

#

molten galleon (Notepad: l-a test)

no singing audio, 5/10

molten galleon Sep 12, 2023, 10:18 AM

#

I make sure to turn my audio off for these so I don't have to re-record them when I finally put together an article

#

although I might have to crunch them down to gifs so I don't accidentally kill my github storage limit

#

(Notepad: l-M test)

#

(Roblox: l-M test center-justified)

#

(Roblox: l-a test left justified)

#

(Chromium: l-M test center justified)

#

(Chromium: l-M vertical alignment preservation)

#

(Chromium: l-a test)

#

study #4: Vertical Text Movement

Like mentioned in The Article, vertical text positioning is a pixel-position-based operation
All 3 samples show similar general behavior: The final cursor position is moved towards the closest character boundary, whether that's in front or behind the current cursor position
Chromium is a little different in that it clearly caches your horizontal position upon a vertical move, and always picks the closest position on each line to this initial cached position. You can see how it differs in the l-a test between the 3. The notepad sample cascades forwards as you move up and down because the cursor always ends up closer to the start of the next character, the Roblox sample does the opposite, likely due to the different font (Source Sans Pro), while Chromium shows no signs of cursor cascading.

livid patrol Sep 13, 2023, 6:54 PM

#

Can you render Egyptian Hieroglyph D053

old harbor Sep 13, 2023, 6:54 PM

#

aka the man-in-the-wall-symbol most likely

livid patrol Sep 13, 2023, 6:55 PM

#

𓀓

old harbor Sep 13, 2023, 6:56 PM

#

yeah exactly what i thunk

#

a man climbing out of a wall

molten galleon Sep 13, 2023, 10:48 PM

#

yeah I just have to map the font

#

#

speaking of which, this is the best tool ever for figuring out fallback schemes if I didn't already post this

old harbor Sep 13, 2023, 10:49 PM

#

thats the browser dev tools neh?

molten galleon Sep 13, 2023, 10:49 PM

#

yeah

#

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Fonts
then navigate to here to find the file

#

#

then I just add it to my generic fallback faces

#

just paste it in my sample.txt (looks tiny in n++)

#

#

and viola

old harbor Sep 13, 2023, 10:52 PM

#

noice

molten galleon Sep 13, 2023, 10:52 PM

#

bonus showing off my cursor nav

livid patrol Sep 13, 2023, 10:54 PM

#

sorry, but that's not Egyptian Hieroglyph D053. I didn't want to spoil it

molten galleon Sep 13, 2023, 10:54 PM

#

https://graphemica.com/𓀓

𓀓 | egyptian hieroglyph a016 (U+13013) @ Graphemica

𓀓 | | egyptian hieroglyph a016 (U+13013) @ Graphemica

#

ah you're right, it's a016

#

https://graphemica.com/𓂺

𓂺 | egyptian hieroglyph d053 (U+130BA) @ Graphemica

𓂺 | | egyptian hieroglyph d053 (U+130BA) @ Graphemica

#

looks like Segoe UI Historic doesn't map it

#

let's see if there's a noto hieroglyph

#

#

💀

#

why did microsoft omit this hieroglyph frog_think

old harbor Sep 13, 2023, 11:08 PM

#

its on the index

molten galleon Sep 13, 2023, 11:16 PM

#

the valve index?

old harbor Sep 13, 2023, 11:32 PM

#

: )

#

the bad things index aka list

molten galleon Sep 13, 2023, 11:41 PM

#

I wonder if this hieroglyph was on the rosetta stone

old harbor Sep 14, 2023, 12:49 AM

#

it might have been a dick move 😛

molten galleon Sep 15, 2023, 12:28 PM

#

wow people make writing articles look easy

#

I've been spending a few days getting all my notes into article form and I'm still barely up to where my code is at

old harbor Sep 15, 2023, 4:56 PM

#

it takes time

molten galleon Sep 15, 2023, 5:04 PM

#

https://utf8everywhere.org/

#

btw good article

old harbor Sep 15, 2023, 5:26 PM

#

i always read graphmemes when i see graphemes

molten galleon Sep 15, 2023, 11:30 PM

#

https://forenoonwatch.github.io/text_editing.html
So much left to say and do but it's already the longest thing I've ever written

livid patrol Sep 15, 2023, 11:39 PM

#

Really cool how you go into details and investigating how each editor handles various "edge cases"

molten galleon Sep 15, 2023, 11:40 PM

#

it's pure paranoia tbh, it all comes from just wanting to make it work right once lol

#

and of course my impression is that there's clearly a de facto "standard" that you'd expect out of a text editor, subconsciously

#

but no one has ever documented it

#

except maybe john chromium and john win32 textbox

livid patrol Sep 15, 2023, 11:42 PM

#

It shows that you have attention for detail

#

It's a big part of what makes or breaks a text editing experience

molten galleon Sep 16, 2023, 12:11 PM

#

Hmm rereading this I realize that my word break checks are nowhere near rigorous enough actually, I haven't tested non-alphabetical characters without whitespace. My dissertation is in shambles

#

also writing this made me realize that I need to change my font family file schema, and that I still don't have a good answer for affinity

molten galleon Sep 17, 2023, 7:31 PM

#

https://pomf2.lain.la/f/10pc7sni.gif

#

today we're gonna talk about implementing mouse selection, but first we're gonna have to talk about parallel universes

verbal surge Sep 17, 2023, 8:51 PM

#

molten galleon today we're gonna talk about implementing mouse selection, but first we're gonna...

Selecting this whole string in 0.5 LMB press

molten galleon Sep 24, 2023, 10:29 AM

#

just 2 sections left of the blog post and I'll have the preliminary research done (and I'll just have to implement everything to match)

#

also noticing in this shitty .NET banking program that some text boxes don't let you CTRL+A so I can't fast-clear them

#

also my work PC and secondary monitor are different resolution and DPI, I now understand why people put so much effort to do DPI-aware scaling because guess what it also doesn't do correctly

#

but that's a story for another day

old harbor Sep 24, 2023, 11:50 AM

#

ctrl+a being disabled could mean nobody wants you to extract (copy paste) from there or paste into

#

could be a sensitive field, password box or there is a need for users to enter manually every time because of various reasons

#

dpi scaling is also a pain in the ass, because there are like 3 or 4 modes and it depends on what OS and version you are

molten galleon Sep 24, 2023, 12:30 PM

#

it's the filter box for table/column names

old harbor Sep 24, 2023, 12:46 PM

#

ah

#

custom wndproc 😛 and ctrl+a simply not handled, is that a component set from ComponentOne/Telerik/TheOtherThirdBigCompetitor?

molten galleon Sep 24, 2023, 12:56 PM

#

hard to say but it's shitty corporate software

#

the worst thing by far is that they disallow you from modifying the databases that back the program (even though you can see them) because it violates their service agreement or whatever, however they don't utilize this assumption at all for performance i.e. for caching, and everything falls through to who knows how many queries to do basic shit

#

and in the new versions they made the latency worse because they queue basic operations into the scheduler of the HPC cluster instead of doing it on the app server

#

which picks up jobs like every 30-60 seconds

#

and now here I am, sunday morning 9am with 2 other guys from my team painstakingly making manual edits through the shittiest UI known to man

old harbor Sep 24, 2023, 1:42 PM

#

the fick

#

why on a sunday

#

are you on some oncall shit?

molten galleon Sep 24, 2023, 1:46 PM

#

just how the crunch goes, it takes so long and we have to get it done before the end of the month because the system is used to do monthly reporting to the government

#

banking in a nutshell

#

we have like ~1 week window per month where analysts aren't using the system

#

so we have to crunch every second

molten galleon Sep 24, 2023, 1:48 PM

#

molten galleon the worst thing by far is that they disallow you from modifying the databases th...

but it all relates to this because we're doing a major schema change

#

and what would be a 5 minute SQL script run and test is a 3 man, week-long death march

old harbor Sep 24, 2023, 2:39 PM

#

: (

#

i hope you get at least well compensated for this weekend bs

old harbor Sep 24, 2023, 2:55 PM

#

hmm stumbled upon this neogfx thing https://neogfx.org/

neoGFX C++ Application/Game Engine and Development Platform

neoGFX Cross-Platform GPU-Oriented C++ Application/Game Framework

molten galleon Sep 24, 2023, 3:04 PM

#

nice, this guy is going pretty hard

#

however the mark of the beast

old harbor Sep 24, 2023, 3:04 PM

#

: D

old harbor Sep 24, 2023, 5:30 PM

#

#

i wonder what ui thing that is

#

its from this project https://github.com/PlasmaEngine/PlasmaEngine

#

seems to be a highly customized dear imgui

molten galleon Sep 24, 2023, 5:40 PM

#

damn not bad

old harbor Sep 24, 2023, 5:41 PM

#

with some sort of layouting engine around it and event/signals ala Qt

verbal surge Sep 25, 2023, 7:36 AM

#

old harbor hmm stumbled upon this neogfx thing https://neogfx.org/

ahh, that thing, their YouTube channel is cursed

old harbor Sep 25, 2023, 10:24 AM

#

verbal surge ahh, that thing, their YouTube channel is cursed

indeed 😄

molten galleon Sep 25, 2023, 10:28 AM

#

yeah its got a tiny amount of schizo particles

#

but that's part of the fun of engine dev I guess

verbal surge Sep 25, 2023, 10:31 AM

#

Most sane "my own GUI" developer

molten galleon Sep 25, 2023, 10:34 AM

#

so true

old harbor Sep 25, 2023, 11:02 AM

#

but he hired a cute chic in front of some cheap ass green screen setup to convey the sales pitch

molten galleon Oct 14, 2023, 12:30 PM

#

thank god I took the time to write my sharticle and document all these dumb details, otherwise I would've forgotten everything by now

#

I've been converting the data model for paragraph layout info to my own instead of coping with ICU's weird ass decisions

#

for some reason ICU opts to reverse the in memory order of RTL text indices, which literally helps nothing

#

every algorithm is only made worse by it

#

and I had to basically branch to 2 different impls for LTR or RTL

molten galleon Oct 14, 2023, 12:58 PM

#

also intra-cluster positioning is a lot more complex than it initially seemed when I trusted ICU's data, ICU's LayoutEngine gives you 1 glyph index per Code Unit (as in per u16 value, not per codepoint), with unmatched mappings being tombstoned with 0xFFFF

#

and this technically almost works out because ICU forces you to deal with UTF16 and none of my test data (Arabic and Latin) is multi-CU in UTF16, but arabic is 2CU in UTF8

old harbor Oct 14, 2023, 1:00 PM

#

its quite fascinating how complex fonts and icuisms actually are

molten galleon Oct 14, 2023, 1:00 PM

#

it's how complex writing is

old harbor Oct 14, 2023, 1:00 PM

#

and that i know 0 about it, besides characters in fonts have a bunch of properties

#

hehe

molten galleon Oct 14, 2023, 1:01 PM

#

because ancient people didn't consult the unicode org before putting symbols on bones and clay tablets

old harbor Oct 14, 2023, 1:01 PM

#

they should have!

#

i wonder how weird all this will be in another 2k years

molten galleon Oct 14, 2023, 1:05 PM

#

yeah I wonder, the digital age has already changed typography pretty significantly

#

things like dictionaries and Unicode and digital fonts definitely do a lot to slow the evolution of written language

#

https://en.wikipedia.org/wiki/Arabic_chat_alphabet
stuff like arabizi is an interesting evolution though

old harbor Oct 14, 2023, 1:20 PM

#

i think you can do PhDs about this stuff

molten galleon Oct 19, 2023, 9:23 AM

#

finally got all the different cursor navigation types working, including positional affinity (picking a side when a position in a string represents 2 distinct visual positions)

#

definitely the majority of the complexity

#

but this time it's using my own layout data model so it shouldn't need any further changes when I convert to UTF-8, which is the next step

old harbor Oct 19, 2023, 9:51 AM

#

fonts and icu in general must be a pain in the ass, i admire your work : )

molten galleon Oct 19, 2023, 9:52 AM

#

trust me they are lmao

#

what I'm about to do is probably the most sketchy and error prone part

old harbor Oct 19, 2023, 9:52 AM

#

i see that on a regular basis when you edit shit in any editor

molten galleon Oct 19, 2023, 9:53 AM

#

which is implementing some of ICU's forced UTF-16 algorithms to work in UTF8 and making sure I don't regress

old harbor Oct 19, 2023, 9:53 AM

#

sometimes ursors go left, sometimes not, somtimes it works majeekally sometimes it does not at all

#

as if there is no consistent rule somehow

#

or a lot of western software vendors dont give a fuck about RTL

molten galleon Oct 19, 2023, 9:53 AM

#

what's funny is that you can tell how much VS Code in particular just did their own thing

#

I strongly suspect they don't have a proper unicode dataset

old harbor Oct 19, 2023, 9:54 AM

#

heh

molten galleon Oct 19, 2023, 9:54 AM

#

but they have interesting decisions that fit only with a code editor vs. a general text editor

old harbor Oct 19, 2023, 9:54 AM

#

isnt MS part of the consortium?

molten galleon Oct 19, 2023, 9:54 AM

#

yeah but I don't think that trickled down to VSC

old harbor Oct 19, 2023, 9:55 AM

#

yeah

molten galleon Oct 19, 2023, 9:55 AM

#

they have some very clearly NIH editor decisions

old harbor Oct 19, 2023, 9:55 AM

#

tbh,it wouldnt surprise me that they data they collect via telemetry just told them that no many RTL isms are happening really, even if its used in major RTL countries

#

and therefore it has no high prio to get it right

molten galleon Oct 19, 2023, 9:56 AM

#

but some are interesting, like instead of using the cursor pixel pos for vertical navigation they use the cursor index in the line

#

before I read about it/tested it I actually thought that's what most programs did

old harbor Oct 19, 2023, 9:56 AM

#

ah

molten galleon Oct 19, 2023, 9:56 AM

#

but I can see why they'd pick it as a code editor feature since code is way more structured by char index

#

but for a 'normal' text editor you use the pixel pos because A) it 'feels' better for the user and B) it's compatible with different line justification

#

like if your lines are all centered in the text box

#

and yeah I think for code editors in general international text is a lower priority

old harbor Oct 19, 2023, 9:58 AM

#

yeah usually text is also monospaced, i would assume

#

besides the 3 excentric people who use some handwrittenstyle font

molten galleon Oct 19, 2023, 9:59 AM

#

in VSC I couldn't tell if it was, it didn't seem monospaced

#

but notepad++ is super monospaced

#

and the arabic text looks super compressed

old harbor Oct 19, 2023, 10:00 AM

#

ah interesting

molten galleon Oct 19, 2023, 10:00 AM

#

one thing I haven't bothered with yet (since it's a huge optimization rabbit hole I'm nowhere close to tackling) is fast paths for huge text

old harbor Oct 19, 2023, 10:00 AM

#

perhaps people usually code in latin, and only comment in CJK/arabic/nonlatin

molten galleon Oct 19, 2023, 10:00 AM

#

monospaced text is convenient because you can use simple scans on the string to roughly figure out what you need to do more expensive computations on

old harbor Oct 19, 2023, 10:01 AM

#

hmm perhaps you could open an issue in vscode and ask how they did or what their train of thought is/was wrt to all this, as a question

molten galleon Oct 19, 2023, 10:02 AM

#

yeah, some of the things I looked into are definitely open source and I could dig into that but also huge and fairly annoying to read

#

like chromium

#

there's also stuff I didn't even consider, like gtk or libreoffice probably both have huge discussion chains about this stuff

#

but they're probably deep in mailing list hell

old harbor Oct 19, 2023, 10:21 AM

#

i can imagine

molten galleon Oct 19, 2023, 8:52 PM

#

oh my god the ICU bidirectional iterator test code is 5000 lines long

#

old harbor Oct 19, 2023, 9:12 PM

#

sheesh hehe

#

but +1 for tests

molten galleon Oct 19, 2023, 9:55 PM

#

no way I can port this crap without tests

molten galleon Oct 20, 2023, 12:24 PM

#

"Java programmer was here"
"how do you know?"

#

hmmm, time to strategerize

#

currently porting the BiDi and script run finder tests to Catch2 so I don't have to mess with putting together a CMakeLists for ICU's ctestfw

#

originally my plan was to take in the uscript and ubidi code as my own and port it to UTF-8 as part of my own lib, but considering how complex they both are, and considering the complexity of updating ICU versions from there on, I should probably do the smart thing and just do the minimum possible work to port them from using a UChar* string to using UText like the rest of their not-awful API does

#

that probably has the highest chance of being accepted as a PR as well, although that's not strictly a priority since I already have a somewhat altered fork I can't upstream in whole

#

I gave their existing PRs a quick look and unfortunately no one has submitted that yet

molten galleon Oct 20, 2023, 6:49 PM

#

tfw some of the BiDi tests actually fail but ICU doesn't emit them properly as failures and just sets some random bool for verbose logging

#

ended up building ctestfw after all

#

maybe it's because I'm using cintltest and not intltest

#

#

yeah awesome, one test case fails, no one catches it

#

thanks ICU

molten galleon Oct 21, 2023, 10:16 AM

#

hmm okay I understand why they never ported it to UText, the implementation is really tied to u16, I might have to go with my original option and just maintain my own secondary set of ubidi_ functions that are implemented as UTF-8 at best

#

man I'm really in for it now

#

#

at least they left some hints

#

you can also tell this is super old (minus the fact the file create date is listed as august 6 1999) because they list "any code point always needs the same number of code units" as an invariant of UTF-16, which was an assumption made back in the day before they realized they were going to need > 65536 codepoints

#

though that 2nd assumption is probably the most annoying thing about UTF-16 from the perspective of porting it to UTF-8, so many 'important' characters fit into 1 CU in UTF-16 that these algorithms are implemented without decoding the codepoints as they iterate

#

https://www.unicode.org/reports/tr9/
what a pain

old harbor Oct 21, 2023, 10:53 AM

#

: (

#

before yielding to insanity, perhaps stop and make the stuff you have so far work better/faster/nicer/ provide better tooling or wrap it in a library of sorts

livid patrol Oct 21, 2023, 10:55 AM

#

deccer wants to steal your work domeboy

#

https://tenor.com/view/gríma-wormtongue-lotr-lord-of-the-rings-hes-not-welcome-gif-16020804

Tenor

#

||jk hehe||

molten galleon Oct 21, 2023, 11:03 AM

#

there's basically enough to go off of so far

#

it's just UTF-16 based

#

which means any interaction with the string has to be describe in U16 code unit indices

#

meaning rich text/formatting, cursor positioning, even if the rest of your app is in UTF-8

#

I just have to do a 1:1 conversion of the script run and bidi run algorithms to UTF-8 and basically drop them in where the originals used to be

#

and everything will be perfect

#

though the bidi algorithm is probably one of, if not the most complex parts of ICU

molten galleon Oct 28, 2023, 11:08 AM

#

I've just finished adding a copy of the paragraph layout builder that doesn't depend on icu-le-hb/layoutex and populates the layout info by invoking harfbuzz and ICU directly, still UTF-16 though, but at least it's written in a way that I can hopefully just replace references to UTF-16 with UTF-8 and everything will magically work

#

I need to really creep up on the UTF-8 thing because porting the bidi/script algorithms will invariably suck

#

#

but at least I'm slowly building up a good test suite

#

the next step is probably to make "emulated" UTF-8 which uses the UTF-16 functions but just reconciles all the character indices after, which will give me a good set of reference data

#

but ultimately that impl is not a good stopping point to get UTF-8 compatibility because both UTF-16 and UTF-8 have indeterminate string index to codepoint mappings so I'll have to do a ton of linear scans through the src and dst strings to fix up the places I use char indices (which are a lot)

#

also, as far as FOSS implementations of the Unicode Bidirectional Algorithm go, there are only 2 I can find period, ICU's which is in UTF-16, and GNU fribidi, which is UTF-32, neither of which are any more useful than the other because text editing is like the one use case where the implementation encoding actually matters

#

but fribidi is LGPL so it's not "avoid looking at" tier, at least

#

also it's worth nothing I'd more or less be done if I just coped and used UTF-16 as my text editing format, as I assume most applications do

molten galleon Oct 29, 2023, 2:08 PM

#

almost done with encoding nonsense, I have a build_paragraph_layout_utf8 that is almost free of linear scans for encoding fixups, just have to deal with the bidi algorithm finally...

#

porting the ICU script run iterator took all of 5 minutes and even helped me save a completely useless malloc it does

#

the bidi iterator will not be so kind

molten galleon Oct 29, 2023, 9:34 PM

#

at about where I expected, not so bad overall, gonna have to start making calls about policy and being careful about data from this point though

#

all that's left to convert are things like this where it expects to read a char16_t and check if it's a bidi control char, which would be a 1-CU value in UTF-16 but not so in UTF-8, so in places like these I'm gonna have to carefully convert the whole algorithm to walk by codepoint

#

and make sure that everything that uses the visual/logical maps also walks the mapping by codepoint

#

some things to look at later
https://github.com/Tehreer/SheenBidi
https://github.com/servo/unicode-bidi
https://www.unicode.org/L2/L2001/01218-Bidi.pdf
https://www.w3.org/International/articles/inline-bidi-markup/uba-basics

#

this w3 article is really good, worth the read

old harbor Oct 29, 2023, 9:50 PM

#

earlier today, i was tempted to start yet another project : (

#

an opengl based text editor : )

#

with code selection and all that shit obviously

#

but, then i had to think about what it means and what rats tail it all is hehe, thats when i remberd icu and DR and his fight-the-icu-isms

molten galleon Oct 29, 2023, 9:52 PM

#

yep

#

but it's soon getting to a happy place

#

this is like the worst part by far

#

...actually there's a ton more

#

this is one of those "oh well this is a tight project I can finish and move on to other stuff" occasions

#

I thought I'd just be writing some glue code between libs

#

but the more you look the more things you can do/should do/want to do with text

#

there's the editing logic, the layout logic (what I'm dealing with now), I still need to take another pass at formatting, another pass at rendering, memory management, threading, etc etc

old harbor Oct 29, 2023, 9:55 PM

#

i got rid of that inception of mine, i can barely code any simple shit

molten galleon Oct 29, 2023, 9:55 PM

#

stuff like threading and memory management can get super out of hand too because your optimization model basically depends on what kind of text you're dealing with

#

e.g. for something like a game, or a program like excel, you have lots of short strings you run layout on

#

but stuff like winword or an IDE have more or less a bigass string you need to process in a completely different way

old harbor Oct 29, 2023, 9:56 PM

#

yeah

#

i can imagine

#

i was also thinking about windows the os, earlier this morning

#

about how it handles all the controls and shit, since those are all handles... and each of them can receive messages

#

even "labels" to represent text

#

its quite unrelated to text rendering itself or icu in any way

molten galleon Oct 29, 2023, 9:57 PM

#

newer versions of windows actually do ship using ICU lol

#

old harbor Oct 29, 2023, 9:58 PM

#

but i was also wondering how the os processes all of these things... moving the mouse must triger som "foreach (handle : visiblehandles) { sendmousemessage(handle);"

#

hehe

#

i think there were even bug reports about the icu parts windows shipped in the past years

#

and people exploiting it

molten galleon Oct 29, 2023, 10:00 PM

#

yeah I definitely think they have all sorts of acceleration structures to filter events spatially and by what's focused though

old harbor Oct 29, 2023, 10:11 PM

#

they have it for paint events too at least

#

still crazy how fluid it all works

#

even with 150k, 200k handles flying around

molten galleon Oct 29, 2023, 10:14 PM

#

I think that's been around since the earliest versions of windows, back when painting was super expensive and they had to be careful about overdraw for simple window rects

old harbor Oct 29, 2023, 10:14 PM

#

yeah

#

since win 1.0

#

if you ignore all the isms windows has/had/willhave, its actually quite a cool system

#

that messaging thing

molten galleon Oct 29, 2023, 10:16 PM

#

yeah

molten galleon Oct 30, 2023, 12:09 PM

#

I have started to feel my hubris of lazy copy-paste porting the ICU bidi algorithm...

#

though I think this lib https://github.com/Tehreer/SheenBidi might be the new area of investigation, and I think I really need to start building good benchmarks, since the point of contention with the ICU algorithm is: "if I am working with natively UTF-8 data, is it faster and/or less memory intensive to work with a UTF-8 native implementation of the BiDi algorithm, rather than allocate and fill a buffer with a different encoding, and fix up indices after"

#

though I would have to figure out if SheenBidi is :

any faster or less memory intensive to begin with
actually has non-naive handling of UTF-8 to begin with (e.g. it doesn't just pack a buffer with its own UTF-16/32 crap before running the algo)
actually up to date with the standard without modification, ICU UBiDi has some generated data tables inside it and I might have to see if I need to feed them through. Since this is disconnected from the unicode standard and the unicode standard is a live spec, basically any decision I make is gonna require continuous fixup

old harbor Oct 30, 2023, 1:00 PM

#

i admire your dedication

#

font/unicodeisms would bore me to death

molten galleon Oct 30, 2023, 1:02 PM

#

I'm trying to weasel my way through it, this is an investigation into how, really

#

perpetual search for "why doesn't a library exist that just does [x]"

old harbor Oct 30, 2023, 1:03 PM

#

: )

#

are you going to write about the findings too?

molten galleon Oct 30, 2023, 1:03 PM

#

yeah its worth noting

#

though at some level it's a sort of triumphant "this is how I figured out how to avoid learning how to implement the bidi algorithm myself in the most optimal way possible"

old harbor Oct 30, 2023, 1:04 PM

#

hehe

molten galleon Oct 30, 2023, 1:09 PM

#

though right now it's also not just "can I find a library to do x for me" it's "can I find a library to do x for me with as little waste as possible" because everything to do with ICU does so many extra conversions, allocations, data copies/shuffles, etc etc

old harbor Oct 30, 2023, 1:15 PM

#

and build up some sort of modular framework perhaps?

molten galleon Oct 30, 2023, 1:17 PM

#

yeah that's the goal

#

though realistically if you want it to be The Best™️ for your use case you'd have to fork it and tear it to shreds

#

making my lib stack not wasteful is the first half of the puzzle, the other half is making it multithreaded and whittling down its execution time further

old harbor Oct 30, 2023, 8:53 PM

#

it feels like i might have posted this one already somewhere, but i am not sure : ) https://github.com/vchelaru/Gum

GitHub

GitHub - vchelaru/Gum: Flexible layout tool for creating UI on any ...

Flexible layout tool for creating UI on any platform - GitHub - vchelaru/Gum: Flexible layout tool for creating UI on any platform

#

its visual editorisms reminded me of ugly uis and of your taste of ugly uis 😄

#

the editor is WinForms

molten galleon Oct 30, 2023, 9:26 PM

#

http://gumui.net/

Gum

Victor Chelaru

Home

#

unfortunately his webpage is modern

old harbor Oct 30, 2023, 9:53 PM

#

: )

molten galleon Oct 31, 2023, 10:08 AM

#

hmm I need to build some good test data

#

luckily the ICU tests are full of nice little strings to copy paste to start with, but to get a good profiling range I need to build some test data in various sizes, since the test strings in ICU are generally classified as "tiny"

#

and from the shitty preliminary benchmarking and profiling I've been doing, in the "tiny" case, basically nothing even registers on the clock except the allocations, which is exactly what I expected to start off with

#

since my main angle of attack was just to find whatever ways to ellide/remove allocations

#

I think the easiest way is to just randomly combine the little test strings into a really long string and then test views into it of different lengths

#

#

I think if I randomly assemble a string up to 16MiB that should be sufficient

#

and run profiles/unit tests on powers of 2 or 4 starting from somewhere around 64-256 bytes

#

I also need to figure out if Catch2 or Google Benchmark let you name benchmarks at runtime, I've only ever seen them use string literals like

BENCHMARK("Bench Name") { do_thing(); };

so I might have to paste each test bucket manually

molten galleon Oct 31, 2023, 1:49 PM

#

already not passing basic equivalency tests on basic strings... fuck fuck fuck

#

the silver lining is that unlike ICU, sheenbidi seems a lot simpler to fork and mess with

#

although it's quickly turning from bold and brash to belongs in the trash

molten galleon Oct 31, 2023, 6:34 PM

#

gonna really have to think about this one

#

I'm slowly starting to understand the UBA though

molten galleon Nov 1, 2023, 4:24 PM

#

Okay I think I've somewhat triaged the extents of the differences:

the UBA Paragraph parsing rules are implemented identically in both, SheenBidi actually does them a little better because it gives me the separator offsets which I need for layout later
Calculating logical level runs is a pretty simple operation to do (although sheenbidi doesn't provide it directly), and it seems to work out identically if I memcpy ICU's levels over SheenBidi's
Level gen is almost identical between both, actually, with 2 differences: ICU has some kind of post-processing called adjustWSLevels which performs some level recalc for non-graphic characters, for these test strings it causes the majority of the level diffs. The other difference is that ICU seems to have a fastpath to early out for monodirectional runs which causes some suboptimal levelgen in SheenBidi but the directionality stays correct
Visual run gen is different in ICU and SheenBidi even using the same level data, I haven't been able to find a root cause for this yet though

molten galleon Nov 2, 2023, 10:30 AM

#

#

hmmm this honestly surprised me, taking a break from figuring out how to reconcile correctness and looking purely at speed, ICU UBiDi is actually significantly slower across the board

#

to the point I actually had to tone down my test string max size from 16MiB to 1MiB in google benchmark, and all the way down to 16KiB in catch2

#

in fact I had to get google benchmark because I thought I broke catch2 because I thought it got stuck somewhere

molten galleon Nov 2, 2023, 12:06 PM

#

Hypothesis 1: The reason for the disparity is because UBiDi calculates across paragraph boundaries, but SheenBidi works per paragraph. Need to check this
Hypothesis 2: Some of the visual run transpositions are pretty far, I might have to completely redo my editor memory model since I've only been testing a max run level of 1

molten galleon Nov 2, 2023, 12:37 PM

#

#

Hypothesis 1: somewhat correct?

#

removing paragraph separators cut ICU execution time in half but minimally affected sheen

#

but ICU is still an order of magnitude slower

#

Things left to investigate

Allocation size/frequency, although it's starting to matter less considering just how much sheen ran laps around ICU
How much should I care about the extent sheen generates differences from ICU
How much of a pessimization will I suffer from attempting to make sheen conformant with ICU's tests

molten galleon Nov 2, 2023, 1:04 PM

#

#

#

top: chromium, bottom: notepad, same file

#

I dumped ICU's test case strings

#

they don't even render them the same order bleakekw

#

in notepad it even messed up the alignment and rendered the dots outside the margin

#

I am going to take the blue pill, use sheenbidi, and pretend all of this was a dream

molten galleon Nov 2, 2023, 5:10 PM

#

molten galleon - Hypothesis 1: The reason for the disparity is because UBiDi calculates across ...

hypothesis 2 was correct bleakekw I cannot rely on runs being globally in logical order and only visually reversed at the run level, I'll have to refactor everything to operate exclusively by visual-ordered glyph index

molten galleon Nov 4, 2023, 8:14 PM

#

so minor losses in my clean "anything editor related can be bounded in O(log strlen)" assumption, and my assumption I wouldn't have to edit the datamodel again... but slowly returning basic functionality

molten galleon Nov 6, 2023, 8:38 PM

#

I definitely underestimated bidirectional text

molten galleon Nov 6, 2023, 9:17 PM

#

okay very important point, I've found a minimal example that you can shake up a line such that it does not end on its highest logical interval

molten galleon Nov 9, 2023, 10:51 PM

#

old harbor Nov 9, 2023, 11:05 PM

#

sweet : )

#

how long do the tests run?

molten galleon Nov 9, 2023, 11:07 PM

#

pretty much instantly, it's not a perf test or a particularly exhaustive test

#

it's just a comparison test to ICU's LayoutEx layout generator vs mine

#

#

couple of mixed run strings to warm it up and then one of the bidi test strings

#

it just has a lot of assertions because I make sure the lines, runs, char positions, char indices, glyphs, etc all match perfectly

#

and this is only the UTF-16:reference test, still have to fix the UTF-8 one back up

#

then it's ready for porting to sheenbidi

#

and then I am finally, finally done with layout logic and encodings

old harbor Nov 9, 2023, 11:15 PM

#

Cool, and yea, unit tests are supposed to finish instantly, or as kwik as possible at least

molten galleon Nov 10, 2023, 1:41 PM

#

#

woah

#

gcc debugs your bidi control chars

#

although I doubt this is wrong because I pulled this test string straight out of the unicode spec

#

actually nvm I removed text7

molten galleon Nov 10, 2023, 4:40 PM

#

and just like that, the layout builder is now fully UTF-8 native, no more converting to UTF-16 and back

old harbor Nov 10, 2023, 5:01 PM

#

noice

old harbor Nov 10, 2023, 8:12 PM

#

hmm DR? did i show you this before? https://github.com/slint-ui/slint im not sure

GitHub

GitHub - slint-ui/slint: Slint is a declarative GUI toolkit to buil...

Slint is a declarative GUI toolkit to build native user interfaces for Rust, C++, or JavaScript apps. - GitHub - slint-ui/slint: Slint is a declarative GUI toolkit to build native user interfaces f...

molten galleon Nov 10, 2023, 8:30 PM

#

don't think so, it looks very mobile/webby though

old harbor Nov 10, 2023, 8:32 PM

#

i need to see what it actually is too, and only saw "c++" tbh, i ignored rust/js

#

ah it has a "Pricing" thing on their website :3

#

https://slint.dev/releases/1.3.0/demos/gallery/

molten galleon Nov 10, 2023, 8:35 PM

#

grabbing my little string of death

#

lays out fine

#

pwned

#

I love all the little things you can see happening

#

the negated text color not matching the highlighted region
a brave (and ill informed) attempt at visual-order highlighting
layout being applied separately in the highlighted region, causing a dynamic reorder

#

lmao

#

I feel vindicated in my struggle

#

I started typing ffi, which ligaturizes in a lot of fonts

#

it shits the bed

#

you can't sub-select, the cursor jumps around as I type

#

can't believe this guy is trying to charge for this

#

though, what this at least tells me is that this guy isn't just transpiling to web controls and this is a fully NIH UI solution that's running in the browser

#

⁉️

old harbor Nov 10, 2023, 8:55 PM

#

haha

molten galleon Nov 10, 2023, 8:57 PM

#

#

interestingly though this is one of the things I'm looking at right now

#

rendering highlighting I mean

old harbor Nov 10, 2023, 8:57 PM

#

looks proper

molten galleon Nov 10, 2023, 8:58 PM

#

that's notepad

old harbor Nov 10, 2023, 8:58 PM

#

if you ask my uninitiated ass

#

ah

molten galleon Nov 10, 2023, 8:58 PM

#

this image just shows off the main things to think about

#

this is 1 glyph, so the visual effect of highlighting needs to be split

old harbor Nov 10, 2023, 8:59 PM

#

im sure you will never be able to use anything but your own stuff

#

with all the background knowledge you have with that font and bidi and IME stuff

#

everything will be shit sort of hehe

molten galleon Nov 10, 2023, 9:00 PM

#

OSes and browsers do just fine, generally

old harbor Nov 10, 2023, 9:00 PM

#

yea i mean custom libs

molten galleon Nov 10, 2023, 9:00 PM

#

only things where people actually went and double checked I guess

old harbor Nov 10, 2023, 9:00 PM

#

did you check out RmlUI yet?

#

i also kinda wished the ui factorio based their ui on was not dead

molten galleon Nov 10, 2023, 9:01 PM

#

I haven't played with rmlui but the idea seems solid at least, just from looking at it

old harbor Nov 10, 2023, 9:02 PM

#

oki

#

perhaps ill just forget about that slint thing, didnt look like it has many controls anyway

molten galleon Nov 10, 2023, 9:02 PM

#

their focus is weird

#

a custom UI for web is inherently kinda sus because fundamentally web has the best UI, pretty much

#

using HTML/CSS as the interface isn't always the most amazing, but there's also plenty of tooling to make it not suck

#

and that's what RmlUI gets right

old harbor Nov 10, 2023, 9:04 PM

#

i see

#

im not a fan of css at all, because of its stupid semantics - naming of those mostly

#

the box model is also unintuitive, instead of top left right bottom they have it other way around and stuff like that

#

the whole flexbox bs 😄

molten galleon Nov 10, 2023, 9:07 PM

#

yeah, it's definitely not great but I think it's the best decision for "UI solutions you expect people to actually use"

#

my actual UI design "ideology" is based on roblox primarily because that's what I'm familiar with and what I'm used to working against, but I don't actually like designing UI so I just need the familiarity for it to suck less

left junco Nov 10, 2023, 9:08 PM

#

I've barely used it but among the real time graphics GUI libraries it seems like the most likely to be usable out of the box for more complicated arbitrary UI

#

RmlUI

molten galleon Nov 10, 2023, 9:09 PM

#

other than text, UI is really just a bunch of image rects and I think it's not too crazy to figure out how you want to do stuff

#

for bigger outfits how you want to do stuff usually means "have something compatible with a webdev's tool stack so you can hire web devs and ignore this inane crap"

left junco Nov 10, 2023, 9:09 PM

#

Yeah it depends on how high level you need

old harbor Nov 10, 2023, 9:10 PM

#

yeah its rectangles with textures, an arrange (to measure all the controls) pass and a layout pass

left junco Nov 10, 2023, 9:11 PM

#

It's creating and arranging those rectangles that's the hard part lol

old harbor Nov 10, 2023, 9:12 PM

#

o make them look neat, yeah 😄

molten galleon Nov 10, 2023, 9:12 PM

#

yeah, at least that has its limits of complexity and it mostly doesn't impact the user when it's suboptimal

#

text is the exception because as soon as you have a text box you have some smartass like me pasting magic runes trying to break it

old harbor Nov 10, 2023, 9:13 PM

#

X_X

molten galleon Nov 10, 2023, 9:13 PM

#

I think you might be in the perfect market for this demongod because you're making a multiplayer game

old harbor Nov 10, 2023, 9:13 PM

#

if i can detect that you are running my stuff, ill just replace all text input controls with labels

molten galleon Nov 10, 2023, 9:13 PM

#

which is really one of the main reasons a game might start to intake arbitrary strings of non-ascii

left junco Nov 10, 2023, 9:14 PM

#

Yeah true hmm

old harbor Nov 10, 2023, 9:14 PM

#

epecially when Shooting Range Simulator expands into the asian market 😄

left junco Nov 10, 2023, 9:14 PM

#

Right now I support ASCII only lol

old harbor Nov 10, 2023, 9:14 PM

#

typical american move 😄

molten galleon Nov 10, 2023, 9:15 PM

#

you'll want well formatted and HD rendered arabic for scenario immersion

old harbor Nov 10, 2023, 9:15 PM

#

im sure you meant iso 8859-1

molten galleon Nov 10, 2023, 9:15 PM

#

windows codepage 1252

left junco Nov 10, 2023, 9:15 PM

#

Haha

old harbor Nov 10, 2023, 9:15 PM

#

in UTF32-BE

left junco Nov 10, 2023, 9:15 PM

#

I'll add it to the extremely long list of things I should add someday

#

But legitimately not a bad idea

molten galleon Nov 10, 2023, 9:15 PM

#

your chat message was dropped because you forgot to send a BOM at the start

old harbor Nov 10, 2023, 9:16 PM

#

hey its on the list at least

molten galleon Nov 10, 2023, 9:16 PM

#

it basically fits the exact problem I'm trying to solve, low friction plug n play solution for arbitrary text to just work in a game engine

#

without imposing UI on you

left junco Nov 10, 2023, 9:16 PM

#

I wouldn't mind having the ability to use text as textures, like so that a sign can just render text rather than needing an ultra HD image on disk

molten galleon Nov 10, 2023, 9:16 PM

#

the thing that you expect you should get from just adding freetype

left junco Nov 10, 2023, 9:17 PM

#

Yeah

molten galleon Nov 10, 2023, 9:17 PM

#

or just freetype+harfbuzz, or just freetype+harfbuzz+icu

#

and so on

old harbor Nov 10, 2023, 9:17 PM

#

instead its just libDR now

molten galleon Nov 10, 2023, 9:19 PM

#

left junco I wouldn't mind having the ability to use text as textures, like so that a sign ...

yeah once I get out of the logical data dungeon I want to do some side by side comparison of some of the scalable text techniques

#

I've found that msdf is extremely suboptimal

left junco Nov 10, 2023, 9:20 PM

#

Which one does star citizen use

#

My friend showed me that game and I was pretty impressed by the text

molten galleon Nov 10, 2023, 9:20 PM

#

that's a very deccer question lol

#

if it's just the 2D text they could just be software rasterizing (e.g. freetype) at the desired size

#

3D text is a bit more challenging because you can't just get good fidelity out of 1 baked size

left junco Nov 10, 2023, 9:21 PM

#

It's like street signs and stuff

#

They look very crisp up close

molten galleon Nov 10, 2023, 9:22 PM

#

right now the one that's looking the most promising is this:
https://wdobbie.com/post/gpu-text-rendering-with-vector-textures/
which is literally just rasterizing the curves on the GPU as freetype would on the CPU before it hands you the glyph

#

and since SC is a modern game I wouldn't be surprised if they just do that, in tehir own way

#

in fact SDF text seems like a bit of a weird hack for the modern day since you should have plenty of computational power to rasterize glyphs in realtime

molten galleon Nov 10, 2023, 9:52 PM

#

I wonder what the easiest way to do this is though, all this is doing is rendering the highlight color and rendering the subregion inverted, I don't think I can do this with changing blend op because it still needs to mix alpha

#

I think I might just emit these as separate rects

#

it's also interesting a really minor detail with notepad/the windows OS text box is that the cursor is double inverted

#

it's 1 - the blue, and inverts the text back to black

old harbor Nov 10, 2023, 9:55 PM

#

isnt it one_src minus 1?

#

err other way around

molten galleon Nov 10, 2023, 9:55 PM

#

yeah but I think I'd mess up the alpha mixing of the font

#

it's 1 - dst color

#

if you set it up as a blend op

#

but then it would operate on the post-blended alpha

old harbor Nov 10, 2023, 9:56 PM

#

i never got blend modes right the first time

molten galleon Nov 10, 2023, 9:56 PM

#

I just remember dst = what's already there

molten galleon Nov 12, 2023, 4:52 PM

#

#

works perfectly, no color inversion yet though

old harbor Nov 12, 2023, 5:03 PM

#

looks great already

molten galleon Nov 14, 2023, 6:43 PM

#

fixed my rendering for text formatting to properly use interval data, so now underlines and strikeouts are continuous

old harbor Nov 14, 2023, 6:45 PM

#

sweet : )

#

outline looks neat too

molten galleon Nov 14, 2023, 6:45 PM

#

just need to combine this with color inversion from highlighting and that should be it for most of the user state machine driven render crap

#

though I still need to implement text formatting features that revolve around glyph substitutions

#

like synthetic bold/italic, or smallcaps

old harbor Nov 14, 2023, 6:47 PM

#

are you going to cook all that into some sort of libDR?

molten galleon Nov 14, 2023, 6:47 PM

#

yeah, I still haven't sorted out how the final API should look but the framework is generally there

old harbor Nov 14, 2023, 6:47 PM

#

noice

#

perhaps as a bonus you can provide some dearimgui integration 😄

molten galleon Nov 14, 2023, 6:48 PM

#

I'll have to see how easy it is to plug into imgui (I doubt it's too hard tbh)

old harbor Nov 14, 2023, 6:49 PM

#

imgui will basically just be the client sending in character data, right?

#

"abc"

molten galleon Nov 14, 2023, 6:49 PM

#

yeah, it's more about sending back the result rects or whatever

old harbor Nov 14, 2023, 6:49 PM

#

and your middleware will know how to cook the font atlas, and then imgui will spit out quads again

#

yeah

molten galleon Nov 14, 2023, 6:51 PM

#

I assume there's some function somewhere in imgui that's its internal layoutText() or drawText() or something

#

and I just have to delete it and drop in calls to my lib

old harbor Nov 14, 2023, 6:51 PM

#

: )

#

lets dig, once you think your stuff is ready

#

over a weekend or so

molten galleon Nov 14, 2023, 6:52 PM

#

yeah, I think I'm getting closer to my goal

old harbor Nov 14, 2023, 6:52 PM

#

probably have to think about/invent some ImGui::PushStyle-esque-isms to handle various text formatting things then

molten galleon Nov 14, 2023, 6:53 PM

#

I just need to

wrap up highlight rendering
add text addition/deletion logic and clipboard support
deal with glyph substitution related stuff (which I've learned includes rendering tabs as well)

molten galleon Nov 14, 2023, 6:55 PM

#

old harbor probably have to think about/invent some ImGui::PushStyle-esque-isms to handle v...

yeah idk how you're "supposed" to handle text formatting

#

as usual my basic playbook comes from roblox https://create.roblox.com/docs/ui/rich-text

#

I have simple XML style markup tags

old harbor Nov 14, 2023, 6:55 PM

#

ah i see

molten galleon Nov 14, 2023, 6:55 PM

#

old harbor Nov 14, 2023, 6:55 PM

#

so some embedded "css" basically, which needs to be interpreted somehow, and turned into style instructures per run, which are sent to imgui somehow to tell your middleware

molten galleon Nov 14, 2023, 6:55 PM

#

this is what the above text looks like when I click on it

#

yeah, CSS is actually so much worse for this

#

I have some equivalent CSS for some of these formats since it's basically the only other place I can test formatted text

#

old harbor Nov 14, 2023, 6:56 PM

#

css is probably the wrong word, it looks like xml 🙂

molten galleon Nov 14, 2023, 6:56 PM

#

and yeah right now I just provide a function to generate runs from inline formatting

old harbor Nov 14, 2023, 6:56 PM

#

ah nice

molten galleon Nov 14, 2023, 6:57 PM

#

but that's something you can generate yourself

#

for example, from code formatting rules or even a LSP

old harbor Nov 14, 2023, 6:57 PM

#

im sure we can cook something up then

molten galleon Nov 14, 2023, 7:13 PM

#

yeah definitely

molten galleon Nov 15, 2023, 1:25 PM

#

in chromium, only the actual glyph colors are inverted and not the strokes or underlines/strikeouts

#

they also have a second set of washed out colors for when focus is lost

#

shall I use this excuse to be lazy?

#

https://tenor.com/view/peter-griffin-perhaps-family-guy-thinking-gif-13587558

Tenor

#

not inverting the strikeout and underlines is definitely kinda crusty and I don't wanna copy that

#

but the stroke can clip outside of the bounds of the highlight box so I'll just conveniently ignore that

old harbor Nov 15, 2023, 5:35 PM

#

illallowit.gif

molten galleon Nov 16, 2023, 1:27 PM

#

super satisfying when it just works

molten galleon Nov 16, 2023, 2:14 PM

#

also small research dump:
https://developer.mozilla.org/en-US/docs/Web/CSS/font-synthesis
CSS provides the ability to toggle 4 different features of font synthesis: weight (think boldness), style (italic), smallcaps, and position (meaning subscript and superscript)
https://github.com/RazrFalcon/resvg/issues/297#issuecomment-667674132
I found this random github issue that directly quotes some code out of Blender for how they call freetype to synthesize bold and italic. I also checked inside MuPDF and they do it pretty much the same way, luckily it seems that for bold and italic you just need to create a subfont of the main font and mess with its master settings. The harfbuzz API has the functions hb_font_set_synthetic_bold and hb_font_set_synthetic_slant as well, I'll have to read the source code on those and see what they do, the implication is that you need to always set them if you're putting together a synthetic font to let the shaper know about the changes, but idk if that's necessary with a hb_ft font.

When it comes to smallcaps, looking in MuPDF I found that they seem to invoke that by passing a HB_TAG('s','m','c','p') tag to the shaper, I believe smcp is an OpenType tag that makes it select a different charmap for looking up smallcaps glyphs in a font assuming you already have them, this I'll definitely have to do some more digging with though, because I dunno how I'll synthesize them if they're missing or if I'm not using an opentype font. I think the issue is largely similar for subscript/superscripts

molten galleon Nov 17, 2023, 11:17 AM

#

old harbor Nov 17, 2023, 11:38 AM

#

noice 🙂

#

it needs the typing sound from this thing... one sec

#

https://github.com/zevv/bucklespring

GitHub

GitHub - zevv/bucklespring: Nostalgia bucklespring keyboard sound

Nostalgia bucklespring keyboard sound. Contribute to zevv/bucklespring development by creating an account on GitHub.

#

ah fuck, windows builds are broken for this thing 😄

balmy gulch Nov 17, 2023, 12:37 PM

#

molten galleon

I can feel my soul being transported to the 2000s

livid patrol Nov 17, 2023, 12:38 PM

#

https://youtu.be/TKfS5zVfGBc

YouTube

TradeSend

Trance - 009 Sound System Dreamscape (HD)

Trance - 009 Sound System Dreamscape (HD) long edit

HD Quality Picture and Sound

▶ Play video

old harbor Nov 17, 2023, 12:45 PM

#

thats not really old trance 🙂 but illallowitjiff.gif

molten galleon Nov 24, 2023, 8:19 PM

#

@old harbor

#

they slide off your active list if they haven't been talked in for a few days

#

'tis what 'tis

old harbor Nov 24, 2023, 8:33 PM

#

yes but

#

it should be visible in the gallery

#

weird : )

#

so my question is, do you know whether cmake knows how to config the compiler when it comes to -j XX

#

will it always pass the maximum available cores/threads or just use whatever is default and not use -j

#

there is CMAKE_BUILD_PARALLEL_LEVEL but im not sure if i should be touching it or not

#

but i would like to have all my cores used when compiling anything

molten galleon Nov 24, 2023, 8:35 PM

#

that's specifically for cmake --build, aka if you wanna be using the ultra platform agnostic build mode

#

you'll have to investigate what it chooses by default

#

I think otherwise it depends on the generator target, make for example defaults to single threaded but ninja is intelligent about being parallel

old harbor Nov 24, 2023, 8:36 PM

#

but it would make sense to expect cmake configging my system it should make use of all cores, neh?

molten galleon Nov 24, 2023, 8:36 PM

#

https://cmake.org/cmake/help/latest/envvar/CMAKE_BUILD_PARALLEL_LEVEL.html
it affects cmake --build, which is the invocation of your build script, not configuration time

old harbor Nov 24, 2023, 8:36 PM

#

or rather, it makes s ense to compile with all cores, with or without cmake

#

yeah i saw that too

molten galleon Nov 24, 2023, 8:37 PM

#

so it uses the tool's default by default

molten galleon Nov 24, 2023, 8:37 PM

#

molten galleon I think otherwise it depends on the generator target, make for example defaults ...

which means you should defer to this

old harbor Nov 24, 2023, 8:37 PM

#

hmm

molten galleon Nov 24, 2023, 8:38 PM

#

if you're using ninja (which you should be) I wouldn't worry

old harbor Nov 24, 2023, 8:39 PM

#

ja ninja is default here

molten galleon Nov 24, 2023, 8:40 PM

#

yeah so then it'll just parallelize by default and everything should be fine

old harbor Nov 24, 2023, 8:47 PM

#

oki

molten galleon Dec 2, 2023, 6:46 PM

#

oh boy I think I am pretty close to the approximate shape of my library API

#

just need to take another pass at my font registry

old harbor Dec 2, 2023, 7:03 PM

#

https://tenor.com/view/cotton-candy-girl-kid-child-excited-gif-27094294

Tenor

olive sapphire Dec 4, 2023, 11:56 PM

#

@molten galleon by the way, do you just re-render for size changes? does this dynamically render text with freetype and put it into the atlas? may I get a resource on doing subpixel aliasing with the GPU, without using framebuffers?

molten galleon Dec 4, 2023, 11:56 PM

#

uhhh

#

I just render it at whatever size you request and put it in the atlas yeah

#

not exactly sure what you mean by subpixel aliasing with the GPU here though, I just use my vertex shader make sure my quads stay pixel aligned so I don't have any smudging

#

the subpixel AA comes baked from however freetype generates it

olive sapphire Dec 4, 2023, 11:58 PM

#

molten galleon the subpixel AA comes baked from however freetype generates it

You need to blend using a specific function with the current pixel to do it properly.

molten galleon Dec 4, 2023, 11:59 PM

#

I just use regular blending so its whatever lol

olive sapphire Dec 5, 2023, 12:01 AM

#

Huh, you should try some other backgrounds because it might get ugly.

molten galleon Dec 5, 2023, 12:02 AM

#

idk I've never had any issues

#

are you rendering glyphs like {r,r,r,r} * (color, 1.0)

#

my blending was shitty when I first tried rendering text because I didn't realize you're supposed to just use the font data as the alpha

#

in which case it works on pretty much every background

#

alpha'd textures blend fine in general, otherwise pngs would always suck

olive sapphire Dec 5, 2023, 12:04 AM

#

molten galleon are you rendering glyphs like {r,r,r,r} * (color, 1.0)

Currently, but I was looking into shaders for subpixel and SDFs(as you know already).

molten galleon Dec 5, 2023, 12:05 AM

#

oh you're looking for subpixel aa for sdfs

#

like the stuff with fwidth?

olive sapphire Dec 5, 2023, 12:05 AM

#

Nah, I think i'm dropping SDFs for now. The cost tradeoff doesn't make sense

#

It's a shame because I spent a lot of time researching and making tools for using them. Oh well.

olive sapphire Dec 5, 2023, 12:06 AM

#

molten galleon like the stuff with fwidth?

That doesn't do subpixel, just regular grayscale. It's pretty bad still.

#

The text rendering is just bad with the shader and the distance field.

molten galleon Dec 5, 2023, 12:07 AM

#

is this raster or sdf

olive sapphire Dec 5, 2023, 12:07 AM

#

SDF

#

~30x30

#

It runs at the same FPS, at about 6-7000 though so it's definitely still efficient, but making it look decent is hard.

#

@molten galleon do you use a 3-channel atlas or a single channel one?

molten galleon Dec 5, 2023, 12:10 AM

#

I have a single channel atlas for most stuff and a second one for color glyphs

olive sapphire Dec 5, 2023, 12:11 AM

#

Oh yeah, and whatever algorithm I use still produces some weird kind of artifact on the SDF itself:

#

(texture was malloced so there's some random heap data in there)

#

this is causing even the straight lines to be wiggly:

livid patrol Dec 5, 2023, 12:15 AM

#

calloc pls

molten galleon Dec 5, 2023, 12:17 AM

#

yeah you should get rid of those weird artifacts

olive sapphire Dec 5, 2023, 12:19 AM

#

molten galleon yeah you should get rid of those weird artifacts

I'm using existing algorithms to render them. I didn't make this algorithm, and am not proficient enough or have enough knowledge to know what is causing that.

molten galleon Dec 5, 2023, 12:19 AM

#

I mean the ones from the malloc to start

#

you should eliminate unknowns

olive sapphire Dec 5, 2023, 12:20 AM

#

Those never even show up on the output, it's just the algorithm's artifacts.

#

That's because I zoom in a tiny bit when I provide UVs, because even hitting the edge of the texture causes there to be lines.

#

Overall, SDFs have been unreliable.

livid patrol Dec 5, 2023, 12:22 AM

#

address mode issue?

olive sapphire Dec 5, 2023, 12:23 AM

#

?

livid patrol Dec 5, 2023, 12:23 AM

#

oh wait this is an atlas, rip

#

you should figure out why there are lines. maybe adding a texel border around the glyphs would help

olive sapphire Dec 5, 2023, 12:24 AM

#

Yeah, that's also been something I've been trying to work on.

molten galleon Dec 5, 2023, 12:31 AM

#

you should absolutely pad your atlas as a start

olive sapphire Dec 5, 2023, 12:32 AM

#

That would be a massive waste of space.

molten galleon Dec 5, 2023, 12:32 AM

#

not really

livid patrol Dec 5, 2023, 12:32 AM

#

would it though

olive sapphire Dec 5, 2023, 12:33 AM

#

olive sapphire The text rendering is just bad with the shader and the distance field.

Well, I already had a solution for it hitting the edges, and the lines still occur:

olive sapphire Dec 5, 2023, 2:27 AM

#

molten galleon the subpixel AA comes baked from however freetype generates it

BTW, I had a couple questions about this too.

#

You said you still use grayscale atlases, how do you do subpixel with them?

olive sapphire Dec 5, 2023, 5:18 AM

#

olive sapphire this is causing even the straight lines to be wiggly:

This was another SDF technique, but without the buffer space so some of the corners at the edges got preserved:

#

It has some flaws, but maybe a shader can fix them

#

#

I'm rendering them at 96 px height this time, but still not the best.

#

Still have not figured out what this fringing is, so I guess my results aren't too reliable though.

#

This is a completely new algorithm so maybe there's some error in how I'm copying them.

molten galleon Dec 5, 2023, 10:11 AM

#

olive sapphire You said you still use grayscale atlases, how do you do subpixel with them?

I don't use subpixel aa, never bothered to investigate it

molten galleon Dec 5, 2023, 1:05 PM

#

lmao it also just occurred to me I've been answering wrong all last night

#

by subpixel aa I originally assumed you meant dealing with adjusting AA to subpixel glyph positions, which is one of the problems you have with cached raster glyphs, and I inded just solve that by pixel-aligning my glyphs

#

but I don't use multichannel subpixel AA like ClearType or something would, just too many issues like you outlined above, lots of complexity for very little gain at 1080p and above

olive sapphire Dec 5, 2023, 1:46 PM

#

molten galleon by subpixel aa I originally assumed you meant dealing with adjusting AA to subpi...

Oh, is there a certain method to pixel alignment? I haven't really gone into the subject.

olive sapphire Dec 5, 2023, 1:47 PM

#

molten galleon but I don't use multichannel subpixel AA like ClearType or something would, just...

I have a 1440p display and non-aliased and even grayscale AA text looks terrible, and every other app that uses Cleartype or their own Subpixel AA Solution looks much better

old harbor Dec 5, 2023, 1:47 PM

#

worst case you turn it into a heightmap generator : )

olive sapphire Dec 5, 2023, 1:47 PM

#

???

old harbor Dec 5, 2023, 1:47 PM

#

olive sapphire Still have not figured out what this fringing is, so I guess my results aren't t...

that thing gives off heightmap vibes if you zoom in a little

#

and i was trying to be funny

olive sapphire Dec 5, 2023, 1:48 PM

#

old harbor that thing gives off heightmap vibes if you zoom in a little

OHH lol

molten galleon Dec 5, 2023, 1:56 PM

#

olive sapphire Oh, is there a certain method to pixel alignment? I haven't really gone into the...

#version 460

layout (location = 0) uniform vec2 u_invScreenSize;
layout (location = 1) uniform vec4 u_extents;
layout (location = 2) uniform vec4 u_texCoords;
layout (location = 3) uniform vec4 u_color;

layout (location = 0) out vec2 v_texCoord;
layout (location = 1) out vec4 v_color;

void main() {
    uint b = 1 << (gl_VertexID % 6);
    vec2 baseCoord = vec2((0x19 & b) != 0, (0xB & b) != 0);
    gl_Position = vec4(floor(fma(baseCoord, u_extents.zw, u_extents.xy)) * u_invScreenSize, 0, 1);
    gl_Position.xy = fma(gl_Position.xy, vec2(2.0, -2.0), vec2(-1.0, 1.0));
    v_texCoord = fma(baseCoord, u_texCoords.zw, u_texCoords.xy);
    v_color = u_color;
}

nothing in particular, I just floor the final pixel positions in pixel-viewport-space to make sure that image sampling doesn't end up blurry

#

it's just how I do my general UI shaders

#

enjoy my highly readable quad shader

olive sapphire Dec 5, 2023, 2:37 PM

#

You really don't care about supporting 3.3 or anything like that do you?

#

What's in texCoords.zw?

#

I don't fully understand what this is doing yet.

molten galleon Dec 5, 2023, 2:44 PM

#

not in the slightest

#

my test program is 4.6 but my 'real' program is vulkan 1.2

#

I just store texcoords (x, y, width, height) in a vec4

#

and I just like using fused multiply-adds

#

somehow more readable to me

olive sapphire Dec 5, 2023, 2:44 PM

#

0b00011001 & b != 0, 0b1011 & b != 0 so you're checking whether vertices are in certain indices here.

molten galleon Dec 5, 2023, 2:45 PM

#

nah I generated those thanks to a neat trick I saw jaker post a while back

#

the first 2 lines generate the 6 vertices of a quad made of 2 triangles

#

so (magic number & b) != 0 just generates either a 0 or a 1 for that component

#

via the bitmask

olive sapphire Dec 5, 2023, 2:46 PM

#

Yeah, I think I understand that part but I don't know why you're specifically checking for those indices

molten galleon Dec 5, 2023, 2:47 PM

#

because doing the bool operation forces it to either be a 0 or a 1

olive sapphire Dec 5, 2023, 2:47 PM

#

I know but what's so special about the first, 4th and 5th vertex

#

and then the first, second and 4th for the second component

molten galleon Dec 5, 2023, 2:47 PM

#

because these are vertices in a quad

olive sapphire Dec 5, 2023, 2:48 PM

#

You're missing the third and the 6th.

#

Maybe I'm being stupid but what do the other 3 lines even do?

molten galleon Dec 5, 2023, 2:50 PM

#

the bitmask generates a vec2 vertex position like (0, 0), (0, 1), (1, 0), (1, 1) based on the vertex ID

olive sapphire Dec 5, 2023, 2:50 PM

#

it seems like you're multiplying by an extents uniform, which involves screensize, and then you multiply everything with 2.0 and -2.0?

molten galleon Dec 5, 2023, 2:50 PM

#

the 3rd like transforms it by rect extents (x, y, width height) and puts it into NDC

#

so -1 to 1

olive sapphire Dec 5, 2023, 2:51 PM

#

What are extents?

molten galleon Dec 5, 2023, 2:51 PM

#

(x, y, width height) in pixels

#

so (50, 50, 120, 60) draws a 120x60 rect at pixel position 50,50 from the top left

#

all the rest of the math just transforms that into GL coordinate space

olive sapphire Dec 5, 2023, 2:52 PM

#

molten galleon the bitmask generates a vec2 vertex position like (0, 0), (0, 1), (1, 0), (1, 1)...

Yes, if you look at the first line, the bitmask is supposed to identify if the gl_VertexID %6 is a certain value, but what I didn't understand is why those particular IDs are focused on

molten galleon Dec 5, 2023, 2:52 PM

#

it doesnt identify anything

#

it generates them

#

I am abusing a boolean operator to force it to be a certain value

olive sapphire Dec 5, 2023, 2:53 PM

#

Wait, are you not batching anything? Are you supplying uniforms for every single quad and then making a separate draw call for them???

molten galleon Dec 5, 2023, 2:53 PM

#

yeah this is my GL test program because I literally don't care

olive sapphire Dec 5, 2023, 2:54 PM

#

Yeah this might not be useful for me, I'm currently batching shapes, text and images all at once.

molten galleon Dec 5, 2023, 2:54 PM

#

#version 450
#extension GL_EXT_scalar_block_layout: require
#extension GL_EXT_buffer_reference: require

layout (location = 0) out vec2 texCoord;
layout (location = 1) out vec4 outColor;
layout (location = 2) out uvec2 outImageIndices;

struct Instance {
    vec4 texLayout;
    vec4 color;
    mat3x2 transform;
    uvec2 imageIndices;
};

layout (buffer_reference, scalar) readonly buffer Instances {
    Instance data[];
};

layout (push_constant) uniform PushConstants {
    vec2 invHalfSize;
    Instances instances;
} constants;

void main() {
    Instance inst = constants.instances.data[gl_VertexIndex / 6];
    uint b = 1 << (gl_VertexIndex % 6);
    vec2 baseCoord = vec2((0x1C & b) != 0, (0xE & b) != 0);

    gl_Position = vec4(inst.transform * vec3(baseCoord - 0.5, 1.0), 0.5, 1.0);
    gl_Position.xy = floor(gl_Position.xy) * constants.invHalfSize - 1.0;
    texCoord = fma(baseCoord, inst.texLayout.zw, inst.texLayout.xy);
    outColor = inst.color;
    outImageIndices = inst.imageIndices;
}

this is my vulkan shader

#

different magic numbers for reasons I forget

olive sapphire Dec 5, 2023, 2:55 PM

#

Do you plan on batching in the future?

molten galleon Dec 5, 2023, 2:55 PM

#

the vulkan one does batching

#

the GL one doesn't

#

because the purpose of my text test program (currently) isn't to show off how fast I can blit quads

#

and one of my renderer goals was to touch GL as little as possible

olive sapphire Dec 5, 2023, 2:57 PM

#

Wait so why is the Vulkan shader referencing OpenGL Extensions and is written in what is basically GLSL?

#

I've never written Vulkan so that's new.

molten galleon Dec 5, 2023, 2:57 PM

#

because the shader is in GLSL so the extensions are still prefixed with GL_

olive sapphire Dec 5, 2023, 2:57 PM

#

molten galleon because the purpose of my text test program (currently) isn't to show off how fa...

What are you even trying to do by developing that then.

molten galleon Dec 5, 2023, 2:57 PM

#

neither of those extensions are actually available in opengl

#

literally everything else

#

text formatting, text editing, text layouting (bidi handling, line breaking)

#

dealing with font fallbacks and caches

#

font synthetics

olive sapphire Dec 5, 2023, 2:58 PM

#

This is overengineered for games, underengineered for being a library, and you don't seem to care about performance, size or quality.

molten galleon Dec 5, 2023, 2:58 PM

#

everything you need for rendering fonts that isn't actually shading them

#

its a little overengineered for games but shader performance isn't my immediate goal

#

writing text shaders comes later

#

when I'm done it will absolutely be good enough for any text workload though

olive sapphire Dec 5, 2023, 2:59 PM

#

Hm what do you mean by "isn't actually shading them"

#

You're using the shaders right there

molten galleon Dec 5, 2023, 3:00 PM

#

yeah because I need to see the glyphs lol

olive sapphire Dec 5, 2023, 3:00 PM

#

Huh, so do you upload the textures and everything yourself, but expect the user to write the shading program and deal with your specific uniforms and data formats?

molten galleon Dec 5, 2023, 3:01 PM

#

what I mean is that it's not currently my objective to R&D the aspect of my text lib that actually puts the symbols on the screen

#

because that's not currently the interesting part or the most confounding problem

#

though obv I'm gonna do that at some point for the sake of my own use cases

olive sapphire Dec 5, 2023, 3:02 PM

#

I don't know what the goal is, because "putting pixels on the screen" is the whole point of text rendering, and making it seamless and correct is why people research it so much. I don't understand what even you will use this for in the future.

molten galleon Dec 5, 2023, 3:03 PM

#

https://faultlore.com/blah/text-hates-you
https://lord.io/text-editing-hates-you-too/
you should read these

olive sapphire Dec 5, 2023, 3:03 PM

#

Almost everyone already uses the libraries you're using, like freetype and harfbuzz, which are relatively easy to use and integrate into your game at the large cost of binary size.

molten galleon Dec 5, 2023, 3:04 PM

#

nah the whole core problem is that even if you have freetype, harfbuzz, and ICU, a lot of your work isn't done

#

if that was all there is to it I would've been done months ago and would've moved on

olive sapphire Dec 5, 2023, 3:04 PM

#

I have already read them, and most people still ignore them and it doesn't make a difference to 99% of the people playing games or even doing serious text editing.

molten galleon Dec 5, 2023, 3:04 PM

#

ok? that's kinda their own problem lol

olive sapphire Dec 5, 2023, 3:05 PM

#

molten galleon nah the whole core problem is that even if you have freetype, harfbuzz, and ICU,...

My text renderer is currently only a total addition of less than 100 lines of code in my project, I don't understand why it would be that hard for professionals to come up with even better solutions than I have.

molten galleon Dec 5, 2023, 3:06 PM

#

they do, they just exist isolated in their own domains

#

that's the source of my surprise

#

it's not that the solutions don't exist, it's that they don't exist in a plug and play way from a single point

#

really doing text right is usually isolated to stuff like browser engines or operating systems

olive sapphire Dec 5, 2023, 3:07 PM

#

They do, they're just usually behind a paywall because they're premium tools and a lot of professional research has been put into them from people working in the graphics industry for decades.

molten galleon Dec 5, 2023, 3:08 PM

#

I don't see your point tho

olive sapphire Dec 5, 2023, 3:08 PM

#

I am just confused on what your overall goal is with this project. You haven't even provided any docs or anything on how to use the API.

molten galleon Dec 5, 2023, 3:09 PM

#

it's still indev lol

#

and mostly non-finalized

olive sapphire Dec 5, 2023, 3:10 PM

#

If you outline the API now, you will know what to build towards and can start recieving feedback immediately.

#

Also, have you finished RTL?

molten galleon Dec 5, 2023, 3:11 PM

#

lol my goal is simple, I need for my personal projects a simple API for providing text formatting and layout data for feeding a list of common text effects and text handling features I care about, and I'm structuring it as its own library because no single gathering point exists for this

#

yeah

olive sapphire Dec 5, 2023, 3:11 PM

#

molten galleon yeah

Ok that is definitely an exceptional part of that project then.

molten galleon Dec 5, 2023, 3:11 PM

#

my layout API is pretty much finalized, I just have to clean up some junk with it and add some documentation to the methods

#

the big thing I'm doing R&D for right now is my font registry

#

because I wanna put a bound on the amount of memory I load for the shitload of font files I want

#

but keep it fast and multithreadable

#

olive sapphire Dec 5, 2023, 3:13 PM

#

Are you going to generate and ship font files?

molten galleon Dec 5, 2023, 3:13 PM

#

nah I wanna handle normal ttf/otf

#

the problem is, in order to have rich text formatting, even if you're only using a single font family, you often need multiple faces

#

and usually the way a program handles this is bespoke

olive sapphire Dec 5, 2023, 3:14 PM

#

molten galleon because I wanna put a bound on the amount of memory I load for the shitload of f...

Hm, wouldn't that just be a question of using freetype to load all of those fonts, it doesn't take much memory for freetype to load something and keep it there.

molten galleon Dec 5, 2023, 3:14 PM

#

e.g. chromium has its own little algorithm for choosing what face it'll render some characters with, windows has its font registry, various linux desktop envs have their own font registries

olive sapphire Dec 5, 2023, 3:15 PM

#

You mean like the third argument for FT_New_Face?

molten galleon Dec 5, 2023, 3:15 PM

#

#

no

olive sapphire Dec 5, 2023, 3:15 PM

#

Wouldn't you just query those and leave it up to the user?

molten galleon Dec 5, 2023, 3:15 PM

#

and no

#

a really good exercise is to write out some text in different languages in a browser and check the Computed section in inspect element

#

the arguments for FT_New_Face are about picking faces from a single font file, which are usually different styles for a single thing

#

but you have to use information about/from the text in order to pick them

olive sapphire Dec 5, 2023, 3:17 PM

#

molten galleon a really good exercise is to write out some text in different languages in a bro...

Huh, I don't remember chrome having anything like that Rendered Fonts window.

molten galleon Dec 5, 2023, 3:18 PM

#

for example, a lot of fonts are partitioned by script (as in language script: latin, cyrillic, han, hiragana, devanagari, etc etc etc)

#

and special symbols have their own fallbacks

#

math symbols, emoji, etc etc

#

on top of that they're partitioned by weight (boldness) and style (italicness)

olive sapphire Dec 5, 2023, 3:19 PM

#

Hm

molten galleon Dec 5, 2023, 3:19 PM

#

so for any given string, you actually have to take some passes at it and figure out what fonts you need, which is also complicated by if you want formatting

#

like bolding some part or doing a smallcaps substitution

#

which is why you need a font registry, rendering a single string with a single font still means you need information about other fonts you might need to pick

olive sapphire Dec 5, 2023, 3:20 PM

#

Are you talking about fallbacks and stuff like that?

molten galleon Dec 5, 2023, 3:20 PM

#

yeah

olive sapphire Dec 5, 2023, 3:20 PM

#

Shouldn't the user worry about them though?

#

Like you can't try to query system fonts on every machine, that's not reliable and can be different depending on platforms.

molten galleon Dec 5, 2023, 3:21 PM

#

if they want to they don't have to use my library, this is for people like me (in the position of the user) where I ust want to build my registry and have a tool that handles it

#

what fonts you put into the registry and their relationship have to be input by the user, yes

#

so

#

if you want to register stuff in C:/Windows/Fonts to your registry, that's your problem

olive sapphire Dec 5, 2023, 3:22 PM

#

Oh ok, so why do you need your own registry?

old harbor Dec 5, 2023, 3:22 PM

#

olive sapphire Like you can't try to query system fonts on every machine, that's not reliable a...

thats a solved problem, platforms have well defined font locations

molten galleon Dec 5, 2023, 3:22 PM

#

to store that information and be able to hand it to the layout engine

#

like the browser does

#

my text is in times new roman, but how does it know what font to pick for an emoji?

#

it checks the registry and gets a font from it

olive sapphire Dec 5, 2023, 3:23 PM

#

old harbor thats a solved problem, platforms have well defined font locations

I was talking to some text/code editor developers and they were saying having default fallbacks is not something they are considering because of how different it can be on different platforms.

molten galleon Dec 5, 2023, 3:23 PM

#

yeah how you build your pool of fonts is up to you

old harbor Dec 5, 2023, 3:23 PM

#

olive sapphire I was talking to some text/code editor developers and they were saying having de...

its still a solved problem

#

think 80/20 cases

#

your nieche x will have problems if it wants nieche-isms

molten galleon Dec 5, 2023, 3:24 PM

#

it's just that a good text renderer/layout engine needs information about more than just 1 font

#

my font registry is a tool to serve my API the data that it needs in a nice way

olive sapphire Dec 5, 2023, 3:24 PM

#

Oh ok

old harbor Dec 5, 2023, 3:25 PM

#

on windows its c:/windows/fonts... on lunix its some /usr/... on mac its idk where /Fonts perhaps

#

plus you can always add fonts from custom paths

molten galleon Dec 5, 2023, 3:25 PM

#

the details are up to "the user"™️ but generally speaking for a proper text rendering solution you need to build metadata that lives at a higher level than just 1 font file

olive sapphire Dec 5, 2023, 3:25 PM

#

So do you just try to see if different fonts have a glyph through a fallback chain, or do you have designated fonts like Segui UI for emojis, Consolas for monospace etc?

molten galleon Dec 5, 2023, 3:26 PM

#

right now fonts are organized into families which have files that represent a weight, a style, and a script (or scripts)

#

with the option to have one of the files be a general fallback for all scripts

#

and on top of that I have a general fallback chain that it queries one at a time to fill in stuff that is common/inherited script like math symbols or emoji

#

https://github.com/forenoonwatch/rich-text/blob/master/fonts/families/NotoSans.json
e.g.

#

https://github.com/forenoonwatch/rich-text/blob/master/fonts/families/SegoeUI.json

olive sapphire Dec 5, 2023, 3:29 PM

#

molten galleon <https://github.com/forenoonwatch/rich-text/blob/master/fonts/families/SegoeUI.j...

Ohh ok, so you're supplying some of those yourself, but the user can also add some of their own in there if they want.

molten galleon Dec 5, 2023, 3:29 PM

#

consider this test/example data

#

you don't need any of mine

olive sapphire Dec 5, 2023, 3:30 PM

#

This is actually a pretty rich and graceful system

molten galleon Dec 5, 2023, 3:30 PM

#

the repo is slowly transitioning from "experimenting monorepo" to a library

#

so I am gonna clean up and explicitly mark more stuff that's given as an example vs stuff that's the core lib

olive sapphire Dec 5, 2023, 5:22 PM

#

If you could do some research into SDFs and shaders for them, I'd really appreciate it by the way @molten galleon. If you want my source for the two generation algorithms I used for single-channel SDFs, I can DM them to you.

molten galleon Dec 5, 2023, 5:23 PM

#

I did my brief digging into MSDF and decided that for now SDF based solutions just kinda suck, lemme find what I have

#

https://medium.com/@calebfaith/implementing-msdf-font-in-opengl-ea09a9ab7e00
here's what I used for the shaders

#

I have a pretty short list of stuff I'm looking to try as my universal rendering solution, I think I've posted it before

olive sapphire Dec 5, 2023, 5:24 PM

#

I did some research into something called Ultralight, and it has a really good SDF solution with less mem and using single channel SDFs. This is what the creator figured out:
https://twitter.com/adamjsimmons/status/611677036545863680

molten galleon Dec 5, 2023, 5:24 PM

#

just the Qin et al stuff Jasper mentioned a while ago, and just rendering vectors

#

I bet you can find a lot of good vector rendering implementations especially if you steal from nanovg or similar

olive sapphire Dec 5, 2023, 5:25 PM

#

nanovg is really slow with them.

#

It still renders them on the CPU.

#

AFAIK.

molten galleon Dec 5, 2023, 5:25 PM

#

https://wdobbie.com/post/gpu-text-rendering-with-vector-textures/
okay then lol not that

#

but htis is a great source

#

comes with implementation notes and a webgl demo

#

#

this entire pdf is 13ms/frame on my machine, but this is webgl and definitely not the best you can do

olive sapphire Dec 5, 2023, 5:28 PM

#

The demo is running sub 10fps on my machine.

molten galleon Dec 5, 2023, 5:28 PM

#

it doesn't give fps it's giving ms per frame I'm pretty sure

olive sapphire Dec 5, 2023, 5:29 PM

#

It's not using GPU?

old harbor Dec 5, 2023, 5:29 PM

#

you have to dig into this a bit more

#

and not wait for DR to give you the solution : )

olive sapphire Dec 5, 2023, 5:29 PM

#

Of course, I have

olive sapphire Dec 5, 2023, 5:31 PM

#

molten galleon it doesn't give fps it's giving ms per frame I'm pretty sure

Well obviously, I can physically see it's not even running at 5 FPS.

#

And it didn't have a frame time counter on my page:

molten galleon Dec 5, 2023, 5:32 PM

#

thats odd

old harbor Dec 5, 2023, 5:35 PM

#

if you are using a laptop

#

check that your dedicated gpu is the primary one

#

if you have one

olive sapphire Dec 5, 2023, 5:36 PM

#

I have the latest browser with full WebGPU, WebGL2, OpenGL 4.6 support and a dedicated GPU.

old harbor Dec 5, 2023, 5:36 PM

#

thats not what i asked

olive sapphire Dec 5, 2023, 5:37 PM

#

I mean I only have one, so it should be the only thing it uses. Other WebGL things like shadertoy use the full gpu.

old harbor Dec 5, 2023, 5:37 PM

#

whatever

olive sapphire Dec 5, 2023, 5:44 PM

#

wow the atlas for that whole page really is just:

#

https://wdobbie.com/pdf/atlas.bmp

molten galleon Dec 5, 2023, 5:50 PM

#

yep that's the benefit of rendering vectors

#

also it's not a bs approximation like you'd get from something generally SDF-based, a vector font is truly a vector font

#

so why not render vectors

olive sapphire Dec 5, 2023, 5:51 PM

#

http://alvyray.com/Memos/CG/Microsoft/6_pixel.pdf

#

This is a cool resource.

molten galleon Dec 5, 2023, 11:54 PM

#

hmm, mupdf has a pretty cool LRU cache actually, I think I might go down a rabbit hole implementing general purpose LRU caches

#

though I need to brush up on thread_local and techniques around it, because I want to figure out how to make something that dynamically scales with thread count

#

actually there's not much to it, except I (obviously) can't make thread_local stuff part of an object, so I probably have to design my font cache as if there can only ever be one (I hesitate to say singleton but that's the word that comes to mind)

left junco Dec 6, 2023, 12:04 AM

#

It's basically exactly the same as static data

#

i.e. static storage duration data

molten galleon Dec 6, 2023, 12:05 AM

#

I wonder if there's some convenient trick with wait free lists to lazily register per-thread data to some central object

#

yeah I was hoping I could squeeze a little more out of it

left junco Dec 6, 2023, 12:06 AM

#

molten galleon I wonder if there's some convenient trick with wait free lists to lazily registe...

What would this entail

#

Or what's stopping you

molten galleon Dec 6, 2023, 12:10 AM

#

ok I'm back, uh, idk I'm just kinda fleshing out the idea in my head

left junco Dec 6, 2023, 12:10 AM

#

I assume you could make the default constructor do the registration

#

With some global registry

molten galleon Dec 6, 2023, 12:18 AM

#

for font caches my main thing would be to keep some thread-local FT_Library/FT_Face/hb_font_t's around to fulfill their various use cases, kinda having a hard time figuring out how much about them I wanna store per-thread and how I'd keep them from sticking around too long

left junco Dec 6, 2023, 12:18 AM

#

With some global registry

molten galleon Dec 6, 2023, 12:18 AM

#

lol nice internet

#

for actual main font data (i.e. loading the file from memory) it's simple enough because that'll just be read into the main LRU cache, but each thread would build its own thread_local cache of FT/hb stuff which would effectively act as a view into it

left junco Dec 6, 2023, 12:24 AM

#

Lmao I blame discord

molten galleon Dec 6, 2023, 12:26 AM

#

though another completely unrelated idea that uses thread local stuff I've been wrestling with was for vulkan deletion queues

#

for that I was thinking you would have lazily-allocated-and-registered queues on whatever threads happened to ditch an object into the deletion queue, but they'd stay centrally registered somewhere in case that thread dies, and for when deletion time comes around

#

you can gather all currently available queues of stuff to maybe delete, dispatch them to n worker threads, and safely iterate them

#

and instead of always contending for a single queue or always negotiating with each thread for each queue, there's just a single atomic flag for whether it's deletion time or not

#

and when that flag is set and you still want to delete something, you just defer that to yet another queue

#

but that's just a random idea I had roughly based on how potrick described what he wants for daxa, it has made me start to wonder if there are some neat established tricks for lazily registering per-thread data to some kind of object with minimal contention, and temporarily taking ownership of it from a different thread or threads to do some kind of gathering work

molten galleon Dec 6, 2023, 1:08 PM

#

hmmm ok I thought about this more and it led me to fun requirements like "wait free hash table" and I think this is way too much for a deletion queue in vulkan, let alone a font cache

livid patrol Dec 6, 2023, 1:11 PM

#

I thought wait-freedom has only been proven to be practical for extremely simple data structures

#

Like a queue or something

#

A hash map seems way too complex according to my intuition but I have 0 facts to back it up

molten galleon Dec 6, 2023, 1:12 PM

#

yeah I have no idea how practical one would be

livid patrol Dec 6, 2023, 1:13 PM

#

I read somewhere that it's technically possible to implement any data structure with a number of wait-free building blocks, but there is no guarantee that it's anywhere close to fast

molten galleon Dec 6, 2023, 1:26 PM

#

yeah, also my rough idea is something like

struct DeletionQueue {
  std::vector<Thing> m_queues[THREAD_COUNT];
  MultiReaderMutex m_mutex; // I forget what these are actually called

  void queue_delete(Thing t) {
    if (m_mutex.try_lock_reader()) {
      m_queues[get_current_thread_queue_index()].push_back(t);
      m_mutex.unlock();
    }
    else {
      // Put it into some kind of thread local secondary backup queue to dump later
    }
  }

  void flush() {
    m_mutex.lock_writer();
    // dispatch flushing the valid handles in all queues on my own worker threads
    m_mutex.unlock();
  }
}

#

this should allow you to build a deletion queue that has no blocking and minimal contention when submitting an object to be queued for deletion

#

but the challenge is how would you generalize it to handle an arbitrary count of threads you don't necessarily control

nocturne hill Dec 6, 2023, 1:28 PM

#

try_lock_reader already creates some contention

molten galleon Dec 6, 2023, 1:29 PM

#

yeah it's like one atomic CAS though and it won't ever block the thread trying to submit something to delete

#

not the end of the world

nocturne hill Dec 6, 2023, 1:29 PM

#

this looks overcomplicated

molten galleon Dec 6, 2023, 1:29 PM

#

yeah I mean I had the idea in my head so I was just investigating how feasible it is

nocturne hill Dec 6, 2023, 1:29 PM

#

you can just collect stuff-to-be-deleted into a list (e.g. std::vector) and then append it to a per-VkQueue delet queue at submit time or whateverrr

molten galleon Dec 6, 2023, 1:30 PM

#

I think daxa has something very roughly resembling this idea, potentially implemented in a completely different way though

#

yeah, though you'd have to lock it to delete something

#

which is probably not too bad in reality, but the possibility seemed intriguing

nocturne hill Dec 6, 2023, 1:31 PM

#

well yes locks are good actually

#

efficient waiting

#

given that submit is a relatively heavy operation this seems like a good time

#

you need to lock stuff anyways because access to VkQueue needs to be externally synced

molten galleon Dec 6, 2023, 1:34 PM

#

I don't think I'd need to sync access to any queue to destroy stuff, I can just calculate when it's safe to delete an object based on frames in flight or timeline semaphore values like you would with any other deletion queue implementation

molten galleon Dec 6, 2023, 3:43 PM

#

also for the hash map thing - I don't think I would need a wait free hashmap, just some kind of fixed size structure like

std::pair<std::queue<Thing>, std::mutex> m_queues[QUEUE_CAPACITY];

void queue_delete(Thing t) {
  auto& [queue, mutex] = m_queues[hash_index(GetCurrentThreadId())];
  std::lock_guard lock(mutex);
  queue.push_back(t);
}

#

if my problem/goal would be to reduce overall contention at least

#

but yeah it's probably overcomplicated, but definitely interesting

olive sapphire Dec 6, 2023, 7:00 PM

#

BTW, don't ever use STL hashmaps. They're extremely slow.

old harbor Dec 6, 2023, 7:27 PM

#

they are fine

#

you dont always need nanosecond-trading-realtime-capabilities everywhere

olive sapphire Dec 6, 2023, 7:36 PM

#

When I used C++ hashmaps, they were taking 30-50 microseconds per query. I know for a fact that hashmaps can be 100x faster than that easily. The reason I found that was that they were being a performance bottleneck when I was querying uniform locations, characters according to their glyphset, etc.

#

Weirdly enough, I made a naive implementation without any optimizations in about 100 lines of C, and it was less than .2 microseconds per query, proving my point.

old harbor Dec 6, 2023, 7:37 PM

#

if you use them in a hotpath sure

#

if you use them to collect resources to be cleared later, its not going to kill anyone

olive sapphire Dec 6, 2023, 7:39 PM

#

I would say if you're using them for uniforms, it might just be better to query the uniform location every call than to use a filthy slow STL hashmap.

molten galleon Dec 6, 2023, 7:40 PM

#

no you'd wanna use explicit uniform locations

olive sapphire Dec 6, 2023, 7:41 PM

#

?

molten galleon Dec 6, 2023, 7:41 PM

#

layout (location = 0) uniform u_whatever; and whatnot

#

also yeah while the STL hashmap is hilariously bad it also... mostly doesn't matter for my use case, not to mention requires propogating an opinionated container since this is a middleman lib

olive sapphire Dec 6, 2023, 7:41 PM

#

I didn't know that was even possible, but I wouldn't trust it as much since shaders can change easily.

molten galleon Dec 6, 2023, 7:42 PM

#

I like maltke skarupke's flat_hash_map, I also like absl::flat_hash_map, eastl has a good one

#

but I'd have to probably set some kind of compile flag to let me override the hashmap if/when I really care

olive sapphire Dec 6, 2023, 7:44 PM

#

molten galleon `layout (location = 0) uniform u_whatever;` and whatnot

Actually, that's a great tip, thanks.

molten galleon Dec 6, 2023, 7:44 PM

#

olive sapphire I didn't know that was even possible, but I wouldn't trust it as much since shad...

in practice it's perfectly fine yeah

#

in vulkan you straight up can't have named uniform locations*

*unless you use shader reflection to go very far out of your way to reimplement it out of spite or something

olive sapphire Dec 6, 2023, 7:45 PM

#

Only in OpenGL? Why would Vulkan regress?

molten galleon Dec 6, 2023, 7:45 PM

#

because it's not a regression, working with strings in deep render code is just an antipattern

#

a good mantra to follow is that a shader isn't a resource, it's part of your code

olive sapphire Dec 6, 2023, 7:46 PM

#

Oh I thought you meant numbered. Opengl has confused me with its terminology by constantly referring to numbers as "names"

nocturne hill Dec 6, 2023, 7:46 PM

#

olive sapphire When I used C++ hashmaps, they were taking 30-50 microseconds per query. I know ...

are you sure you didn't compile in debug

#

idk what your query is but 30-50 microseconds per .find is not realistic even with shitty impls

olive sapphire Dec 6, 2023, 7:47 PM

#

I was, it was required for performance querying.

nocturne hill Dec 6, 2023, 7:47 PM

#

so it likely was a mistake on your side

#

but ye whatever

old harbor Dec 6, 2023, 7:47 PM

#

olive sapphire Oh I thought you meant numbered. Opengl has confused me with its terminology by ...

hehe thats indeed super sutpid

molten galleon Dec 6, 2023, 7:47 PM

#

you always profile in release

#

debug numbers don't really matter

olive sapphire Dec 6, 2023, 7:47 PM

#

It doesn't provide symbols when I'm in release.

nocturne hill Dec 6, 2023, 7:48 PM

#

idk if ur using ms toolchain but youk now...... generally you can compile select TUs with optimizations

#

and keep your debug info

olive sapphire Dec 6, 2023, 7:48 PM

#

It also actually inlines my inline functions, which makes it hard to do anything regarding them.

nocturne hill Dec 6, 2023, 7:48 PM

#

clang -O3 -g -c IMPORTANT.c

olive sapphire Dec 6, 2023, 7:48 PM

#

yeah I'm using MSVC

nocturne hill Dec 6, 2023, 7:49 PM

#

well with msvc I'm sure you could've figured something out as well

old harbor Dec 6, 2023, 7:49 PM

#

doesnt cmake come with a configuration for that scenario already?

olive sapphire Dec 6, 2023, 7:49 PM

#

I mean I didn't even know release could have symbols.

nocturne hill Dec 6, 2023, 7:49 PM

#

old harbor doesnt cmake come with a configuration for that scenario already?

no

old harbor Dec 6, 2023, 7:49 PM

#

Relsomething

nocturne hill Dec 6, 2023, 7:49 PM

#

that's indeed opts with debug info yes

#

but opts destroy debug experience either way

olive sapphire Dec 6, 2023, 7:50 PM

#

CMake doesn't for my setup.

nocturne hill Dec 6, 2023, 7:50 PM

#

RelWithDebInfo

#

what

old harbor Dec 6, 2023, 7:50 PM

#

thats what i meant

nocturne hill Dec 6, 2023, 7:50 PM

#

nocturne hill but opts destroy debug experience either way

so the idea is to have select TUs compiled in debugoptimized and others in debug

#

I actually always use debugoptimized for my programs because I don't have a lot of ceepeepee components and so it doesn't blow out blob size with debug info

#

and at my previous job we also always used debugoptimized for our ue projects because debug was unusable

#

900MiB .pdb 💀

#Rich Text - Uncreatively Named Text Handling Prototype Tool