next hatch
Today we go live with our Gleam application to our first 10 external users
from the client. It's invite only and their private data, so I cannot share a
link.
It's a Gleam/Lustre front end, a lot of LLM calls via AWS Bedrock, and a lot
of Postgres calls.
Excluding our public external libaries it's about 28K+ lines of Gleam which compiles to almost 300K lines of Erlang.
Regarding libraries:
aws4_request already written for us,pog library super reliable and feature complete,We are currently on Gleam 1.13 due to breaking changes in the stdlib after
version 0.67.1 - but will get onto Gleam 1.14 ASAP. I believe we are waiting for
https://github.com/gleam-lang/gleam/issues/5286 to make a release.
Here is the size of the bare umbrella app, before build:
cloc packages
github.com/AlDanial/cloc v 2.04 T=0.13 s (3625.5 files/s, 354906.6 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Gleam 280 3024 1206 28072
JSON 24 1 0 7857
Markdown 30 642 0 1594
SQL 75 336 278 1315
CSS 2 126 34 439
JavaScript 9 35 17 216
Erlang 2 2 0 10
-------------------------------------------------------------------------------
SUM: 477 4274 1692 40728
-------------------------------------------------------------------------------
^ the markdown are chaplin templates
Here is the Lustre web app, after the build:
cloc build/
github.com/AlDanial/cloc v 2.04 T=1.30 s (1503.2 files/s, 327961.3 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Erlang 1513 27152 8297 293391
Gleam 219 5921 18547 50077
Markdown 94 2681 11 7397
JavaScript 42 902 867 5829
JSON 1 0 0 2498
Elixir 8 150 280 425
-------------------------------------------------------------------------------
SUM: 1956 37131 28107 361499
-------------------------------------------------------------------------------
^ not sure what the Elixir is - perhaps a test case or a glitch detection
Here is the DATA PIPELINE APP, after the build
cloc build/
github.com/AlDanial/cloc v 2.04 T=0.39 s (2061.2 files/s, 387872.9 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Erlang 624 10627 5170 91462
Gleam 79 2044 10164 14249
Markdown 48 1441 6 3953
JavaScript 16 313 567 2379
Elixir 6 143 278 389
-------------------------------------------------------------------------------
SUM: 803 14659 16187 120258
-------------------------------------------------------------------------------
vapid flare
That’s exciting! I’m curious are you using squirrel too? Or pog directly?
next hatch
That’s exciting! I’m curious are you using squirrel too? Or pog directly?
We want to use squirrel - I found it after we wrote our first DB calls
So will refactor to it soon!!
vapid flare
Aha I see! Let me know if you run into any problems with it!! 😁
next hatch
We have about 230 DB unit tests - so a bit of rework
One issue - which could become a blocker - in CI we are getting this itermittently - on a library which parses large JSON responses :- https://github.com/gleam-lang/gleam/pull/5252
Currently it fails occasionally in both Gleam 1.13 and 1.14
vapid flare
Mhm that’s interesting, what’s the library and how big are the json responses?
next hatch
I put TRACE=ON in CI, here is a failure :- https://github.com/iindyverse/assemblyai/actions/runs/22915419765/job/66499305631#step:5:1142
Here is the JSON (right hand side) :- https://www.assemblyai.com/docs/api-reference/transcripts/get
next hatch
Mhm that’s interesting, what’s the library and how big are the json responses?
It it's cheap and low risk, perhaps bump that Rust param again before 1.15 released 😉
still shuttle
I still want to figure out a permanent solution to that problem
I thought stacker would do the trick, but I guess it doesn't work how I thought it did or I misunderstood where the problem occurred
next hatch
So we create Gleam decoders programattically from large OpenAPI specs - so we are probably pushing some boundaries here
Workaround for us would be just delete all the decoders for the parts of the API we don't use
Interesting that it is intermittent also - a re-run of failed builds (normally) succeeds
onyx hawk
I thought `stacker` would do the trick, but I guess it doesn't work how I though...
i have a hunch that it might allocate quadratic amounts of memory
next hatch
i have a hunch that it might allocate quadratic amounts of memory
I will see if I can monitor the compiler process locally and see how much memory it tops out at
Minimal compile of a gleam project:
$ gleam new minimal
$ cd minimal
$ /usr/bin/time -l gleam build
Resolving versions
Downloading packages
Downloaded 2 packages in 0.03s
Added gleam_stdlib v0.70.0
Added gleeunit v1.9.0
Compiling gleam_stdlib
Compiling gleeunit
Compiling minimal
Compiled in 0.47s
0.78 real 0.72 user 0.31 sys
143638528 maximum resident set size
0 average shared memory size
0 average unshared data size
0 average unshared stack size
14445 page reclaims
336 page faults
0 swaps
0 block input operations
0 block output operations
30 messages sent
32 messages received
0 signals received
2766 voluntary context switches
19018 involuntary context switches
863254865 instructions retired
275369613 cycles elapsed
17499584 peak memory footprint
The troublesome library:
$ /usr/bin/time -l gleam build
Downloading packages
Downloaded 21 packages in 0.06s
Compiling gleam_stdlib
Compiling gleam_json
Compiling non_empty_list
Compiling gleam_http
Compiling gleam_javascript
Compiling oas_generator_utils
Compiling castor
Compiling filepath
Compiling glam
Compiling splitter
Compiling glexer
Compiling glance
Compiling glance_printer
Compiling gleam_crypto
Compiling gleeunit
Compiling justin
Compiling snag
Compiling midas
Compiling oas
Compiling simplifile
Compiling oas_generator
Compiling assemblyai
Compiled in 0.21s
0.30 real 0.18 user 0.10 sys
79052800 maximum resident set size
0 average shared memory size
0 average unshared data size
0 average unshared stack size
5072 page reclaims
14 page faults
0 swaps
0 block input operations
0 block output operations
0 messages sent
0 messages received
0 signals received
66 voluntary context switches
87 involuntary context switches
2672009668 instructions retired
813577429 cycles elapsed
62359232 peak memory footprint
Also, I never saw it fail locally (zsh, macOS) - it fails on github actions (Ubuntu).
Could be something differing in OS settings / stack size etc
next hatch
Hmm perhaps jump to 1.15 will be the solution - haven't seen a build fail on 1.15 yet. Only 1.13 and 1.14
onyx hawk
the limit was bumped by some amount yeah, but i think it will still fail if you push it far enough