hushed plank Jan 4, 2025, 3:16 PM

#

my Dockerfoo is lacking

shrewd jackal Jan 4, 2025, 3:16 PM

#

so the previous steps should be cached in the future

#

I accidentally had it at the very top, so it pulled down everything again for each update

shrewd jackal Jan 4, 2025, 3:17 PM

#

hushed plank my Dockerfoo is lacking

well, it'd be alpine foo to make sure apk pulls the same thing

hushed plank Jan 4, 2025, 3:18 PM

#

https://superuser.com/questions/1055060/how-to-install-a-specific-package-version-in-alpine

#

oof, that is mostly from 2016

#

The official recommendation for pinning exact package versions is to keep your own mirror / repository with all the specific package and their versions that you may want to use.

fresh remnant Jan 4, 2025, 3:20 PM

#

GitHub allows you to set up webhooks, which can send a message whenever a repo is updated (amongst a great many other things...). Might be worthwhile checking into that, instead of regular polling.

hushed plank Jan 4, 2025, 3:21 PM

#

that will need setup by @icy agate

#

afaik only he can add/change hooks

fresh remnant Jan 4, 2025, 3:22 PM

#

Yes, but it would only need to be set up once. I did it for work repos... they gave me permissions to set it up, I did, then they took the permissions away... which didn't affect the webhook at all.

hushed plank Jan 4, 2025, 3:23 PM

#

nice

shrewd jackal Jan 4, 2025, 3:24 PM

#

it pulling newer versions should no longer happen, since docker caches old build steps
is it time to reprocess all 15k commits again? 😄

fresh remnant Jan 4, 2025, 3:24 PM

#

I mirrored the company public repos internally (this was before we went all-in on github). Even kept track of deleted archives by moving the internal mirror to an "archived" tree.

hushed plank Jan 4, 2025, 3:24 PM

#

well I hope he will do it; having notifications here in discord for all github repo events will be nice too, not just for the commits to master, in a separate channel

hushed plank Jan 4, 2025, 3:25 PM

#

shrewd jackal it pulling newer versions should no longer happen, since docker caches old build...

can you do it, while keeping the old data until it is updated?

fresh remnant Jan 4, 2025, 3:25 PM

#

Of course, I wrote that code in Go... this was a few years before V got started.

shrewd jackal Jan 4, 2025, 3:25 PM

#

hushed plank can you do it, while keeping the old data until it is updated?

yes, I added a queued state a while ago

hushed plank Jan 4, 2025, 3:25 PM

#

i.e. update the graphs gradually, not just as a clean slate

#

nice

shrewd jackal Jan 4, 2025, 3:25 PM

#

enum ResultState {
    pending
    success
    failed                   // generic error during measurements
    failed_too_old           // no cmd/v
    failed_vc_compile        // vc compile failed by oldv
    failed_vboostrap_compile // vc boostrap failed
    failed_vself_compile     // vself compile failed
    queued                   // success, but queued for re-processing
}

hushed plank Jan 4, 2025, 3:26 PM

#

I'll have to modify commits.state = 1 in the report query too?

fresh remnant Jan 4, 2025, 3:26 PM

#

"failed_vboostrap_compile" -> "failed_vbootstrap_compile"? 🙂

shrewd jackal Jan 4, 2025, 3:27 PM

#

hushed plank I'll have to modify `commits.state = 1` in the report query too?

yes

fresh remnant Jan 4, 2025, 3:27 PM

#

Probably not worth the change now, though.

shrewd jackal Jan 4, 2025, 3:27 PM

#

fresh remnant Probably not worth the change now, though.

well it doesnt matter, in the database is just an int

hushed plank Jan 4, 2025, 3:27 PM

#

are the numbers stable?

#

commits.state = 1 or commits.state = 7 ?

fresh remnant Jan 4, 2025, 3:27 PM

#

int in the database is fine, but if it changes the column name...

shrewd jackal Jan 4, 2025, 3:27 PM

#

hushed plank are the numbers stable?

yes, thats why its at the end

hushed plank Jan 4, 2025, 3:27 PM

#

cool

#

changing

shrewd jackal Jan 4, 2025, 3:28 PM

#

fresh remnant int in the database is fine, but if it changes the column name...

why would it? that's defined somewhere else

#


@[table: 'commits']
struct Commit {
    commit_hash string @[primary]
    date        time.Time
mut:
    tested                 time.Time
    state                  ResultState
        // ...
}

fresh remnant Jan 4, 2025, 3:29 PM

#

Back to the docker thing... yes, it caches all the layers from the previous build, and only creates new ones from any change point to the end. It can't just do one in the middle because of how the layers are built on top of the previous.

hushed plank Jan 4, 2025, 3:29 PM

#

oh, the DB is ~14.5MB now

shrewd jackal Jan 4, 2025, 3:29 PM

#

fresh remnant Back to the docker thing... yes, it caches all the layers from the previous buil...

that's why I reordered the steps, so the dynamic part (fast.v) is at the end #1312494584151933001 message

fresh remnant Jan 4, 2025, 3:30 PM

#

Yup

#

I was just confirming. 🙂

hushed plank Jan 4, 2025, 3:31 PM

#

sqlite supports name in (1,7), nice

shrewd jackal Jan 4, 2025, 3:32 PM

#

name renxSUS

hushed plank Jan 4, 2025, 3:33 PM

#

pushed

#

#

it seems to work locally

#

just merged a commit

#

lets see when it will be visible in the graphs

#

495adc9cee03aa711b96f1b41b32073670375b91

shrewd jackal Jan 4, 2025, 3:40 PM

#

hushed plank lets see when it will be visible in the graphs

its still processing the last few days

#

so once its done with those and then a 10min delay

hushed plank Jan 5, 2025, 8:52 AM

#

https://fast.leodev.xyz/v_hello.120.html#csize

#

#

I think that changing gcc/libc versions are not enough to explain the last jump 🤔

#

because while they could potentially explain the speed differences, that can not be, for the size of the generated .c file

#

I think V for some reason started to produce larger .c files in the container

#

according to that graph

#

but locally, I can not see it:

#0 10:55:11 ^ master /v/oo>./v -o - examples/hello_world.v |wc
   1943    5225   45548

#

i.e. ~46KB, not 200KB as it is in the graph for the latest commits

#

on the other hand:

#0 10:55:13 ^ master /v/oo>./v -cc musl-gcc -o - examples/hello_world.v |wc
   5665   21589  200000

#

which is a lot closer

#

it could be, that V started to detect the environment as a musl one, instead of a libc one 🤔

#

which could potentially be caused by changes to Alpine and its packages 🤔

#

is there a /etc/alpine-release file inside the container?

#

and what is the output of ls -l /proc/self/map_files/ ?

#

oh, interesting ... ./v -cc gcc -o - examples/hello_world.v |wc is also close to 200KB

#

there is definitely something weird going on ... the size for the generated .c file for -cc tcc and -cc gcc on the same system, should be ~ equal

#

passing -d no_backtrace reduces the size a lot

#

I think the backtrace code for compilers != tcc, has string interpolation, and other v features that later can not be eliminated by markused

shrewd jackal Jan 5, 2025, 10:15 AM

#

It processed the 2 new commits automatically

shrewd jackal Jan 5, 2025, 10:15 AM

#

hushed plank I think that changing gcc/libc versions are not enough to explain the last jump ...

gcc version did not change

#

I have an idea

#

I added —fresh_tcc=false

#

So because no compiler is explicitly passed

#

It now builds for gcc instead of tcc (because it can’t find tcc)

hushed plank Jan 5, 2025, 10:17 AM

#

do you pass -cc when doing the measurements for the generated c size for the hello world?

shrewd jackal Jan 5, 2025, 10:18 AM

#

No

shrewd jackal Jan 5, 2025, 10:18 AM

#

shrewd jackal So because no compiler is explicitly passed

.

hushed plank Jan 5, 2025, 10:19 AM

#

but then it should be closer to what ./v -o - examples/hello_world.v |wc produces (i.e. 46KB)

hushed plank Jan 5, 2025, 10:19 AM

#

shrewd jackal It processed the 2 new commits automatically

that is excellent news

shrewd jackal Jan 5, 2025, 10:20 AM

#

hushed plank but then it should be closer to what `./v -o - examples/hello_world.v |wc` produ...

Why? Why would it compile for TCC when it can’t find TCC?

hushed plank Jan 5, 2025, 11:16 AM

#

the problem is that the generated code for gcc/clang is big, not that the one for tcc is small

#

I've improved it a bit

#

but there are still some problems to solve

hushed plank Jan 5, 2025, 3:27 PM

#

#

seems to be working fine now

#

for tcc, the generated .c file continues to be ~45KB

#

while for != tcc, the generated .c file for hello world is now not 200KB, but just ~55KB

#

which hopefully will speed up more the CI jobs that run with clang/msvc/gcc too

#

because of the many small _test.v files there

#

I still have to fix the windows jobs though

#

before merging https://github.com/vlang/v/pull/23380

shrewd jackal Jan 5, 2025, 4:45 PM

#

currently working on the "design" of regression/improvement messages

#

example:

#

5f3ced7213

Performance has improved 📈
-# Commited <t:1597435052>, Tested <t:1735589209>

self is as expected

-# 189642857 total

hello world has improved 📈

total 45.285ms -> 39.642ms 📈
-# 39642857 total

#

or for a more recent commit:

#

84fbe2728d

Performance has improved 📈
-# Commited <t:1716271916>, Tested <t:1733831379>

self has improved 📈

- total 1.272s -> 1.087s 📈
- check 281.941ms -> 205.179ms 📈
- cgen 587.563ms -> 483.025ms 📈
- total_stages 1.185s -> 999.627ms 📈
-# 1087539885 total, 59539285 scan, 250174929 parse, 205179217 check, 29616142 transform, 483025641 cgen, 999627716 total_stages

hello world has improved 📈

- total 112.274ms -> 101.812ms 📈
- check 17.431ms -> 13.819ms 📈
- cgen 46.303ms -> 39.813ms 📈
- total_stages 101.984ms -> 91.609ms 📈
-# 101812714 total, 6860071 scan, 30981357 parse, 13819785 check, 3133357 transform, 39813571 cgen, 91609071 total_stages

hushed plank Jan 5, 2025, 4:48 PM

#

do you use a hysteresis and some smoothing window?

shrewd jackal Jan 5, 2025, 4:48 PM

#

no; it's currently just >10% improvement over previous commit

hushed plank Jan 5, 2025, 4:50 PM

#

you do have dedicated hardware, so it is probably good enough

#

the noise is probably smaller than 10%

#

and the values are already averaged, so should be relatively stable

shrewd jackal Jan 5, 2025, 4:57 PM

#

why did twemoji make the chart upwards red 📈 and trend down blue 📉

#

renNotLikeThis

hushed plank Jan 5, 2025, 5:03 PM

#

what is twemoji ?

#

perhaps the association is up -> hot -> red, and down -> cold -> blue

#

is there a == chart?

#

although without it (since most commits will be like that) the output may be clearer to scan

shrewd jackal Jan 5, 2025, 5:49 PM

#

hushed plank what is twemoji ?

The library they use for rendering emojis

#

Made by twitter

#

Because each OS does their own rendering, and it standardizes things

shrewd jackal Jan 5, 2025, 5:50 PM

#

hushed plank is there a == chart?

I searched for one but didn’t find one

fresh remnant Jan 5, 2025, 5:54 PM

#

red up makes sense from a "it's heating up!" meaning getting better/faster, where blue down implies "cooling down"... losing the upward momentum.

#

So ... twemoji == TWitterEMOJIs. Does help to standardize... if everyone uses them.

hushed plank Jan 5, 2025, 5:56 PM

#

as the old saying goes, the good thing about standards is that there are many to choose from

shrewd jackal Jan 5, 2025, 6:08 PM

#

Looks like Apple made the same choice

#

So it probably is indeed related to temperature

#

Im thinking of replacing it with a green/red up/down arrow

hushed plank Jan 5, 2025, 6:09 PM

#

please no

#

red/blue is a better combo

shrewd jackal Jan 5, 2025, 6:10 PM

#

I mean like 🔼🔽 but colored

hushed plank Jan 5, 2025, 6:10 PM

#

red/blue is both mnemonic (two people independently came to the same conclusion in this very thread)

#

and it is less likely to be mistaken, unlike red/green (red/green color blindness affects 8-10% of the male population - see https://en.wikipedia.org/wiki/Color_blindness)

idle furnace Jan 5, 2025, 6:23 PM

#

hushed plank

Error opening link

hushed plank Jan 5, 2025, 6:24 PM

#

refresh the page

shrewd jackal Jan 5, 2025, 6:28 PM

#

it will work eventually after a few refreshes

idle furnace Jan 5, 2025, 6:43 PM

#

Worked the 2nd time

shrewd jackal Jan 7, 2025, 9:46 AM

#

2dd65872dc

Performance has degraded 📉
-# Commited <t:1730802922>, Tested <t:1736180569>

self has degraded 📉

- Total 1.102s -> 1.261s (+13%) 📉
- parse 296.536ms -> 330.806ms (+10%) 📉
- check 186.961ms -> 211.311ms (+12%) 📉
- cgen 458.623ms -> 555.326ms (+17%) 📉
- Total Stages 1.008s -> 1.165s (+13%) 📉
-# 1.261s Total, 65.121ms scan, 330.806ms parse, 211.311ms check, 37.975ms transform, 555.326ms cgen, 1.165s Total Stages

hello world has degraded 📉

- Total 105.054ms -> 121.348ms (+13%) 📉
- parse 34.624ms -> 39.102ms (+11%) 📉
- check 14.343ms -> 18.079ms (+21%) 📉
- cgen 37.935ms -> 45.647ms (+17%) 📉
- Total Stages 94.414ms -> 110.601ms (+15%) 📉
-# 121.348ms Total, 7.422ms scan, 39.102ms parse, 18.079ms check, 3.592ms transform, 45.647ms cgen, 110.601ms Total Stages

#

(regression was fixed a few commits later)

hushed plank Jan 7, 2025, 9:48 AM

#

can you please make it process the last 2 days?

#

for https://fast.leodev.xyz/v_hello.7.html

shrewd jackal Jan 7, 2025, 9:49 AM

#

done, it's not picking up new commits by itself because it's reprocessing old ones

hushed plank Jan 7, 2025, 9:50 AM

#

I do not care about the old ones (they are already done, and are curious only for finding anomalies)

#

I care about the new ones

#

can it be setup to always process the new ones, at least once a day?

#

the current last commit is af1ef92917ae318bff759d945d439582e67fac84 from 2025-01-05

#

and today is 2025-01-07

shrewd jackal Jan 7, 2025, 10:06 AM

#

making it sync every 10th commit now 👍

hushed plank Jan 7, 2025, 10:07 AM

#

excellent

#

is that ~ every 30 minutes?

shrewd jackal Jan 7, 2025, 10:09 AM

#

more like once an hour

#

it depends wildly

#

iirc it's more like 4 mins per commit atm, so every 40mins

hushed plank Jan 7, 2025, 10:10 AM

#

nice

#

I hoped for 1 day, but once per hour is even better

shrewd jackal Jan 7, 2025, 10:21 AM

#

time per commit: 3:35.530

#

so every 35mins

#

(assuming my code works)

#

the regression "detector" is now also running (the one that produces the messages comparing new commits to the previous one)

hushed plank Jan 7, 2025, 10:24 AM

#

nice

shrewd jackal Jan 7, 2025, 10:24 AM

#

will test it for a bit and then ask alex for a webhook in here

shrewd jackal Jan 7, 2025, 11:19 AM

#

hushed plank can you please make it process the last 2 days?

processed

#

nice cgen drop

#

from your gcc markused

hushed plank Jan 7, 2025, 11:25 AM

#

thanks

shrewd jackal Jan 7, 2025, 1:35 PM

#

V panic: : failed to get latest commit: db.sqlite.SQLError: out of memory (21) (SELECT commit_hash, date, tested, state, sysname, nodename, release, version, machine, cc_version, compiler_size, v_self_default_id, v_self_skip_unused_id, v_self_no_skip_unused_id, v_hello_default_id, v_hello_skip_unused_id, v_hello_no_skip_unused_id FROM commits WHERE state = ?1 OR state = ?2 ORDER BY date DESC LIMIT ?3;)

#

welp

#

there's 60gb free 😄

hushed plank Jan 7, 2025, 1:36 PM

#

does it work, if you restart it?

shrewd jackal Jan 7, 2025, 1:36 PM

#

and it works locally

#

well it crashes the container and it restarts automatically

#

so no

hushed plank Jan 7, 2025, 1:36 PM

#

perhaps the container has some limit, and it hits it

shrewd jackal Jan 7, 2025, 1:37 PM

#

just for more context: this is the new regression detector

hushed plank Jan 7, 2025, 1:37 PM

#

what do the system logs say (on the host)?

shrewd jackal Jan 7, 2025, 1:37 PM

#

nothing

hushed plank Jan 7, 2025, 1:38 PM

#

they may have more details about what exact limit was reached

shrewd jackal Jan 7, 2025, 1:38 PM

#

and it works locally

#

these are all the logs:

2025-01-07T13:36:22.973420057Z 2025-01-07T13:36:22.973350Z [FATAL] failed to get latest commit: db.sqlite.SQLError: out of memory (21) (SELECT `commit_hash`, `date`, `tested`, `state`, `sysname`, `nodename`, `release`, `version`, `machine`, `cc_version`, `compiler_size`, `v_self_default_id`, `v_self_skip_unused_id`, `v_self_no_skip_unused_id`, `v_hello_default_id`, `v_hello_skip_unused_id`, `v_hello_no_skip_unused_id` FROM `commits` WHERE `state` = ?1 OR `state` = ?2 ORDER BY `date` DESC LIMIT ?3;)
2025-01-07T13:37:23.251752447Z 2025-01-07T13:37:23.251682Z [FATAL] failed to get latest commit: db.sqlite.SQLError: out of memory (21) (SELECT `commit_hash`, `date`, `tested`, `state`, `sysname`, `nodename`, `release`, `version`, `machine`, `cc_version`, `compiler_size`, `v_self_default_id`, `v_self_skip_unused_id`, `v_self_no_skip_unused_id`, `v_hello_default_id`, `v_hello_skip_unused_id`, `v_hello_no_skip_unused_id` FROM `commits` WHERE `state` = ?1 OR `state` = ?2 ORDER BY `date` DESC LIMIT ?3;)
2025-01-07T13:37:23.251792671Z V panic: : failed to get latest commit: db.sqlite.SQLError: out of memory (21) (SELECT `commit_hash`, `date`, `tested`, `state`, `sysname`, `nodename`, `release`, `version`, `machine`, `cc_version`, `compiler_size`, `v_self_default_id`, `v_self_skip_unused_id`, `v_self_no_skip_unused_id`, `v_hello_default_id`, `v_hello_skip_unused_id`, `v_hello_no_skip_unused_id` FROM `commits` WHERE `state` = ?1 OR `state` = ?2 ORDER BY `date` DESC LIMIT ?3;)
2025-01-07T13:37:23.251812276Z v hash: ca6e3a3
2025-01-07T13:37:23.251817179Z backtrace_symbols is missing => printing backtraces is not available. 2025-01-07T13:37:23.251821396Z Some libc implementations like musl simply do not provide it. ```
(double timestamps due to `-t` in `docker logs`)

hushed plank Jan 7, 2025, 1:39 PM

#

try state in ( x, y )

#

instead of state = x or state = y

shrewd jackal Jan 7, 2025, 1:40 PM

#

don't think that is supported by the orm

hushed plank Jan 7, 2025, 1:40 PM

#

do not use the orm

shrewd jackal Jan 7, 2025, 1:40 PM

#

sqlite cli has no issue:

sqlite> SELECT `commit_hash`, `date`, `tested`, `state`, `sysname`, `nodename`, `release`, `version`, `machine`, `cc_version`, `compiler_size`, `v_self_default_id`, `v_self_skip_unused_id`, `v_self_no_skip_unused_id`, `v_hello_default_id`, `v_hello_skip_unused_id`, `v_hello_no_skip_unused_id` FROM `commits` WHERE `state` = 1 OR `state` = 7 ORDER BY `date` DESC LIMIT 1;
63db71b650a4467e007a2752a76f807e084f17be|1736255999|1736256566|1|Linux|e97439752b4b|6.8.0-47-generic|#47-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 21:40:26 UTC 2024|x86_64|cc (Alpine 14.2.0) 14.2.0|4138064|63897|63897|63896|63899|63899|63898

hushed plank Jan 7, 2025, 1:40 PM

#

pass the query directly to db.exec()

#

to make sure that it is well formed, and to eliminate 1 level of indirection

shrewd jackal Jan 7, 2025, 1:41 PM

#

the query is well formed, I just ran it manually

hushed plank Jan 7, 2025, 1:41 PM

#

some DBs do not support binding parameters in some parts of a query

#

you ran an expanded version probably

#

without bindings

#

try running db.exec() with that exact query (the one you ran locally)

#

but on the container

#

I suspect that the sqlite version on the container is != the one you have locally

#

and that will help eliminate one possible cause

#

although, if that was the case, it would not have worked till now

#

so it is most likely to be something else, related to more recent changes

shrewd jackal Jan 7, 2025, 1:45 PM

#

query also failed

#

I exec'ed into the normal fast container and sqlite in there and the query worked

#

so it's not the sqlite version of alpine

#

I have this:

    println(db.exec('SELECT `commit_hash`, `date`, `tested`, `state`, `sysname`, `nodename`, `release`, `version`, `machine`, `cc_version`, `compiler_size`, `v_self_default_id`, `v_self_skip_unused_id`, `v_self_no_skip_unused_id`, `v_hello_default_id`, `v_hello_skip_unused_id`, `v_hello_no_skip_unused_id` FROM `commits` WHERE `state` = 1 OR `state` = 7 ORDER BY `date` DESC LIMIT 1;') or { panic(err) })

and it panics with the same OOM

hushed plank Jan 7, 2025, 1:46 PM

#

are you using the same sqlite file, but shared to different containers?

shrewd jackal Jan 7, 2025, 1:46 PM

#

locally, it prints

[sqlite.Row{
    vals: ['af1ef92917ae318bff759d945d439582e67fac84', '1736064711', '1736078401', '1', 'Linux', '8af2e92c43f5', '6.8.0-47-generic', '#47-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 21:40:26 UTC 2024', 'x86_64', 'c
c (Alpine 14.2.0) 14.2.0', '4198144', '60589', '60589', '60588', '60591', '60591', '60590']
}]

shrewd jackal Jan 7, 2025, 1:46 PM

#

hushed plank are you using the same sqlite file, but shared to different containers?

yes

#

the webserver has no issues

hushed plank Jan 7, 2025, 1:47 PM

#

try copying the file beforehand to a different place/name

#

and then let one of the containers operate on the copy

hushed plank Jan 7, 2025, 1:47 PM

#

shrewd jackal the webserver has no issues

the webserver sometimes returns 500

shrewd jackal Jan 7, 2025, 1:48 PM

#

made a copy, same issue

hushed plank Jan 7, 2025, 1:48 PM

#

I think that one of the containers may be locking the file

#

hm, then it is not that

shrewd jackal Jan 7, 2025, 1:48 PM

#

hushed plank I think that one of the containers may be locking the file

sqlite handles that and simply waits

shrewd jackal Jan 7, 2025, 1:49 PM

#

hushed plank the webserver sometimes returns 500

I do not see any errors in the webserver logs of the past 2 days

#

the 502s are it not accepting connections

hushed plank Jan 7, 2025, 1:49 PM

#

it could in the same container, but it may have problems doing it across several

shrewd jackal Jan 7, 2025, 1:53 PM

#

ok the CLI is close to OOM too

#

I used .stats on and got

#

sqlite> .timer on
sqlite> .stats on
sqlite> .mode list
sqlite> SELECT `commit_hash`, `date`, `tested`, `state`, `sysname`, `nodename`, `release`, `version`, `machine`, `cc_version`, `compiler_size`, `v_self_default_id`, `v_self_skip_unused_id`, `v_self_no_skip_unused_id`, `v_hello_default_id`, `v_hello_skip_unused_id`, `v_hello_no_skip_unused_id` FROM `commits` WHERE `state` = 1 OR `state` = 7 ORDER BY `date` DESC LIMIT 1;
63db71b650a4467e007a2752a76f807e084f17be|1736255999|1736256566|1|Linux|e97439752b4b|6.8.0-47-generic|#47-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 21:40:26 UTC 2024|x86_64|cc (Alpine 14.2.0) 14.2.0|4138064|63897|63897|63896|63899|63899|63898
Memory Used:                         2188120 (max 2287320) bytes
Number of Outstanding Allocations:   833 (max 842)
Number of Pcache Overflow Bytes:     2022120 (max 2030320) bytes
Largest Allocation:                  87360 bytes
Largest Pcache Allocation:           4368 bytes
Lookaside Slots Used:                74 (max 123)
Successful lookaside attempts:       181
Lookaside failures due to size:      4
Lookaside failures due to OOM:       124
Pager Heap Usage:                    2102616 bytes
Page cache hits:                     2
Page cache misses:                   1101
Page cache writes:                   0
Page cache spills:                   0
Schema Heap Usage:                   3320 bytes
Statement Heap/Lookaside Usage:      41056 bytes
Fullscan Steps:                      19099
Sort Operations:                     1
Autoindex Inserts:                   0
Virtual Machine Steps:               173472
Reprepare operations:                0
Number of times run:                 1

#

it's probably the ORDER BY that is causing the entire commits table to be loaded into memory to be sorted

#

and the V sqlite wrapper is using prepared statements or something that uses 100kb more RAM

hushed plank Jan 7, 2025, 1:55 PM

#

but it is not very big, it is under 100MB

#

I wonder what limits it

shrewd jackal Jan 7, 2025, 1:55 PM

#

oh, the max isn't the limit is it

#

renxICANT

hushed plank Jan 7, 2025, 1:56 PM

#

I think that only the available RAM should be the limit

#

unless it is hitting some docker/containerization thing before that

idle furnace Jan 7, 2025, 1:56 PM

#

You cannot share a single sqlite file among processes or threads

shrewd jackal Jan 7, 2025, 1:56 PM

#

idle furnace You cannot share a single sqlite file among processes or threads

you are wrong

shrewd jackal Jan 7, 2025, 1:57 PM

#

hushed plank unless it is hitting some docker/containerization thing before that

by default there are no memory limits

idle furnace Jan 7, 2025, 1:57 PM

#

I had to put locks around mine to prevent deadlocking

hushed plank Jan 7, 2025, 1:57 PM

#

https://github.com/go-shiori/shiori/issues/546

GitHub

Docker image cannot be used with SQLite in a mounted volume · Issue...

Data Shiori version: latest docker image (1.5.3) Database Engine: SQLite Operating system: N/A CLI/Web interface/Web Extension: N/A Describe the bug / actual behavior If you run the Docker image wi...

idle furnace Jan 7, 2025, 1:57 PM

#

Be interesting if the V version is safe from that.

hushed plank Jan 7, 2025, 1:58 PM

#

I think it is some docker/sqlite interaction

#

the locking may not work well across containers

#

and have edge cases

idle furnace Jan 7, 2025, 1:58 PM

#

May not

#

I had a single thread/process handle all DB calls so the locking could be centralized

hushed plank Jan 7, 2025, 1:59 PM

#

shared filesystems are notorious for locking bugs

idle furnace Jan 7, 2025, 2:00 PM

#

Setup a restful api to get and save data to a central container handling the DB calls. That way other containers in theory could talk to it.

hushed plank Jan 7, 2025, 2:00 PM

#

but still, it is a bit surprising that the error is not a more sensible one

#

and it also happened when the .db file was copied to a separate one

#

so presumably only one embedded sqlite in its own processs tried to use it

idle furnace Jan 7, 2025, 2:04 PM

#

So long as the copy does not have an active lock. My experience though is SQLite creates .journal file while processing

hushed plank Jan 7, 2025, 2:07 PM

#

according to this: https://www.sqlite.org/lang_createindex.html

#

sqlite supports indexes for sorting too

#

so another thing to try is to add one, for date desc

#

so that most queries will hopefully use that, and limit the amout of used resources/locking time

#

something like this:
CREATE INDEX date_idx on commits (date desc);

#

after that, if I run:

explain SELECT `commit_hash`, `date`, `tested`, `state`, `sysname`, `nodename`, `release`, `version`, `machine`, `cc_version`, `compiler_size`, `v_self_default_id`, `v_self_skip_unused_id`, `v_self_no_skip_unused_id`, `v_hello_default_id`, `v_hello_skip_unused_id`, `v_hello_no_skip_unused_id` FROM `commits` WHERE `state` = 1 OR `state` = 7 ORDER BY `date` DESC LIMIT 1;

#

I get 4 OpenRead 2 3893 0 k(2,-,) 0 root=3893 iDb=0; date_idx

#

beside many other lines that I do not understand, but probably it means that the index is used

#

the query plan at least, is certainly shorter than before

idle furnace Jan 7, 2025, 2:16 PM

#

Excellent plan

hushed plank Jan 7, 2025, 2:16 PM

#

it fits in 1 page now

#

while it was 2 screens before

#

and before it had also: 4 Explain 4 0 0 SCAN commits 0

#

while now it is: 5 Explain 5 0 0 SCAN commits USING INDEX date_idx 0

#

before the index:
Memory Used: 2185744 (max 2286176) bytes
after the index, for the same query (in a fresh sqlite3 instance):
Memory Used: 168992 (max 169864) bytes

fresh remnant Jan 7, 2025, 2:26 PM

#

I do prefer "state in (1, 7)" to "state = 1 OR state = 7"... if for no other reason that it is shorter. SQL statements are crazy long anyway, so any method of shortening them is a good method. 🙂

#

Also... is there really a need to list every field individually? Other than having to make the struct slightly larger with any other fields (if you're using ORM), I don't see why you can't just "ignore" the ones you don't care about in that query.

hushed plank Jan 7, 2025, 2:28 PM

#

the query was generated by the ORM

fresh remnant Jan 7, 2025, 2:28 PM

#

Ah, gotcha. Ugh. ORM definitely needs work...

hushed plank Jan 7, 2025, 2:28 PM

#

and it selects only those columns, that are present in the struct on the V side

#

potentially the table can have many more columns

fresh remnant Jan 7, 2025, 2:29 PM

#

That last one could've been just

SELECT * FROM `commits` WHERE `state` in (1, 7) ORDER BY `date` DESC LIMIT 1;

hushed plank Jan 7, 2025, 2:29 PM

#

select * is shorter, but can transfer a lot more data

#

imho the * form is suitable for human written queries

fresh remnant Jan 7, 2025, 2:29 PM

#

Yep, it can... but it's certainly unlikely compared to all those individual fields. 😕

hushed plank Jan 7, 2025, 2:30 PM

#

while the version with the explicitly listed fields is easy for machines to generate

#

and even if more verbose, it is a one time cost, while sending the multiple results can dominate the processing/decoding time

fresh remnant Jan 7, 2025, 2:31 PM

#

Possibly, yes. Possibly no, depending on the efficiency of the SQL parser, etc.

hushed plank Jan 7, 2025, 2:32 PM

#

most likely yes - you could have a query that returns many millions of rows

fresh remnant Jan 7, 2025, 2:32 PM

#

Or only 1.

hushed plank Jan 7, 2025, 2:32 PM

#

anything that can save you from processing even 1 field in them

#

will have disproportionate effects

fresh remnant Jan 7, 2025, 2:32 PM

#

Yep. Trade-offs everywhere.

hushed plank Jan 7, 2025, 2:32 PM

#

fresh remnant Or only 1.

yes, or only 1

#

in which case the parsing time increase can be more

#

but it also depends on whether the DB decided to cache the query, and how it did so

fresh remnant Jan 7, 2025, 2:34 PM

#

One of the goals of optimizing database stuff is to improve queries such that fewer rows are returned, meaning less processing of rows after the query. If you only get back exactly what you wanted, you don't have to have if/else blocks to decide what to do with the rows.

hushed plank Jan 7, 2025, 2:36 PM

#

the same applies to the columns

fresh remnant Jan 7, 2025, 2:36 PM

#

In this particular case, you do something with all of them, but it could likely still be cut down by querying `date` BETWEEN...

#

Especially on the "last 30 days", etc. charts.

hushed plank Jan 7, 2025, 2:37 PM

#

those use "AND commits.date > ${cutoff_ts}" in the WHERE part of the query

#

hopefully they will also benefit from the new index

fresh remnant Jan 7, 2025, 2:38 PM

#

Yes indeed. Anywho, nice side discussion... back to the regularly scheduled fast2 talk. 😉

hushed plank Jan 7, 2025, 2:45 PM

#

#

hushed plank Jan 7, 2025, 3:12 PM

#

I've added the .sql files in the oldv_charts repo, and pushed

#

hushed plank Jan 9, 2025, 9:46 AM

#

fast2.v works very nicely now, when it comes to the latest commits

#

I've merged some this morning, and they are already visible in the graphs

#

excellent work @shrewd jackal, thank you

shrewd jackal Jan 9, 2025, 10:47 AM

#

@hushed plank NVM, that did not actually fix it

#

what fixed it was that I did not specify -prod

#

with -prod it still OOMs sqlite

#

but works without

hushed plank Jan 9, 2025, 10:48 AM

#

are you using gcc or clang?

shrewd jackal Jan 9, 2025, 10:48 AM

#

gcc

#

it's alpine

#

on my mac it works with both on clang

hushed plank Jan 9, 2025, 10:49 AM

#

try also with -prod -cflags -fno-inline-small-functions

shrewd jackal Jan 9, 2025, 10:49 AM

#

i'll try with gcc on my mac

hushed plank Jan 9, 2025, 10:49 AM

#

in the alpine container

shrewd jackal Jan 9, 2025, 10:49 AM

#

well, nix-shell -p libgcc doesn't work for some reason, so no gcc testing on mac ig

hushed plank Jan 9, 2025, 10:50 AM

#

you can try passing -d dynamic_boehm on macos

#

I have not used nix on either linux nor macos

#

(personally I consider it a waste of time, and mostly a source of bugs)

shrewd jackal Jan 9, 2025, 10:51 AM

#

hushed plank try also with `-prod -cflags -fno-inline-small-functions`

that works

hushed plank Jan 9, 2025, 10:51 AM

#

on alpine?

shrewd jackal Jan 9, 2025, 10:51 AM

#

yes

hushed plank Jan 9, 2025, 10:51 AM

#

thanks

#

I think that not passing -prod at all is the better solution

#

the script itself is not very performance sensitive afaik

#

it only measures external programs

shrewd jackal Jan 9, 2025, 10:52 AM

#

that's an entirely different script

hushed plank Jan 9, 2025, 10:53 AM

#

do you pass -prod for the webserver?

shrewd jackal Jan 9, 2025, 10:53 AM

#

we have 3 scripts now:
- fast; does all the measurements
- web; the webserver
- regret; the new regression "detector" (which OOMs sqlite and now works)

#

all 3 are with -prod

hushed plank Jan 9, 2025, 10:54 AM

#

oh, ok

shrewd jackal Jan 9, 2025, 10:55 AM

#

(on my private git instance)

shrewd jackal Jan 9, 2025, 3:56 PM

#

renxPOPCORN

#

Waiting for a new commit

hushed plank Jan 9, 2025, 3:56 PM

#

2020?

#

or for 2025?

shrewd jackal Jan 9, 2025, 3:57 PM

#

A new one for the regressions

hushed plank Jan 9, 2025, 3:57 PM

#

I'll merge https://github.com/vlang/v/pull/23417 in a minute

shrewd jackal Jan 9, 2025, 3:58 PM

#

renxPOPCORN

hushed plank Jan 9, 2025, 3:59 PM

#

done

shrewd jackal Jan 9, 2025, 4:00 PM

#

I’ll let you know how it goes

shrewd jackal Jan 9, 2025, 4:22 PM

#

cc7caf4384

Performance is as expected
-# Commited <t:1736438369>, Tested <t:1736439109>

#

renxHMM

#

The exact stats for hello/self are missing

shrewd jackal Jan 9, 2025, 5:38 PM

#

even weirder, all printlns are missing from the logs somehow
it worked since the webhook fired, so renxSHRUG

#

hello world/self stats are missing if they're unavailable (eg the column is NULL)

#

erm

hushed plank Jan 9, 2025, 5:43 PM

#

try adding unbuffer_stdout() at the start of your program

shrewd jackal Jan 9, 2025, 5:43 PM

#

Commit{
    commit_hash: 'ba9d358d5f170f40e46234ae6b9ad788270189be'
    date: 2025-01-09 16:15:05
    tested: 2025-01-09 16:44:44
    state: success
    sysname: Option(none)
    nodename: Option(none)
    release: Option(none)
    version: Option(none)
    machine: Option(none)
    cc_version: Option(none)
    compiler_size: Option(none)
    v_self_default: Option(none)
    v_self_skip_unused: Option(none)
    v_self_no_skip_unused: Option(none)
    v_hello_default: Option(none)
    v_hello_skip_unused: Option(none)
    v_hello_no_skip_unused: Option(none)
}

#

I think sqlite stil has issues

#

all of those are set

#

works localy

hushed plank Jan 9, 2025, 5:43 PM

#

does it work without -prod?

shrewd jackal Jan 9, 2025, 5:44 PM

#

it does indeed

#

so sqlite with prod on alpine is just broken?

#

but it works for fast and the webserver

#

renxSHRUG

#

renxPOPCORN

#

time to wait for next commit

hushed plank Jan 9, 2025, 5:48 PM

#

shrewd jackal so sqlite with prod on alpine is just broken?

perhaps it is for a specific version of gcc/musl combination

#

you could also try -cflags -O2

#

but imho just do not use -prod - I doubt the benefits are worth the trouble for those programs, since they are not realtime, and most of the time spend in them is in executing other programs anyway

#

you will get faster feedback, and less problems like this one without it

shrewd jackal Jan 9, 2025, 5:50 PM

#

that works

#

so does -O3

#

-O3 was suspiciously fast (2.4s)

hushed plank Jan 9, 2025, 5:51 PM

#

I think that -prod uses -O3 -flto

#

-0 != -O

shrewd jackal Jan 9, 2025, 5:52 PM

#

O3 obv

#

-O3 -flto was 2.7s and also works

hushed plank Jan 9, 2025, 5:53 PM

#

do you pass -cc gcc too?

#

without -prod, you will have to specify the compiler explicitly

shrewd jackal Jan 9, 2025, 5:53 PM

#

good point, it may be using tcc

hushed plank Jan 9, 2025, 5:53 PM

#

otherwise you will get them passed to tcc

#

and it mostly just ignores them

shrewd jackal Jan 9, 2025, 5:54 PM

#

yeah it was using tcc, I will try base gcc first

fresh remnant Jan 9, 2025, 5:54 PM

#

That would explain why it was "suspiciously fast". 🙂

shrewd jackal Jan 9, 2025, 5:54 PM

#

ok base gcc works, time to tune it

fresh remnant Jan 9, 2025, 5:55 PM

#

-prod is the equivalent to -cc cc -O3 -flto, so it will use whatever /usr/bin/cc links to.

shrewd jackal Jan 9, 2025, 5:55 PM

#

even just base gcc takes 14s

fresh remnant Jan 9, 2025, 5:56 PM

#

For me, it is lrwxrwxrwx 1 root root 3 Sep 10 09:07 /usr/bin/cc -> gcc

hushed plank Jan 9, 2025, 5:56 PM

#

on alpine?

shrewd jackal Jan 9, 2025, 5:56 PM

#

gcc with -O3 -flto causes sqlite OOMs again

#

trying -O2 now

#

-O2 works fine

#

I get lrwxrwxrwx 1 root root 3 Jan 9 10:31 /usr/bin/cc -> gcc on alpine

#

what does -flto do?

#

ah link time optimizations

#

trying just -O3 now

#

that also works

fresh remnant Jan 9, 2025, 6:02 PM

#

So a problem in linking on alpine? Odd...

shrewd jackal Jan 9, 2025, 6:02 PM

#

which somehow doesn't affect the other scripts renxSHRUG

#

anyway

#

renxPOPCORN

#

time for new commits

hushed plank Jan 9, 2025, 6:04 PM

#

you should ping @tall isle 🙂

#

or we should find & fix some doc typos/inconsistencies - they are usually fast to pass review/merge, since they do not trigger too many CI jobs

shrewd jackal Jan 9, 2025, 6:07 PM

#

TLDR; when compiled with -prod (or -cflags "-O3 -flto") sqlite throws OOM errors: #1312494584151933001 message
with -cflags -fno-inline-small-functions SELECT returns partial results #1312494584151933001 message

tall isle Jan 9, 2025, 6:09 PM

#

hmm

shrewd jackal Jan 9, 2025, 6:12 PM

#

and only on alpine

fresh remnant Jan 9, 2025, 6:13 PM

#

... which is musl based.

#

Perhaps it might hit other musl based systems... if there are any. 🙂

shrewd jackal Jan 10, 2025, 9:55 AM

#

3b31699a0d

Performance is as expected
-# Commited <t:1736500936>, Tested <t:1736502314>

self is as expected

check 206.469ms -> 203.574ms (+1%)
markused 65.957ms -> 63.833ms (+3%)
-# 1.184s Total, 61.383ms scan, 313.491ms parse, 203.574ms check, 40.677ms transform, 432.296ms cgen, 63.833ms markused, 1.012s Total Stages

hello world is as expected

cgen 10.068ms -> 10.647ms (-5%) 📉
-# 82.539ms Total, 7.058ms scan, 35.791ms parse, 15.169ms check, 3.876ms transform, 10.647ms cgen, 1.049ms markused, 68.831ms Total Stages

#

The +/- for the % are inverted

#

So is the emoji

hushed plank Jan 10, 2025, 9:56 AM

#

I think it is too sensitive - in my experience, launching small executables on linux can vary within 1ms

#

not sure about cgen

shrewd jackal Jan 10, 2025, 9:56 AM

#

It is saying it’s as expected

#

The tolerances for showing as a difference and counting as improvement/degradation are different

hushed plank Jan 10, 2025, 9:58 AM

#

oh, ok

shrewd jackal Jan 10, 2025, 10:16 AM

#

3b31699a0d

Performance is as expected
-# Commited <t:1736500936>, Tested <t:1736502314>

self is as expected

- check 206.469ms -> 203.574ms (-1%)
- markused 65.957ms -> 63.833ms (-3%)
-# 1.184s Total, 61.383ms scan, 313.491ms parse, 203.574ms check, 40.677ms transform, 432.296ms cgen, 63.833ms markused, 1.012s Total Stages

hello world is as expected

- cgen 10.068ms -> 10.647ms (+5%) 📈
-# 82.539ms Total, 7.058ms scan, 35.791ms parse, 15.169ms check, 3.876ms transform, 10.647ms cgen, 1.049ms markused, 68.831ms Total Stages

#

fixed it

hushed plank Jan 10, 2025, 10:16 AM

#

nice

shrewd jackal Jan 10, 2025, 10:16 AM

#

no wait, the emoji was correct NotLikeThis

hushed plank Jan 10, 2025, 10:17 AM

#

do you want them to always show the better direction for the metric?

#

or to reflect 100% the actual change data?

#

both could be valuable alternatives, as long as they are consistent

shrewd jackal Jan 10, 2025, 10:17 AM

#

it shows whatever changed by more than 1%

#

eg for when skip-unused became default, there were positive and negative changes (cgen down, markused up)

hushed plank Jan 10, 2025, 10:19 AM

#

if they show "the better direction" (i.e. up is always whatever is good/more performant/smaller code size etc), the report would be easier to scan at a high level

#

if they always reflect 100% the actual change, you would have to think about what each emoji infographic actually means for each metric

shrewd jackal Jan 10, 2025, 10:21 AM

#

hushed plank if they always reflect 100% the actual change, you would have to think about wha...

already done

#

there's 3 kinds of metrics internally:
- performance (lower is better)
- reverse performance (higher is better, just vlines/s)
- size (lower is better)

#

📎 v-fast-master.zip

#

entire current source code

tall isle Jan 10, 2025, 10:24 AM

#

no change in the current table vision of results?

shrewd jackal Jan 10, 2025, 10:24 AM

#

you mean this?

#

`hello (examples/hello_world.v)` on 3b31699a0d (2025-01-10 09:22:16)

      names      mean                stddev       min       max  samples  noisy  unit
 ----------  --------  --------------------  --------  --------  -------  -----  ----------------
    timings
       scan   7.058ms   ± 54.507us (0.77%)   6.988ms   7.152ms       14
      parse  35.791ms  ± 260.206us (0.73%)  35.283ms  36.314ms       14
      check  15.169ms  ± 100.008us (0.66%)  15.021ms  15.336ms       14
  transform   3.876ms   ± 29.303us (0.76%)   3.835ms   3.936ms       14
   markused   1.049ms    ± 5.229us (0.50%)   1.041ms   1.058ms       14
       cgen  10.647ms  ± 113.253us (1.06%)  10.426ms  10.801ms       14
 all stages  68.831ms  ± 464.683us (0.68%)  68.154ms  69.843ms       14
      total  82.539ms  ± 507.705us (0.62%)  81.696ms  83.551ms       14

  v  source
  v    size    469235          ± 0 (0.00%)    469235    469235       20         bytes
  v   lines     17666          ± 0 (0.00%)     17666     17666       20         lines
  v lines/s    214037       ± 1314 (0.61%)    211439    216240       14         lines per second
  v   types       150          ± 0 (0.00%)       150       150       20         types
  v modules         5          ± 0 (0.00%)         5         5       20         modules
  v   files        65          ± 0 (0.00%)        65        65       20         files

    c  code
    c  size     57141          ± 0 (0.00%)     57141     57141       20         bytes
    c lines      2342          ± 0 (0.00%)      2342      2342       20         lines

tall isle Jan 10, 2025, 10:24 AM

#

on https://fast.leodev.xyz/ I just see charts

shrewd jackal Jan 10, 2025, 10:25 AM

#

ah yes, the website doesn't have tables currently

#

only the charts

hushed plank Jan 10, 2025, 10:25 AM

#

you can download the database itself

#

and analyze the numbers as you wish

#

the link to it is at the bottom of the https://fast.leodev.xyz/index.html page

#

perhaps we should put it at the top too

#

since it is hard to see now without scrolling

shrewd jackal Jan 10, 2025, 12:37 PM

#

36154b8631

Performance is as expected
-# Commited <t:1736511315>, Tested <t:1736512160>

self is as expected

- check 203.574ms -> 205.670ms (+1%)
- markused 63.833ms -> 65.590ms (+3%)
-# 1.186s Total, 61.198ms scan, 312.992ms parse, 205.670ms check, 40.780ms transform, 430.703ms cgen, 65.590ms markused, 1.012s Total Stages

hello world is as expected

-# 82.347ms Total, 7.021ms scan, 35.720ms parse, 15.088ms check, 3.895ms transform, 10.590ms cgen, 1.044ms markused, 68.590ms Total Stages

#

lgtm

tall isle Jan 10, 2025, 12:45 PM

#

good work

shrewd jackal Jan 10, 2025, 1:11 PM

#

@icy agate can we get a channel with a webhook similar to #git-commits for performance of each commit?

icy agate Jan 10, 2025, 10:22 PM

#

shrewd jackal <@592103446384214044> can we get a channel with a webhook similar to <#592363023...

sure

shrewd jackal Jan 11, 2025, 10:16 AM

#

bed28d1e8b

Performance is as expected
-# Commited <t:1736588673>, Tested <t:1736588961>

self is as expected

-# 1.184s Total, 60.918ms scan, 310.808ms parse, 206.737ms check, 40.453ms transform, 429.738ms cgen, 65.435ms markused, 1.010s Total Stages

hello world is as expected

-# 82.250ms Total, 6.988ms scan, 35.575ms parse, 15.096ms check, 3.883ms transform, 10.623ms cgen, 1.044ms markused, 68.461ms Total Stages

shrewd jackal Jan 11, 2025, 10:17 AM

#

icy agate sure

So? 😄

#

It didn’t pick up the other commit, I think I forgot a desc in the sql query

hushed plank Jan 11, 2025, 10:19 AM

#

the other one does not change the compiler, it should be more or less the same

shrewd jackal Jan 11, 2025, 10:47 AM

#

Indeed, but it’s supposed to post a message for every commit and it got skipped because they were too close together

#

No wait, it got skipped because the other one got processed first

#

Uh, this will be annoying. I need to update fast to process older commits first

hushed plank Jan 11, 2025, 10:48 AM

#

can it make a new query, after each commit is re-processed?

shrewd jackal Jan 11, 2025, 10:48 AM

#

Id prioritise pending over queued in that case

hushed plank Jan 11, 2025, 10:49 AM

#

or sort them by 2 fields - first the recent ones (by date), and then by status

#

so that the most recent new ones will be always first

idle furnace Jan 11, 2025, 1:27 PM

#

Priority queue is that a possibility?

shrewd jackal Jan 11, 2025, 1:50 PM

#

it's a SQL select statement...

idle furnace Jan 11, 2025, 1:52 PM

#

Ahh okay

shrewd jackal Jan 11, 2025, 2:07 PM

#

better prioritization is implemented 👍

#

entirety of 2024 is reprocessed with gcc instead of tcc now

tall isle Jan 11, 2025, 2:15 PM

#

cool

#

how long it was?

shrewd jackal Jan 11, 2025, 2:17 PM

#

around 6 days

#

± one or two hours

fresh remnant Jan 11, 2025, 2:18 PM

#

Once it's all finally settled, hopefully will NOT have to go back and reprocess older ones.

shrewd jackal Jan 11, 2025, 2:19 PM

#

renxPRAY

#

ideally, c gen wouldn't be different for tcc/gcc

fresh remnant Jan 11, 2025, 2:28 PM

#

Ideally... but it is in at least a few places.

hushed plank Jan 11, 2025, 5:46 PM

#

did it pickup all the recent commits?

#

although it is probably too early to tell 🤔

#

I'll wait for a few hours

shrewd jackal Jan 11, 2025, 6:00 PM

#

hushed plank did it pickup all the recent commits?

It’s working on them now

#

1c2f1a3504

Performance is as expected
-# Commited <t:1736616349>, Tested <t:1736618238>

self is as expected

-# 1.186s Total, 61.036ms scan, 310.829ms parse, 206.588ms check, 40.626ms transform, 431.259ms cgen, 66.032ms markused, 1.011s Total Stages

hello world is as expected

markused 1.044ms -> 1.056ms (+1%)
-# 82.316ms Total, 7.014ms scan, 35.604ms parse, 15.161ms check, 3.874ms transform, 10.589ms cgen, 1.056ms markused, 68.596ms Total Stages

#

Not gonna post every commit unless there’s an actual impact

shrewd jackal Jan 13, 2025, 8:17 AM

#

added a min delta of 1ms instead of only >1% difference to reduce noise

#

since a lot of the listed changes is something being 10 microseconds faster than last time

icy agate Jan 14, 2025, 3:28 PM

#

shrewd jackal So? 😄

#fast

#

how do i set up the hook?

shrewd jackal Jan 14, 2025, 3:31 PM

#

icy agate how do i set up the hook?

Edit Channel > Integrations > create webhook
Give it a name and then dm me the link

#

I’ll set it up when I’m home

icy agate Jan 14, 2025, 3:56 PM

#

done 🙂

shrewd jackal Jan 14, 2025, 4:12 PM

#

it's live

idle furnace Jan 14, 2025, 4:33 PM

#

Can we all see it?

fresh remnant Jan 14, 2025, 4:36 PM

#

For now, it's a private channel. Will likely be opened after it's proven to show what it should.

icy agate Jan 14, 2025, 5:28 PM

#

done, it's public now

shrewd jackal Jan 14, 2025, 5:52 PM

#

For anyone who didn’t read the entire history: the delay is currently up to 40mins for new commits
This is because we’re also retesting old commits at the same time and only check for new commits every 10 processed commits

Once the backlog is once again cleared (in 2-3 weeks), the delay should be consistently below 15mins

#

10min Sleep between git pulls of the repo, 4 mins for processing and 1min sleep between checking for new results
10mins can be optimized in the future using GitHub webhooks, but we can tackle that in 3 weeks when it becomes relevant

hushed plank Jan 14, 2025, 5:54 PM

#

I just merged a PR.

#

it triggered

#

#fast message

shrewd jackal Jan 14, 2025, 6:00 PM

#

That was quick

fresh remnant Jan 15, 2025, 2:28 PM

#

That's the thing about checking on a schedule... if the merge happens just before the check is done, it triggers right away. If the merge happens just after the check it done, it has to wait the full period until the next check is done.

#

That's why events are a good thing... no time will be wasted getting the signal that a merge has happened, and no cpu time will be wasted checking for merges to happen when they didn't.

shrewd jackal Jan 15, 2025, 4:14 PM

#

it's currently not wasting time checking for merges, it's busy benchmarking old commits

#

and every 10 commits it does a check for new ones

#

each commit taking ~3 to 3.5mins makes the worst case 35mins

fresh remnant Jan 15, 2025, 4:17 PM

#

That's expected until it finally all settles down. Once all the old stuff is benchmarked, and it is all caught up with the current commits, all it will have to do is wait for now ones after that. And that's much simpler with a listen on a port waiting for an event.

shrewd jackal Jan 15, 2025, 4:17 PM

#

that's a problem for future me

shrewd jackal Jan 17, 2025, 8:06 AM

#

we finished yesterday with 2023

hushed plank Jan 17, 2025, 8:07 AM

#

nice

#

it seems to be a bit faster than before

shrewd jackal Jan 17, 2025, 8:08 AM

#

took 6 days and 2 hours

hushed plank Jan 17, 2025, 8:09 AM

#

but it may be only my impression

shrewd jackal Jan 17, 2025, 8:09 AM

#

sounds about the same as last time

hushed plank Jan 17, 2025, 8:09 AM

#

I like how there are no gaps now

#

and everything looks mostly smoother now, except for the rarer spikes

#

it makes it much easier to find the outliers

#

we should soon work on optimizing parsing and removing support for old features like fn f<T>() etc

#

and moving more checks from it to the checker

#

I think it can improve its performance a lot

#

and especially for small programs, it is now the dominant factor

#

because of all the reparsing of builtin, strconv and strings that happens for even just hello world

#

#

not doing it at all, based on some caching mechanism, could be even better

#

but it would have to be very reliable

icy agate Jan 17, 2025, 5:02 PM

#

that's a lot more even than i expected

hushed plank Jan 17, 2025, 5:05 PM

#

that is the value of graphs

#

throughout the history of the project, the relative slowness of the different stages changed several times, but currently the parser dominates for small programs

shrewd jackal Jan 20, 2025, 7:32 PM

#

didn't get to adding commit messages

#

rescheduled to this saturday

shrewd jackal Jan 22, 2025, 6:58 AM

#

2022 done since 6h ago

hushed plank Jan 22, 2025, 7:57 AM

#

the graphs seem nice to me, without obvious surprises

shrewd jackal Jan 25, 2025, 1:09 PM

#

working on it now

shrewd jackal Jan 25, 2025, 1:24 PM

#

done

#

also backfilled all old commits

hushed plank Jan 25, 2025, 1:24 PM

#

nice

shrewd jackal Jan 25, 2025, 1:25 PM

#

so you could add it to the graphs now too

#

the column is just commit_title

shrewd jackal Jan 28, 2025, 8:11 AM

#

2021 done as of 12h ago

hushed plank Jan 28, 2025, 8:12 AM

#

nice

hushed plank Jan 29, 2025, 1:33 PM

#

@shrewd jackal I have a local patch, that will change the output of -stats like this:

#0 15:33:15 ^ master ~/code/v>./v -stats examples/hello_world.v
        V  source  code size:      17627 lines,      82343 tokens,     468173 bytes,   148 types,     5 modules,    64 files
generated  target  code size:       1923 lines,      45410 bytes
compilation took: 260.961 ms, compilation speed: 67546 vlines/s, cgen threads: 3

#

(note the new tokens metric)

#

will it be a problem for the current setup?

shrewd jackal Jan 29, 2025, 1:34 PM

#

no

hushed plank Jan 29, 2025, 1:34 PM

#

ok

#

thanks

shrewd jackal Jan 29, 2025, 1:34 PM

#

but if you want to track it, I should add it first or we'll need to reprocess

hushed plank Jan 29, 2025, 1:35 PM

#

no, I am not interested in tracking it for now

shrewd jackal Jan 29, 2025, 1:36 PM

#

hushed plank Jan 29, 2025, 1:36 PM

#

it is mostly useful for now, to consider the impact of changing some of token.Token's fields to use less bits

shrewd jackal Jan 29, 2025, 1:36 PM

#

the parser should be fine if you add anything, as long as you don't rename anything

hushed plank Jan 29, 2025, 1:37 PM

#

very nice

#

the rest are not changed, so it will work

shrewd jackal Jan 30, 2025, 7:05 AM

#

We’re done

hushed plank Jan 30, 2025, 7:44 AM

#

no 2020?

shrewd jackal Jan 30, 2025, 8:09 AM

#

up to 2020-04-21

#

that's the last supported one

hushed plank Jan 30, 2025, 8:10 AM

#

my bad, I should have looked at the Total, ms chart

#

I forgot that the Stages, ms one depends on stuff that became available later

shrewd jackal Jan 30, 2025, 8:13 AM

#

technically the same thing is happening with total as well, but it falls back to measuring the full execution time

shrewd jackal Feb 15, 2025, 10:00 AM

#

somehow the container is leaking thousands of zombie git processes thonk

#

a restart fixes it but it's weird

hushed plank Feb 15, 2025, 11:02 AM

#

how do you call git?

#

with os.system, os.execute or something else?

shrewd jackal Feb 15, 2025, 12:20 PM

#

combination of os.system and os.execution depending on if I need the output

#

they might also be from oldv

#

I simply don't know

#

shouldn't ps -A have listed the arguments too? 🤔

hushed plank Feb 15, 2025, 12:21 PM

#

can it show the full command path?

#

I think it should

#

shrewd jackal Feb 15, 2025, 12:22 PM

#

it does on macos but not on ubuntu renxSHRUG

hushed plank Feb 15, 2025, 12:22 PM

#

for me it does

#

perhaps the docker ps has different options?

#

or the alpine one, not sure where you are running it

shrewd jackal Feb 15, 2025, 12:22 PM

#

I am running it on the host which is ubuntu

#

it's not logging the commands for any process though

#

so it's not related to docker

hushed plank Feb 15, 2025, 12:23 PM

#

no idea then

shrewd jackal Feb 15, 2025, 12:23 PM

#

cat /proc/xxx/cmdline is also empty

#

for the zombie git processes

hushed plank Feb 15, 2025, 12:24 PM

#

try running /usr/bin/true abcdef

#

with both os.system, and with os.execute

#

to see if you can reproduce it on the host

#

in a loop, say 20 times

#

import os
for _ in 0..20 {
    os.execute('/usr/bin/true abcdef execute')
}
for _ in 0..20 {
    os.system('/usr/bin/true abcdef system')
}
println('done')

#

this for me passed, without creating zombies

#

shrewd jackal Feb 15, 2025, 12:27 PM

#

what if the program doesn't exit

hushed plank Feb 15, 2025, 12:28 PM

#

if the program does not exit, it will not be a zombie

shrewd jackal Feb 15, 2025, 12:28 PM

#

I mean the v program which spawns them

hushed plank Feb 15, 2025, 12:28 PM

#

it will be either running or paused

hushed plank Feb 15, 2025, 12:29 PM

#

shrewd jackal I mean the v program which spawns them

not sure

#

but it is easy to test

#

it can just wait at the end

shrewd jackal Feb 15, 2025, 12:29 PM

#

everything is empty

#

I fear the kernel already removed everything useful

#

:/

hushed plank Feb 15, 2025, 12:30 PM

#

proc files are often like that

#

try ls -lart /proc/self/

#

the content is generated on access

#

so it can not show the filesize in advance

shrewd jackal Feb 15, 2025, 12:30 PM

#

oh true

#

but the fd file has a size on self

#

renxSHRUG

#

but cmdline and env are empty for the git processes but not for self

hushed plank Feb 15, 2025, 12:31 PM

#

for me, everything is 0s

#

try catting /proc/PID/cmdline

shrewd jackal Feb 15, 2025, 12:32 PM

#

I also get 0 except for fd, regardless

shrewd jackal Feb 15, 2025, 12:32 PM

#

hushed plank try catting /proc/PID/cmdline

it's empty

#

/proc/PID/env too

#

and every file I tried aswell

#

all symlinks are dead

#

eg cwd, exe

#

renxSHRUG

hushed plank Feb 15, 2025, 12:35 PM

#

why is /bin/busybox ?

#

is that on Alpine?

shrewd jackal Feb 15, 2025, 12:35 PM

#

yes this is inside the container

#

but from outside the container it's the same thing

hushed plank Feb 15, 2025, 12:35 PM

#

try running the program from above in the container

#

perhaps the behaviour of glibc is different than that of musl

shrewd jackal Feb 15, 2025, 12:36 PM

#

sh: /usr/bin/true: not found

#

😄

hushed plank Feb 15, 2025, 12:36 PM

#

try with ls

#

I hope it has at least that

#

or busybox true abcdef

shrewd jackal Feb 15, 2025, 12:37 PM

#

it's /bin/true, not /usr/bin/true

#

no zombies

hushed plank Feb 15, 2025, 12:37 PM

#

try running git status

shrewd jackal Feb 15, 2025, 12:38 PM

#

kept the program alive (os.input()) and checked with different shell, also no zombies

hushed plank Feb 15, 2025, 12:38 PM

#

could it be that git itself starts sub processes

#

that it does not wait for

#

but that would have been caught much earlier 🤔 so it is unlikely

shrewd jackal Feb 15, 2025, 12:39 PM

#

no zombies

#

I'll try running oldv

#

because oldv with -s is used for updating git

#

oh dear lord

#

yeah it's caused by oldv

hushed plank Feb 15, 2025, 12:41 PM

#

try ps auxwww | grep 'Z'

shrewd jackal Feb 15, 2025, 12:41 PM

#

that only greps itself

#

I don't think alpine ps supports those options

hushed plank Feb 15, 2025, 12:42 PM

#

it does not on Ubuntu

shrewd jackal Feb 15, 2025, 12:42 PM

#

i'll run from ubuntu

#

yep that gets all the zombies

#

root     3084130  0.0  0.0      0     0 ?        Zs   13:41   0:00 [git] <defunct>
root     3084137  0.0  0.0      0     0 ?        Zs   13:41   0:00 [git] <defunct>
root     3084160  0.0  0.0      0     0 ?        Zs   13:41   0:00 [git] <defunct>
root     3084167  0.0  0.0      0     0 ?        Zs   13:41   0:00 [git] <defunct>
root     3084191  0.0  0.0      0     0 ?        Zs   13:41   0:00 [git] <defunct>
root     3084198  0.0  0.0      0     0 ?        Zs   13:41   0:00 [git] <defunct>

#

small snipped

hushed plank Feb 15, 2025, 12:44 PM

#

shrewd jackal because oldv with -s is used for updating git

afaik oldv waits for the synchronisation to finish

shrewd jackal Feb 15, 2025, 12:44 PM

#

each oldv -s adds 2 new zombies

hushed plank Feb 15, 2025, 12:44 PM

#

I'll test locally

#

lol

#

on ubuntu it does not 🤔

#

I'll setup an alpine container

shrewd jackal Feb 15, 2025, 12:46 PM

#

is VERBOSE=1 supposed to work for -s?

hushed plank Feb 15, 2025, 12:46 PM

#

I'll use Dockerfile.alpine

#

yes, since it uses the scripting module

#

you can also pass -v explicitly

#

but that shows less details

shrewd jackal Feb 15, 2025, 12:48 PM

#

oh it was because I was using .execute facepalm

#

it runs exactly two git processes and there's 2 new zombies...

hushed plank Feb 15, 2025, 12:48 PM

#

shrewd jackal Feb 15, 2025, 12:49 PM

#

but it waits for them to finish and they're successful

hushed plank Feb 15, 2025, 12:49 PM

#

the zombies are for git pull --quiet ?

shrewd jackal Feb 15, 2025, 12:49 PM

#

there are no other git commands being ran

hushed plank Feb 15, 2025, 12:49 PM

#

weird

#

is git an executable or a script on alpine?

shrewd jackal Feb 15, 2025, 12:49 PM

#

/usr/bin/git: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, BuildID[sha1]=ce140b8e440c6869ada79dfd2d4427fc18c8e280, stripped

hushed plank Feb 15, 2025, 12:50 PM

#

can you try running git pull --quiet in the script above

#

to see what will happen (on Alpine)

#

it may need to be in a folder, for which you have run git init first

shrewd jackal Feb 15, 2025, 12:51 PM

#

yeah that adds a zombie

shrewd jackal Feb 15, 2025, 12:51 PM

#

hushed plank it may need to be in a folder, for which you have run `git init` first

I am doing -C and pointing it to vc

hushed plank Feb 15, 2025, 12:51 PM

#

for both os.execute and os.system, or for just os.execute?

shrewd jackal Feb 15, 2025, 12:52 PM

#

rewrote it just to call os.system once because speed

hushed plank Feb 15, 2025, 12:52 PM

#

(I am still waiting for the alpine container to build, so I can not yet test myself)

shrewd jackal Feb 15, 2025, 12:52 PM

#

…………………………………

#

ok just running git -C /root/.cache/oldv/vc pull myself also does it

#

renxDEAD

hushed plank Feb 15, 2025, 12:53 PM

#

i.e. git itself zombifies?

shrewd jackal Feb 15, 2025, 12:53 PM

#

yes

#

status works fine

hushed plank Feb 15, 2025, 12:53 PM

#

or perhaps busybox does it

#

in certain situations

#

can you try installing bash in alpine

#

and then do the same check, after starting bash

shrewd jackal Feb 15, 2025, 12:53 PM

#

done and it also does it

hushed plank Feb 15, 2025, 12:54 PM

#

so it is a git specific problem, on alpine

#

interesting

shrewd jackal Feb 15, 2025, 12:54 PM

#

appears to be

#

guess I will just restart the container once a week

#

or do nothing at all, don't think it's causing any issues

hushed plank Feb 15, 2025, 12:58 PM

#

no, it just clutters your ps output afaik

#

I wonder if it does it for all git pull commands, or just for those that make network connections

#

the build process finished for me

#

#

there are no zombies

#

7f2354e1f09e:/opt/vlang$ cat /etc/alpine-release
3.20.6

#

7f2354e1f09e:/opt/vlang$ git version
git version 2.45.3

#

I've build the container with: docker build -t vlang_alpine - < Dockerfile.alpine

#

from the main V repo

#

perhaps for that particular alpine/git combination, the bug is fixed?

#

I'll try running oldv too

#

(in docker run -it vlang_alpine bash)

#

oof, I do not have rsync in that container

shrewd jackal Feb 15, 2025, 1:22 PM

#

hushed plank no, it just clutters your ps output afaik

And I get a warning when sshing

hushed plank Feb 15, 2025, 1:29 PM

#

because of a "message of the day" ?

#

afaik you can touch ~/.hushlogin to stop them

hushed plank Feb 15, 2025, 2:06 PM

#

sorry for the delay; just finished rebuilding the container and then running oldv in it

#

#

I think that for that particular version of alpine, git is ok

#

there are no zombies after it runs

shrewd jackal Feb 16, 2025, 8:31 AM

#

hushed plank

you need to do ps -A | grep git

#

alpine ps doesn't support those options

#

or do the ps command from outside the container (preferably using a second shell to make sure the container doesn't exit)

shrewd jackal Feb 16, 2025, 8:54 AM

#

nvm ps -a should've logged them

#

I would upgrade the image but then you'll complain that numbers change

lofty monolith Feb 16, 2025, 9:33 AM

#

My comment is late, it seems you have already found a solution. At my work we once faced a similar problem with zombies, which only appeared in Alpine, only calling python subprocesses. I don't remember the details exactly, but it was in musl, we started using tini https://github.com/krallin/tini as PID 1 in the container and this solved the problem

hushed plank Feb 16, 2025, 10:04 AM

#

wait for it to exit all the while reaping zombies and performing signal forwarding.

#

yeah, that would solve it

#

perhaps the init process in alpine was fixed in the later versions too

#

or musl changed their wait4 implementation

hushed plank Feb 16, 2025, 10:07 AM

#

shrewd jackal I would upgrade the image but then you'll complain that numbers change

imho if it does not bother you, leave it as it is

#

or reboot the vps once a week

#

since upgrading the container will probably change the compiler versions too etc

hushed plank Mar 14, 2025, 7:13 AM

#

@shrewd jackal will it be a problem, if I change the shortened hash in the vlang/vc commit messages to a full hash?

#

i.e. have for example not:
[v:master] a826feb - v.builder: remove forgotten b.show_parsed_files/0 call
but:
[v:master] a826febf5e3fc716c9478b4c4a3ca2f26a8531c9 - v.builder: remove forgotten b.show_parsed_files/0 call

shrewd jackal Mar 14, 2025, 7:15 AM

#

Idk

hushed plank Mar 14, 2025, 7:15 AM

#

currently it is produced by git rev-parse --short HEAD

shrewd jackal Mar 14, 2025, 7:15 AM

#

fast is always passing the full hash

hushed plank Mar 14, 2025, 7:15 AM

#

so it should be unique, according to its documentation

#

but the full hash is probably more useful for automation tools anyways

shrewd jackal Mar 14, 2025, 7:17 AM

#

The only time it’s truncated is in #fast (Link itself still has full length)

#

Otherwise always full length

hushed plank Mar 14, 2025, 7:18 AM

#

excellent, I will change gen_vc_ci.yml then

hushed plank Mar 14, 2025, 7:40 AM

#

perfect ... everything works fine

static scarab Mar 16, 2025, 9:58 AM

#

Also trying to fill a struct because of a custom sql query. Does this code actually work for you? I'll have this one: error: cannot assign an Option value to a non-option variable in line 11

shrewd jackal Mar 16, 2025, 10:33 AM

#

https://github.com/spytheman/oldv_charts/blob/master/utils.v#L51-L69

GitHub

oldv_charts/utils.v at master · spytheman/oldv_charts

Drawing performance graphs from previous V versions, using cmd/tools/oldv.v and https://plotly.com/javascript/ . - spytheman/oldv_charts

#

consider using #help

shrewd jackal Mar 25, 2025, 6:03 AM

#

A

shrewd jackal Mar 25, 2025, 8:45 AM

#

looks like we're leaking a bit of memory

#

and 7349 zombie processes

#

but it's manageable

#

will restart on next week's sunday anyway (first sunday each month is patch sunday)

hushed plank Mar 25, 2025, 8:57 AM

#

is it the old issue, or a new one?

shrewd jackal Mar 25, 2025, 11:27 AM

#

The zombie processes is old (git on alpine)

#

And never really cared about memory, so might be old aswell

shrewd jackal Apr 1, 2025, 7:05 AM

#

started monitoring cpu & memory usage

hushed plank Apr 1, 2025, 7:15 AM

#

are the spikes for the RAM usage corresponding to -prod compilations?

#

hm, the last graph looks a bit weird to me 🤔 - it had only one big peak, and the level at the end was lower than that at the start

shrewd jackal Apr 1, 2025, 7:36 AM

#

hushed plank are the spikes for the RAM usage corresponding to -prod compilations?

yes

#

both cpu and ram spikes are it doing benchmarking

#

the last spike is when it runs oldv -sync every 10mins to check for new commits

#

because it's recompiled on every run (v run'ed), I should probably AOT that

shrewd jackal Apr 1, 2025, 11:39 AM

#

zoomed out version

#

memory consumption just jumps by 200mb "randomly"

#

seems to coincide with the fetches every 10mins

#

renxSHRUG

#

set the memory limit to 2gb, so the container should get killed once it exceeds that

#

(just for testing)

hushed plank Apr 1, 2025, 11:43 AM

#

does it have swap?

#

if it does, it may not be

shrewd jackal Apr 1, 2025, 11:46 AM

#

looks like that's off by default

#

nvm it defaults to same amount as RAM

#

so it'd swap 2gb as well

#

looks like there's no way to change the swap limit using docker compose :/

#

oh there is, but it's different than the normal memory limit because docker moment

#

#

renxICANT

shrewd jackal Apr 2, 2025, 12:10 PM

#

oldv is now compiled in the build process, so the spikes every 10mins should be gone

#

#

did someone break time.Duration.str() IIRC this should show 10m instead of the ns

#

renxSHRUG

shrewd jackal Apr 2, 2025, 12:33 PM

#

spikes remain, so it's not the compilation done by v run that's causing it

#

probably just git coping with 18k+ commits

hushed plank Apr 2, 2025, 3:50 PM

#

import time
d := time.Duration(600_000_000_000)
dump(d)
dump(d.str())

this shows for me:

[a.v:4] d: 10:00.000
[a.v:5] d.str(): 10:00.000

#

I guess there could be an i64 cast in the sleeping for message 🤔

#

because if I add:

i := i64(d)
dump(i)
dump(i.str())

I get the long literal too

shrewd jackal Apr 2, 2025, 5:24 PM

#

Oh it’s d := 10 * time.minute

#

I could swear that returns a time.Duration though

fresh remnant Apr 2, 2025, 6:08 PM

#

time.Duration is just a type alias for u64.

shrewd jackal Apr 2, 2025, 6:30 PM

#

With a special str() method

hushed plank Apr 3, 2025, 1:46 PM

#

shrewd jackal I could swear that returns a time.Duration though

it could have ... there were changes to the alias handling, but I could not find a test specifically for that (literal * alias)

#

if you happen to remember when did it work differently, we could try bisecting to find the change

shrewd jackal Apr 3, 2025, 1:47 PM

#

I could also just be imagining things because I wrote too much go

hushed plank Apr 3, 2025, 1:47 PM

#

and then think about if it is an improvement or a silent regression

shrewd jackal Apr 3, 2025, 1:48 PM

#

#

looks like we leak ~170mb per test

#

renxSHRUG

#

that could be anywhere, from the ORM to os.execute

#

after the first commit it dropped down to before levels

#

so only the 2nd test leaked? renxSHRUG

hushed plank Apr 3, 2025, 1:50 PM

#

hm, why was not it here #1312494584151933001 message ?

hushed plank Apr 3, 2025, 1:51 PM

#

shrewd jackal so only the 2nd test leaked? <:renxSHRUG:1168858910581604412>

weird 🤔

shrewd jackal Apr 3, 2025, 1:51 PM

#

I have no idea

#

I am grasping at straws

hushed plank Apr 3, 2025, 1:51 PM

#

lets see if it will remain after more merged commits

hushed plank Apr 3, 2025, 2:37 PM

#

does it keep leaking after the latest commits?

shrewd jackal Apr 4, 2025, 7:30 AM

#

#

no but then it decided to leak on its own

#

renxSHRUG

hushed plank Apr 4, 2025, 9:52 AM

#

try doing free && sync && echo 3 | sudo tee /proc/sys/vm/drop_caches && free in the running container

#

and on the host too

shrewd jackal Apr 4, 2025, 11:12 AM

#

tee: /proc/sys/vm/drop_caches: Read-only file system

#

outside of the container:

$ free && sync && echo 3 | sudo tee /proc/sys/vm/drop_caches && free
               total        used        free      shared  buff/cache   available
Mem:        65617100     4980336    50350996      230152    11249168    60636764
Swap:       33520636         768    33519868
3
               total        used        free      shared  buff/cache   available
Mem:        65617100     4666344    60308892      230236     1528980    60950756
Swap:       33520636         768    33519868

#

memory cached dropped to 0, but used memory stays the same

#

cache dropped by 9gb system wide

hushed plank Apr 4, 2025, 11:18 AM

#

if you run htop outside the container, and sort by memory, which processes are at the top?
(F6, then choose PERCENT_MEM)

shrewd jackal Apr 4, 2025, 11:19 AM

#

btop:

hushed plank Apr 4, 2025, 11:19 AM

#

is fast compiled without gc?

shrewd jackal Apr 4, 2025, 11:19 AM

#

htop agrees

shrewd jackal Apr 4, 2025, 11:20 AM

#

hushed plank is `fast` compiled without gc?

no special flags, just RUN /data/v/v /data/fast.v -prod -o /data/fast

hushed plank Apr 4, 2025, 11:20 AM

#

if you restart it, what is the initial RAM usage?

shrewd jackal Apr 4, 2025, 11:21 AM

#

339M

shrewd jackal Apr 4, 2025, 7:32 PM

#

we're up to 1gb again

shrewd jackal Apr 5, 2025, 2:00 PM

#

#

just realized I accidentally removed the memory limit

shrewd jackal Apr 29, 2025, 8:57 AM

#

looks like we stabilized at 2.18 GiB of leaked memory? renxSHRUG

#

(spikes are GCC)

fierce jetty Apr 29, 2025, 4:22 PM

#

shrewd jackal looks like we stabilized at 2.18 GiB of leaked memory? <:renxSHRUG:1168858910581...

maybe something 32 bit if it stops at around two gigs

shrewd jackal Apr 29, 2025, 4:23 PM

#

It has leaked to 5gb+ in the past
Also 2.18B is above 31bit limit

fierce jetty Apr 29, 2025, 4:26 PM

#

shrewd jackal It has leaked to 5gb+ in the past Also 2.18B is above 31bit limit

yeah but system also counts towards it doesn't it? also it probably depends on a single component somewhere

shrewd jackal Apr 29, 2025, 4:29 PM

#

No, that’s the memory usage of only the fast container

shrewd jackal Jul 17, 2025, 9:30 AM

#

Thinking of adding a thread in #fast with the full details of each commit (which are currently posted to a private channel)

#

Talking about these messages:

#

`hello (examples/hello_world.v)` on c3dfe62d7b (2025-07-17 07:14:01)

      names      mean                stddev       min       max  samples  noisy  unit
 ----------  --------  --------------------  --------  --------  -------  -----  ----------------
    timings
       scan   7.410ms   ± 63.149us (0.85%)   7.272ms   7.517ms       14
      parse  36.092ms  ± 277.716us (0.77%)  35.583ms  36.644ms       14
      check  14.869ms  ± 112.774us (0.76%)  14.668ms  15.042ms       14
  transform   3.986ms   ± 32.058us (0.80%)   3.927ms   4.039ms       14
   markused   1.107ms    ± 6.355us (0.57%)   1.100ms   1.118ms       14
       cgen  10.111ms  ± 124.674us (1.23%)   9.891ms  10.276ms       14
 all stages  68.560ms  ± 390.802us (0.57%)  67.815ms  69.143ms       14
      total  83.385ms  ± 433.914us (0.52%)  82.500ms  84.037ms       14

  v  source
  v    size    489074          ± 0 (0.00%)    489074    489074       20         bytes
  v   lines     18241          ± 0 (0.00%)     18241     18241       20         lines
  v lines/s    218761       ± 1139 (0.52%)    217059    221103       14         lines per second
  v   types       154          ± 0 (0.00%)       154       154       20         types
  v modules         5          ± 0 (0.00%)         5         5       20         modules
  v   files        67          ± 0 (0.00%)        67        67       20         files

    c  code
    c  size     57264          ± 0 (0.00%)     57264     57264       20         bytes
    c lines      2361          ± 0 (0.00%)      2361      2361       20         lines

#

There’s 4 per commit with the different benchmarks

#

Extra thread because it’s just a lot of noise…

hushed plank Jul 17, 2025, 9:31 AM

#

imho it looks nice

#

on desktop

shrewd jackal Jul 17, 2025, 9:32 AM

#

Yeah let’s not talk about mobile…

hushed plank Jul 17, 2025, 9:34 AM

#

what is the noisy column for?

shrewd jackal Jul 17, 2025, 8:20 PM

#

hushed plank what is the noisy column for?

If the stddev is too high

#

But we just add more samples if it’s too high (limited to 60 or 80)

#

And that noisy is after discarding the top 1/3rd

#

So it pretty much is always false

hushed plank Jul 18, 2025, 7:49 AM

#

if you remove it, and remove the unit columns from the report thread on discord, it may be more readable on mobile

shrewd jackal Jul 25, 2025, 11:56 AM

#

removed, will let you know how details look in the future

shrewd jackal Jul 25, 2025, 1:02 PM

#

or not, guess I broke things

#

panic!

#

I forgot to change the length of the fixed array used for calculating the length of the --- header underline

#

and then it tried to access the info table OOB

hushed plank Jul 25, 2025, 1:04 PM

#

as Douglas Adams says, "Don't panic."

#

and make sure to have a towel

shrewd jackal Jul 25, 2025, 1:05 PM

#

hushed plank as Douglas Adams says, "Don't panic."

it panic!'ed, not me 🙂

hushed plank Jul 25, 2025, 1:05 PM

#

v should get a towel

shrewd jackal Jul 25, 2025, 1:05 PM

#

i hope there's a restaurant somewhere at the edge of the universe or something

hushed plank Jul 25, 2025, 1:06 PM

#

with a good enough towel, every place is at the edge of the universe restaurant

shrewd jackal Jul 25, 2025, 1:07 PM

#

fixed 🙂 (hopefully)

#

we'll know in ~5mins when its done benchmarking

#

`hello (examples/hello_world.v)` on 38beb23c6a (2025-07-25 12:15:05)

      names      mean                stddev       min       max  samples
 ----------  --------  --------------------  --------  --------  -------
    timings
       scan   7.492ms   ± 83.658us (1.12%)   7.332ms   7.579ms       14
      parse  36.684ms  ± 279.844us (0.76%)  36.158ms  37.078ms       14
      check  15.118ms  ± 182.528us (1.21%)  14.808ms  15.420ms       14
  transform   4.042ms   ± 34.838us (0.86%)   3.991ms   4.112ms       14
   markused   1.523ms   ± 12.188us (0.80%)   1.506ms   1.545ms       14
       cgen   9.837ms   ± 77.581us (0.79%)   9.741ms   9.968ms       14
 all stages  69.319ms  ± 505.317us (0.73%)  68.218ms  70.215ms       14
      total  84.844ms  ± 594.103us (0.70%)  83.602ms  85.726ms       14

  v  source
  v    size    489574          ± 0 (0.00%)    489574    489574       20
  v   lines     18248          ± 0 (0.00%)     18248     18248       20
  v lines/s    215086       ± 1511 (0.70%)    212864    218272       14
  v   types       156          ± 0 (0.00%)       156       156       20
  v modules         5          ± 0 (0.00%)         5         5       20
  v   files        67          ± 0 (0.00%)        67        67       20

    c  code
    c  size     51508          ± 0 (0.00%)     51508     51508       20
    c lines      2051          ± 0 (0.00%)      2051      2051       20

#

i don't think we can really improve it for mobile

hushed plank Jul 25, 2025, 1:13 PM

#

yeah, and even there, in landscape mode, for me it looks nice

#

it wraps only in portrait mode

#

(I suppose there is no way to reduce the font size)

shrewd jackal Jul 25, 2025, 1:15 PM

#

nope

#

best UX would be rendering it to an image but……………

#

anyway, I created #1398292135047856229

#

which will be populated from now onwards

hushed plank Jul 25, 2025, 3:36 PM

#

in the latest table, the number of samples for hello, jumped to 27

#

(from 14 before that)

shrewd jackal Jul 25, 2025, 7:39 PM

#

hushed plank (from 14 before that)

Its 40/20 before discarding the top 1/3

#

Essentially it does more samples if the stddev is too high

#

(Before discarding the top 1/3rd)

#

After discarding the top 1/3rd, the stddev is almost always no longer noisy

#

Why is it sometimes too high? Dunno. The server isn’t that busy renxSHRUG

#

Usually it’s very rare to go beyond the default 20

hushed plank Jul 26, 2025, 4:48 AM

#

self (-no-skip-unused cmd/v) on 318394dd47 (2025-07-26 04:15:08)

#

can it also show the commit title after the hash?

#

(perhaps on its own line)

#

(or perhaps as a header, before the rest of the tables)

hushed plank Jul 26, 2025, 2:07 PM

#

I'll turn off the notifications for the thread for me, otherwise it sounds like the beginning of a spam bot raiding 😄 ...

astral ore Jul 31, 2025, 8:36 AM

#

hushed plank perhaps the association is up -> hot -> red, and down -> cold -> blue

I've never seen anyone use a chart this way. Always the opposite. Red for bad, yellow for warning, green for good, and blue for excellent

shrewd jackal Jul 31, 2025, 8:41 AM

#

Now you have 🙂

hushed plank Jul 31, 2025, 8:41 AM

#

#

the creators of the "twemoji" decided to confuse us

shrewd jackal Jul 31, 2025, 8:42 AM

#

Twemoji is very inspired by apple.

hushed plank Jul 31, 2025, 8:42 AM

#

I guess Tim Cook liked his charts colored that way

#

or Jobs, but I doubt it

shrewd jackal Jul 31, 2025, 8:46 AM

#

Apple, Google, Samsung, Microsoft Teams, Twemoji, Huawei, Joypixels, LG, Telegram, Skype, SerenityOS, Toss Face, Play Station, OpenMoji, Icons8, emojidex, Facebook Messenger, Softbank, Mozilla: red.
Microsoft 3d Fluent, Microsoft, Docomo: blue.
Whatsapp, Facebook, HTC: green.
Noto, au: black.

#

Consensus seems to be red renxSHRUG

astral ore Jul 31, 2025, 8:49 AM

#

shrewd jackal

Well, so far you see a different color pattern for the bar chart and the line chart. Maybe we're seeing the line charts wrong and should be seeing it from right to left. Hahahah

shrewd jackal Jul 31, 2025, 8:50 AM

#

I think the temperature association is correct.

#

Probably was meant for weather applications.

tall isle Jul 31, 2025, 11:47 AM

#

trolling charts

shrewd jackal Oct 18, 2025, 9:27 AM

#

should now no longer run if the 1min loadavg is >1

#

with a friendly message in #1398292135047856229

#

(because I need to check if running a measurement would delay the next because it thinks the server is busy)

#

rescheduled 30 commits for testing

hushed plank Oct 18, 2025, 9:31 AM

#

latest 30?

shrewd jackal Oct 18, 2025, 9:32 AM

#

latest 27 and 3 from a few days before

hushed plank Oct 18, 2025, 9:32 AM

#

nice

shrewd jackal Oct 18, 2025, 9:33 AM

#

data is already gone from the charts and will be repopulated once server is no longer busy

#

note for future self: set state to 0 (pending) in sqlite to reprocess commits

shrewd jackal Oct 18, 2025, 9:33 AM

#

shrewd jackal note for future self: set state to 0 (pending) in sqlite to reprocess commits

shrewd jackal Oct 18, 2025, 3:30 PM

#

welp, I accidentally made the check into a busyloop, so the load average never drops below anymore

shrewd jackal Oct 20, 2025, 5:11 AM

#

oldv -s is for some reason returning a non-zero status code

shrewd jackal Oct 20, 2025, 12:56 PM

#

#

no idea why

hushed plank Oct 20, 2025, 12:57 PM

#

network issue?

shrewd jackal Oct 20, 2025, 12:57 PM

#

no

hushed plank Oct 20, 2025, 12:58 PM

#

can you run it manually with -s and -v?

shrewd jackal Oct 20, 2025, 12:58 PM

#

no because the container exits and I can't shell into it

hushed plank Oct 20, 2025, 12:58 PM

#

oh, check the available free disk space too

shrewd jackal Oct 20, 2025, 12:59 PM

#

hundreds of gb free

#

I added -v and will recompile

hushed plank Oct 20, 2025, 1:00 PM

#

and potentially the permissions of the cache folder (or just remove it)

shrewd jackal Oct 20, 2025, 1:01 PM

#

now it works

#

renxSHRUG

hushed plank Oct 20, 2025, 1:02 PM

#

then imho most likely it was a network issue

shrewd jackal Oct 20, 2025, 1:02 PM

#

it was retrying once a minute for the past day

hushed plank Oct 20, 2025, 1:03 PM

#

they are usually transient in nature

shrewd jackal Oct 20, 2025, 1:03 PM

#

I can give you hundreds of lines of the same error

#

since the entire container exits, all temporary storage (including the cloned vc history) is wiped everytime

#

and it's the same oldv binary

shrewd jackal Oct 20, 2025, 1:45 PM

#

My suspicion was right, the benchmarking will make it think it’s busy: #1398292135047856229 message

#

Oh well

shrewd jackal Oct 20, 2025, 4:10 PM

#

I forgot to restart the container that sends the messages in #fast

#

oh well

#

it will* work from now on again!!

-# should*

shrewd jackal Oct 20, 2025, 7:23 PM

#

It might be a little too sensitive

hushed plank Oct 21, 2025, 4:19 PM

#

shrewd jackal no idea why

I wonder if there is a chance that AWS's problems were also indirectly related

shrewd jackal Oct 21, 2025, 4:20 PM

#

no this issue was well before that + I doubt microsoft hosts on aws instead of azure

hushed plank Oct 21, 2025, 4:26 PM

#

docker however may

#

V's own CI was affected mainly in the jobs that made requests to https://codeberg.org/ (for vinix) and to hub.docker.com (for the various qemu docker containers for s390, termux, the websocket tests etc)

#

what is the /etc/resolv.conf file in the container?

shrewd jackal Oct 21, 2025, 4:44 PM

#

docker hub is in no way related once the container is built and everything was already cached locally since the start of the year

#

and again, this has been going on for days BEFORE the aws outage

hushed plank Oct 21, 2025, 4:58 PM

#

the aws outage may have had earlier symptoms, affecting only some DNS requests

#

(and afaik you started to make changes in order to fix the oscillating up/down measurements in Saturday, not sure if that may have affected the container)

#

anyway it is just speculation; if you think it is not related, ok

shrewd jackal Nov 1, 2025, 9:08 AM

#

sent the two commits back to be oven

#

idk what happened, the server was indeed busy at the time but it thought it wasn't renxSHRUG

hushed plank Nov 1, 2025, 1:32 PM

#

are they now re-calculated?

#

I think the spike was due to using unsigned arithmetic in a very hot loop

#

but it was very surprising to me too

#

my hypothesis is that gcc on the host could not optimise the loop (which has division in it) as well, when it used unsigned numbers

#

i.e. it was inherent to the commit/host/gcc version, not due to the host load

#

with tcc, the diff does not seem to be big

#

I have not tried with clang yet

shrewd jackal Nov 1, 2025, 2:08 PM

#

hushed plank I think the spike was due to using unsigned arithmetic in a very hot loop

no, the server was just busy

#

I edited the wrong sqlite file....

#

they should rerun now

hushed plank Nov 1, 2025, 4:19 PM

#

yep, the spike is indeed gone now 🤔

shrewd jackal Dec 13, 2025, 9:29 PM

#

gonna remove the "Server appears to be busy..."/"Server is no longer busy..." messages

#

it does a few random pauses through the day but I don't think it matters that sometimes measurements are delayed by a few mins

hushed plank Dec 13, 2025, 9:31 PM

#

yes

shrewd jackal Dec 16, 2025, 8:49 AM

#

Updated the bot that posts in #fast:

suppresses embeds
explicitly disallows pings (@ everyone was only implicitly disallowed)
resolves github usernames! doing @anything will now update it to a github url (@leo -> [@leo](github.com/leo)); I may need to add exceptions to this rule, for e.g. @VCOMMITHASH or similar (being entirely uppercase might be a good heuristic?)

hushed plank Dec 16, 2025, 9:48 AM

#

@shrewd jackal https://github.com/vlang/v/commit/1b334a9702b7bce7a87ca1d750a2f490bc80f22b is missing

#

docs: fix from feedback in https://youtu.be/IuE6Bo1klK0?t=2051

shrewd jackal Dec 16, 2025, 9:48 AM

#

it might've panicked

#

no it didn't 🤔

#

it sent the request to discord

#

failed to post webhook: SSL handshake failed

hushed plank Dec 16, 2025, 9:49 AM

#

is it compiled with -d use_openssl ?

shrewd jackal Dec 16, 2025, 9:50 AM

#

no

hushed plank Dec 16, 2025, 9:50 AM

#

or perhaps there is/was a Discord network glitch

#

the SSL handshake failed is from:

vlib/net/openssl/ssl_connection.c.v:255:return error('SSL handshake failed (OpenSSL SSL_get_verify_result = ${res})')
vlib/net/mbedtls/ssl_connection.c.v:300:return error_with_code('SSL handshake failed', ret)
vlib/net/mbedtls/ssl_connection.c.v:469:return error_with_code('SSL handshake failed', ret)
vlib/net/mbedtls/ssl_connection.c.v:505:return error_with_code('SSL handshake failed', ret)

#

so if it is missing the OpenSSL, it is most likely from net.mbedtls and -d use_openssl could potentially fix it

shrewd jackal Dec 16, 2025, 9:51 AM

#

it never happened before, might've just been really bad timing?

#

also keep in mind that this is running an ancient V version IIRC

hushed plank Dec 16, 2025, 9:52 AM

#

we should probably make the errors more unique 🤔

#

the vlib/net/mbedtls and vlib/net/openssl wrappers have not changed much

shrewd jackal Dec 16, 2025, 2:27 PM

#

right so the alpine image didn't have openssl headers, so I had them which means all the caches were invalidated and we also get latest V 🙃

#

time to see if that magically fixes it

shrewd jackal Dec 16, 2025, 3:28 PM

#

thoughts on upgrading the fast runner to the latest V and latest GCC/alpine packages?
we'll need some kind of process, or do we want to keep using jan 2025 V/GCC forever?

#

I don't think it'll affect much tbh. At the time, we were confused why the graph jumped 20% and we (incorrectly) attributed it to GCC changing, but it was because I changed the code to skip compiling the generated C code in the benchmarks

fresh remnant Dec 16, 2025, 3:42 PM

#

Interesting idea... does V need to be changed to run these checks? If not, there's no real reason to update.

#

Could do it anyway on, say, quarterly basis - just in case there's a change in V that makes the tests run faster/more cleanly/etc.

shrewd jackal Dec 16, 2025, 3:53 PM

#

fresh remnant Interesting idea... does V _need_ to be changed to run these checks? If not, th...

I don't see why V would need any changes at all

#

it's just a byproduct of how docker image layer caching works, if I update the alpine registry which is step 3, all steps after that need to re-run, which includes V itself (in step 5)

#

so unless I explicitly chose to checkout a specific commit (instead of master), it'll get the latest V at the same time

#

not updating is more annoying, because that means I need to keep all the layer caches around.
I haven't been able to run docker image prune since the beginning of the year, I probably accumulated like 20-40gb of unused images by now

hushed plank Dec 16, 2025, 3:58 PM

#

shrewd jackal thoughts on upgrading the fast runner to the latest V and latest GCC/alpine pack...

imho it can be updated without too many problems - it will just introduce a hopefully small jump at the time it is done, so I think that it may be best to do it at the start of 2026

#

how about 1-2 times per year, once in January, and once in June?

#

the influence/overhead of the measuring script itself should be miniscule (single ms per commit), compared to the actual compilations, that are done anyway (tens or hundreds of ms)

hushed plank Dec 16, 2025, 4:02 PM

#

fresh remnant Could do it anyway on, say, quarterly basis - just in case there's a change in V...

that seems fine too

shrewd jackal Dec 16, 2025, 4:03 PM

#

hushed plank the influence/overhead of the measuring script itself should be miniscule (singl...

the main thing would be GCC. it might do more optimizations to the per-commit V exe

hushed plank Dec 16, 2025, 4:03 PM

#

can it be done in stages?

#

the first one just building the executable of the measuring script

#

and later the second one (build from scratch), having just git and the executables from the first

fresh remnant Dec 16, 2025, 4:04 PM

#

If you really want to get crazy with it, the docker image could clone v and do v run fast.v - it would always be using latest V, then.

hushed plank Dec 16, 2025, 4:05 PM

#

would not that produce more garbage that has to be collected later?

shrewd jackal Dec 16, 2025, 4:05 PM

#

yeah but that means dynamically running containers which it doesn't currently. it's a single long lived container currently

hushed plank Dec 16, 2025, 4:05 PM

#

afaik the goal currently is to remove those 40gb

#

are they from the accumulated oldv cache btw?

shrewd jackal Dec 16, 2025, 4:06 PM

#

no.

#

I both clean those up in real time and even if I didn't, they would be in-ram only and be deleted when the container restarts (once a month when I do system upgrades)

hushed plank Dec 16, 2025, 4:07 PM

#

then I do not understand what grows

shrewd jackal Dec 16, 2025, 4:07 PM

#

V itself doesn't

hushed plank Dec 16, 2025, 4:07 PM

#

are all images layered on top of each other?

shrewd jackal Dec 16, 2025, 4:07 PM

#

all the other stuff I run on the server get updates

hushed plank Dec 16, 2025, 4:07 PM

#

oh

shrewd jackal Dec 16, 2025, 4:07 PM

#

which leads to old images

#

the setup itself in isolation works fine, but in combination with the rest of that runs on the server, it doesn't

#

also

#

this isn't an issue unless I need to change the fast.v script

#

since I can reuse the old container indefinitely

#

only changing the script needs the old caches

hushed plank Dec 16, 2025, 4:10 PM

#

can you make a backup of just the V container/image as it is

#

then docker image prune

#

then import back just the backup?

#

I am generally of the mind that things that work, should not be changed

#

just backed up/migrated

#

i.e. avoiding updates in production systems, unless there is a compelling reason for them

#

and not just because some number changed somewhere

shrewd jackal Dec 16, 2025, 4:13 PM

#

dockler image prune only clears unused images, it wouldn't affect the currently running fast instance.
However, it will remove all the caches, so if I need to rebuild it because I change fast.v, it will need to start from scratch and get the newest V/GCC

hushed plank Dec 16, 2025, 4:14 PM

#

for me, that is not a problem, just say when it is done, and preferably do it at round-ish intervals, so that the corresponding jumps in the graphs, can be distinguished from the regular stream of changes

fresh remnant Dec 16, 2025, 4:16 PM

#

Perhaps put the caches in a persistent data store, so they won't be affected by the main image changing.

#

https://docs.docker.com/engine/storage/volumes/

#

"Volumes are often a better choice than writing data directly to a container, because a volume doesn't increase the size of the containers using it."

shrewd jackal Dec 16, 2025, 4:18 PM

#

Volumes have nothing to do with this.

#

This is in the build process of an image, docker's own caching

fresh remnant Dec 16, 2025, 4:19 PM

#

I was looking at the "However, it will remove all the caches, so if I need to rebuild it because I change fast.v, it will need to start from scratch and get the newest V/GCC" statement.

#

Unless a docker image prune would wipe out your volumes, too.

shrewd jackal Dec 16, 2025, 4:27 PM

#

volumes have nothing to do with docker's build caching

#

they are for running containers

#

again, nothing to do with building

fresh remnant Dec 16, 2025, 4:38 PM

#

So you want the image layers that are cached, but they keep expanding in size?

#

That's... not exactly caching, if they can change size...

#

Yes, it is faster to build a new image by using cached layers, but if this is something you don't do often, why would that matter? 10 min vs 10 sec is nothing when you rebuild as infrequently as this image is changed.

#

It's really only significant if you need to rebuild multiple times, very quickly.

shrewd jackal Dec 16, 2025, 4:58 PM

#

this gotta be ragebait

hushed plank Dec 16, 2025, 4:59 PM

#

imho no; we just do not understand the intricacies of the situation, to be more helpful

shrewd jackal Dec 16, 2025, 5:00 PM

#

I don't want them cached. You want them cached.
The caches here aren't for speed. They exist so all builds of fast.v use the same GCC version and base alpine image.

fresh remnant Dec 16, 2025, 5:00 PM

#

Correct. I'm throwing out ideas, in case one of them helps.

shrewd jackal Dec 16, 2025, 5:01 PM

#

if the apk install gcc wasn't cached, it would get a new GCC version every time I need to recompile fast.v

hushed plank Dec 16, 2025, 5:03 PM

#

can it pin a specific version instead?

shrewd jackal Dec 16, 2025, 5:03 PM

#

I don't know, I just used what docker already gives me.

hushed plank Dec 16, 2025, 5:03 PM

#

then it would not change, even if it is downloaded each time

fresh remnant Dec 16, 2025, 5:04 PM

#

So you want to "pin" the version of gcc and V so that you know they aren't changing, and that would mean the only changes would be because of fast.v. Correct? If that is the case, yeah, pinning to a specific version/sha of gcc and V would be the simplest way to ensure that it kept those the same while rebuilding the image from scratch.

hushed plank Dec 16, 2025, 5:04 PM

#

although that would cost more bandwidth sometimes as a negative

#

the version of V itself does not matter much

#

but the version of gcc does

#

since it does the optimisations, and even small changes can cause big jumps in the measured results

#

i.e. if we treat the V version as the independent variable, all the rest is preferable to stay the same

#

so that the results could be more easily comparable

#fast2.v

self is as expected

hello world has improved 📈

self has improved 📈

hello world has improved 📈

self has degraded 📉

hello world has degraded 📉

self is as expected

hello world is as expected

self is as expected

hello world is as expected

hello (examples/hello_world.v) on 3b31699a0d (2025-01-10 09:22:16)

self is as expected

hello world is as expected

self is as expected

hello world is as expected

self is as expected

hello world is as expected

hello (examples/hello_world.v) on c3dfe62d7b (2025-07-17 07:14:01)

hello (examples/hello_world.v) on 38beb23c6a (2025-07-25 12:15:05)

`hello (examples/hello_world.v)` on 3b31699a0d (2025-01-10 09:22:16)

`hello (examples/hello_world.v)` on c3dfe62d7b (2025-07-17 07:14:01)

`hello (examples/hello_world.v)` on 38beb23c6a (2025-07-25 12:15:05)