its reprocessing them now
#fast2.v
shrewd jackal
hushed plank
ok
shrewd jackal
its reprocessing them now
shrewd jackal
because it already did those
hushed plank
it did them with different measurement methodology
shrewd jackal
no
#1312494584151933001 message
this already has the new measurements for the past week
hushed plank
how many hours did it take to get to it?
shrewd jackal
1.5h
started 16:50 and took until 18:20
then went on to 2023 commits
it took a week last time for 1.5 years, so it will take at least a week this time too
fresh remnant
Just for fun... I tried to run fast2.v locally.
2024-12-07T17:55:26.801609Z [INFO ] uname: Linux 7950x 6.12.1-4-MANJARO #1 SMP PREEMPT_DYNAMIC Mon, 25 Nov 2024 05:36:03 +0000 x86_64
2024-12-07T17:55:26.801616Z [INFO ] lexec: cc --version | head -n1
2024-12-07T17:55:26.805348Z [INFO ] benching commit: 11dc600561d33e1547309098188b0a7894076956
2024-12-07T17:55:26.805358Z [INFO ] changing wd to:
2024-12-07T17:55:26.805362Z [INFO ] failed to chdir to:
??
shrewd jackal
this is latest
you need to set VDIR to where you have a full clone of the v repo
fresh remnant
Much better. Thanks. 🙂
shrewd jackal
for context this is my docker setup
fresh remnant
I already see one way you could speed it up at least a little bit. You are checking the existence of git, make, etc. every time through the loop. That only needs to be done once before you start processing the commits.
fresh remnant
Oh, snap. That's... not so good, then. Maybe it needs an option to just not check. 😕
fresh remnant
37s is 37s wasted. And the amount of time will vary per machine.
shrewd jackal
I dont think anyone other than me is insane enough to test all 18k commits
fresh remnant
Heh. Insanity can be catching... after all, I decided to try it, just to compare how long it took to process a week's work of commits on my machine.
shrewd jackal
oh probably faster
this machine has roughly the same speed as the t2.micro fast.v is currently on
so you should be like 2-3x faster
fresh remnant
Dang... oldv really goes to town. Not only does it check for the existence of a bunch of stuff, it downloads a complete copy of tcc every time, too. That likely takes a bit more than 2ms...
Especially depending on your network link, etc.
shrewd jackal
takes my server 1.5s for the tcc clone
fresh remnant
So, 1.5s X 18800 commits...
shrewd jackal
the killer is the prod compile of v however, takes 2mins
fresh remnant
That's... a long time.
[jalon@7950x v]$ time v -cc clang -prod self
V self compiling (-cc clang -prod -o v2)...
V built successfully as executable "v".
real 0m38.885s
user 0m38.594s
sys 0m0.552s
[jalon@7950x v]$ time v -cc gcc -prod self
V self compiling (-cc gcc -prod -o v2)...
V built successfully as executable "v".
real 0m44.155s
user 0m42.939s
sys 0m1.446s
[jalon@7950x v]$
For me, the tcc download was taking ~2 full seconds every time.
fresh remnant
I got 1G line, but that doesn't mean 1G all the way.
shrewd jackal
got them on the same chart
shrewd jackal
figured it out
.reverse() is not in place
@hushed plank does it re-render on every request?
I was thinking of running it in another container hooked up to the same db
so it has live data
oldv_charts/ > v run .
internally_generated_v_code:34:37: error: invalid character `\`
ah yes
(there is no \ in the code)
shrewd jackal
<@344410584684560385> does it re-render on every request?
looks like it doesnt
oh yeah its cached
valid fog
I must say graphs look nice!
shrewd jackal
I must say graphs look nice!
you can view them yourself now too!
valid fog
One thought, showing graph in ms and than you have 1,15B on scale ... I find it dificult to grasp as you have to calc units 😄
valid fog
hmm shows Total, ms
valid fog
nice!
shrewd jackal
you can actually see the stddev on the hello benchmark
fresh remnant
https://fast.leodev.xyz should work
Bad gateway Error code 502
Visit cloudflare.com for more information.
2024-12-07 21:14:33 UTC
shrewd jackal
Bad gateway Error code 502
Visit cloudflare.com for more information.
2024-12-07...
Retry until it works
The http server is single threaded
So if there is more than one request it doesn’t handle it well
(Your browser requests /favicon.ico)
At least that’s my theory for the random 502s
fresh remnant
Heh. It did work this time.
fresh remnant
Oh... I didn't know his site was running on Windows... 🙂
hushed plank
I've added a Commit label on hover
@shrewd jackal can you please update the dynamic service with the new version?
the index page has now also a link to download the current .sqlite file
so that everyone can have the updated db locally to try different analyses
shrewd jackal
Can you add pages for the skip-unused and no-skip-unused charts
hushed plank
ok
post it/update the service when it is convenient for you
I'll use the one from yesterday
sorry if I woke you up 😐 with the ping
hushed plank
added more links on the index page, to make navigation easier:
shrewd jackal
wait thats how for loops work?
someone needs to update https://github.com/vlang/v/blob/master/vlib/v/TEMPLATES.md
hushed plank
in the templates, yes
shrewd jackal
the docs say it uses {} instead of @end
hushed plank
when you want to iterate over maps
I have not tried that; I got the syntax from the existing tests
shrewd jackal
I tried to make it work using @for with the {} but it just didnt work
updating to latest now
updated
hushed plank
same domain?
I think that the server may be failing sometimes, when the db is read
because it may be locked by the other batch process
for modification
shrewd jackal
I think that the server may be failing sometimes, when the db is read
yep
shrewd jackal
because it may be locked by the other batch process
no, I think its just that the proxy is trying to open a 2nd tcp connection
but its single threaded so can only open a single one
hushed plank
connections are supposed to block in this case 🤔
shrewd jackal
net/http: HTTP/1.x transport connection broken: malformed HTTP version \"HTTP/\""
from the proxy
shrewd jackal
agreed
hushed plank
but I do not see how could they affect the results so much
2dd65872dc9f38443b1f139c5cca04e68797fac6
and the next one after it
07eed88df7b07bf52ce62a319292e30273dcf824
shrewd jackal
those were processed at 8am today
I'd slightly understand if I was compiling the chart server at that time, but I was asleep
hushed plank
oh, there is also another one in that area (just zoomed)
4a0a9fba99fde675758ee69fa6096f7cf8491457
Plotly is very nice
shrewd jackal
hushed plank
can you invalidate/re-run just those 3 commits?
shrewd jackal
sure I should be able to just set them to pending in the db
hushed plank
excellent
shrewd jackal
done
yep, its reprocessing them
they're gone from the graph for now since the state is pending
hushed plank
I like the nice separation of concerns, allowed by the DB
shrewd jackal
spike persists
hushed plank
btw, what is the memory usage of the web server currently?
shrewd jackal
fast is at 1.4GB
which is a lot worse than before, its only been running for a day
web server is at 132MB
hushed plank
spike persists
it is for a single commit now ... 4a0a9fba99fde675758ee69fa6096f7cf8491457
shrewd jackal
it is for a single commit now ... 4a0a9fba99fde675758ee69fa6096f7cf8491457
since it hasnt processed the others yet
it takes ~2-3mins per commit
total took 3:27.723, time per commit: 3:21.016
hushed plank
which is a lot worse than before, its only been running for a day
it may be ~ to the number of columns
shrewd jackal
07eed is now processed too, also spike
hushed plank
or number of measurements
shrewd jackal
it may be ~ to the number of columns
I'd expect the webserver to have a similar high usage then?
unless its only when you use ORM
hushed plank
I'd expect the webserver to have a similar high usage then?
good point
locally the server uses ~30MB with the new DB
I'm running it with v watch run . -dynamic
shrewd jackal
the 3 commits are all reprocessed and same spike
hushed plank
I'll check if it will grow with load
hushed plank
the 3 commits are all reprocessed and same spike
thanks, so it is not a fluke... very interesting
shrewd jackal
https://fast.leodev.xyz should work
The total graph should be the 1st or 2nd
Anything but last
Also cgen going up
hushed plank
Anything but last
why?
imho it can be combined with the rest in the Stages chart
but I do not see a reason for it not to be last
shrewd jackal
it’d disturb the scale too much
hushed plank
Ctrl-End is very convenient for going to it on desktop
or scrolling fast with your finger on mobile
shrewd jackal
but I do not see a reason for it not to be last
If you check the new skip-unused as default and just see the first few graphs, you’d think it got faster. But if you check total, it’s slower (for v self)
shrewd jackal
or scrolling fast with your finger on mobile
Prone to accidentally zooming in a graph
hushed plank
imho combining it is fine - it puts everything else in perspective
shrewd jackal
And will make scan+parser unreadable
hushed plank
and will make it very clear the potential reason why something got faster/slower at a glance - just look at the total, see a spike, then look at the lines below to see where it is coming from, if it is not there, then it must be from the C compiler
there are separate charts for each stage for a reason
so that you can also focus on each in isolation
on that note, we may need a chart for the C compilation too
shrewd jackal
Theres no c Compilation
hushed plank
oh ... right, you skip it
with the -o x.c
I forgot
then it is from markused + transformer
which can be made into charts too
shrewd jackal
- makes total more stable
- required to get c size for commits before -stats
hushed plank
shrewd jackal
Total from before -stats will also be higher, since then it’ll take the full execution duration
Is the total from -stats and -show-timings different? 
hushed plank
it is a bit, the TOTAL also includes argument parsing
it is safe to assume they are the same though
the diff is usually much less than 1ms (on my i3 machine ... which is aging a bit)
but markused and transform are their own separate stages
shrewd jackal
Scan/parse/check/cgen is taken from current fast.v, am open to tracking transform and mark_used in the future (next weekend)
hushed plank
that are currently not included in the charts
shrewd jackal
Because they’re not tracked
hushed plank
they usually take very little time, but recently since -skip-unused is the default, they are significant
for longer programs like V itself
shrewd jackal
Since they’ve only become relevant now, less to reprocess
hushed plank
yes
the tradeoff with -skip-unused being the default, is that markused does more work, so that later cgen can do less, and produce less C output, which after that leads to faster C compilation times and smaller executables
shrewd jackal
When was -skip-unused added? I could skip the double processing on (very) old commits
hushed plank
years ago
it just was not the default, because it failed for lots of test cases until recently
6c728cf 2021-10-12
I would not bother with filtering it
fresh remnant
Actually caused some regular code to crash, as I recall.
hushed plank
no
the failures were always at compile time (C programs not compiling, since they missed too much stuff, that cgen decided to generate calls to, and markused could not see), not crashing
unless you extend the definition of crashing to include that, but then it becomes a bit meaningless
it could have affected modules compiled to shared libraries though
and then programs that load those, could have crashed
if some symbols were missing, and they did not handle that situation in a more robust way
the vast majority of tests and examples though are not compiled as shared libs
so they just got the C compile time errors
oh, it is even earlier than what I posted above
40fff7b 2021-02-05
│ 40fff7b 2021-02-05 10:42 +0200 DAngelov ∙ v.pref: supportv -skip-unused run examples/hello_world.v``
is the first mention in the git log
after that it was extended to cover more and more cases
shrewd jackal
daily progress report: up to 2024-09-07
hushed plank
is the memory usage of the webserver growing?
shrewd jackal
fast is at 1.5GB
server is at 208MB
hushed plank
yes, because it replaces stuff like g.write('abc ${def} ${xyz}') which uses interpolation
hushed plank
with a series of calls, that do not use interpolation at all
shrewd jackal
yes, because it replaces stuff like `g.write('abc ${def} ${xyz}')` which uses in...
yeah but thats huge
hushed plank
when it can prove that the interpolated things can be serialised safely
hushed plank
yeah but thats huge
there were a lot of those in cgen
the main thing is that string interpolation allocates a new string builder, that is then discarded pretty much right away
and the result is copied to a new string (which is also discarded right after use), that is also then copied to the cgen string builder
so you get 2 allocations and 2 memory copies at least, as pure overhead
compared to the case, where you can just append to the already existing cgen string builder
g.write('abc ${def} ${xyz}') however does look a lot nicer (more readable and compact) than directly writing the serialised calls without the interpolation
-> the optimisation
hushed plank
https://github.com/vlang/v/commit/5a33c36ab79efd32b5fdcafc22228fe5f57f9a83
that was also a case, where reducing the allocations helped a lot
the error messages there were only used in a particular branch, but created everytime for a relatively frequently called function
It is very nice that the chart shows both improvements and regressions so clearly ... Reading a table with numbers does not have the same impact.
while with a chart, you can spot the spikes/slopes right away
I am currently mostly puzzled by the scan chart
since the recent drops there are by commits that do not seem related, but clearly had a big impact
hushed plank
273adcc35c06f0d71086c96cd40f1d2f62b4bf7a and 19318110d95519a9b6aeaa90f00b2d83eaa395e1
hushed plank
"huge" is relative, that's like a millisecond
yeah, but the absolute value is 64ms, so 1ms is > 1% improvement
shrewd jackal
the commit before (which is slightly on the downwards slope) is https://github.com/vlang/v/commit/2c3ad8ddf61d0fe72cf6dba3770d13433041966d
which is a fun coincidence, but should be unrelated since latest oldv is used
hushed plank
imho it is not relevant - oldv's code is not used by the compiler
I think we should offset the Stages chart
so that it starts at 0ms for the y values
to make the relative sizes more comparative visually
as it is, it uses the Scan line as a minimum, but it is ~63ms, not 0ms
Can we make the points/hover labels linkable/clickable?
shrewd jackal
hushed plank
showing the commit messages would be also nice
but may be too much data for the pages
shrewd jackal
commit messages would require more context, aka we'd need a full git clone available for the server
shrewd jackal
imo not worth it for now
in the future, I will add a tool which automatically finds spikes outside of stddev with more context
hushed plank
that is an interesting idea
I personally tend to just rely on my visual cortex
but that does not scale
shrewd jackal
and you dont check the graph after every commit
icy agate
amazing stuff @shrewd jackal !
shrewd jackal
oh boy, thats a lot of messages
shrewd jackal
this thread
hushed plank
@shrewd jackal please also add the commits from today/yesterday, when you have some time
shrewd jackal
currently needs a full recreation of the container
the git clone is baked into the image
shrewd jackal
the db is a mapped directory
hushed plank
how much time does it take to create a new container?
shrewd jackal
and collects all the stats
hushed plank
I did not know it is CPU bound
shrewd jackal
its at 1.5GB of ram ccurrently
the fast process isnt CPU bound
if Im gonna do a re-redeploy, let me add transform/markused as well
hushed plank
ok, as you wish
it is not urgent
there were just some more markused improvements done recently, and I am curious about their impact
shrewd jackal
what's all_fn_const_and_global
hushed plank
a function
it is a part of markused
there is no need to track it separately
it is included in it
shrewd jackal
can "markused" be uppercased like the other "main stages"?
hushed plank
only mark_used and TRANSFORM are the interesting ones
it could, but that would be in the future, and you will have a special case
it is currently reflecting the name of the function
that is its entry point
shrewd jackal
it could, but that would be in the future, and you will have a special case
special case would just be an extra name in the match case
it looks like its not part of it
hushed plank
ok, I can replace it in a few minutes, then push the change
shrewd jackal
👍
hushed plank
done
it is in latest V
I think the names are not covered by tests, so it should not have broken anything on the CI
oh, you will have to change the match again - it is MARKUSED in my change
hushed plank
with it process the latest commits?
shrewd jackal
like 3mins
125s
hushed plank
not bad
shrewd jackal
it will resync the oldv cache at that time?
there is no oldv cache at the start of the container, first use of oldv will clone it again
oldv cache isnt in the image
hushed plank
even better
shrewd jackal
the issue isnt oldv cache
it doesnt check the oldv cache for the commits
I do a full clone of V, compile fast.v using it and then fast uses that clone to get the commit history
not the oldv cache's git clone
its going through them 👍
hushed plank
very nice
shrewd jackal
markused N/A
I totally forgot I added that failsafe 😄
I hope the charts handle NULL values correctly
shrewd jackal
anything but total and csize/clines can and will be NULL once we go back far enough
hushed plank
the charts are fine - they will be just 0 for the nulls
I've pushed the changes to the webserver
as a bonus, we also got the 0 anchoring for the combined chart indirectly 🙂
because of the nulls
so that now the Scan line is not on the x axis
oh ... I can click on the name of each line in the legend on the right, to turn it on/off
very convenient
shrewd jackal
In like 3-4 days when we’re done with 2024, you need to remove the WHERE date > in utils.v
Since it’s currently filtering to 2024+ only
hushed plank
can I do it now?
shrewd jackal
Graph will get skewed
hushed plank
because of the empty space at the start?
shrewd jackal
Since there’s like 10 commits in mid 2023
shrewd jackal
Try it and you’ll see 😄
hushed plank
there is no rush for the older stats
I wonder what the html size will be for all the points, per page
it is <200KB currently
shrewd jackal
For how many commits? 1000?
shrewd jackal
for all ~20000 of them
??
shrewd jackal
But those aren’t in the html
hushed plank
currently no, but the question is if the charts will be able to handle all of them, and what the payload for the html will become, to transfer all of them
shrewd jackal
Yes, so if we knew how many commits it currently is, we can extrapolate
Hence me asking how many commits it currently is
hushed plank
19:11:37.715 commits.length
19:11:37.742 852
so roughly 22x more data will be the full amount
shrewd jackal
Id guess 16x more, since a lot of the commits are too old to measure
hushed plank
a single page will be 3.6MB at most, even for the full dataset
without compression
and ~2.6x smaller so ~1.3MB at most, compressed with gzip -9
not too bad
imho pagination will not be necessary at first
and we can add one based on years easily
if needed
shrewd jackal
<@922118393178517545> please also add the commits from today/yesterday, when you...
All processed
shrewd jackal
Tomorrow
hushed plank
I'll just look at them locally
hushed plank
I think that total does
shrewd jackal
yes, total is everything
total_stages is my own invention
fn (r Run) total_stages() time.Duration {
return (r.scan or { 0 }) + (r.parse or { 0 }) + (r.check or { 0 }) + (r.cgen or { 0 })
}
from before I tracked total
hushed plank
markdown and transformer are not strictly necessary for most programs, they optimize the behavior of the compiler itself
so it makes sense to have something like total_stages as it is
perhaps just renaming in the charts only to "fundamental stages" or something like that
shrewd jackal
I dont think its even in the charts currently
hushed plank
it is not, since I did not know what it is, before you explained it
I can add it right away
shrewd jackal
progress update: we're pretty much done with june (2024-06-01 10:49:19)
it does around 2 months per day
shrewd jackal
the spikes?
hushed plank
yeah
shrewd jackal
the stddev is within 10 microseconds for markused normally
so should be fairly accurate
shrewd jackal
I've updated the webserver
deployed
hushed plank
perhaps we did made changes to builtin, and it affected it indirectly, not sure
shrewd jackal
misc should exclude markused and transform
hushed plank
no, I want them
hushed plank
yes
misc is the slack/remainder, so to speak
for stuff that does not have its own graph
but also for markused and transform even though they do
they are optimization stages, with potentially different implementation, and with flags to turn them off etc
and unlike the fundamental ones, that can be done relatively easily
having another chart, but with them removed too, may be also useful though
done
shrewd jackal
I will need to rerun again in a week or so, so we get the correct transform timings
hushed plank
it is a bit bad when I look at it like that - we seem to have 5-7% of overhead, that is not in the stages
hushed plank
I will need to rerun again in a week or so, so we get the correct transform timi...
that would be nice, but it is not urgent at all
hushed plank
yeah, that is a bit of a mess
shrewd jackal
well on my M4, that part is like 1/3 of the entire build process
hushed plank
because the builder has to find where the imports are
so it does parsing, but that then gets mixed up with the separate PARSE cumulative timer (sorry for the word salad ... I just re-read it 😄 , it did made sense to me at the time, but not anymore)
shrewd jackal
hushed plank
it will get cleaned up hopefully
shrewd jackal
because i am updating it
hushed plank
ah, ok
shrewd jackal
give it a minute
deployed with all the new charts
they're missing from the "quick links" on the index page, but I dont think thats an issue
hushed plank
no, I left them out of there deliberately
it gets too cluttered with them and imho they are useful only sometimes
I think we may need a second page with them, or some kind of dynamic loading of their data in the future, since the stats page is getting bigger faster than I anticipated
and was already ~2MB yesterday at one point
or we can just compress it with gzip and call it a day
shrewd jackal
pretty sure cloudflare adds gzip compression origin <-> cloudflare and then zstd/brotli cloudflare <-> you
hushed plank
I am more concerned about the bandwidth of your server
hushed plank
ah, ok
hushed plank
that is a bit surprising 🤔
although I like square signals like this one in principle, since they are relatively easier to analyse
shrewd jackal
almost done with 2024, just one week remaining
fresh remnant
Lots of sudden drops to 0 on the charts. 😕
hushed plank
to 0?
hushed plank
imho only the markused and transform have sudden drops, because some of the data is missing for them, and will be added after a while
fresh remnant
Sudden vertical lines to the top/bottom of the chart, near the end.
hushed plank
the ones with the drops at the end, are showing the effects of changing the default for -skip-unused afaik
fresh remnant
For example, look at https://fast.leodev.xyz/v_hello.html#vlines_ps, the cgen, ms chart.
hushed plank
hushed plank
yes, that is expected - that is the effect of -skip-unused being the default
we tried to enable it on Nov 30, and it was turned off again right after that
then after more fixes, for more tests, it was turned on and stays on
fresh remnant
Gotcha. Just looks really odd, if you don't know that.
hushed plank
that is why on the home page, there are 3 variants
the one that is the default
no matter what that was at any given time
and 2 more, that correspond to an explicit -skip-unused and -no-skip-unused switches
hushed plank
Gotcha. Just looks really odd, if you don't know that.
I agree, but it reflects accurately what happened
personally I use the explicit charts more
that avoid those discontinuities
shrewd jackal
almost done with 2024, just one week remaining
2024 done!
More results won’t be on the charts due to the cutoff
hushed plank
I've added more time limited pages - last 7 days, last 30 days, last 120 days, last 365 days, and everything
the more recent ones are very small and quick to query/load
and that output/ folder is easy to rsync to some static web hosting, even github pages
once a day or so
shrewd jackal
Im visiting family this weekend, so tomorrow morning is the last chance for deployments until Monday
The SQLite file has a lot of dangling measurements now
hushed plank
I've pushed my changes so far
shrewd jackal
We can likely shave off a megabyte or two by removing them
hushed plank
I would not worry about it - it is just 5MB for an entire year
and it is better to have more data, and not use it, than need it, and not have it
shrewd jackal
5mb for 2000 commits out of 18800
shrewd jackal
and it is better to have more data, and not use it, than need it, and not have i...
The data is contextless, we don’t know which commit the measurement belonged to
hushed plank
is that only old data, or is the measuring tool currently producing such rows that will not get used?
shrewd jackal
is that only old data, or is the measuring tool currently producing such rows th...
Old data from before the reprocessing
hushed plank
then it will not grow while it is processing more commits
shrewd jackal
Indeed it won’t, and?
hushed plank
then it is not time critical when it will be purged
shrewd jackal
I just need to write a delete query once
hushed plank
as you wish 🤷🏻♂️
shrewd jackal
We were talking about size, and I mentioned how there’s a bunch of space wasted in the db
hushed plank
I understand that, but that wasted space will not grow proportionally to the work that the current measuring tool does
in time, its relative size will shrink
so the current 5MB for 1 year
will not grow to become 45MB for 9x more commits, but probably less
personally I would not worry about it, but if you wish to purge it, more power to you
shrewd jackal
Gotcha. Just looks really odd, if you don't know that.
itd be nice if we could add "annotations" to the graphs for such commits
eg like this
(https://radar.cloudflare.com/?dateRange=14d, that particularly customer supposedly being fortnite releasing a big game update)
(cloudflare built their own graphing library on top of d3.js, so we cant just yoink their impl)
shrewd jackal
recreating the fast container rn so we get coverage of new commits
icy agate
eg like this
looks lovely
shrewd jackal
Almost done with 2023
shrewd jackal
Almost done with 2023
shrewd jackal
total took 3:27.723, time per commit: 3:21.016
time per commit went down by almost 30s?
memory usage is at 3.5gb
web server is at 159mb
shrewd jackal
looks like it just skipped over all commits from may to october 2022?
it failed to build them
because tcc was missing? 🤨
there's hundreds of commits that all had that issue
1f3336c9d34278ccf7c8e579e792377139ce6121
hushed plank
looks like it just skipped over all commits from may to october 2022?
imho check the logs if there were transient network problems during the time
for me, oldv, could checkout 1f3336c9d34278ccf7c8e579e792377139ce6121, and compile it just fine
https://www.githubstatus.com/incidents/fnq063tqh7cc may be related
github had issues on 2024/12/17
and they could have affected cloning from https://github.com/vlang/tccbin
they do not say anything about 2024/12/19 though yet
shrewd jackal
imho check the logs if there were transient network problems during the time
No issues
hushed plank
check the logs on your machine
hushed plank
a -> b -> c -> d -> e
at any point, the link can be broken, and e could then fail to get data from a
for any reason
checking the logs on e, usually provides better info, about what it experienced, than checking the logs/states of the intermediaries
shrewd jackal
hushed plank
I think we should remove --quiet from the clone operations
to help diagnose such issues
#0 10:44:16 ^ master /v/oo>git clone --depth 1 --quiet --single-branch --branch thirdparty-linux-amd64 https://github.com/vlang/tccbin xxx
#0 10:44:36 ^ master /v/oo>file xxx/tcc.exe
xxx/tcc.exe: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=1ba1add6a3e9f4377204ff16379d2ea227c2ec28, for GNU/Linux 3.2.0, with debug_info, not stripped
#0 10:45:08 ^ master /v/oo>xxx/tcc.exe --version
tcc version 0.9.28rc 2024-07-31 HEAD@1cee0908 (x86_64 Linux)
cloning locally worked for me
shrewd jackal
I'll rerun 1f3336c9d34278ccf7c8e579e792377139ce6121
it failed again with the same reason
# 2024-12-20 08:45:52.130 scripting.run : make fresh_tcc
# cmd.exit_code : 0 cmd.output:
# 1: rm -rf ./thirdparty/tcc
# 2: git clone --depth 1 --quiet --single-branch --branch thirdparty-linux-amd64 https://github.com/vlang/tccbin ./thirdparty/tcc
# 3: The executable './thirdparty/tcc/tcc.exe' does not work. ```try git clone --depth 1 --quiet --single-branch --branch thirdparty-linux-amd64 https://github.com/vlang/tccbin abcdef manually
while it is happening
(and potentially remove the --quiet flag)
shrewd jackal
wait, it always outputs that
even with working commits
# ----------------------------------------------------------------------
# 2024-12-20 08:43:17.232 scripting.run : make fresh_tcc
# cmd.exit_code : 0 cmd.output:
# 1: rm -rf ./thirdparty/tcc
# 2: git clone --depth 1 --quiet --single-branch --branch thirdparty-linux-amd64 https://github.com/vlang/tccbin ./thirdparty/tcc
# 3: The executable './thirdparty/tcc/tcc.exe' does not work.
# ----------------------------------------------------------------------
# 2024-12-20 08:43:18.525 compile_oldv_if_needed : vc_v_bootstrap_flags: -no-parallel
# 2024-12-20 08:43:18.525 compile_oldv_if_needed : vc_v_cpermissive_flags: -Wno-error=incompatible-pointer-types -Wno-error=implicit-function-declaration -Wno-error=int-conversion
# 2024-12-20 08:43:18.525 compile_oldv_if_needed : vgit_context.commit_v__ts: 1643719812
# 2024-12-20 08:43:18.525 scripting.run : cc -Wno-error=incompatible-pointer-types -Wno-error=implicit-function-declaration -Wno-error=int-conversion -std=gnu11 -I ./thirdparty/stdatomic/nix -w -o cv "/root/.cache/oldv/v_at_vc/v.c" -lm -lpthread
# cmd.exit_code : 0 cmd.output:
# ----------------------------------------------------------------------
# 2024-12-20 08:43:28.139 scripting.run : ./cv -no-parallel -o v cmd/v
# cmd.exit_code : 0 cmd.output:
# ----------------------------------------------------------------------
# 2024-12-20 08:43:38.624 scripting.chdir : cd /root/.cache/oldv/v_at_a01484405081fa00eb5dd06260ced2acc5289c6d
# v commit hash: a014844050 | folder: /root/.cache/oldv/v_at_a01484405081fa00eb5dd06260ced2acc5289c6d
try running it manually
shrewd jackal
$ git clone --depth 1 --quiet --single-branch --branch thirdparty-linux-amd64 https://github.com/vlang/tccbin abcdef
$ ll abcdef/
total 2756
drwxr-xr-x 6 root root 4096 Dec 20 09:48 ./
drwxr-xr-x 6 root root 4096 Dec 20 09:48 ../
drwxr-xr-x 8 root root 4096 Dec 20 09:48 .git/
-rw-r--r-\- 1 root root 546 Dec 20 09:48 README.md
-rwxr-xr-x 1 root root 2253 Dec 20 09:48 build.sh*
-rw-r--r-\- 1 root root 30 Dec 20 09:48 build_on_date.txt
-rw-r--r-\- 1 root root 41 Dec 20 09:48 build_source_hash.txt
-rw-r--r-\- 1 root root 61 Dec 20 09:48 build_version.txt
drwxr-xr-x 2 root root 4096 Dec 20 09:48 include/
drwxr-xr-x 3 root root 4096 Dec 20 09:48 lib/
drwxr-xr-x 5 root root 4096 Dec 20 09:48 share/
-rwxr-xr-x 1 root root 2775576 Dec 20 09:48 tcc.exe*
that's in ubuntu though, outside the docker container
hushed plank
run abcdef/tcc.exe --version
shrewd jackal
tcc version 0.9.28rc 2024-07-31 HEAD@1cee0908 (x86_64 Linux)
hushed plank
does it run inside the container?
shrewd jackal
clone worked, but it says not found when running
/ # ./abcdef/tcc.exe --version
sh: ./abcdef/tcc.exe: not found
$ ls -al abcdef/
total 2756
drwxr-xr-x 6 root root 4096 Dec 20 08:49 .
drwxr-xr-x 1 root root 4096 Dec 20 08:49 ..
drwxr-xr-x 8 root root 4096 Dec 20 08:49 .git
-rw-r--r-\- 1 root root 546 Dec 20 08:49 README.md
-rwxr-xr-x 1 root root 2253 Dec 20 08:49 build.sh
-rw-r--r-\- 1 root root 30 Dec 20 08:49 build_on_date.txt
-rw-r--r-\- 1 root root 41 Dec 20 08:49 build_source_hash.txt
-rw-r--r-\- 1 root root 61 Dec 20 08:49 build_version.txt
drwxr-xr-x 2 root root 4096 Dec 20 08:49 include
drwxr-xr-x 3 root root 4096 Dec 20 08:49 lib
drwxr-xr-x 5 root root 4096 Dec 20 08:49 share
-rwxr-xr-x 1 root root 2775576 Dec 20 08:49 tcc.exe
no, I clean it up
I can just try to cd into one while it's working though
~/.cache/oldv/v_at_310969a057250a886251f8dd12c1a3e81daf91ab/thirdparty/tcc # ./tcc.exe --version
sh: ./tcc.exe: not found
nope they don't work either, might be a shell issue?
hushed plank
try file ./tcc.exe
shrewd jackal
./tcc.exe: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=1ba1add6a3e9f4377204ff16379d2ea227c2ec28, for GNU/Linux 3.2.0, with debug_info, not stripped
hushed plank
do you have /lib64/ld-linux-x86-64.so.2 in the container?
shrewd jackal
/lib64 is not a thing, no
shrewd jackal
yes
FROM alpine:3
COPY ./fast.v /data/fast.v
RUN apk add build-base
RUN apk add --no-cache git make rsync sqlite-dev
RUN git clone https://github.com/vlang/v /data/v
RUN cd /data/v && make
RUN /data/v/v /data/fast.v -prod -o /data/fast
ENTRYPOINT ["/data/fast"]
well, that potentially explains why it does not work
but it does not explain the gap
shrewd jackal
it works for all commits except those specific ones
hushed plank
it should have continued to work/not work the same
I still think that it may be a intermittent network/cloning issue
shrewd jackal
```
~/.cache/oldv/v_at_310969a057250a886251f8dd12c1a3e81daf91ab/thirdparty/tcc #...
yes but it's building just fine with this
hushed plank
it works for all commits except those specific ones <:renxSHRUG:1168858910581604...
can you try repeating only them?
shrewd jackal
commit 310969a057250a886251f8dd12c1a3e81daf91ab was tested successfully
hushed plank
although imho making it get the new commits is more valuable
since the newest is from 2024/12/13 afaik
shrewd jackal
can you try repeating only them?
I reran 1f3336c9d34278ccf7c8e579e792377139ce6121 and it failed again
hushed plank
given that it is an Alpine container
it is using musl
so perhaps it may be related to that
I've ran the above oldv run manually on Ubuntu, which is using glibc
for 1f3336c9d34278ccf7c8e579e792377139ce6121
│ 0e5c1ce 2022-05-03 09:17 +0300 DAngelov ∙ builtin: improve musl/Alpine support (define weak backtrace/backtrace_symbols/backtrace_symbols_fd symbols) (#1425
that seems to be close to the start of the gap
and afaik different versions of Alpine moved backtrace_symbols in/out of musl into its own package
libexecinfo or something
shrewd jackal
I wouldn’t expect such a TCC issue though…
hushed plank
the tcc issue is unrelated (from the makefile)
since you said that it happens even for oldv runs that succeeds
shrewd jackal
The „error“ in the git clone, yes. But then it doesn’t find TCC in the next step
shrewd jackal
successful commit: #1312494584151933001 message
failed commit: #1312494584151933001 message
hushed plank
can you run md5sum on both tcc.exe files?
latest one is:
b38b6b28e0872c41f026345195bfba7e /home/delian/.cache/oldv/v_at_1f3336c9d34278ccf7c8e579e792377139ce6121/thirdparty/tcc/tcc.exe (on Ubuntu)
shrewd jackal
b38b6b28e0872c41f026345195bfba7e abcdef/tcc.exe for ubuntu
b38b6b28e0872c41f026345195bfba7e tcc.exe inside the container
for the commits that dont work, I dont know. since its cleaned up and fails too quickly
manually running oldv in the container now
it worked? 🤨
nvm it didnt
# ----------------------------------------------------------------------
# 2024-12-20 09:43:18.400 scripting.run : ./cv -no-parallel -o v cmd/v
# cmd.exit_code : 1 cmd.output:
# 1: ==================
# 2: sh: /root/.cache/oldv/v_at_1f3336c9d34278ccf7c8e579e792377139ce6121/thirdparty/tcc/tcc.exe: not found
# 3: ...
# 4: ==================
# 5: (Use `v -cg` to print the entire error message)
# 6:
# 7: builder error:
# 8: ==================
# 9: C error. This should never happen.
# 10:
# 11: This is a compiler bug, please report it using `v bug file.v`.
# 12:
# 13: https://github.com/vlang/v/issues/new/choose
# 14:
# 15: You can also use #help on Discord: https://discord.gg/vlang
# 16:
# ----------------------------------------------------------------------
# 2024-12-20 09:43:21.464 scripting.chdir : cd /root/.cache/oldv/v_at_1f3336c9d34278ccf7c8e579e792377139ce6121
# v commit hash: 1f3336c9d3 | folder: /root/.cache/oldv/v_at_1f3336c9d34278ccf7c8e579e792377139ce6121
/data/v # md5sum /root/.cache/oldv/v_at_1f3336c9d34278ccf7c8e579e792377139ce6121/thirdparty/tcc/tcc.exe
b38b6b28e0872c41f026345195bfba7e /root/.cache/oldv/v_at_1f3336c9d34278ccf7c8e579e792377139ce6121/thirdparty/tcc/tcc.exe
not found but file is there with correct hash
file output is also identical
file size is identical
permissions are also the same
Im perplexed
Is there a way to get oldv to use gcc or something for bootstrap instead of TCC?
hushed plank
set VFLAGS="-cc gcc" in the container
but imho do not bother with it
making it show the most recent commits is much more important
than making it work for arbitrary commits from 2 years ago
shrewd jackal
gcc shows the actual errors
# 2024-12-20 10:30:26.265 scripting.run : ./cv -no-parallel -o v cmd/v
# cmd.exit_code : 1 cmd.output:
# 1: ==================
# 2: /tmp/v_0/v.14567018460197361098.tmp.c: In function 'print_backtrace_skipping_top_frames_linux':
# 3: /tmp/v_0/v.14567018460197361098.tmp.c:17613:39: error: implicit declaration of function 'backtrace' [-Wimplicit-function-declaration]
# 4: 17613 | int nr_ptrs = backtrace(&buffer[0], 100);
# 5: | ^~~~~~~~~
# 6: /tmp/v_0/v.14567018460197361098.tmp.c:17620:43: error: implicit declaration of function 'backtrace_symbols' [-Wimplicit-function-declaration]
# 7: 17620 | char** csymbols = backtrace_symbols(((voidptr)(&buffer[v_fixed_index(skipframes, 100)])), nr_actual_frames);
# 8: | ^~~~~~~~~~~~~~~~~
# 9: /tmp/v_0/v.14567018460197361098.tmp.c:17620:43: error: initialization of 'char **' from 'int' makes pointer from integer without a cast [-Wint-conversion]
which as you suspected were related to backtrace on alpine
hushed plank
try passing VFLAGS="-cc gcc -cflags -lexecinfo"
and perhaps modify the container to add that package
shrewd jackal
give me 3mins since I am recreating the image to have new commits
shrewd jackal
and perhaps modify the container to add that package
which one specifically?
hushed plank
apk add libexecinfo-dev
and perhaps elfutils-dev
(reading from https://github.com/awslabs/aws-lambda-cpp/issues/124)
(musl tends to be PITA for development, even though it is good for end users, because of the ability to do a fully static deployment)
shrewd jackal
libexecinfo-dev doesnt seem to be a package
hushed plank
try libunwind-dev and libunwind-static
there is also something called libc6-compat
from our Dockerfile.alpine
shrewd jackal
still cant find execinfo
changed -lexecinfo to -lunwind
and it worked
well it ran oldv
but still same backtrace error
hushed plank
did you add libunwind-dev ?
shrewd jackal
yes
wait, this is a new issue separate from the TCC one
because it no longer fails to bootstrap, just when self-compiling using the bootstrapped cv
hushed plank
there seems to be no surprises in the data for the latest commits; all the trends continue as expected
shrewd jackal
Done with 2022 now thanks to the gap
shrewd jackal
we're before -stats showed v lines & v size now
shrewd jackal
Done with 2021
hushed plank
9899 commits processed 🥳
it will probably speed up even more for the older commits
because the source was much smaller
shrewd jackal
I do wish we’d have the 2022 data…
Even if I had to patch oldv to modify the v.c file to add missing headers/fix it to run on alpine
hushed plank
imho running it on an ubuntu container will be less painful (afaik no one develops V itself on Alpine, it is only used for deployment, while plenty of people, including me, do use Ubuntu as main development machines)
but given the slopes and the general shape, I do not expect big surprises in the 2022 line segment
I wish we had introduced the V lines/s metric earlier
I suspect that it would have peaked somewhere before April 2020
in the V versions that were one pass without AST
although this suggests, that the peak was around May, 2021
and it is interesting how sharp it is 🤔, i.e. it was increased for a short time, and then it dropped just as sharply
shrewd jackal
imho running it on an ubuntu container will be less painful (afaik no one develo...
Bit late for that, no? 
hushed plank
yes
shrewd jackal
Do old V versions error out when passed unrecognised arguments?
Current 2020 commits with -skip-unused finish within a millisecond and have no scan/parse/check/cgen phases
icy agate
in the V versions that were one pass without AST
2 passes actually (2 parser runs, one cgen)
icy agate
Do old V versions error out when passed unrecognised arguments?
it's true for new versions as well 🙂
shrewd jackal
it's true for new versions as well 🙂
Ah that’ll be slightly annoying
hushed plank
Check for the exit code.
shrewd jackal
fast is at 1038MB
web server is at 226MB
this is my fix for now
so the _skip_unused fields in the database will just be NULL if unsupported
instead of having nonsensical values
redeploying, so new commits from the past week will be processed now
shrewd jackal
web server has been running for 2 weeks straight
b09338005b6f v-fast2-web "/data/server -dynam…" 2 weeks ago Up 2 weeks
hushed plank
nice
shrewd jackal
fast was running for a week straight
wait I did the wrong > in my check nvm i did not
set the v_self_skip_unused_id and v_hello_skip_unused_id columns to NULL for commits before -skip-unused was added, so the graphs start at the commit where it was added
welp, that just made them invisible from all charts due to an old check
@hushed plank can you replace AND commits.v_self_skip_unused_id IS NOT NULL with AND commits.${kind} IS NOT NULL in https://github.com/spytheman/oldv_charts/blob/master/utils.v#L100
shrewd jackal
thanks, updating the server
shrewd jackal
we're before all timings now
and looks like the total in -stats was restricted to ms precision back then
nvm
hushed plank
I think it used .ticks()
shrewd jackal
it's so old, that -stats didnt have timing
it's falling back to execution time now
hushed plank
@shrewd jackal, try recompiling with latest V, the latest version of the webserver, when you have some time
it should reduce the memory usage a bit
shrewd jackal
I saw the commit
156MB
commits before e6828560d115c22e0b48dc7ea99ad776a10c53b8 (Nov 6, 2020) dont build
backtrace issues
# 2024-12-27 16:47:17.451 scripting.run : ./cv -o v cmd/v
# cmd.exit_code : 1 cmd.output:
# 1: ==================
# 2: /tmp/v_0/v.tmp.c: In function 'print_backtrace_skipping_top_frames_linux':
# 3: /tmp/v_0/v.tmp.c:9087:33: error: passing argument 1 of 'backtrace' from incompatible pointer type [-Wincompatible-pointer-types]
# 4: 9087 | int nr_ptrs = backtrace(buffer, 100);
# 5: | ^~~~~~
# 6: | |
# 7: | unsigned char **
# 8: /tmp/v_0/v.tmp.c:280:47: note: expected 'void **' but argument is of type 'unsigned char **'
# 9: 280 | int backtrace (void **__array, int __size) { return 0; }
# 10: | ~~~~~~~^~~~~~~
# 11: /tmp/v_0/v.tmp.c:9094:47: error: passing argument 1 of 'backtrace_symbols' from incompatible pointer type [-Wincompatible-pointer-types]
# 12: 9094 | charptr* csymbols = backtrace_symbols(&buffer[skipframes], nr_actual_frames);
# 13: | ^~~~~~~~~~~~~~~~~~~
# 14: ...
can oldv add -d no_backtrace or whatever the flag is?
if those old versions even support that
hushed plank
try passing -Wno-incompatible-pointer-types in CFLAGS
shrewd jackal
oldv already passes -Wno-error=incompatible-pointer-types
ah nvm
thats for the v.c compile
shrewd jackal
yes, v.c to cv worked
hushed plank
so the first part of the bootstrap seems to have succeeded
shrewd jackal
the first line says its running ./cv -o v cmd/v, which fails
hushed plank
try setting VFLAGS to -d no_backtrace
afaik VFLAGS support was added very early
in 5d0cb14 2019-07-16
shrewd jackal
added VFLAGS="-d no_backtrace" CFLAGS="-Wno-error=incompatible-pointer-types"
lets see
hushed plank
so it may work
hushed plank
try VFLAGS="-d no_backtrace -cflags -Wno-error=incompatible-pointer-types"
shrewd jackal
hushed plank
CFLAGS is also good, for the v.c compilation/bootstrap
but it may not be respected by older V versions
shrewd jackal
nope, still not working
hushed plank
for e6828560d115c22e0b48dc7ea99ad776a10c53b8 ?
hushed plank
what are the oldest ones
shrewd jackal
I dont know, its still going through them
hushed plank
I need 2-3 hashes
hushed plank
ok
hushed plank
the main trouble is using a more modern (stricter) C compiler
that has more warnings turned into errors
by default
you can try passing -fpermissive in CFLAGS, and with VFLAGS="-cflags -fpermissive" too
shrewd jackal
probably just need to pass vc_v_cpermissive_flags in here too?
https://github.com/vlang/v/blob/master/cmd/tools/modules/vgit/vgit.v#L260
assuming -cflags is supported
shrewd jackal
cv managed to compile
hushed plank
try with latest V
for me it compiled 1bc906357319d2fd72399868b18bcc353639ab59
with clang-18
shrewd jackal
try with latest V
trying
ok so oldv works now!
but my own builds not because I am missing the option!
shrewd jackal
it uses whatever is latest when I build the image
so I deleted it and rebuilt it
I recompile V with -prod (after oldv), which failed because of the missing cflags
added them now and we'll see in a bit
it works!!
hushed plank
nice
shrewd jackal
re-queued 2.3k failed commits
oh half of those are probably the ones from the gap in 2022
anyway…
TCC cache when
hushed plank
as a flag for oldv?
fresh remnant
Still wondering why oldv checks out tcc every time it runs. If it's always checking out latest, and tcc should already be available before oldv even gets built...
shrewd jackal
the make fresh_tcc step is making a new TCC clone for every commit
shrewd jackal
Still wondering why oldv checks out tcc every time it runs. If it's always chec...
it runs make fresh_tcc
maybe really old versions put it somewhere else/used a different source?
fresh remnant
I thought it was always in thirdparty/tcc.
shrewd jackal
hushed plank
it is just simpler, and so far it was used mostly for manual bisecting of a handful of commits
-> the additional time was not as big a factor as it is for thousands
shrewd jackal
it's ~1.1s per commit
fresh remnant
Maybe it needs a new option for this... -no-tcc-update or something. It would do what it does now by default, but with this switch, it skips the tcc update, and just uses what's there.
hushed plank
the bigger impact is that it makes more network requests
and those are inherently unreliable
I do not like making complexity/reliability tradeoffs, but sometimes they are not avoidable 😐
shrewd jackal
Maybe it needs a new option for this... `-no-tcc-update` or something. It would...
well nothing is there because it copies the v folder, checks out the old commit and then does a git clean
preferably, just clone TCC into the .cache/oldv directory and then link thirdparty/tcc to it?
hushed plank
you can already do that with setting TCCREPO to some file:// url
it is just more complex 😐
and will require a bit of scripting
given that oldv does not do it itself
fresh remnant
oldv may need to do it, as this only affects oldv.
oldv could simply be changed to use -cc VROOT/thirdparty/tcc/tcc.exe
hushed plank
the trouble is that the makefile itself has also evolved
and the support for TCCREPO with it too
so for older commits, you can still get into weird situations
fresh remnant
Maybe with a -use-v-tcc?
hushed plank
which just running make avoids
at the cost of fetching it from the network (or doing whatever the current makefile for the commit specifies)
since v supported make as the official way to build itself since the start
even though the actual commands changed
the interface/top level command is stable
fresh remnant
It's worth a bit of effort to figure out a way around this, though. Tens of thousands of commits @ 1.1s per commit works out to be a whole lot of wasted time.
And network accesses, and...
shrewd jackal
im not sure why it even uses TCC anyway, why not use GCC?
hushed plank
because tcc is faster
shrewd jackal
i.e. the 1s is <1% of the total
commits that fail to compile usually take ~10s, so it's 10% of the runtime
hushed plank
and gcc is not available on macos by default
and on freebsd too
and probably others that have GNU phobia
shrewd jackal
# 2024-12-27 19:25:31.608 scripting.run : cc -Wno-error=incompatible-pointer-types -Wno-error=implicit-function-declaration -Wno-error=int-conversion -fpermissive -std=gnu11 -I ./thirdparty/stdatomic/nix -w -o cv "/root/.cache/oldv/v_at_vc/v.c" -lm -lpthread
it looks like its running cc and not tcc anyway though?
fresh remnant
Except Windows.
shrewd jackal
yes, so the bootstrap itself requires cc to be installed, so when would it ever not be available after the bootstrap?
hushed plank
and has several rules depending on what compiler is present there
including ones for msvc, for gcc and for tcc
hushed plank
yes, so the bootstrap itself requires `cc` to be installed, so when would it eve...
cc is slower, tcc is faster
fresh remnant
I've always thought v doctor on Windows should look up which compiler will be used, same as V itself, rather than just reporting "no CC found" (or whatever the message is...).
hushed plank
unless cc is tcc, but there are not many distros with that
you are free to set VFLAGS=-cc cc if you want to btw
shrewd jackal
ok so:
- oldv (vgit) is currently running
make fresh_tcc - oldv is not using TCC by itself
so why not add a -no-tcc flag that simply skips the make fresh_tcc step?
fresh remnant
Using anything other than tcc would've increased the time to update the database by probably 10X or more.
hushed plank
because it will complicate oldv and the makefile
for the sake of a task, that has not been done before, and that still needs to run with the older makefiles
shrewd jackal
vgit is explicitly calling make fresh_tcc, its not running the makefile
hushed plank
as they were at the time of the commits
shrewd jackal
oh the flag already exists: https://github.com/vlang/v/blob/master/cmd/tools/modules/vgit/vgit.v#L167
nvm then lol
fresh remnant
So there's an option to turn it on... how do you turn it off? Or what is supplying the option to turn it on?
shrewd jackal
there's a boolean option and it defaults to on
hushed plank
pass --fresh_tcc=false
hushed plank
Using anything other than tcc would've increased the time to update the database...
tcc is not used already
because of the unfortunate choice of an Alpine container
instead of an Ubuntu one
and the runs are done with V compiled with cc
fresh remnant
Never mind, then. But at least the constant downloads of tcc can be stopped. 😕
As for base image... ubuntu isn't nearly as bad, these days. Used to be at least 1/2 Gig or more, but their latest are down around 75 Megs. They no longer include XWindows, etc., etc. in the base image.
Still bigger than Alpine base, but more "standard" (as in, matching more distros that people normally use).
hushed plank
it is not so bad, because most of the measurements are done with flags that prevent running the C compiler
fresh remnant
There you go... just did docker pull ubuntu:
ubuntu latest b1d9df8ab815 5 weeks ago 78.1MB
like -o x.c or -check-syntax etc
shrewd jackal
all checks do -o x.c
fresh remnant
All good, then. No C compiler called... so definitely no need for tcc to be pulled down.
hushed plank
also forcing cc, instead of tcc, will produce more reliable results hopefully (since there were times where tcc could not compile yet V, and times where it could, but not fully, and then it fall back to cc anyway, but that still affected the timings)
shrewd jackal
I think the additional cflags also fixed the 2022 gap
shrewd jackal
total took 2:53.200, remaining: 7969, time per commit: 2:52.317
ETA: 381h
time per commit will go down once we're back in 2020 again
in like a week or so
hushed plank
how is the RAM usage of the web server?
shrewd jackal
111M*
shrewd jackal
it goes up a lot when its actively serving requests
shrewd jackal
2022 gap closed
hushed plank
nice
2022/05/08 seems interesting 🤔
b53b1cc 2022-05-07, reverted later in cf536b8 2022-05-11
other than that, the data seems pretty normal
shrewd jackal
It stopped testing new commits
Last successful was https://github.com/vlang/v/commit/155891a4e0728396357f4db5b3bb8f064fceff12
Someone remind me tomorrow to take a look
shrewd jackal
the commit before (08fac28c520c1446dfc6aa1897fbbb780e731ff7) errored with
# ----------------------------------------------------------------------
# 2024-12-31 21:18:27.265 scripting.run : ./cv -cflags " -Wno-error=incompatible-pointer-types -Wno-error=implicit-function-declaration -Wno-error=int-conversion -fpermissive" -o v cmd/v
# cmd.exit_code : 0 cmd.output:
# 1: warning: import cycle detected between the following modules:
# 2:
# 3: * v.table -> v.ast
# 4: * v.ast -> v.table
# 5:
# ----------------------------------------------------------------------
true!
lexec returned with 4
so it errors with status code 4 when compiling hello_world/v
2024-12-31T21:18:48.022175Z [INFO ] warmup...
2024-12-31T21:18:48.022177Z [INFO ] lexec: /tmp/v_0/v-fast/v-08fac28c520c1446dfc6aa1897fbbb780e731ff7 cmd/v -show-timings -stats -o '/tmp/v_0/v-fast/out.c'
2024-12-31T21:18:48.136299Z [INFO ] lexec returned with 4
unsupported option, probably?
hushed plank
could be 🤔 I'll check locally
seems to work locally, it exits with 0
I'll try changing the working folder
still works:
shrewd jackal
hmmm
~/.cache/oldv/v_at_08fac28c520c1446dfc6aa1897fbbb780e731ff7 # ./v -cflags " -Wno-error=incompatible-pointer-types -Wno-error=implicit-function-declaration -Wno-error=int-conversion -fpermi
ssive" -o v cmd/v
warning: import cycle detected between the following modules:
* v.table -> v.ast
* v.ast -> v.table
~/.cache/oldv/v_at_08fac28c520c1446dfc6aa1897fbbb780e731ff7 # echo $?
0
that particular version also does no argument validation at all
it just ignores any that it does not know
(the #0 at the start of my prompts, show the exit code of the previous command)
shrewd jackal
~/.cache/oldv/v_at_08fac28c520c1446dfc6aa1897fbbb780e731ff7 # ./v cmd/v -show-timings -stats -o '/tmp/v_0/v-fast/out.c'
warning: import cycle detected between the following modules:
* v.table -> v.ast
* v.ast -> v.table
~/.cache/oldv/v_at_08fac28c520c1446dfc6aa1897fbbb780e731ff7 # echo $?
0
I cant repro in the same alpine container
hushed plank
perhaps there were VFLAGS set ?
hushed plank
#0 14:18:56 ^ (HEAD detached at 08fac28c5) ~/.cache/oldv/v_at_08fac28c520c1446dfc6aa1897fbbb780e731ff7>rg exit.*4
vlib/builtin/bare/linuxsys_bare.v:11:wexited = 0x00000004
cmd/tools/modules/scripting/scripting.v:25:println('# cmd.exit_code : ${x.exit_code.str():-4s} cmd.output:')
cmd/tools/oldv.v:109:println('# command: ${context.cmd_to_run:-34s} exit code: ${cmdres.exit_code:-4d} result:')
cmd/tools/vcreate.v:86:exit(4)
any other important env variables that'd be nice to log?
hushed plank
exit(4) seems to be only used by v create, so I do not know what could have led to it 🤔
shrewd jackal
if it exits with > 0, it should log the entire result (stdout, stderr?) too, but it was empty
hushed plank
could it be killed somehow?
shrewd jackal
hushed plank
how is the RAM usage of the container?
shrewd jackal
I reprocessed and it did the same
half a gigabyte
in total, 2.7gb out of 64gb used
so it shouldnt OOM, ever
hushed plank
so an OOM is very unlikely 🤔
hushed plank
any other important env variables that'd be nice to log?
I think not
there is also VOSARGS, but it is very unlikely to be used
(it is a legacy one, that acts like VFLAGS, and we should probably deprecate it)
shrewd jackal
ok, I'm logging VFLAGS & VEXE before every lexec() call now
ok we gotta wait a bit, its benchmarking new commits first
hushed plank
interesting, the latest commits seems to have some regression for the size of the generated C code for hello_world.v 🤔
but locally, it seems to be still small
#0 14:52:48 ^ master ~/code/v>./v -o x.c examples/hello_world.v && wc x.c
1938 5208 45403 x.c
I should probably try it in an Alpine container
#0 14:53:39 ^ master ~/code/v>./v -cc musl-gcc -o x.c examples/hello_world.v && wc x.c
5659 21556 199334 x.c
musl-gcc was enough 🤔
interesting
although that may not be enough, since ./v -cc musl-gcc -o x.c examples/hello_world.v && wc x.c produces the same for the commits from 2024/12/27 too
I'll just wait till it shows where exactly it jumped
shrewd jackal
I'll just wait till it shows where exactly it jumped
it should be there now
VFLAGS is unset
2025-01-01T18:17:02.981127Z [INFO ] warmup...
2025-01-01T18:17:02.981144Z [INFO ] lexec: /tmp/v_0/v-fast/v-08fac28c520c1446dfc6aa1897fbbb780e731ff7 cmd/v -show-timings -stats -o '/tmp/v_0/v-fast/out.c'
2025-01-01T18:17:02.981153Z [INFO ] lexec VEXE: /root/.cache/oldv/v_at_08fac28c520c1446dfc6aa1897fbbb780e731ff7/v
2025-01-01T18:17:03.097168Z [INFO ] lexec returned with 4 in 116.022ms
only VEXE is
shrewd jackal
ok found the problem
./v -prod self
cannot compile ‘/root/.cache/oldv/v_at_08fac28c520c1446dfc6aa1897fbbb780e731ff7/cmd/tools/vself.v:
==================
/root/.cache/v/vself.tmp.c: In function 'os__exec':
/root/.cache/v/vself.tmp.c:8461:72: error: passing argument 1 of 'vstrlen' from incompatible pointer type [-Wincompatible-pointer-types]
8461 | strings__Builder_write_bytes(&res, &/*qq*/buf, vstrlen(&/*qq*/buf));
| ^~~~~~~~~~
| |
| byte (*)[4096] {aka unsigned char (*)[4096]}
/root/.cache/v/vself.tmp.c:5424:21: note: expected 'byteptr' {aka 'unsigned char *'} but argument is of type 'byte (*)[4096]' {aka 'unsigned char (*)[4096]'}
5424 | int vstrlen(byteptr s) {
| ~~~~~~~~^
/root/.cache/v/vself.tmp.c:8461:52: error: passing argument 2 of 'strings__Builder_write_bytes' from incompatible pointer type [-Wincompatible-pointer-types]
8461 | strings__Builder_write_bytes(&res, &/*qq*/buf, vstrlen(&/*qq*/buf));
| ^~~~~~~~~~
...
==================
no idea how my earlier checks dont catch it though, it returns exit code 1
and if the v compiler wasnt successfully compiled, it should exit with 127 instead of 4
hushed plank
you may have to make cc a wrapper, that always passes -fpermissive to the actual compiler
shrewd jackal
making a live version rn 👍
hushed plank
live as in always updating from the latest commits?
shrewd jackal
since I do not understand the mystery of pre-april 2020 commits
hushed plank
that will be very nice, thank you!
hushed plank
I could not figure out what happened after 2024/12/27
is it possible that the jump is related to some flag/hardware change?
the commit 60258937688384f7a24d73291b9357f5ccdf2552 is for vlib/net/smtp/smtp.v
it should not have affected the compiler at all
shrewd jackal
is it possible that the jump is related to some flag/hardware change?
no
the server has been running non-stop for 34 days
hushed plank
the one before that dcc1a6c226aebd8a467fbea7fae477d0a0cbf845, is for .github/workflows/v_apps_and_modules_compile_ci.yml
shrewd jackal
check if the hostname changed in the db
hushed plank
which also could not possibly affect the compiler and its speed
shrewd jackal
the hostname is equal to the container name, so a different name would indicate a new image with a possibly newer alpine/gcc version
shrewd jackal
yeah, different container names
hushed plank
can it be pinned to a specific version?
shrewd jackal
if you tell me how, sure 😄
FROM alpine:3
RUN apk add build-base
RUN apk add --no-cache git make rsync sqlite-dev
RUN git clone https://github.com/vlang/v /data/v --depth 1
RUN cd /data/v && make
COPY ./fast.v /data/fast.v
RUN /data/v/v /data/fast.v -prod -o /data/fast
ENTRYPOINT ["/data/fast", "-watch"]
it should no longer happen however, since I moved the COPY ./fast.v /data/fast.v down the stack