viper

[lethal quest]

lol

thanks

i need to stop putting random tuples and pairs

hm, still getting the same linker issues

annoyingly i cant use objdump on windows

[cold plaza]

why not? the one in msys2 works just fine

and also llvm-readelf for seeing the symbols/relocs

[lethal quest]

it just says format not recognized

possibly because the obj is completely fucked

[cold plaza]

what if you do file on the object

[lethal quest]

does msys2 have that?

[cold plaza]

yes

[lethal quest]

hold on

ok i have some weird version of those programs

that isnt msys2

ill install it now

[cold plaza]

also one funny thing I found is that if the input file doesn't exist you are going to get an empty object file as the output (well not completely empty but with only the header and no sections)

[lethal quest]

yeah i saw that

ill fix that eventually

[cold plaza]

what is the issue that you get

[lethal quest]

link.exe gives some weird error about disk

and golink says an error occurred

[cold plaza]

what about mingw gcc

[lethal quest]

let me find out

just installing base-devel

ah mingw objdump actually works

i get undefined reference to `no symbol'

heres llvm-readelf output

[cold plaza]

thats weird, there are literally no symbols in that

[lethal quest]

yeah

[cold plaza]

what is the asm that it was generated from?

[lethal quest]

message: db "Hello World!\0"

extern OutputDebugStringA

main:
    mov rcx, message
    call OutputDebugStringA

    mov rax, 0
    ret

ah, one issue is the lack of a short name for that symbol

afaict, theres no handling for anything other than a short name

[cold plaza]

oh yeah

[lethal quest]

for (const auto& sym : mSymbolTable) {
    PEWrite(stream, sym.mShortName.name, 8);
    PEWrite(stream, sym.mValue);
    PEWrite(stream, sym.mSectionNumber);
    PEWrite(stream, sym.mType);
    PEWrite(stream, sym.mStorageClass);
    PEWrite(stream, sym.mNumberOfAuxSymbols);
}
``` i assume something here?

yep

[cold plaza]

yeah if its larger than 8 bytes the first 4 bytes are zeroes and then the following 4 contain a string table offset

[lethal quest]

ah you've got the union in there already

[cold plaza]

actually wait it should be supported

        if (name.size() < 8)
        {
            memcpy(symbol.mShortName.name, name.c_str(), name.size() + 1);
        }
        else
        {
            symbol.mShortName.offset.zeros = 0;
            symbol.mShortName.offset.offset = mStringTable.size();
            mStringTable += name + '\0';
        }

[lethal quest]

yes i just realized that

thats the entire point of the union

yea

unless some padding is being added to the struct?

[cold plaza]

looks like there are indeed 2 bytes of padding at the end

[lethal quest]

ill just make it a uint64_t

[cold plaza]

but that padding shouldn't really affect anything as the struct isn't being straight up written to the file

[lethal quest]

it kinda is

bcs we are treating it as a char[8]

still no symbols either way

the symbol names are in the file, so i dont know why its not showing any

[cold plaza]

it looks to be somehow related to the long symbol name because even if you have just ```x86asm
extern OutputDebugStringA

main:
ret

[lethal quest]

ill try just with main

that still doesnt give me any symbols

hmm

[cold plaza]

At the beginning of the COFF string table are 4 bytes that contain the total size (in bytes) of the rest of the string table. This size includes the size field itself, so that the value in this location would be 4 if no strings were present.

[lethal quest]

ah

do we have that

[cold plaza]

no

[lethal quest]

no we dont

let me add that now

should be easy enough

PEWrite(stream, static_cast<uint32_t>(mStringTable.size() + 4));

ok that fixed it

[cold plaza]

did you also change the offset

[lethal quest]

wdym

for symbols?

oh wait i see what you mean

[cold plaza]

size_t usableStart = symbolTableOffset + symbolTableSize + mStringTable.size();

[lethal quest]

ah

ill do that now

also needed to change it in the symbol index into the strtab

that looks better

nice, it segfaults

my guess is that its something to do with the relocation

[cold plaza]

well that makes sense as the call instruction only expects a 4 byte relative relocation

so doing a 8 byte absolute relocation is going to overwrite something after it (and not really be right for the call either)

[lethal quest]

mVirtualAddress is only 32-bits

ah i see we are using IMAGE_REL_AMD64_ADDR64

nasm emits a IMAGE_REL_AMD64_REL32 there

[cold plaza]

yeah the relocation type should actually depend on the instruction

[lethal quest]

hm

thats a bit annoying

especially abstracting that

i can maybe just get away with passing a parameter which is absolute or relative

that looks better

though the first relocation name is .text from nasm and not message

yea, i think thats something we are missing

symbol table entries for sections

[lethal quest]

today i will try to flesh out the type system a bit more

add void type and pointer types

[lethal quest]

ok i did all that

not sure what i want to do next

maybe some control flow

[lethal quest]

i now have if statements

[lethal quest]

also added while loops

i think thats a good place to stop for tonight

[lethal quest]

adding more operators today, ie < > >= <=

ok i finished adding all of those

func @main() -> i32 {
    let testvar: i32 = 1;

    while(testvar <= 13) {
        testvar += 1;
    }

    return testvar;
}``` about the limit of what i can do currently

but still progress

i will probably add function calling next

which is something ive been avoiding

because of the complexity it adds to regalloc

im not sure whether i want to just throw a value into a new register if i encounter something that needs a specific register, or if i want to allocate around that

[cold plaza]

I guess the optimal solution would be to somehow reconsider the previous register allocations for the operands/a variable that is using some kind of result register if you encounter something like that

but it might not be that simple to do

the simplest solution would probably be to just spill the registers for the duration of the instruction and do moves to them before the instruction

[lethal quest]

yeah that would be simplest

i was thinking about some kind of tracking of which registers are going to be required

then not allocate them for any values that are live ever at the same time as the values requiring those registers

but that isnt the easiest thing

[cold plaza]

yeah

then there are even more things you could do to optimize it further like if there are other long living variables and you don't have any other registers for them than the specific needed ones it might still make sense to put them to there and spill them for the duration of the other variable/move the other variable to some other register at some point

but I wouldn't worry about things like this to begin with

[lethal quest]

it shouldnt be too difficult to change later

as long as i dont make it too crap

[lethal quest]

i will have very little if any time to do anything today

so if i do have some time i'll probably just add some more operators or something

[lethal quest]

wont be doing anything tonight, im too tired

[lethal quest]

nvm^^

[lethal quest]

ok i will attempt to add function calls today

or at least begin them

[lethal quest]

func @test() -> i32 {
    return 69;
}

func @main() -> i32 {
    let testvar: i32 = 1;

    while(testvar <= test()) {
        testvar += 1;
    }

    return testvar;
}``` i am able to do this now

one thing though is i dont think it would work if you call multiple functions that have overlapping intervals

as it would try to put both of them into eax

so i need to find a solution to that

not sure what i want to do with the 30 minutes i have to work on this tonight

[lethal quest]

we have pointers now

func @test() -> i32 {
    return 69;
}

func @main() -> i32 {
    let testvar: i32 = 1;

    while(testvar <= test()) {
        testvar += 1;
    }

    let ptr: i32* = &testvar;

    return *ptr;
}```

[astral fractal]

neat

[astral fractal]

question: why * after type not before

[lethal quest]

im not sure

its slightly easier to parse

[astral fractal]

is it?

[urban bane]

* before is easier

because let a: i32* = &testvar can be parsed as let a: i32 *= &testvar

[astral fractal]

nah

thats irrelevant

*= is special in the same way != and ++ are

[urban bane]

both arent assignments

who tf renamed me rustc

#staff message

ah nvm

[astral fractal]

doesnt matter

[urban bane]

if ur grammar is wrong with something like

let <symbol>: <type> <assignment>

[astral fractal]

it doesn't matter because both have the same sort of semantics

++ is different from + +

just as *= is different from * =

and let a: i32 *= lol isnt even legal anyway

[urban bane]

ok fair

[empty pendant]

hmmmmmm

[lethal quest]

yes because i parse the type then check for star tokens immediately after

and set the type to a pointertype to the previous one

type =
gettype(currentTokenString)
while currentToken is star
    type = pointertype(type)

its not much easier

but a bit

[lethal quest]

this is fixed now

i will add arguments next

[lethal quest]

ok i added arguments to vipir

now i will add them to the lang itself

then parameters for call expressions

func @main(argc: i32) -> i32 {
    let testvar: i32 = argc;

    let ptr: i32* = &testvar;

    return *ptr;
}```  thats done

the codegen is horrendous

[lethal quest]

added parameters to vipir

[lethal quest]

func @max(a: i32, b: i32) -> i32 {
    if (a > b) return a;
    return b;
}

func @main(argc: i32) -> i32 {
    return max(argc, 5);
}``` most advanced viper program

the build times for the entire project are getting a bit out of hand

ok you can now declare and call external functions, so you can print an integer by calling a C function

i will add strings next

[lethal quest]

for some strange reason this assembly: x86asm 5d: 48 c7 45 f8 00 00 00 mov QWORD PTR [rbp-0x8],0x0 64: 00 65: b8 00 00 00 00 mov eax,0x0 6a: c9 leave 6b: c3 ret is getting linked to this assembly x86asm 11bd: 48 c7 45 f8 60 11 00 mov QWORD PTR [rbp-0x8],0x1160 11c4: 00 11c5: 00 00 add BYTE PTR [rax],al 11c7: 00 00 add BYTE PTR [rax],al 11c9: 00 c9 add cl,cl

ive got a feeling my relocations are off

ah yeah i think thats it

[lethal quest]

wasn't able to get that done tonight, there are some more deep rooted issues i will need to look into

[lethal quest]

i might just make global things always be PIE

i dont really see any reason not to

it should make relocations and suchlike a lot simpler

[lethal quest]

think i've found some issues with how im doing the relative offsets

fixed that

next thing i need to add is SIB, which ive kinda been avoiding

[lethal quest]

ok that wasnt as bad as i initially thought

though the next thing is to add it to vipir for pointer arithmetic, which might be a bit more involved

[lethal quest]

added pointer addition to viper

im not exactly sure what i want to do next

[lethal quest]

ok i added extremely basic structs

they barely function

[lethal quest]

added -> operator

[lethal quest]

added global variables

[lethal quest]

one thing im not exactly sure how i want to do is pointer addition for struct pointers

since you cant just use the scale part of sib

what gcc seems to do is multiply the index to the number of fields then multiply that by the alignment

[lethal quest]

i also need to stop throwing everything into the text section

[lethal quest]

added the data section and now globals other than functions are put in it

[lethal quest]

im debating what the syntax should be for struct initializing

something with { ... } but i feel like a prefix to that would make it a bit clearer

maybe something like struct foo { ... }

[cold plaza]

just a personal opinion but I hate that :p for the same reason that I hate using non-typedef'd structs in c, imo its mostly just useless prefixes for the sake of it when you can easily guess what it is from the usage of it (or just use an ide and look it up there)

[lethal quest]

im also coming at it from an ease-of-parsing perspective

{ ... } would be ambiguous between a compound statement and an initializing block in that case

and i guess i could just make it depend on if its coming after an assignment or not but i dont really like that kind of context-sensitive stuff

[cold plaza]

how would foo {...} be ambiguous?

[lethal quest]

sorry i thought you just meant { ... }

yeah i guess i could do that

ill have a think

[cold plaza]

yeah

[lethal quest]

the build times are getting a bit out of hand

[lethal quest]

i think not too long from now i will be able to begin writing viperos

maybe a week or two

the big things i need to implement are arrays and type casting

[lethal quest]

added pointer casts

struct abc {
    test: i64;
    foo: i32;
}

global a: struct abc = struct abc { 11, 33 };

func @main() -> i8 {
    let ptr: i32* = &a.foo;
    return *(i8*)ptr;
}```

the next thing i want to implement is casts of integral types to other integral types

then integral -> pointer and vice versa

[lethal quest]

added casts from smaller to larger integral types

now i need truncation

done that also now

last bit of casting for the time being done, which is ptr -> int and int -> ptr

[lethal quest]

arrays also done now

[astral fractal]

arrays as in slices?

or as in c-style pointers

also why is global separate from let?

seems a bit weird

and also why struct tags?

they are there in C to ensure unambigous parsing

and even then not really

[lethal quest]

as in C style arrays

[lethal quest]

i dont know actually

ill allow them to be declared inside functions in future though

so it would be ambiguous then unless i have some kind of static keyword

[astral fractal]

you can do a bit of a meme with constexpr semantics i think

hmm, not for mutable statics

[lethal quest]

ive just realized my parsing logic for vasm doesnt work across the board

due to the fact that there are instructions that take one or two operands

or three even

[lethal quest]

continuing on from #1095033465440841799, im rewriting the regalloc

because its bad

function @main() -> i32 {
1:
        alloca VREG1: i32
        store i32 12 -> i32* VREG1
        load i32 %VREG2, i32* VREG1
        imul %VREG3, %VREG2, i32 1
        store %VREG3 -> i32* VREG1
        load i32 %VREG2, i32* VREG1
        ret %VREG2
}```  so far i have this kind of virtual reg allocation

obviously if that alloca is in a register the loads can be eliminated

[lethal quest]

one thing im not sure about here is

lets say the alloca is put in rax

then vreg2 is rcx, vreg3 is rdx

since vreg2 isnt needed, the loads can be omitted

but then vreg3 should be vreg2

because vreg2 isnt used for anything

i guess i can leave that for now

[lethal quest]

right

now i need to deal with precolored thingies

and function calls destroying everything

the way ive done this wont work too well with that

i kinda need to break the 1:1 translation of vreg to physical reg/stack slot

because rax would be usable before and after a function call, so long as the value isnt live over it

but currently i'd have to make that reg never used

[lethal quest]

hm i found another annoying bug

previously if an alloca was in a register, loads from that alloca would do nothing and instead just use that register directly

but in a special case where that is the last time that alloca was used, the register would then be available

and be reused by another value

i might just make alloca's live for the entire function for the time being

albeit a bit of a hack

[lethal quest]

another issue i've found is if i have spilled a value to the stack, but i need to store something from memory into it

like in theory if i had a loadinst that had been spilled, that was loading a spilled alloca

it would attempt to do mov [rbp-off1], [rbp-off2]

which is obviously not possible

[lethal quest]

i could just keep a register free for this kind of thing

but that seems a bit bad

[lethal quest]

im working on a lower level IR that the main IR compiles to

it'll let me do optimizations much easier i think

LABEL main
LABEL 1
MOVE $69 -> %T1
MOVE %T1 -> %T2
MOVE %T2 -> eax
RET```

like this has a bunch of easy optimizations that can be done

[lethal quest]

currently just porting over all of the existing IR to the new lower level IR instead of generating asm directly

then ill work on some kind of isel layer

[lethal quest]

im not sure how i want what will eventually become lea to work in this IR

[lethal quest]

i would also need to force the thing its taking the address of to actually be in memory

[lethal quest]

ok thats done

next thing is either some optimizations or loading a value from an address stored in like a register

which im not entirely sure how i want to do either

i cant just check if its a register since an alloca could be in a register

[lethal quest]

i guess i can just have a different ir instruction for indirect loads and the main loadinst resolves which one to emit

if its an alloca or not

[lethal quest]

im getting fairly close to where i was before now

with the new lower level IR

[lethal quest]

i think its just mul now left to do

[lethal quest]

ok new backend is all done

[lethal quest]

i managed to break everything by changing the liveness analysis

but i think i may have fixed it for the most part

[astral fractal]

how bad is the new codegen?

did you implement store/load forwarding yet

[lethal quest]

no

im still fixing everything

as in making it work at all

let alone optimizing it

the current issue is when i try to gep from the stack back onto the stack

[astral fractal]

why do people implement gep

it seems rather silly

[lethal quest]

as opposed to what?

[astral fractal]

just inlining pointer math

[lethal quest]

what like

+ for an i32* just does + thing * 4

[astral fractal]

well i mean like

on structs

[lethal quest]

same story

* alignment * membercount or whatever

[astral fractal]

as in to get a struct member from struct

[lethal quest]

along those lines

yea

[astral fractal]

well yeah but why getelementptr

[lethal quest]

idk

llvm does that

[astral fractal]

like in general

you dont have to be a sheep

actually its because strict aliasing i think

[lethal quest]

yes but i didnt think of another solution

so i just went with that

[astral fractal]

pointer cast and add?

[lethal quest]

i guess

so if we have ```c
struct foo {
int a;
char b;
short c
};

foo x;
x.a; // (int*)(&x)+0```

essentially this?

[astral fractal]

yeah pretty much i think

[lethal quest]

i guess that works

[astral fractal]

this may be slightly worse due to strict aliasing, which i think is why llvm doesnt do that

[lethal quest]

but i would still need special logic for pointer adding

[astral fractal]

not really?

[lethal quest]

well like

multiplying by scale

[astral fractal]

just multiply by scale beforehand

[lethal quest]

yeah

[astral fractal]

yourself

[lethal quest]

oh

hm

this would require slightly more work to turn it into like [rax+8] or whatever but i guess thats fine

[astral fractal]

kinda?

i mean yeah it makes instruction selection a bit harder

theres a simple fix

just use arm :^)

[lethal quest]

i would

except no

[astral fractal]

understandable

noone wants to use arm

tbh this isnt even that bad

its pretty simple isel stuff

[lethal quest]

yeah i guess

i am bad though

ok tomorrows job will be to remove gep then

and replace it

[lethal quest]

we have modules

well imports

[lethal quest]

ok now i will attempt to start adding some proper optimizations

[lethal quest]

i added some optimizations

they are not amazing by any means

but its definitely a step in the right direction

[astral fractal]

did you do load-store forwarding yet

[lethal quest]

i tried looking that up

i couldnt really find anything about it

[astral fractal]

hm

i dont know what the real name for it is

the idea is that you combine *X = Y and Z = *X and turn it into Z = Y

and you can completly discard the stores if nobody reads from the alloca

[lethal quest]

ohh

CSE?

[astral fractal]

eh kinda

you should cse too tbh

oh and the reason gep is useful is because of strict aliasing

you know two addresses a1 and a2 do not alias if you obtained them by accessing different fields in one struct

iiuc

[lethal quest]

i do not currently make use of strict aliasing

and im not sure if im going to

[lethal quest]

i will try to get something done today

not sure what

[astral fractal]

alias analysis

[lethal quest]

is that a particularly useful optimization?

[astral fractal]

well its like half of that thing so yes

[lethal quest]

fair

[lethal quest]

i have discovered a bug

if you need to pass a series of parameters to a function

but some of the values to pass are in a different parameter register

they will get trashed

[lethal quest]

ok im gonna move the regalloc layer to the LIR

[lethal quest]

i have discovered a very subtle bug

and im not entirely sure how i want to go about fixing it

if a register is smashed, any active values using that register must have their register switched to a different one

but it just searches for registers from the pool of free registers

however, one of those may be used previously in the value's lifetime

for some other value

which would cause them to use the same register at the same time

which obviously cannot happen

[lethal quest]

https://discord.gg/ahTsCwFZ viper discord for anyone who would like to join

[lethal quest]

fixed a bunch of issues in the compiler

[lethal quest]

we got vscode syntax highlighting

its pretty basic

just the shitty textmate thing

but eventually ill write a language server

[candid hollow]

language servers seem so fun

[lethal quest]

if you want to write one for viper be my guest

[candid hollow]

unfortunately they contain one of the hard problems of computer science (cache invalidation)

[lethal quest]

i think the LSP makes it much easier

not specifically what you just said but the whole thing in general

[lethal quest]

the allocator now runs without segfaulting

it does not work just yet

[lethal quest]

this is getting quite difficult to debug without debug info(or at the very least backtraces) so im going to attempt to implement some portion of DWARF

[lethal quest]

wow it has been a while

currently rewriting as per usual

this time with a focus on things actually working

most importantly the type system

so we have implicit casting and such from the get go

the current state is quite basic, although we do have function pointers and some conditional stuff

func main(argc: i32) -> i8 {
    let f: (i8)* -> i32 = &test;
    let fp: (i8)** -> i32 = &f;
    let g: (i8)* -> i32 = *fp;
    return g(argc);
}``` heres some example code

func thing(test: i32) -> i32 {
    return test;
}

func thing(test: i8) -> i32 {
    return test;
}

func main() -> i32 {
    let test2: i16 = 3;
    let test3: i8 = 12;
    test3 = 2;

    let test: i32 = thing(test2);
    if (false)
        test = test2 + test3;
        
    let ptr: i32* = &test;

    return -*ptr;
}``` and some more

[lethal quest]

gonna work on writing some tests today

a combination of fuzzing, end-to-end and unit testing

[lethal quest]

wrote some simple lexer tests

guess the logical next step is the parser

[lethal quest]

ok parser tests are done

also added extern functions

im debating now how i want to handle character types and strings

obviously i could just have i8 also be a char

and i8* for string

[lethal quest]

considering rewriting the IR as well because it is really bad

[lethal quest]

viper is no longer dead(again)

just finished implementing the module system

export import example.test2;

export func x() -> i32 {
    return y();
}``` some example code, should all be pretty self-explanatory

[empty pendant]

[lethal quest]

some of classes finished tonight

you can define classes and use them as a type, but no member access yet

so they dont do a huge amount

[lethal quest]

today if time permits i will be adding member access to classes

which needs slightly more work than i would have liked due to how variables exist inside the compiler

i think for now structs will only be allocad instead of being ssa

mem2reg probably cant do much with them at the moment either

but thats something i can deal with later

after member access i'll make it so you can actually import/export classes

which should be fairly straightforward

then if i have time, member functions

[lethal quest]

ok member access was actually relatively easy to implement

[lethal quest]

not sure how i want to handle exported classes that have fields of types that are not exported

something like this ```
class foo {
...
}

export class bar {
foo thing;
}```

to someone importing this file, bar is visible but foo wont be

[urban bane]

it's an error with rustc

if a field is marked pub but not the type then it errors

[astral fractal]

really?

no, its a warning (private_interfaces)

[urban bane]

ah

idk i have werror

[lethal quest]

ok we have exporting classes

was relatively painless

not sure what i want to do next

either member functions or some kind of symbol file emission for libs

[lethal quest]

decided i will actually be adding namespaces

[lethal quest]

we have basic namespaces now

they dont function in imported files just yet

but i want to redesign the import system anyway

[lethal quest]

better import system is done

[lethal quest]

we have namespace lookups for types now

so now im just gonna add a shit ton of small simple features like enums, using declarations etc etc

[lethal quest]

member functions are a thing now

they have implicit this parameter

but you can also just type the name of a member

[lethal quest]

we also have access modifiers now

well just one actually

everything defaults to private but you can put public in front

im not sure if i want a c++ style public: or not

[lethal quest]

enums also done now

[lethal quest]

added cast expressions, the syntax is just cast<X>(y)

going to attempt to add templates next

[lethal quest]

we have basic templated functions

[lethal quest]

taking a small break from viper to add a compiler for the IR language so i can work on some more optimizations

[lyric skiff]

me when this project

[lethal quest]

what on earth does this mean

[lyric skiff]

iykyk

[lethal quest]

i guess i dont know

is it some italian thing

[lyric skiff]

no

[lethal quest]

i see

[valid dove]

Viper is the stage name of Houston-based hip hop artist Lee Carter who has gained a cult online following for his unconventional style of rapping and prolific output of records, with the most notable release being his 2008 rap album You'll Cowards Don't Even Smoke Crack.

[lethal quest]

clearly im too young for these kind of references

but thank you

[upbeat latch]

https://www.youtube.com/watch?v=-XazLR5UFWM

[lethal quest]

made a little lexer generator since i basically have the same lexer written 3 times now

now all the projects will use the lexer generator

its not exactly a generator ig, more just configuring the lexer

but i'll extend it in future

[lethal quest]

doing a bit more vlex(lexer generator) stuff today

a friend of mine is using it for one of their projects and theres a few issues to iron out so ill be doing that first

then adding some qol features

after that ill go back to vipirc and get that going

then probably spend a bit more time doing optimizations

[lethal quest]

finally fixed all the vlex errors

[placid dust]

how much progress has been made since 2023?

[lethal quest]

probably quite a lot

why do you ask?

ive rewritten the compiler probably multiple times since 2023

[placid dust]

just curious, when I click the channel that's the first thing that shows

[lethal quest]

ah i see

yeah its changed quite a bit

not all exactly "progress" though

[lethal quest]

been working on a slightly different project recently, its basically just viper but simplified and with a slightly less garbage compiler

func main() -> i32 {
    let x: str = "hello";
    let array: char[5];
    let y: str = array[0:5];
    bmemcpy(y, x);
    test(x);
    test(y);
    return 0;
}``` some example code

has its own build system and library format