How to preprocess go code with `toolexec`?

[finite hamlet]

Can anyone please share how to modify go code by calling go build -a -toolexec <mytool>

What's the goal?

Log what function was called.
So that moontrace will add a print at the beginning of each function body, for example:
fmt.Println(fmt.Sprintf("Calling %s func", funcName))

Motivation?

At the moment just for the sake of experimentation and trying to reduce some of the routine when developing and debugging personal projects.

What I tried so far

Prerequisite: unfortunately, I have found mercilessly little information regarding the use of the toolexec flag, so all attempts made are made more on the basis of indirect inference.

My final result so far: https://go.dev/play/p/Jk_vWDY7fO4

Everything looks as if the approach itself is incorrect. My personal feeling is that compilation should be handled by tool, which is now proxied through toolexec, rather than an explicit call to go build inside moontrace code, but i might be wrong.

Of interest: a video was recently released from a developer at Datadog where they used toolexec to autoinstrument code. And at the stage of explaining how it all works, they showed the example of a commandline tool thats run for toolexec tool:
https://youtu.be/5l-W7vPSbuc?si=tDocHZREZPgIUpWb&t=1924
(timecode included)

But when I output the result of os.Args inside moontrace code, all I see is the following:

[/Users/mark/Projects/moontrace/moontrace /Users/mark/sdk/go1.22.0/pkg/tool/darwin_arm64/compile -V=full]

So even if I want to use go build call, at the moontrace stage of execution, I lose all the options that were passed to build command.

I've also found projects that actively use toolexec, like xgo or burrowers/garble, but honestly I don't have enough experience to get the basics out of them.

I would like to know only the most basic things, for modifying code at compile time with toolexec, without modifying the source code.

Thank you in advance!

[finite hamlet]

Seems like I've figured out how to solve my problem. Even learned how to change imports in importcfg during build. I'll try to describe an approximate solution here tomorrow.

[mild berry]

I want to create go superset so I'm very interested in reading this.

👏

[finite hamlet]

Okay, here's what I ended up with:
https://gist.github.com/pijng/835de8d34799e911c6a38a49f2ebff7c

I have added comments there to all the key points, as well as tried to explicitly describe the approach itself and some of the nuances.

Let's say we have this code:

package main

import "fmt"

func main() {
    fmt.Println("Original print")
}

If you compile it and run it, it will output the expected "Original print":

$ go build main.go
$ ./main 
Original print

But if i compile the main.go by specifying the injector as an argument to toolexec:

$ go build -a -toolexec=$PWD/injector main.go
$ ./main
Calling [main] func
Original print

This will print out "Calling [main] func" – the very code we added before compilation. In this case, there are no changes in the original file.

Now, i'm pretty sure there are hella lot of pitfalls. But at least I figured out how to make the simplest preprocessor possible

Also, I really recommend to watch Jon Bodner talks (link in original post).
And also a video by Daniel Martí (member of go team) about obfuscating the go code at the compile time:
https://www.youtube.com/watch?v=-uDnciABNOQ

Not necessarily so important in our case, but as additional information very useful.

[finite hamlet]

Also, I’m not really sure you can create a go superset with this approach as @mild berry stated, since it’s required to write a valid go code, otherwise a compilation will obviously fail.

I can only think of some go dialect written in the comments (duh), that will be transformed to plain go code with toolexec, but I'm just fantasizing at this point 😅

How to preprocess go code with toolexec?

[mild berry]

I was thinking of parsing superset ast and then transforming it to actual go's ast and then forwarding it to go's toolchain

That's hard to make it maintainable tho, I checked gopls source and other go tooling and it would just require enormous effort to add support for new ast

Not speaking of just compilling, rather also autocompletion, code highlighting, all that good stuff

#1217138616095080498 message

mainly implement unofficially some of language changes/go2 that could benefit people

[finite hamlet]

Ah, I see. Yeah, I have no idea how to achieve that without an enormous amount of work tho 😦

[mild berry]

Yeah, go's toolchain is very complex under the hood, soo I dont know

I wish there was something like extensions api to go's toolchain

But they stated explicitly in docs that they will not add it

[finite hamlet]

I can only think of introducing some custom directives that the user can specify in the comments above the functions or other declarations, so that toolexec can parse it and… do something with it, I guess? But still looks like a pretty limited option

[finite hamlet]

Yup, fr. I’m surprised we can even hook onto compile steps, but I think it’s mostly used by go team themselves.

[mild berry]

Yeaaaaa, that's not fun at all

I was seriously considering making this, but when I saw go's toolchain code

I was like nah that's a lot of stuff and I already have work to do irl

[finite hamlet]

The toolchain is something else for sure, my first thoughts were "nah, i give up" with every new investigation of it 😬
I'll stick to the trivial stuff for now.

[solid patrol]

cool thread, thanks for sharing your solution

[finite hamlet]

Soo... I decided to go further and turn the whole thing into a wrapper in the form of a separate library:
https://github.com/pijng/moonject

The library has just one public function Process, to which you need to pass a structure that satisfies the Modifier interface with a single Modify method - this is where you can make all the necessary changes to the file.

Example implementation:
https://github.com/pijng/moonjectlog

Honestly, learning how to properly resolve imports was a nightmare, especially when it came to imports with dots in the path - like gopkg.in/yaml.v3 - or imports that have a different import path than the namespace - like github.com/pressly/goose/v3, whose namespace is goose.

In these cases, the resolver did not understand what was going on, and I couldn't find any normal examples on the Internet. I finally figured out how to solve this issue after dozens of attempts and by reading tests in dave/dst.

And also the preprocessor now has a mandatory single argument - the absolute path to the project. To distinguish project files from 3rd party package files (I couldn't find another way to do it.)