tree-sitter-mojo

[near kernel]

I had half an hour today, and Mojo is close enough to Python, so I forked the Python Tree Sitter grammar and got it working for mojo.

https://github.com/lsh/tree-sitter-mojo

[spice warren]

Nice, I will try to use your variant with Helix editor. You can also look at https://github.com/garam-kim1/tree-sitter-mojo The idea is to automatically update using GitHub Actions.

[near kernel]

Oh if I had seen that maybe I would have just used that lol

[spice warren]

I haven't looked into the details, but perhaps your mojo part of the grammar is more relevant.

But without automation it will be difficult to maintain such a fork. https://github.com/garam-kim1/tree-sitter-mojo/blob/master/.github/workflows/tree_sitter_maker.yml

[manic sapphire]

Epic thanks both helix works now!

[near kernel]

Whoops I forgot to add alias, should do that now

[spice warren]

It would be nice to push Helix PR forward at some point. https://github.com/helix-editor/helix/pull/8657

[manic sapphire]

If you raise a PR from your repo @spice warren it might avoid that test failure?

[spice warren]

It's not mine, but it could definitely be used in PR and then maybe the build and tests will be ok.

It seems better to create a new PR. The current one is already the second 🙂

[hidden bay]

Congrats @spice warren, you just advanced to level 1!

[near kernel]

FWIW this is my helix config and it works for me

[language-server.mojo-lsp]
command = "mojo-lsp-server"
config = { provideFormatter = true }

[[grammar]]
name = "mojo"
source = { git = "https://github.com/lsh/tree-sitter-mojo", rev = "e645ad097257a29bed5ee6213fef47e26b280f63" }

[[language]]
name = "mojo"
grammar = "mojo"
scope = "source.mojo"
roots = ["__init__.mojo"]
file-types = ["mojo", "🔥"]
indent = { tab-width = 4, unit = " " }
auto-format = true
language-servers = ["mojo-lsp"]
formatter = { command = "mojo", args = ["format", "-q", "-"]}

[random spoke]

This is what I get while trying to compile the parser in Neovim

Edit: I just failed to include scanner.c, it is compiled now.

[manic sapphire]

Only thing stopping me from using Helix as my main editor is it still doesn't have persistent undo history. I tried building this but doesn't work on mac: https://github.com/helix-editor/helix/pull/9154

[near kernel]

Just pushed better support for parameter functions and decorators. Also wrote a Zed extension

[manic sapphire]

Awesome cheers @near kernel!

[near kernel]

I’ll try to publish the zed extension later today

[near kernel]

Ooh wait, is there a tree sitter grammar for MLIR? We could have it so that things like:

__mlir_op.`1: i1`

uses MLIR syntax highlighting injection within the backticks

[misty lake]

can you share the steps? thx

[random spoke]

https://github.com/nvim-treesitter/nvim-treesitter?tab=readme-ov-file#adding-parsers

[spice warren]

My PR has been merged and Helix now supports Mojo. https://github.com/helix-editor/helix/pull/10743
For now, users have to build from master, but future releases should have out of the box support.

[near kernel]

We’ll probably have to patch in some of the newer syntax/tree sitter stuff. Lately I’ve been playing with special highlighting for MLIR stuff

[spice warren]

Yea, it will be cool to support MLIR. I initially added this:

((identifier) @type.builtin
  (#match?
    @type.builtin
    "^(__mlir_type)$"))

But this is a workaround, it's better to have built-in support in tree-sitter. Traits are also need special handling, I will try to add them later.

[near kernel]

This is my WIP

I’m not sure how I want to highlight dialect names though. Or builtins

I haven’t pushed it yet

[spice warren]

Good progress. I thought that from a Helix highlighting perspective, __mlir_attr and __mlir_type should be @type.builtin, and __mlir_op - @function.builtin or @function.special

[near kernel]

Yep, that’s what I’m doing (except I’m using Zed in the above which might have slightly different tree sitter setup)

I’m probably gonna start on __mlir_op sometimes this weekend or next week. It’s tricky to get right since it requires custom grammar parsing and not just a highlight token. At least that’s what it requires if we want more granular control over things like bracket vs plain backtick

[spice warren]

Yes, MLIR parsing is not an easy part. I will try to add traits.

[near kernel]

It should just be the same as how I added struct. Pr for that is welcome. I also should add tests

[spice warren]

So expect PR from me. It looks like your tree-sitter version will be used in Helix, Vim and Zed. VS Code might someday switch to tree-sitters - https://github.com/microsoft/vscode/issues/50140. This way, the user can have a consistent experience with the major popular editors.

[cinder dawn]

@near kernel can you link me to your zed extension? I was working on adding mojo support to zed too and want to exchange notes

[near kernel]

Ah I haven't put it up yet since I'm still tweaking a lot, but I can publish it with a YMMV warning I guess

[cinder dawn]

no rush whenever you're ready

maybe you've just saved me some work XD

[near kernel]

It's mostly all working, aside from a few corner cases in the tree sitter grammar

[random spoke]

Neovim 0.10 just dropped with support recognising Mojo files. This is how Mojo is highlighted

[thorn epoch]

I'm gonna make the leap from VSCode to neovim just for this. I feel like typing, "May God have mercy on you" or something. I have no idea what I'm signing up for though I'll eventually have to go linux (because reasons), and this just feels inevitable.

[thorn epoch]

Yep, stood on that landmine already. Probably about my sixth time trying (n)vim and it was a much smoother experience this time. From what I can gather, the program parsers are .dll in windows and well, there's no mojo yet. Sigh...

[random spoke]

Hoping we get Mojo in the remaining major platforms soon

[woeful vine]

Having an OSS LSP for Mojo would be nice

[thorn epoch]

I feel dirty (mostly joking), I ended up installing nvim on WSL then open another WSL window to build and I actually (serious surprise) that I was able to write with nvim in one (simple open a file) and run in another. Will continue learning nvim without code completions, hell, will continue to learn to code and learn mojo in the meantime. 💪

[viscid crest]

Hi Dmitry. I followed your setup instructions and this is the output of hx --health mojo. But autocomplete doesn't work for mojo files and all the diagnostics go away once I enter the insert mode. The only way to bring them back is to re-open the file. Interestingly, I changed the comment token in languages.toml and it takes effect, but can't seem to figure out the autocomplete and diagnostics. Appreciate any feedback.

[spice warren]

Hi. These setup instructions are for an older version of tree-sitter. The latest version of helix has out of the box Mojo support https://github.com/helix-editor/helix/blob/master/CHANGELOG.md?plain=1#L134
Can you try it?

[viscid crest]

This is the output of Helix log when I try with/without language.toml:

2024-07-28T10:33:34.919 helix_lsp::transport [ERROR] mojo-lsp err <- "I[10:33:34.918] --> initialize(0)\n"
2024-07-28T10:33:34.919 helix_lsp::transport [ERROR] mojo-lsp err <- "I[10:33:34.919] --> reply:initialize(0)\n"
2024-07-28T10:33:34.919 helix_lsp::transport [ERROR] mojo-lsp err <- "D[10:33:34.919] >>> {\"id\":0,\"jsonrpc\":\"2.0\",\"result\":{\"capabilities\":{\"codeActionProvider\":{\"codeActionKinds\":[\"quickfix\",\"refactor\",\"info\"]},\"completionProvider\":{\"allCommitCharacters\":[\"\\t\",\".\"],\"resolveProvider\":false,\"triggerCharacters\":[\".\"]},\"definitionProvider\":true,\"documentSymbolProvider\":false,\"foldingRangeProvider\":true,\"hoverProvider\":true,\"inlayHintProvider\":true,\"notebookDocumentSync\":{\"notebookSelector\":[{\"cells\":[{\"language\":\"mojo\"}],\"notebook\":{\"notebookType\":\"jupyter-notebook\",\"scheme\":\"file\"}}]},\"referencesProvider\":true,\"renameProvider\":true,\"semanticTokensProvider\":{\"full\":{\"delta\":true},\"legend\":{\"tokenModifiers\":[],\"tokenTypes\":[\"variable\",\"specialVariable\",\"parameter\",\"function\",\"method\",\"property\",\"class\",\"interface\",\"type\",\"namespace\"]},\"range\":false},\"signatureHelpProvider\":{\"triggerCharacters\":[\"(\",\"[\",\",\"]},\"textDocumentSync\":{\"change\":2,\"openClose\":true,\"save\":true}},\"serverInfo\":{\"name\":\"mojo-lsp-server\",\"version\":\"0.0.1\"}}}\n"

I used https://github.com/helix-editor/helix/blob/master/languages.toml#L377C1-L392C1

I also did the grammar fetch/build for mojo.

[spice warren]

I'm trying these binaries https://github.com/helix-editor/helix/releases/tag/24.07

And on Linux everything works fine. My log:

2024-07-28T18:08:50.071 helix_lsp::transport [ERROR] mojo-lsp err <- "I[18:08:50.071] --> reply:initialize(0)\n"
2024-07-28T18:08:50.071 helix_lsp::transport [ERROR] mojo-lsp err <- "D[18:08:50.071] >>> {\"id\":0,\"jsonrpc\":\"2.0\",\"result\":{\"capabilities\":{\"codeActionProvider\":{\"codeActionKinds\":[\"quickfix\",\"refactor\",\"info\"]},\"completionProvider\":{\"allCommitCharacters\":[\"\\t\"],\"resolveProvider\":false,\"triggerCharacters\":[\".\"]},\"definitionProvider\":true,\"documentSymbolProvider\":false,\"foldingRangeProvider\":true,\"hoverProvider\":true,\"inlayHintProvider\":true,\"notebookDocumentSync\":{\"notebookSelector\":[{\"cells\":[{\"language\":\"mojo\"}],\"notebook\":{\"notebookType\":\"jupyter-notebook\",\"scheme\":\"file\"}}]},\"referencesProvider\":true,\"renameProvider\":true,\"semanticTokensProvider\":{\"full\":{\"delta\":true},\"legend\":{\"tokenModifiers\":[],\"tokenTypes\":[\"variable\",\"specialVariable\",\"parameter\",\"function\",\"method\",\"property\",\"class\",\"interface\",\"type\",\"namespace\"]},\"range\":false},\"signatureHelpProvider\":{\"triggerCharacters\":[\"(\",\"[\",\",\"]},\"textDocumentSync\":{\"change\":2,\"openClose\":true,\"save\":true}},\"serverInfo\":{\"name\":\"mojo-lsp-server\",\"version\":\"0.0.1\"}}}\n"```

There was a bug with LSP in the old Mojo nightlies, but it has been fixed. https://github.com/modularml/mojo/issues/2835
I use most recent Mojo nightly build.

[viscid crest]

Most recent nightly build fixed the problem. Thanks much.

[hidden bay]

Congrats @viscid crest, you just advanced to level 1!

[woeful vine]

Will we ever get any pypi releases of this?

[near kernel]

That’s an interesting question. Maybe when magic gets a little better I’ll write a Mojo wrapper around the C and publish

[woeful vine]

For the python bindings?

[near kernel]

Ah I hadn't considered publishing the python bindings. Is there a specific use case you have in mind that the bindings would be able to accomplish?

[woeful vine]

Allowing me to use it with the python tree sitter library?