For me, the main pain point about Nova’s custom engine is that it has bad debugging support. To this day I haven’t found a solution to my problem writing a Zig grammar. A proper grammar engine should recognize the possibility of infinite recursion in a grammar and make this an error. Nova’s apparently doesn’t. Also because it’s a custom, proprietary engine, I have no possibility of inspecting what the engine does when this happens. This makes me unable to finish that plugin without direct support from Panic.
Here’s my first impression of Tree Sitter:
- Tree Sitter says that it is robust, i.e. produces useful results even in case of errors. This is a huge improvement to the current engine, which requires grammar authors to manually add things like
<cut-off>to handle errors graciously. - Regarding symbolication which Tree Sitter doesn’t support, I recommend having a look at what JetBrains does in their IDEs: The have a Program Structure Interface on top of their parsers which provides such features. This also serves as example that separating parsing from analysis works well in an IDE.
- While the usage of JS for grammars (really?) is a questionable decision, the fact that these are compiled and not interpreted directly makes it quite likely that TS will be faster than the current regex-based engine.
- Tree Sitter grammars are not overly complicated. Their complexity is necessary for doing a good job. Sure we all would like to just throw some keywords into a magic tool that produces a readily usable syntax that does everything we want, but that’s impossible. Decades of research into parsers and grammars have provided knowledge how to parse input quickly and handle error cases well, and the sad fact is that for the most part, this knowledge is ignored by overly simple syntax highlighting engines in editors. Those will get you quick adoption because writing plugins for additional languages is simple, but it will also impose a limit on features that your editor can support.
So yeah, I am wholly supporting the adoption of Tree Sitter.