Nova 10+ Completions from built-in HTML language don’t work when used as subsyntax inside Regex-based language (Twig)

Bug Reports
1

Hello,
I’m the developer of the Twig language plugin for Nova. Twig is a PHP-based templating language used by many projects, e.g. Drupal or Craft CMS.

My language plugin has two syntax definitions: Twig-HTML (Twig code alongside regular HTML tags) and Twig (Twig code inside bracktes). It’s pretty similar to the original PHP-syntax (before the Tree-sitter update).

The Twig-HTML syntax definition includes the original HTML syntax definition and expands it with a few <template-scopes>.

<scopes spell-check="true" lookup="dictionary">		
    <include syntax="html" />
</scopes>

See: https://github.com/tpmatthes/Twig.novaextension/blob/main/Completions/Twig-HTML.xml

This make it possible to use HTML tags and Twig syntax alongside each other in .twig files.

In Nova 9.x, everything works just fine: HTML tags inside .twig files will be highlighted correctly, tags will be closed automatically if you type </, and you can rely on the full set of built-in completions for HTML tags and attributes.

In Nova 10+, the syntax highlighting for HTML tags works as expected. However, tags won’t be automatically closed anymore and only a small fraction of built-in completions for HTML tags and attributes will be triggered.

In both versions, syntax highligting and completions for Twig work as expected, so it seems like this problem is caused by the new version of the built-in HTML syntax definition (using Tree-sitter).

My Twig language plugin is still regex-based, but I’d like to keep using the built-in HTML syntax definition which is now based on Tree-sitter. Is this possible?

As I can see, the new version of the built-in HTML syntax definition includes a few regex rules for backwards compatibility. Maybe that’s the reason why syntax highlighting for HTML tags still works?

I’d be happy to learn more about how Nova handles this case internally, so I can address this issue accordingly. Thanks!

2

Hello,

Thank you for the info.

We’ve received other reports about autoclosing of tags in regex-based grammars not working as expected, and are looking into it.

As you mention, the HTML regex rules were left in for backward compatibility to allow developers time to move their grammars to tree-sitter where appropriate. This is indeed why HTML tag highlighting would still be working in your extension. At a future point, we plan to remove that backwards compatibility in a major Nova version release (as of yet which is undecided, and we will make an announcement well beforehand when we solidify those plans).

If you are working to embed HTML within another language, we recommend migrating to a Tree-sitter grammar when possible, as development work on languages in Nova will only be targeting it going forward, and no additional work will be done on the regex engine except bug fixes.

Unfortunately, Completions for HTML in Nova 10 and later can only target the tree-sitter grammar, as there is not a facility built-in to the engine to allow it to work with the backwards-compatibility regex rules because, technically they are no longer part of a top-level syntax (they’re just left behind to be importable by other grammars). There isn’t a good solution to this as I see at the moment, but there might be a couple of temporarily solutions you could do in the interim, such as taking the HTML Completions file from Nova 9.6 and copying it into your extension, changing the syntax of each provider to your language, and then targeting it at only imported HTML elements (which it might just be set up to do automatically).

I’m sorry there isn’t an easy cut-and-dry solution to this. The way that grammars were importable in the regex engine caused a lot of constraints around how we could extend them going forward (and backed us into some corners as to not break other third-party grammars’ syntax highlighting), which we think will be easier in the future with Tree-sitter due to how it more strictly boxes each embedded language.

3

Hi @logan,
thanks for your fast and detailed response. This is a great reference for future compatibility issues.

I just shipped a new version of my plugin with a copy of the built-in HTML completions from Nova 9.6 as suggested. This will do the job until I eventually migrate to Tree-sitter.

Thanks again!

1 Like