Programmatic Setting of Syntax / Scope using extension API (NOT the XML)

Hi Everybody! :wave:

I hope you are all very well, so, it has been a while that I have been thinking about implementing a feature in my current extension (AlpineJS) and of course prob the other extension that I will be working on in the future. Now I seem to get no where by just reading the API and it looks like it might be because of the API shortcoming or maybe just the way parser engine work? However I think it is doable if Panic sort of create an API for it? Ok so…

The Problem/Feature Request
A lot of JS libraries frameworks are used in a lot of languages, such as Vue, Svelte, AlpinJS etc. Something very common, mixing JS with HTML, PHP, Ruby etc. This sort of relationship has existed for decades let’s say like in PHP we have this exact relationship where the syntax highlighting changes when Nova sees the php tags.

HTML parser
<?php
 "changing to PHP Parser, here is the LSP syntax highlighting etc"
?>
HTML parser

The way historically editors dealt with these situations have been simply creating a language grammar for example in Nova using XML and say "ok whenever you are in .php file and see <?php , read it as php, then read the rest as html. This worked because you could only see php in .php file or ruby in the .erb files (although this is not true anymore! we have Laravel and so forth using different template tags).

With Javascript though, this is a whole new beast. Millions of libraries and frameworks and in order to get it to work, this means extending all these language grammars by manipulating their XML! which is ok but there is no API for us to do so on a case by case basis (we have got the completion API to extend the XML hence me thinking this is a solvable problem!:eyes:). The only way to achieve this is to either finding the original language grammar and hack the support directly (bloating the XML indiscriminately, whether the users want it or not) which is absolutely impossible to maintain and so forth :exploding_head: OR writing a brand new Language grammar, which is impossible to do so considering the number of JS libraries/framework and how each one uses different templating engine, lots of copy and paste work boiler plate. To make the matter worse sometimes they are all even mixed up in the same file! :exploding_head::exploding_head:. This approach however has been ok for stuff like laravel Blade Template extension. But the author could have had an easier time, and used an API, if available at the time of writing and add it on top of the php/html language grammer file rather than writing the whole grammar… But this is not possible at the moment, not only this would be easier to maintain as an extension, it would also decouple the syntaxes both maturing separately without breaking each other. Having said that, at least in Nova, when this extension is on, it works very well! In VS Code this does not even work, the editor does not change syntax inside the template tags and does not give php autocompletion inside {{I am php}}. In Nova however:

HTML syntax 
{{Change to PHP syntax  }}
HTML Syntax

For this reason all javascript libraries or frameworks extensions, lack syntax support inside their template tags in all editors. To give you an example for my case, I would like to be able to inform Nova to switch to Javascript under these certain circumstances in all languages

<div x-data="switch to javascript parser! I am not a TEXT or HTML I am javascript!" ></div>

Sorry for the long post, I didn’t know a better way to get my point across without laying all this out! :sweat_smile: I know it might sound like a trivial problem, but trust me it would have a huge impact on user experience, as I said the fact that VScode did not pick up php inside the {{}} tags it really bugged me as a user and I was so pleased that this was not the case in Nova! I also think it is something that a lot of extension developers would find handy!

Please do let me know if there are any work around of course, even if it is like a hack haha 🫡
Thanks
Emran