Provide a standard and extensible way to discover external tools

Extensions often want or need to use external tools. This includes tools that can be explicitly invoked by the user, like a compiler, but also tools that are implicitly used by an extension, such as a language server or a code formatter. Unless I’m overlooking something, Nova’s extent of support for discovering such tools is currently restricted to querying the login shell for the $PATH variable.

Now some build tools and package managers allow you to specify development dependencies per project. Sometimes these allow for installing dependencies locally for the project, for example Ruby’s bundle can install dependencies locally so that you need to call bundle exec <tool> afterwards to execute them. There are also language-independent tools like Nix Flakes, which can provide all required tools inside a nix develop shell.

Obviously, when this mechanism is not tightly tied to a specific language or environment that will be supported by a single extension, it would be great if an extension could inject some kind of discovery mechanism so that another extension can transparently query for a tool. For example, assume I want to write an extension for the Jekyll static site generator that can run jekyll serve so that I can preview my website. jekyll may be available from the PATH (e.g. when I install it via Homebrew), it may instead be available via bundle exec (when I use a gemfile for my project), or it may be available in nix develop (when I use a flake.nix to specify dependencies of my project).

I propose an API where an extension can query for an external tool (e.g. by giving "jekyll" as parameter) and which returns an absolute path and a set of environment variables. This can then be used to start a Process. Other extensions can register a tool provider that implements this query, e.g. by calling bundle exec which <tool>. This way, extensions can transparently provide tool resolution for other extensions. Nova would provide a basic resolver that simply searches the PATH for the queried tool.

4 Likes

Hm. I’m sorry, I’m a bit confused with your proposed feature request, perhaps because since I’m an amateur programmer, some of the things aren’t clear to me (hopefully, however, the Panic developers understood your issue!).

As far as I know, there is no way of ‘discovering tools’ in macOS, so to speak, since, AFAIK, there is no central registry for ‘tools’. There might be a central registry for applications — i.e. anything that is installed under /Applications or ~/Applications or, well, wherever Apple has designated a ‘standard’ folder for applications; there are a few deeply hidden inside /System/Applications/ etc… — but I’m not aware how that mechanism works.

But ‘tools’… well, command-line tools can be installed anywhere; macOS doesn’t place any ‘restrictions’ on where these are. Homebrew puts them under /usr/local/Cellar/ (usually with symbolic links to /usr/local/bin). I don’t remember where MacPorts put them, but it had its own structure as well. Apple’s command-line developer tools seem to go into /usr/bin, but macOS keeps track of where these are, in the case the user wants to remove them all. I have lots of tools developed with Go, which usually are placed (by convention) under ~/go/bin. Node’s own tools go into ~/node_modules; I forget where Python-based tools go, but usually they go ‘anywhere the user wants’.

And the list goes on and on…

Since there isn’t a common repository of ‘tools’, I don’t understand how Panic could develop an universal mechanism, built into Nova, that ‘discovers’ where such tools might be.

The best that could be accomplished, IMHO, would be an extension or plugin (I believe it would not require being built-in into Nova) that has the ability to search ‘popular’ repositories, such as those provided by MacPorts and Homebrew (and possibly Nix or even Snapcraft — I don’t know if those work under macOS), or maybe even those provided by Node via npm, Python, Perl (CPAN), LuaRocks, Go… — and possibly install whatever tools are required by an extension, on demand.

In turn, plugin/extension/theme developers would require to use an API to automatically register with the above-mentioned extension, ask it if some tool is found on one of the ‘registered’ repositories, and, if the user agrees, install it (if not already installed).

Theoretically, this could be provided by an external extension; all you need is a willing volunteer to write it :slight_smile: And then you need to encourage other extension developers to use the exposed API — which will require some patience and persistence.

Is that what you’re proposing? If so, it seems that it’s something that could easily be done without hacking at Nova’s code.

But I might not be fully understanding what you actually have in mind, because, as far as I can see it, this is something costly to maintain — you would probably need a meta-database of all possible repositories (which are, in turn, written using all possible combinations of programming languages and have all sorts of backends), which needs to be regularly updated to be in sync with each and every one of the repositories. Just handling the dozen or so more popular ones would be enough of a nightmare!

Alternatively, of course, this extension could not have its own database, but rather query each repository independently — each of which of course requires handling such queries in completely different and unrelated ways — and sort of provide a ‘universal’ view to be presented inside Nova: there would be something like a ‘search tool’ where one would be able to make a keyword-based query, and the extension would show, among all repositories it knows about, where a matching tool might be found, and suggest to install it.

Now here is my point: assuming, indeed, that this is what you have in mind, I wonder what the cost/benefit would be for doing something so complex. After all, if you take a look at the number of people who have written some sort of extension/theme/plugin/etc. for Nova in the past two years, that number is relatively small. You can also note how many regular participants there are inside this Developer Forum. While it’s conceivable that such a number might grow in the near future — especially as new Nova users figure out that ‘there is no tool that provides X so perhaps I should just write it and release to the community’. But in any case I wouldn’t bet that we’re talking about millions of users, which is what all these independently maintained repositories have. Some might even have hundreds of millions — taking into account that CPAN, Pypi, LuaRocks, Go, Rust, Gems etc. etc. etc. are used by developers in all platforms, not just macOS, and they all share the same package repositories. As such, the return on investment on making more sophisticated repositories, or meta-repositories on top of existing ones (such as Canonical’s apt that works on top of the established dpkg manager for Debian; over which, in turn, GUIs have been written to handle package management visually…), is worth the effort, since any kind of ‘improvement’ will directly benefit millions or hundreds of millions of users.

But for Nova, how many people would directly benefit from such a meta-database for all repositories, integrated into Nova?

Also consider that each extension, by itself, might not require a lot of external tools, and those that are used will come from a very restricted and limited pool of such tools. In other words, if someone is doing a Perl extension for Nova, they will benefit from a direct connection to CPAN (for example) to retrieve packages on demand; similarly, the C# extension could benefit from a direct package retrieval facility from NuGet or other popular .NET repositories. However, it’s quite unlikely that the Perl extension will require any access to the NuGet repository; it’s not reasonable to assume, therefore, that whoever developed the Perl extension will benefit from whatever useful tools might exist on NuGet, and vice-versa. It is conceivable that some tools might, indeed, be useful for both — such as, say, Prettier. But on the flip side of the coin, it rather makes more sense to do an extension specifically for Prettier, which handles Prettier’s own extensions and themes and whatnots, instead of getting all programming language extensions use their own tweaked version of Prettier, coming from a centralised meta-repository integrated into Nova.

So, what I’m attempting to say is that it makes a bit more sense (in my own perspective, mind you!) to encapsulate whatever external tools might be required from within each extension/theme that will actually need such tools. And it will be up to the extension’s provider to make sure that such tools will be available, optionally asking the Nova user to put the path to a locally-installed tool on the Preferences window/tab.

In fact, one might even argue that such an encapsulation model — where external tools are only ‘known’ by the extensions that actually use them — is a good way to make sure that an old extension, which hasn’t been updated in a while, can nevertheless benefit from a newer version of its external tools, if the end-user has kept them up to date; if for some reason that newer version breaks compatibility with the extension, well, then the extension developer will most likely have provided its own, slightly outdated version of the required tool, but which was guaranteed to work at the time the extension was last updated.

This scenario obviously also works the other way round: I have dreaded to tinker around one of my language extensions that works with an external LSP. Nova’s support and integration with LSP has improved so dramatically that even without doing anything to the extension itself, both the improvements on Nova as well as on the externally-called LSP made ‘everything work as it should’ — without even requiring me to write a single line of code :slight_smile: I totally admit that this is a lazy excuse, of course, but it nevertheless corresponds to the truth…

Contrast that with a central meta-repository which would handle all external tools on behalf of all the extensions. That way, I could never be 100% sure if the tests I made, at some point in time, with one specific version of a tool that was found on the meta-repository at the time of last updating the extension, will continue to work if the meta-repository, suddenly and unexpectedly, reinstalls a new version of the external tool before I have any chance to evaluate it. Since such changes are made at the macOS level — even if ‘moderated’ via Nova — it might not possible to reverse such a change (or even to be aware that any change occurred!). Or, alternatively, one might have a complex mechanism where the meta-repository’s API inside Nova also exposes methods to query for version numbers, check if the tool’s version is ‘compatible’ with what the extension expects, and optionally prevents the system to update such a tool if there is the risk of a breaking change. This is certainly possible to achieve, and, in some scenarios, it might even be a good approach; but it just makes the meta-repository, and the Nova extension that hypothetically would need to manage it, even more complex.

Consider also the case of having multiple repositories with not only the same external tool, but, more worrying, with long lists of dependencies that are mutually conflicting. Homebrew, for instance, takes some care not to overwrite any so-called ‘system’ tools, i.e. those that are already installed and being managed by macOS itself (via the Command-Line Developer Tool). But AFAIK this mechanism is not in place for other repositories. For instance, in some cases, I can use Homebrew to install a specific Perl package directly from Homebrew’s repository, because someone was kind enough to properly add such functionality, collaboratively sharing the management with CPAN itself. But this is by no means an universal requirement: in many cases, package managers, running as fully independent ‘magic black boxes’, are not aware of what other package managers in the same system are doing. Again, this might not be an issue with CPAN and NuGet installed on the same device — because most likely these repositories do not host the same packages) — but it will certainly be an issue with running Homebrew and MacPorts at the same time, having access to the same files on disk without being aware of the other’s existence.

I have no doubt that one advantage of having a meta-repository would be to make it so much easier to figure out conflicting installations from different package managers. But, again, that comes at the tremendous costs of maintaining the meta-repository in sync with every individual repository…

Then again, I might have completely misunderstood your feature request…

This is the point I also tried to make.

This is not what I suggest. macOS does indeed have a common way of discovering tools, coming from its Unix heritage: The PATH variable. Regardless of what tools you might use to install your software, they either append to the PATH explicitly or tell you to do so in your shell configuration. And Nova takes that into account by querying the login shell for your PATH.

What remains, and what my request is about, is software that is installed on a per-project basis. An increasing number of mechanisms are used today to specify a machine-processable description of dependencies of your project. Sometimes, these will be installed permanently on your system/user (for example, go.mod from Go will install dependencies in your GOPATH when you first compile a project), but other mechanisms do indeed install dependencies per-project.

I am suggesting to enable Nova to discover such dependencies. There are two actors in the discovery scenario: The actor that requires the tool and the actor that knows where to look for the tool. These can be different extensions. And there is currently no way for one extension to say „hey I know an additional place to look for tools“, so it is unable to provide knowledge to the other extension.

You might be overthinking this a bit. I certainly don’t want Nova to build up an extensive database about repositories and dependency sources. Quite the opposite: I want it to have a layer as thin as possible to interact with tools that already have that knowledge. MacPorts, Homebrew etc are tools that install packages system-wide, so these are already covered by the Unix PATH discovery method. Additional support specifically for those package managers is not subject of my request.

The minimal implementation of my feature request would be an API function that takes a list of strings which Nova would append to the PATH variable in this workspace. i.e. now any extension in the same workspace has access to additional paths where tools might be found, and discovery would be as simple as calling /usr/bin/env or /usr/bin/which via a Process. This is where I’m coming from, of course a sane API would be able to do some housekeeping to avoid duplicates etc.

The question here is whether a dependency is more likely to break its interface to the editor or its interface to the input. For example, a TreeSitter grammar has a complex querying mechanism, while the grammar it implements is relatively stable. Therefore, it makes sense to bundle it with an extension. On the other hand, a typical command-line tool like jekyll has a rather stable command line interface but changes the way it processes input from time to time to implement new features. Therefore, it should rather not be bundled with an extension. Your example of a language server is middle ground: It does have a complex interface, but has been engineered to alleviate this problem by using JSON-RPC and a discovery mechanism for capabilities of both the server and the client.

This is exactly what project-specific dependencies try to solve, and why I am suggesting this feature. If you have two projects that use the same tool but require different versions of it (again, think of jekyll), you would need to uninstall && reinstall it every time you switch between the projects. If you use a Gemfile with installation into the project directory, you avoid this at the cost of not having the tool available in your login-shell PATH.

A typical project only has one mechanism in place to define such dependencies. So if you are working on a project with a Gemfile in it, the Gemfile extension would provide additional tool lookup; if your project had a Nix Flake instead, it would use the Nix Flake extension to look up tools. There is, of course, the question of how Nova would handle the case where multiple extensions say „I know where to find tools“ because they could be conflicting. Let me draft a rough API interface I am envisioning:

The AssistantsRegistry would have an additional EnvironmentAssistant. An extension could register such an EnvironmentAssistant and could also query a reload of this assistant, similar to a TaskAssistant. The EnvironmentAssistant would provide a function whose single argument is an object containing all environment variables. The function is to return an object with environment variables, or a Promise resolving to one. The EnvironmentAssistant may also be queried for a user-facing name.

The Nova user can then select an environment to use. The base environment would be the one Nova extracts from the login shell. If the project had a Gemfile and an extension supported that, an additional environment would be available whose name might be „Gemfile“. When the user selects this, all Process invokations will get the environment variables that have been returned by the EnvironmentAssistant, including a potentially modified PATH. The EnvironmentAssistant would get the base set of environment variables as input, so that it can modify parts of it, but can also completely replace it (Nix Flakes take this approach, to ensure that you did actually specify all dependencies you’re using).

I hope this explanation addresses your concerns.

1 Like