Language Server: Markdown not rendering properly

logan · April 8, 2021, 4:50pm

You are correct. Nova currently does not support rendering Markdown in hover tooltips.

This is due in small part to the Markdown engine we currently use, which was written with the intent to be rendered in a web context (so it currently only outputs HTML fragments). Nova’s UI is not a web view, and is rendered using native macOS controls that expect Cocoa’s attributed string formatting API. I experimented a long while back with embedding a small web view into the hovers, but its performance for appearing and sizing was… really not that good.

I sadly haven’t had enough time to devote to work on implementing a Markdown parser that could output into something the native controls can handle. But it’s definitely a thing we need to do.

As for documentation, if you notice any missing documentation, please, please do let us know. We try to keep our documentation up to date and complete for the entire extension API, so if something is absent (or incorrect), we’d love to know.

GwynethLlewelyn · April 8, 2021, 8:21pm

Hi @logan — thanks so much for looking into this!

If there is something I’m totally clueless about is developing code for the Apple ecosystem. As such, maybe my humble suggestion is completely stupid, so please forgive me in advance.

When you described that the controls you use expect attributed strings, I immediately thought… well, in that case, and given my limited knowledge of the subject, it seems to me that all you need is to parse Markdown directly into attributed strings, not really creating mini-webpages hovering around the editor…

So, my first idea was just to do some clever regular expressions to capture basic Markdown (which is what the Language Servers emit, anyway), and, instead of spewing out HTML, it would construct an attributed string.

@jbroadway implemented the Smallest Markdown Parser Ever (Slimdown) with exactly that approach in mind. Sure, the code is in PHP, but as you can see from the comments there, the concept is what matters (as well as the regex!), so people ported it into all sorts of programming languages (a curiosity: it seems to be a popular college assignment for computer science students — port Slimdown to your favourite programming language!)

Slimdown naturally uses HTML as the output format, but it lays it out nicely on a table that is very easy to understand, so, in theory, you could get it to render basically anything you wish — including (why not?) Cocoa’s attributed string formatting.

Ok. But I understand that you guys have a zillion issues to work on, and no time for implementing very complex ‘goodies’ that are used by just a fraction of your clients. Fine! What about generating HTML from Markdown and then converting it directly to attributed strings, since Apple so generously provided you with an initializer method to do just that? Basically, get the Slimdown parser to generate it into Data, and then pass it along to the special HTML-based initializer, and that’s all.

O-kay. Perhaps even that is too difficult, I don’t know. Fortunately for all of us, someone already kindly created a full implementation of the paragraph above to do exactly that, parse Markdown into HTML and then let Apple’s methods do the rest of the boring job (released under the Apache License 2.0).

Granted, this Swift-MarkDownKit is a heavyweight, full-blown Markdown-to-HTML-or-Attributed-String parser, including a lot of extra tools that you might not need at all, so you might just use the juicy bits of the kit and let your own quick & dirty implementation of Slimdown drive the actual parsing.

I hope that the suggestions above make any sense to you and are useful in any way!

logan · April 8, 2021, 8:30pm

Hi Gwyneth,

Thanks for the links and the ideas. You are right—what I plan to do is get a parser that can output into this format properly. Apple’s initializer for HTML, while with the right intention, has its own faults and issues that we have dealt with in the past, which sort of wrote it off as an option right away.

Generally, though, this comes down to: at the moment, Nova’s development team is about 80% just me as our engineering teams have been strained with other projects, so I just haven’t had the time to fully move through all of the work that our developers have been interested in on top of the bug-fix workload that we’ve been coordinating with our support and QA teams. Note: I definitely don’t want that to appear as an excuse, though. I hope to be able to swing back around to more work on the LSP side soon, as I definitely understand how important it is to a lot of our extension developers.

As always, thank you for your feedback and I’m happy you all are so invested and willing to work with us to make Nova better.

simondeeley · April 8, 2021, 10:00pm

I had a quick search on the interweb and one result that stood out to me was this repo CocoaMarkdown - it’s not seen an update in a couple of years but also may form the basis of an idea to get Markdown into a native-compliant format.

apexskier · April 9, 2021, 10:29am

That makes sense. I dealt with a similar markdown issue at work but with React/React Native and our custom cross-platform component library.

kopischke · April 9, 2021, 11:26am

Oh, wow, … kudos to you and the workload you handle then, @logan ! I’m probably not stating anything new to you by saying that is so not sustainable. Judging from recent activity levels on this forum, I have the impression Nova adoption is picking up steam, but I daresay either you or the interest in Nova will inevitably burn out if Panic cannot find a way to assign more resources to it, esp. re. extensibility …

GwynethLlewelyn · April 9, 2021, 3:36pm

Oops, sorry, @logan, I guess we shouldn’t be burdening you with so many requests then… you must be already working 20 hours/day (I’m also counting the time spent on the forums, GitHub, etc. answering questions/requests).

I sincerely hope that you guys are able to get a handful of developers to help you out. Let me second @kopischke’s impression — we can only ‘watch from outside’, of course, and have no clue about the real numbers, but I have certainly noticed that after half a year or so, Nova seems to have more extensions & themes than Coda 1 & 2 ever did… and many of those get updated (i.e. they’re not ‘one-shot’ attempts that have quickly been abandoned).

As this thread shows, there are already many extensions for the same language (addressing different issues). This is good. It means lots of people happy to try new things out, even starting from scratch, because one can always do better!

Last but not least, during the Beta testing, I remember seeing a comment from someone at Panic which really captured my attention: the notion that many programmers (especially those not conditioned at the workplace by a uniformly imposed workspace) are moving away from all-in-one IDEs — behemoths that will consume every available resource on the computer and then some — and, instead, pick light-weight editors, pushing all IDE-y things into plugins/external tools, and relying on editors supporting some form of integration with those tools. This might just be a fad, a trend, something in fashion right now, but the truth is that if this trend persists, you nailed it. And I’m not really surprised that even Microsoft figured out the same thing — pushing a much lighter VS Code as an alternative to the full-blown Visual Studio.

As mentioned early, I have no experience in developing things for the Apple ecosystem. One of the main reasons for that is Xcode. Again, I have personally nothing against Apple’s free IDE — except, of course, that it’s insanely heavy. A few times in the past I launched it only to edit a .plist or similar bit of configuration file that Xcode opens by default and makes a good job of doing of displaying it. Before I installed any other editor, Xcode even insisted on opening JavaScript files. This would drag the Mac down to the point that it was impossible to work with. I used a free version of Sublime Text for some time just to quickly edit files with some syntax highlighting (as opposed to opening JavaScript on, say, TextEditor…); I toyed around with one or another editor… until I got Coda I remember that I was a bit sceptic with the overall concept of ‘Projects’, because, well, I thought that I would be getting a huge amount of overhead until I was able to type my first line of code.

It turns out that Panic’s philosophy — best illustrated in Nova, of course — relies on an old Unix aphorism: don’t try to do everything in a monolithic application, but, instead, spread out the load to several bits, each of which doing just one or two things very well (and quite quickly!), and focus on how to interact with those loose bits.

In other words, Emacs.

Ok, jokes besides — no, I don’t want to start another vi vs.Emacs war! — my point here is that, for many decades, old-time programmers using 80x25 VT100 terminals faced the same issues, and the solution was to have an editor which can optionally run all sorts of extensions and external tools, and which was designed for being extended and expanded that way. I’d claim that Nova follows that concept, too (sure, it’s not the only one — as said, it’s a current trend), but, instead of 80x25 VT100 CRTs, we have Macs with GUIs on Retina displays (well, and JavaScript is perhaps easier to learn than LISP…).

Nova is fast enough for me to use it to do a quick edit on some kind of text file. TextEdit is faster, but usually lacks a lot of functionality; but, as a curiosity, I do have GNU Emacs installed on my Mac as well (old habits die hard…), and it’s just slightly faster at opening a file from scratch than Nova. Emacs is well-known for its overhead (compared to so many millions of text editors out there), and on my old, mid-2014 PowerBook, the slight delay in launching all that LISP is perceptible (naturally enough, I mostly use other alternatives that are much faster than good old rms Emacs…). What I mean is — nice work, guys, Nova is really quite fast at doing its job, in spite of all the overhead for being a graphical editor-cum-extensions.

But I’m completely out of topic and will shut up now. I just wanted to give a heads-up, and hope that Panic’s board is willing to get a few extra hands to help out @logan…

jrf · April 9, 2021, 10:24pm

At the end of the day, tooltips in raw Markdown fall pretty low on my priority list. And as a whole, I’ll bet the bug/feature priorities expressed in this forum are not representative with the feedback Panic gets via other channels. Whatever development methodology you pick, at some point you end up with a prioritized list of stuff to do. I’d love a peak at that list for Nova, just to see where my pet feature requests are, but as someone whose day job is writing commercial software, I’m familiar assorted reasons why complete transparency isn’t realistic.

GwynethLlewelyn · April 12, 2021, 11:34am

Ultimately I have to agree with you, there certainly are other priorities.

Also — who knows? — one day Apple might even release an upgrade to their API, allowing Markdown to be directly converted into attributed strings, rendering the issue moot (and having wasted @logan’s time in searching for a solution and hand-crafting it)… so we ought to pester Apple instead, they have vastly more resources, human and otherwise, and if they managed to give developers a method to convert HTML directly into attributed strings, it shouldn’t be hard to do it for Markdown as well — considering that ‘bare bones’ Markdown is orders of magnitude simpler to parse than HTML.

Then again, I remember that it was Panic which found out about bugs on the text-rendering mechanism provided by Apple’s SDKs, and after grumbling with Apple to fix the bugs they had uncovered, instead of waiting for a fix, Panic simply went ahead and rewrote their own text renderer… so sometimes we cannot rely solely on Apple

But on a scale of importance, I’d rate this feature 3 out of 10. Nice and convenient to have, sure, especially when comparing with the competition (as well as compared to some of the extensions which do not rely on a Language Server), but it’s not a critical functionality.

logan · April 20, 2021, 10:52pm

Just wanted to update folks here—I’ve implemented support for basic Markdown rendering for Nova 6 for the LSP hovers. If the LSP you’re wrapping declares support for it, we should render it automatically. Nova 6 is planned to be released sometime this week.

It should cover things like headings, lists, strong/em, etc. No tables support for now, but so far I haven’t come across anything in that realm that’s using it. We can always look into extending it further if needed.

mike.bronner · April 21, 2021, 1:33pm

That is awesome, @logan, I’ll be looking forward to that! My LSP extension is suffering from the same problem.

GwynethLlewelyn · April 21, 2021, 5:44pm

I have to say, @logan, you’re simply a marvel! I hope that Panic pays you your weight in gold — every week You’d still be underpaid.

Thanks very much for that! I’m really, really looking forward to Nova 6 now!

simondeeley · April 22, 2021, 3:25pm

As the original poster for this topic I know I’ve been absent a while but the discussions in this thread are really constructive.

I certainly wasn’t expecting a response from Panic to say they’ve implemented a markdown parser - such fantastic news and can’t wait to test out the feature! Thanks @logan

mike.bronner · April 22, 2021, 3:45pm

Here’s a little snapshot of it working with my LSP extension:

kopischke · April 28, 2021, 8:49am

Apparently, that was grounded in fact.

apexskier · July 3, 2021, 1:09pm

I’m running up against this again. I’m looking at improving the documentation rendering in GitHub - typescript-language-server/typescript-language-server: TypeScript & JavaScript Language Server, and the experience in Nova is pretty bad.

The markdown:

```typescript
constructor Dot(x: number, y: number, width: number): Dot
```

Create a dot.

* _@param_ ``` x ``` — The x value.
* _@param_ ``` y ``` — The y value.
* _@param_ ``` width ``` — The width of the dot, in pixels.
* _@example_ — testing
  ```typescript
  new Dot(x, y, z)
  ```

Nova's rendering:

Note the odd paddings, the lack of a break above the code block in the last <li>, the additional bullet after that, and the lack of syntax highlighting.

Here's what it renders like in this forum:

constructor Dot(x: number, y: number, width: number): Dot

Create a dot.

@param x — The x value.
@param y — The y value.
@param width — The width of the dot, in pixels.
@example — testing
```
new Dot(x, y, z)
```

I can get syntax highlighting if I use MarkedString (which is what the server currently uses), but that type is now deprecated in the language server protocol in favor of MarkedContent, so I don’t want to keep using it.

Nova also only renders the first of the set of MarkedStrings returned in the onHover response.

apexskier · July 3, 2021, 1:15pm

Interesting…

If I return this MarkedString: { language: 'markdown', value: '...' }, the code fence’s syntax highlighting works. (In some ways, this is more readable than the native markdown rendering).

GwynethLlewelyn · July 5, 2021, 4:56pm

We’re @logan’s nightmare…

@apexskier, what you’re describing is not yet on the latest version of your extension, right? It’s just because I only get a much simpler output than what you show above (I’m using your extension as listed on the Library).

My guess is that, as @logan explained in the past, current Markdown support on the hover boxes is limited (and it’s not really rendered into HTML, but directly to the UI). I was pretty sure he had posted something about it here on the dev forums, but I guess I’m just confused since a search returned nothing… except that it was being worked on (and it is working — with limitations ).

apexskier · July 5, 2021, 6:00pm

Yeah, this is running against a branch on my fork of the underlying language server.

apexskier · October 15, 2021, 2:13am

Hopefully this can be taken advantage of to improve this. GitHub - apple/swift-markdown: A Swift package for parsing, building, editing, and analyzing Markdown documents.