BUG: completion selectors broken at line end (blocker for extension)

[kopischke (Martin Kopischke)]

I have been banging my head against the fact that many of the completions in my upcoming syntax extension are non-functional when the cursor is at the end of the line. I have managed to isolate and replicate the issue showing that subscopes are invisible to selector expressions when the cursor is at the end of the outer scope on any new line terminated line.

Yeah, this is a mouthful. Took me a while, too.

To reproduce, assume the following simplistic syntax extension:

<?xml version="1.0" encoding="UTF-8"?>
<syntax name="buggsy" xmlns="https://www.nova.app/syntax">
	<meta>
		<name>Syntax Bug Collection</name>
		<type>structured</type>
		<preferred-file-extension>buggsy</preferred-file-extension>
	</meta>

	<scopes>
		<scope name="buggsy.statement">
			<symbol type="statement" />
			<starts-with>
				<expression><![CDATA[(?x) (?=[^[:space:]]) ]]></expression>
			</starts-with>
			<ends-with>
				<expression><![CDATA[(?x) (?=[[:space:]]) ]]></expression>
			</ends-with>
			<subscopes>
				<scope name="buggsy.identifier.function">
					<expression><![CDATA[(?x) \w+ ]]></expression>
				</scope>
			</subscopes>
		</scope>

		<include syntax="self" collection="functions" />
	</scopes>
</syntax>

with the following completions:

<?xml version="1.0" encoding="UTF-8"?>
<completions>
	<provider>
		<syntax>buggsy</syntax>
		<selector>buggsy.statement</selector>
		<expression><![CDATA[(?x) \w+ ]]></expression>
		<set>buggsy.statements</set>
	</provider>

	<provider>
		<syntax>buggsy</syntax>
        <selector>buggsy.identifier.function</selector>
        <expression><![CDATA[(?x) \w+ ]]></expression>
        <set>buggsy.functions</set>
    </provider>

    <set name="buggsy.statements">
        <completion string="compstate" />
    </set>

    <set name="buggsy.functions">
        <completion string="compfunc" />
    </set>
</completions>

follow these steps:

1. Activate the above as a development extension.
2. Create a new document and assign it the syntax type “Syntax Bug Collection”.
3. In the blank document, type “c”: the completion “compfunc” is suggested, which matches the highlighting scope stack of buggsy.statement > buggsy.identifier.function.
4. Place the cursor at the begin of the line and insert a new line.
5. Jump to the start of the document, right before the new line character.
6. Type “c”: the completion “compstate” is suggested, which does not match the highlighting scope of buggsy.statement > buggsy.identifier.function. Instead, it matches the outer scope only.
7. Add “ word” (with the whitespace) on the same line. Jump back to behind the “c” at the start of the line, repeat 6. Same result.

As a side note, I suppose this has gone unnoticed until now because most static completions, including the ones included with Nova, are either scoped to delimited expressions, where the cursor ends up inside the outer scope, or use a root level exclusionary approach, i.e. some variant of *:not(string, comment). This is not an option for my extension, as the syntax it highlights is highly positional; completions need to be scoped precisely to specific scope stacks to be useful.

[kopischke (Martin Kopischke)]

@logan I am frightfully sorry to be this pushy, but as that bug is blocking the release of my extension, I would be very grateful if you could take a peek. If push comes to shove, I need to release the extension without completion support (better none at all than a broken one); but that would be rather sub-optimal.

[logan (Logan Collins)]

Hello Martin,

I am tracking this internally; I’m afraid I don’t have an immediate answer available, as I’ll need to dig a bit deeper into the completions code to see why it might not be behaving as you’d need for your extension.

I hope to have some more news on this soon. Thank you for your patience and we’re always grateful for what you and the other devs are doing.

[kopischke (Martin Kopischke)]

@logan

I am tracking this internally; I’m afraid I don’t have an immediate answer available, as I’ll need to dig a bit deeper into the completions code to see why it might not be behaving as you’d need for your extension.

Hey Logan, I wouldn’t expect you to have an immediate answer: this looks like a gnarly one. I mean, Nova’s selector system shouldn’t really work differently depending on if a line is new line terminated or not …

@logan

hope to have some more news on this soon. Thank you for your patience and we’re always grateful for what you and the other devs are doing.

I hope you didn’t gather the impression I was reproaching you personally when I shook the tree (I expressly wrote that that was not my intent – hope support communicated that part), and I understand you are probably swamped, which is not your fault (you don’t do the resource allocation inside Panic, I assume – I also wrote that that [the resource allocation, not you taking charge of it] really needs to be re-evaluated).

If I may: it would greatly help if, in the future, you could post just the shortest of ACKs on bug reports. It’s perfectly fine to sit on them a day or two, just by way of seeing if the community provides an answer, but after that, something like “whelp, that one looks gnarly, let me get back to you, might be a while, bit swamped at the moment” will do wonders relieving the feeling of speaking into the void. In my experience this kind of minimal “please hold” communication offsets the annoyance of doing it by generating an inordinate amount of goodwill and patience.

(I’m an educator and [non-IT] project manager when I don’t dick around software, so forgive me for the lecture. Leopard, meet spots)

[logan (Logan Collins)]

No worries at all. I definitely dropped the ball in not responding after the notification came in, and then it dropped down in my list so I very unfortunately didn’t revisit. I will definitely work to be more proactive in responding even if I don’t have a whole lot to offer in the moment, just to let you all know I’m here.

[kopischke (Martin Kopischke)]

@logan congrats on getting Nova 7.5 out of the door! May I trouble you for a candid assessment of how likely the completion issue is to be worked on for the next release? I am trying to decide if I should release a preview version of the fish extension (with a polite, but necessarily very prominent warning that completions will be MIA until this issue is addressed – I really, really don’t want to field complaints about this) or rather sit this out to release a feature complete 1.0. Any input would be greatly appreciated!

[logan (Logan Collins)]

I hope to be able to, yes. Although I’m note entirely sure when our next release will be (things are busy at Panic and our QA and support teams are a bit overloaded right now, so the release pipeline is stretched a bit thin.) I’m hoping to target our next release for early or mid October.

[kopischke (Martin Kopischke)]

So … the next releases have come and gone and Nova’s completion system is still broken. I have shelved my fish extension entirely for the time being, but I am curious to know if there is any realistic timeline for this issue being addressed.

I understand from your boss’ recent comments on this forum Panic will not invest more resources into Nova, so your time constraints are not going away; I sympathise with that, I really do. My issue is that fish’s development pace has picked up quite a bit lately, syntax features are changing fast, and I am faced with a choice between investing a considerable amount of my sparse free time to keep the extension current without the ability to release it, or playing catch-up to umpteen fish releases when things turn around for Nova. Neither is a perspective I relish.

I would greatly appreciate a candid assessment if this issue is on your plate for one of the next two releases, or not. Either is fine (though I would of course prefer the former), but the cycle of hoping for a fix in a new release and finding it missing every time is slowly grinding me down. At this point I am seriously considering cutting my losses, cancelling my extension, and finding an alternative to Nova before my license period runs out. This is not a threat, I hasten to add – that would be childish; I am one single customer and certainly not entitled to your time –, just a plea to help me avoid making a decision out of frustration.

[logan (Logan Collins)]

Hi Martin,

A potentially long post delving into other things, so sorry for the verbosity.

I wish I had a better answer for you. Things have been busy, some life things have happened taking even more of my time, and I haven’t been able to devote the time that is definitely warranted to issues like this.

After your report, I did do an initial pass on this issue, but in order to fully address it I’d need to make behavioral changes to Nova’s parser and taking those changes in a bug fix release (like Nova 8.3) introduces more risk of regression. I’d much rather take those changes in a major version release, and I wasn’t able to start work for this with Nova 8.

It’s most definitely possible I will be able to include changes for this in Nova 9, which is currently scheduled for early next year. I’m currently in the process of re-evaluating my list of targeted issues in the hope that I can focus more on those that pertain to our extensions, as that would unstick a lot of things down the line.

I understand your frustration, I really do, and share no ill will at how you (and our other devs) must be feeling. I only wish I could provide exactly what you need now. But, with that in mind, I fully encourage you to do what is best for you and your workload. If that means taking a break from Nova, then we only hope you’ll consider us again down the line.

While I’m forever grateful for our extension developers and the work they all do, I cannot ignore the fact that we will never have the resources that other companies do in supporting a large ecosystem of extensions, and with providing rapid updates and changes. VSCode has a core team of 20+ engineers plus direct community volunteers. We’ve never set out to compete directly with editors like it, but the comparison will forever be there, because of the overlap of features. (This is definitely not an attempt to make an excuse, just a mental note that I have to remind myself of regularly.)

Whatever the case, thank you as always for the incredible patience you (and all) have shown, and for wherever things take us down the line. It really means the world to me that so many folks have been supportive with Nova, as it is definitely in part a labor of love for me personally (and a lot of folks at Panic). My hope is that as we come out of the current state of affairs around the world and in the industry, we’ll be able to expand our engineering and QA support to make work on Nova smoother all around for everyone involved.

[kopischke (Martin Kopischke)]

Hey @logan, thanks for the candid answer. I must admit it took me a while to come to terms with it. I understand that, what with Panic’s obvious disinclination to give their editor the resources it needs (and I am not comparing Nova to VSCode – I think Sublime is closer to the mark. But that is a discussion to be had with your bosses, not you.), you need to prioritise in a way that makes addressing gnarly bugs in core functionality an unattractive prospect. Still, I would be lying if I said that that wasn’t a frustrating answer when talking about a paid product.

I have taken your advice to “take a break from Nova” to heart and had a long, hard look at the competition out there, and I am afraid moving on is not working for me. VSCode is a monstrosity, and anything that uses TM 2 syntax grammars is out anyway, as there is not way to create a decent fish syntax in it; which also disqualifies things like Code Runner (actually not quite the toy anymore it started as). The newly LSP-enabled BBEdit … uh, I think I started using Macs too late to accept that kind of UI, and anyway, the syntax format doesn’t seem amenable to good fish support either. Which leaves Sublime, which has a meh fish syntax plugin for version 3 (it failed every single one of the syntax tests I wrote for the Nova extension) that could probably become quite good leveraging the new features in version 4, but what I see there is massive teething trouble, plus I’d end up with a thing you can theme just about enough to bring it into the uncanny valley of macOS UIs, which, honestly, yikes.

Long story short, considering the cost (not literal; I’m talking work hours and mindshare) I already sunk into Nova, you’re stuck with me and I will keep going on your nerves about this issue. This means two things:

1. I will be releasing my fish extension with the piecemeal, overeager-where-it-works completion support I could cobble together with the currently broken selector API. That in turn means clearly and publicly labelling all support issues around completion out of scope because they are due to a Nova bug, but hey … ¯_(ツ)_/¯.
2. I will be creating a new Discourse thread about this issue, partly as a FYI for other devs and reference for users of my extension, but mainly because I think I have, by now, pinpointed the misbehaviour in question far more precisely than I did in the original post. I’ll close this one as resolved to declutter things.