I’m currently fine-tuning a language/syntax extension I’ve created for a pretty simple “structured” data file type. It’s just to provide syntax highlighting (for now at least), so no symbolication or completions or anything.
The syntax definition is pretty much fully functional and complete, but I was trying to diagnose one last situation that wasn’t receiving proper syntax highlighting… that’s when odd behavior started occurring.
When I tried to start typing a new indented line (new subscope) it appeared that all syntax parsing after initial file opening had been completely disabled. Saving, closing, and re-opening the file was the only way to get the newly added text to become highlighted again. Notably while typing on this unparsed new line, holding the space bar down resulted in way slower repetition than normal (which I initially assumed to be the result of unintended recursion in my syntax’s regex or scoping definitions).
Even stranger, when trying to use the inspection tool to see what it thought of this new, un-highlighted text… it apparently just did not function anymore. I tried hovering over all different parts of correctly parsed text and could not get anything to pop up.
As I went back to figure out the issue I was greeted with these repeated popup warnings that made it impossible to do anything without force quitting Nova:
This is on a MacBook Pro (M1 Pro, 16GB unified memory)
Now, I’m not well-versed in low-level programming or memory allocation issues… but that said, I’ve never encountered a system alert on macOS repeatedly telling me to force quit an app due to memory concerns (not even on the old 2012 MBP I was using up until earlier this year).
I don’t see what else this could be other than an internal memory leak within Nova… most likely kicked off by some strange edge case involving a specific recursive scenario present in my syntax definition.
Is this a known issue? Anyone else encounter anything similar?
I’ve been cautiously poking this issue with a stick today and found that entering a certain subscope consistently maxes the CPU and triggers Nova’s used memory to gradually and irreversibly (without deactivating the extension or closing its targeted files) start increasing at around 1-2 GB/s … according Activity Monitor.
Convinced that this was the result of an infinite regex parsing loop that I got myself into, I tried all sorts of variations and alternative setups to the relevant parsing/scoping rules and initiated the same trigger to see if it still made the memory balloon.
I’m really at a loss now, literally nothing i’ve tried has been able to break to consistency of this memory leak trigger.
The exact trigger is to enter a certain subscope, which matches a Capitalized word followed some other characters, and then entering just one capital letter kicks off the leak. Bizarrely even testing regexes for the scope which don’t even accept capital letters… entering a capital letter still triggers the memory leak. Typing anything once the memory use has begun increasing seems to do nothing to break it.
However, everything else works fine, even if I paste a full matching string, it seems to bypass this trigger, so only when manually typing out the same matching string does it occur upon the first keystroke.
Not sure what to make of this but, I’m just gonna work around it for now.
I came across this eclecticlight.co article which describes a persistent macOS memory leak bug related to Finder searches… the description sounds very similar to what I’ve already described above:
For example, let’s say you start typing the word syzygy, which should have very few hits. When you’ve just typed the first character s there are likely to be hundreds of thousands of hits. As those are being delivered to the Finder, its memory use grows rapidly. Add the second character y and the rate of memory use slows sharply, as there are far fewer hits, and after the third character z there should be even fewer.
What should happen is that, as the search is refined, the results of earlier partial searches should be cleared from memory, which should reduce the amount of memory used by the Finder. That doesn’t happen, though, and memory use continues to rise, never to be freed, even when you close that window to finish using Find.
I suppose that if Nova uses the same underlying APIs/frameworks for syntax parsing that Finder uses to search files for specific text strings, perhaps this problem originates at the OS level? Just an educated guess though as, like i’ve said, I’m not really qualified to speculate within this realm.
By the way, I eventually ended up finding a scoping setup that seems to avoid the memory leak and haven’t had any trouble with it since then. Not sure what exactly I did to bypass the issue… but I know better than to question an inexplicably functional workaround to an inexplicable problem.
Nothing specific unfortunately. This was before Nova added tree-sitter as the primary syntax engine, so there’s probably a lot of different factors involved.
That said, while way more performant, writing/modifying a tree-sitter grammar can be somewhat of a foot gun and I’ve accidentally ballooned the memory like this many times, all of which just due to programming errors rather than bugs.
I guess my only advice would be that when testing a syntax extension if you notice a slight lag or delay in highlighting… quickly open up the activity monitor and force quit Nova if it’s memory usage is rising rapidly
When you had this issue could you pinpoint to a specific extension by like deactivating them, one by one? Like was it “reversible” as soon as you closed the targeted files or extension?
Also by any chance do you know if in Activity Monitor, if it is an extension related memory leak, would it appear under NovaExtensionService or Nova?
I am just wondering if it is the tree-sitter-blade that is causing the problem, as that is the only syntax related extension I can see there. I tried to open a few duplicate blade files to push it but did not see a dramatic increase anywhere apart from Nova maybe 20-30MB, then when I closed the tabs and deleted the test files it went up another 80-100MB which is a strange behaviour as I am reducing the workload by removing the files and even deleting them out of the workspace? But again nothing out of ordinary.
The only time I’ve ever had runaway memory leaks like this was when I was actively developing/testing my own syntax extensions locally and the cause always came from the extension I was working on (even if I didn’t know what that cause actually was).
Majority of the time, the cause turned out to be an infinite parsing loop I created or something like that. But I personally have, never seen an external extension creating a similar situation.
If I did though I’m not sure it would be straightforward to diagnose that using activity monitor or alternatives, since the enormous memory size always seemed (for both regex and tree-sitter syntaxes) to get attributed to the primary Nova.app process (with the logo). Which I guess makes sense if the syntax parsing engine eating up all that memory is an internal component that just executes instructions given by extensions.
Deactivating the extension usually breaks the runaway memory consumption (at least in dev mode), closing the file seemed to be a little less likely to reverse it.
So I suppose you could try turning extensions off one by one as long as you have time to do that before your system locks up haha.
But if you’re not seeing memory use rising very rapidly on the order of gigabytes, that’s probably an unrelated issue.
