It’s a bit disappointing to me that the power of modern Macs to run good local LLMs is not being taken advantage of among the Mac-native editors, so I’d really like something that made that possible. It seems like a natural fit, but might require more glue code.
Preface: my engineering background does not intersect with LLMs in the slightest, so take what I say next with an adequite number of grains of salt. My knowledge is mostly pieced together by reading high-level overviews and whitepapers, as well as talking to colleagues with far more understanding of these things.
My understanding was that while Apple’s CoreML on-device works extremely well for language models that can operate on, say, your personal photo library or a local audio library, accomplishing the same effects as services like GitHub Copilot would be quite a bit more difficult locally. Not necessarily because of the lack of processing power, but because of the lack of diverse input data as a user.
Microsoft has very proudly stated that Copilot’s LLM runs across massive datacenters feed by the bulk of public repositories hosted by GitHub, which greatly drives how it can provide such accurate code completion and reasoning for a breadth of programming languages and scenarios.
As far as I am aware, attempting to train a similar code intelligence model purely locally against a user’s own projects would definitely not have have the depth necessary to make it viable for general coding assistance. It would need to be trained and fed on a reasonably large, preferably public domain or fair use input set with a great enough language range.
There are a couple of additional models I’ve seen in passing that were (potentially) trained in such a way (like Meta’s various CodeLlama iterations, and Tabby which I believe uses CodeLlama internally by default), but I am not fully aware of the implications of how these would behave for a user locally.
Tabby, in particular, advertises itself as “self-hostable” and mentions using CoreML when running on Mac hardware, but also has somewhat hefty local GPU requirements as well as the need for the Rust runtime. Looking at its installation requirements, it also requires multiple tens of gigabytes of hard drives space to store the predetermined training data, which is obviously not something we could ship in a Mac app unless it were a separate download of some kind (completly ignoring the ethical / legal implications of these two particular models, of which I have not researched to the point of being able to say “oh, we could use this.”)
