Even with smaller models & more optimized hardware, I think edge compute is going to be power-limited first. Batteries today just won’t support constantly running LLMs. But I joked recently that as long as they prove useful then consumers would be willing to swap their iPhone for the old car battery with a phone handle attached.
The first step is for it to be viable for smaller models on desktops. The rest will follow, as hardware catches up. Hybrid analog-hybrid NN hardware is on the horizon, maybe in 3-4 years. This would allow GPT-4 level performance on an iPhone with plausible battery life.
The current hardware of course can't pull anything like this yet. But iPhone supports on-device facial recognition, object recognition, dictation and translation, so small steps...
