So it will be locally hosted on the phone? I seriously doubt it will be very useful in offline only mode. Even relativity small language models (7B or 13B) struggle on even desktop pc’s if you don’t have a high end graphics card with 12+ GB of vram. Analyzing can be relatively fast, but generating will be terribly slow, especially images.
Edit: So after some reading, the snapdragon gen 3 has some impressive specs, but can someone explain how a phone can generate fast ai content while a PC needs let’s say 24GB’s of vram? I get the phone has an ai-specialized chip, but you still need to load the model into memory.
Did they say, it would run locally?
They said it would be available on their upcoming flagship product, sure, that’s marketing. To me, this does not imply that the heavy-lifting is on the phone’s hardware.
(And maybe a special-purpose AI, which requires significantly smaller models, can run on the phone, which is a high-end mobile computer.)
If you don’t need an incredible amount of creativity, quantization can get it to work with relatively little RAM.
