Desert Ant Labs announced itself with the post “Introducing Desert Ant Labs”: the application is fast models running locally on the device. The discussion collected 459 votes. Neither a list of models, nor measurements, nor access conditions follow from the reason: this is a post-presentation of the company, and there is nothing to dissect in it yet, except for the direction itself.
The direction is worth it to once honestly calculate what you get and what you pay. Local inference is sold with the words “fast and private.” Both parts are true - with reservations that are written in small print in announcements or not written at all.
What do you buy and how do you pay?
You get: there is no network trip to someone else’s server, so the response is predictable and does not depend on someone else’s accident; the data does not leave the device, eliminating a whole class of conversations with a lawyer and with a client in a regulated industry; works offline - in the basement of a warehouse, on an airplane, in a field without communication; After purchasing hardware for tokens, no one issues an invoice.
Pay: with battery and heating, this is immediately visible on the phone; memory - the model takes up space and competes for it with the application itself; scattered hardware - the demo flies on a new laptop, while on a three-year-old phone the user watches a different movie; delivery of updates, because a new version of the model means rolling out the application and downloading hundreds of megabytes, and not changing a line in the config; quality on long and complex tasks, where cloud models are still stronger.
Where it pays off today
Narrow tasks with short input and predictable output. Tagging and searching local notes and files. Recognition and normalization of text from document photographs. Input hints. Offline classification of outbound requests. Pre-filtering: filter out garbage on the device and send to the cloud only what requires a heavy model, with the explicit consent of the user.
Does not pay off where you need a long context, multi-step reasoning, or fresh data from the Internet. The hybrid scheme is more honest than any of the extremes: locally - fast and private, in the cloud - the rest, and the user sees in the interface what exactly goes outside.
My reading. There have been many companies promising “fast and local” over the past year, and according to their announcements they are indistinguishable from each other. I look at one thing: do the measurements show this on cheap hardware? Everyone shoots demos on a top-end laptop; almost no one shoots numbers on a budget phone with a strict memory limit, and this is where the bulk of users of a mass product live.
Desert Ant Labs currently only has a post-presentation, and the technology cannot be evaluated based on it. A useful action now is to put the link on your watchlist and come back when there are weights, an SDK, or a demo that you can install and try yourself. At the same time, write down which tasks in your product would generally withstand moving to the device: short input, clear output, tolerance for slightly worse quality. This list will be useful no matter who ends up making the quick local model.