Why Now Daily.

Published

On-Device AI: Running Models on Phones and Computers

On-device AI runs model inference on a phone, computer, vehicle or embedded product instead of sending every input to a remote model service.

Timeline

  1. 2017: TensorFlow Lite debuted as a runtime for deploying machine-learning models on mobile and embedded devices.
  2. 2024-09-04: Google announced the LiteRT name for TensorFlow Lite and an expanded on-device runtime direction.
  3. 2025-05-14: Google updated ML Kit privacy terms describing local input processing and separate update and metrics traffic.
  4. 2026-10-12: Current Apple Core ML and Google on-device ML documentation was reviewed for this guide.

On-device AI means that a model performs inference—the calculation that turns a new input into a prediction or generated output—on the user’s phone, computer, vehicle or embedded product. The model may still have been trained in a data center and downloaded later. A feature can also be hybrid, doing one stage locally and sending another stage to a server. The useful question is therefore which model runs where, what data each stage receives and whether the feature still works when the network is unavailable. [1][2][5]

Local inference can improve responsiveness because an app does not need a network round trip for every request. Google says ML Kit’s on-device processing supports real-time camera use and offline operation; Apple says Core ML uses a device’s CPU, GPU and Neural Engine while optimizing memory and power. These properties suit tasks such as barcode scanning, speech recognition, image analysis, translation and short text assistance. Actual latency still depends on model size, input length, device hardware, thermal conditions and competition from other applications. [1][2]

Privacy is a major benefit when raw inputs never leave the device, but “on device” is not a complete privacy policy. Google’s ML Kit terms say images, video, text and resulting outputs processed by its APIs are not sent to Google servers, while also stating that the APIs may contact servers for fixes, model updates and accelerator compatibility information and may send performance and utilization metrics. The surrounding app can have other analytics, backups, synchronization or cloud features. Users should examine the product’s data disclosures, permissions and settings rather than infer every data flow from one label. [2][3]

The hardware imposes a strict resource budget. Model weights and temporary calculations consume storage and memory; continuous camera, audio or generative workloads consume energy and can heat the device. Specialized neural-processing units can execute supported operations faster and with less energy than general processors, but phones and computers vary widely. A model that is smooth on a recent flagship may be slow, unavailable or handled by a fallback on an older device. Developers therefore test across device classes and define clear minimum requirements. [1][4]

Making a model fit often changes it. Developers may use smaller architectures, quantization, pruning or other compression to reduce download size, memory and computation. Apple’s Core ML guidance describes lower-precision weights as one way to shrink neural-network models, while model files can also be downloaded after installation instead of bundled with the app. These choices can lower resource use, but quality and compatibility must be measured on the final converted model; a smaller numerical representation is not automatically equivalent on every task. [1][4]

Model delivery creates another trade-off. Bundling a model makes the first use available offline but increases the application download. Fetching it later keeps the initial app smaller and allows updates, yet requires a network connection, storage space and a safe fallback while the file is missing or incompatible. Updates should be authenticated, versioned and reversible. A local model can also become stale as language, products or risks change, so teams need a plan for evaluation, replacement and deletion rather than assuming an embedded model will remain adequate forever. [3][4][5]

On-device personalization can adapt certain models with a person’s own labeled data without uploading that material. Apple documents runtime updates for models explicitly built to be updatable. This does not mean every local model learns continuously, and uncontrolled learning could create unpredictable behavior or preserve sensitive information. Product descriptions should distinguish ordinary inference, a downloaded model update and local fine-tuning. Each has a different security, storage and quality lifecycle, and users should have appropriate controls for personal data and reset behavior. [1][6]

To assess an on-device AI claim, check whether core inference is fully local or hybrid, which devices and operating-system versions qualify, how large the model download is and which features work offline. Also ask what telemetry leaves the device, how models are updated, whether outputs are retained, how battery and thermal limits affect performance and how quality was tested across languages and hardware. Local execution can deliver fast, resilient and privacy-preserving features, but the real outcome comes from the entire application design, not simply the physical location of one model. [1][2][3][4][5]

Sources

  1. Apple Developer Documentation — Core ML
  2. Google for Developers — ML Kit
  3. Google for Developers — ML Kit Terms and Privacy
  4. Apple Developer Documentation — Reducing the Size of Your Core ML App
  5. Google AI Edge — TensorFlow Lite Is Now LiteRT
  6. Apple Developer Documentation — Personalizing a Model with On-Device Updates

Related stories