Architecting ofln: Zero-Cloud On-Device LLM Chat on Mobile

1. The Problem: Cloud LLM Fatigue & Privacy Leakage

Most AI chat applications today are glorified API clients. Every token you generate leaves your phone, travels through a third-party server, gets logged into cloud storage, and incurs recurring latency and subscription costs. If you enter airplane mode or step into a subway with zero signal, the application becomes useless.

With ofln, our goal was simple: Download a model once, chat forever with zero cloud dependencies. Your conversations never touch a server, your data never leaves device flash, and inference executes directly on the phone's silicon.

2. High-Throughput Mobile Runtime Architecture

Running transformer models with hundreds of millions or billions of parameters on smartphones requires bypassing standard JavaScript runtimes. The architecture of ofln relies on a clean 3-tier model:

1. UI Layer (React Native / TypeScript): Manages conversation tree state, markdown streaming renderers, and responsive user gestures.
2. Bridge Layer (JSI / C++ Bindings via llama.rn): Uses React Native's Javascript Interface (JSI) to call compiled C++ functions directly without JSON serialization overhead across the old asynchronous bridge.
3. Compute Engine (llama.cpp compiled with NEON / Metal): Directly executes optimized tensor matrix multiplications using ARM NEON SIMD intrinsics on Android and Apple Metal Performance Shaders (MPS) on iOS.

// Direct JSI invocation for streaming tokens without bridge overhead
import { initLlama, LlamaContext } from 'llama.rn';

const context: LlamaContext = await initLlama({
  model: 'file:///data/user/0/com.ofln/models/Llama-3.2-1B-Instruct-Q4_0.gguf',
  use_mlock: true,        // Pin weights in RAM to prevent OS swapping
  n_ctx: 2048,           // Context window constrained for mobile thermal envelope
  n_gpu_layers: 99,      // Offload all compatible layers to Metal/OpenCL
});

3. Quantization Choices: Q4_0 vs. FP16

A 3B parameter model in standard 16-bit floating point precision (FP16) requires ~6GB of RAM—exceeding the available memory budget on most consumer smartphones and triggering immediate Out-Of-Memory (OOM) kills by the mobile OS watchdog.

That is why ofln's curated starter shelf ships Q4_0 GGUF weights. Four-bit quantization shrinks phone-sized models into a footprint that fits typical device RAM while staying compatible with llama.cpp on ARM. Users can still import other GGUF quants from Hugging Face or local storage; the shelf itself standardizes on Q4_0 so downloads stay predictable and verifiable via SHA-256.

4. Real-Time Mobile Inference Telemetry

To maintain transparency, ofln streams live inference telemetry directly underneath every response bubble:

• Inference Speed (tokens/sec): Real-time generation velocity across decode and prompt evaluation stages.
• Time to First Token (TTFT): Measuring prompt processing latency.
• Resident Set Size (RAM in MB): Ensuring model memory footprint stays strictly below OS warning thresholds.

5. Conclusion & Open Source Code

On-device AI is moving from an experimental curiosity into standard consumer expectation. The source code for ofln is completely open-source on GitHub:

Explore ofln on GitHub ↗