Compiling YOLOv11-Nano and llama.cpp on Android NDK for Sub-30ms Field Inference

We ship three offline AI apps to $100 Android phones — government scheme eligibility, crop pest advisory, and an NCERT tutor. Every model runs natively on the device with zero network calls. This post documents the exact compilation pipeline: cross-compiling llama.cpp and YOLOv11-Nano for ARM64 via NDK/CMake, building the JNI bridge that connects C++ inference to Kotlin UI, and the quantization decisions that let a 1.5B parameter model fit within 1.5 GB of working RAM.

The Hardware Constraint

Our target devices are entry-level Android phones: Qualcomm Snapdragon 4-series or MediaTek Helio G-series with 2–3 GB total RAM. After the OS, system services, and the app's Kotlin layer, we have roughly 1.5 GB available for model loading, KV-cache, and inference buffers. Thermal throttling kicks in after 15–20 seconds of sustained compute on these chips, so batch-heavy strategies that work on desktop fall apart on-device.

The dual-model requirement compounds this: we need both a vision model (pest detection from camera frames) and a language model (generating advisory text in Hindi) loaded concurrently. We cannot afford to unload and reload models per query — cold load is 348ms even with mmap, and users expect sub-second responses.

Cross-Compiling llama.cpp via NDK/CMake

llama.cpp is designed for desktop and server inference. Getting it to compile for ARM64 Android requires a CMake toolchain file that targets the NDK's clang cross-compiler. The key settings:

Two gotchas consumed hours of debugging:

The Quantization Pipeline: F32 → F16 → Q4_K_M

Getting the model small enough for a 2 GB device requires aggressive quantization, but the wrong path corrupts the vocabulary and produces garbage output. We learned this the hard way.

The correct pipeline for Qwen 2.5:

Why not Q8 → Q4? We initially tried quantizing F32 → Q8_0 → Q4_K_M. The resulting model loaded fine but generated std::length_error on the first decode call. The issue: Q8 quantization alters the embedding table layout, and re-quantizing from Q8 to Q4 corrupts the vocabulary metadata. The F32 → F16 → Q4_K_M path preserves the original token table intact.

Final model size: ~380 MB for 1.5B parameters at Q4_K_M. Loaded via mmap, this maps directly into the process address space — the kernel pages in blocks on demand, keeping resident memory well under the 1.5 GB ceiling.

The JNI Bridge: UTF-8 Devanagari Handling

The Android runtime is Kotlin/JVM. The inference engine is C++. The bridge is JNI. The critical subtlety: Devanagari text in llama.cpp tokens is raw UTF-8 bytes, not Modified UTF-8 that JNI's NewStringUTF() expects.

Hindi text contains 3-byte UTF-8 sequences (e.g., "ह" = 0xE0 0xA4 0xB9). The JNI NewStringUTF() function interprets these as Modified UTF-8 and occasionally misparses multi-byte sequences, producing garbled output or crashes.

The fix: bypass NewStringUTF() entirely. Copy the raw C bytes into a Java byte array, then construct the String on the JVM side:

This pattern handles all Unicode planes correctly — Devanagari, Latin, CJK, emoji — without the Modified UTF-8 ambiguity. Every token from the GGUF vocabulary decodes cleanly.

YOLOv11-Nano for Pest Detection

For real-time pest identification, we run an int8-quantized YOLOv11-Nano through the same NDK/CMake pipeline. The model detects Mustard Aphids, Fall Armyworm, and early leaf blights from the phone camera.

The vision model feeds into the same advisory pipeline: detected pest → ETL threshold check → biological or chemical prescription → dosage from verified registry. The camera replaces the farmer's verbal description — more accurate identification, faster response.

ChatML Tokenization: parse_special=true

Qwen 2.5 uses ChatML format with special tokens (<|im_start|>, <|im_end|>). If these are tokenized as regular text instead of special tokens, the model never recognizes the turn boundaries and generates incoherent output.

The fix is in the tokenize call: set parse_special = true and pass explicit strlen() for the text length parameter instead of -1. With -1, llama.cpp infers the length, but certain special token sequences contain null bytes that truncate the string prematurely.

Production Benchmarks

Key Takeaways

Quantize from F16, never from Q8. The F32→F16→Q4_K_M path preserves vocabulary metadata. Skipping to Q8 first corrupts the embedding table and produces length errors on decode.

Never use NewStringUTF for non-ASCII tokens. JNI's Modified UTF-8 is a landmine for any language outside ASCII. Copy raw bytes and decode on the JVM side.

Thermal throttling is the real bottleneck. On $100 phones, the CPU throttles after 15–20 seconds. Design your inference pipeline for short bursts — batch what you can, but keep individual generations under 10 seconds.

mmap is your memory strategy. Don't load the entire GGUF into heap. mmap lets the kernel page in model weights on demand and evict under memory pressure, keeping your process well under the OOM killer threshold.