Phase-aware characterization of LLM inference across three generations of edge SoCs
Deploying a language model on an edge SoC means choosing a platform, an inference backend, and a quantization level — and the consequences of those choices are entangled. Aggregate metrics like tokens per second hide where time and energy actually go.
Hydra is a common-schema, phase-aware characterization framework: it instruments HuggingFace Transformers and llama.cpp with a shared per-prompt timing schema, fuses those records with hardware telemetry, and attributes latency, power, energy, and memory behavior to the prefill and decode phases of inference. Using Hydra we characterized 13 instruction-tuned LLMs (1B–8B) across five execution formats on three consecutive NVIDIA Jetson generations — AGX Xavier, AGX Orin, and AGX Thor — producing a corpus of roughly 107K per-prompt records that will be released with the tool.
Key findings:
A backend can win end-to-end while being slower per token: runtime structure (orchestration, de-tokenization) shifts where latency is introduced.
Quantization reduces memory traffic and energy, but bit-width does not predict power — Q6_K can draw more power than Q8_0.
Identical utilization readings mean different things across SoC generations because of DVFS behavior.
Deployability can be limited by allocator behavior rather than nominal memory capacity.
Edge LLM deployment is shaped by more than model size and precision: inference backend, hardware platform, memory traffic, and power management all affect latency and efficiency. We present Hydra, a common-schema, phase-aware workload characterization framework for LLM inference on edge SoCs. Hydra instruments HuggingFace Transformers and llama.cpp with a shared per-prompt timing schema and fuses those records with hardware telemetry, enabling a multi-dimensional characterization of performance, system-resource utilization, and efficiency across prefill and decode phases. Using Hydra, we evaluate three consecutive edge System-on-Chip (SoC) generations (AGX Xavier, AGX Orin, and AGX Thor), 13 instruction-tuned LLMs from seven families, five execution formats, and consider input/output-length sensitivity. The resulting artifact contains roughly 107K per-prompt records and will be released with Hydra. Our analysis shows that aggregate latency alone hides key deployment effects: backend structure changes where latency is introduced, quantization reduces memory traffic and energy but does not predict power monotonically, and SoC generation changes how utilization and efficiency should be interpreted. By connecting phase-level timing with system-resource utilization and efficiency metrics, Hydra enables reproducible, phase-aware characterization of edge LLM inference.
@inproceedings{taherin2026hydra,title={Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levels},author={Taherin, Amir and Taghipour Anvari, Sana and Amante, Charles and Chen, Yixiao and Noroian, Ruben and Feric, Zlatan and Bohm Agostini, Nicolas and Zhao, Pu and Cano, José and Ren, Bin and Wang, Yanzhi and Kaeli, David},booktitle={IEEE International Symposium on Workload Characterization (IISWC)},year={2026},}