Measuring where LLM and RAG inference spends time and energy on edge hardware — and adapting it at runtime
Goal: make large-model inference efficient and predictable on hardware with hard limits on power, memory, and heat. The recurring method: characterize phase by phase, attribute costs to where they arise, then turn what the measurements expose into runtime adaptation — always charging the adaptation’s own overhead against its savings.
Hydra — phase-aware LLM characterization
A common-schema characterization framework that instruments HuggingFace Transformers and llama.cpp with a shared per-prompt timing schema and fuses it with hardware telemetry. Applied to 13 instruction-tuned LLMs across five execution formats on three consecutive Jetson SoC generations (Xavier, Orin, Thor), yielding a ~107K-record corpus. Aggregate latency hides key effects: backend structure shifts where latency is introduced, quantization cuts memory traffic and energy but does not predict power monotonically, and utilization means different things on different SoC generations. To appear at IISWC 2026 (Taherin et al., 2026).
RAGMark — stage-level RAG benchmarking
Joint work with Zlatan Feric: benchmark the RAG pipeline the way a systems person profiles a program — attributing latency, GPU utilization, memory, power, and answer quality to retrieval, reranking, compression, and generation individually. Reranking and compression compound, cutting energy by up to 66%; small upstream context reductions cascade through downstream cost. To appear at IISWC 2026 (Feric et al., 2026).
Adaptive compression for edge RAG
Once stage-level costs are measurable, context compression stops being a preprocessing step and becomes a runtime control knob. On the Jetson AGX Thor, intermediate compression rates cut GPU energy by up to 53.2% with negligible quality loss — but the best rate moves with workload and system state, motivating telemetry-informed adaptive control. To appear in the ACM AI Leadership Summit proceedings (Feric* et al., 2026).
Edge LLM deployment is shaped by more than model size and precision: inference backend, hardware platform, memory traffic, and power management all affect latency and efficiency. We present Hydra, a common-schema, phase-aware workload characterization framework for LLM inference on edge SoCs. Hydra instruments HuggingFace Transformers and llama.cpp with a shared per-prompt timing schema and fuses those records with hardware telemetry, enabling a multi-dimensional characterization of performance, system-resource utilization, and efficiency across prefill and decode phases. Using Hydra, we evaluate three consecutive edge System-on-Chip (SoC) generations (AGX Xavier, AGX Orin, and AGX Thor), 13 instruction-tuned LLMs from seven families, five execution formats, and consider input/output-length sensitivity. The resulting artifact contains roughly 107K per-prompt records and will be released with Hydra. Our analysis shows that aggregate latency alone hides key deployment effects: backend structure changes where latency is introduced, quantization reduces memory traffic and energy but does not predict power monotonically, and SoC generation changes how utilization and efficiency should be interpreted. By connecting phase-level timing with system-resource utilization and efficiency metrics, Hydra enables reproducible, phase-aware characterization of edge LLM inference.
@inproceedings{taherin2026hydra,title={Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levels},author={Taherin, Amir and Taghipour Anvari, Sana and Amante, Charles and Chen, Yixiao and Noroian, Ruben and Feric, Zlatan and Bohm Agostini, Nicolas and Zhao, Pu and Cano, José and Ren, Bin and Wang, Yanzhi and Kaeli, David},booktitle={IEEE International Symposium on Workload Characterization (IISWC)},year={2026},}
We present RAGMark, a modular benchmarking framework for advanced Retrieval-Augmented Generation (RAG) systems targeting small-scale multi-GPU environments. RAGMark evaluates diverse RAG components, including retrievers, vector databases, prompt-processing methods, and generator models, while collecting detailed per-stage metrics such as latency, GPU utilization, memory consumption, power usage, time to first token (TTFT), throughput, and answer quality. The framework is highly extensible, separating RAG stages, timing, and resource monitoring into modular components, and is designed to efficiently sweep large configuration spaces while minimizing repeated model and database initialization overhead. Using RAGMark, we characterize five RAG workloads on open-domain QA datasets across varying retrieval depths, model scales, reranking, compression methods, and vector database configurations. We show that while autoregressive generation dominates latency in naive pipelines, context-reduction techniques shift bottlenecks across compute, memory bandwidth, and preprocessing stages. Reranking and compression produce compounding benefits: reranking reduces compression workload itself, while both jointly reduce prefill and KV-cache traversal costs, lowering energy consumption by up to 66%. We further observe strong cross-stage interactions, where small upstream context reductions cascade through downstream latency, memory traffic, and energy consumption.
@inproceedings{feric2026ragmark,title={RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systems},author={Feric, Zlatan and Taherin, Amir and Ren, Bin and Wang, Yanzhi and Dy, Jennifer and Kaeli, David},booktitle={IEEE International Symposium on Workload Characterization (IISWC)},year={2026},}
Retrieval-augmented generation (RAG) improves language-model responses by grounding generation in external passages, which comes with overhead: retrieved context lengthens the prompt, increasing prefill work, KV-cache footprint, memory traffic, latency, and energy. Context compression offers a natural remedy by pruning retrieved text before generation. However, state-of-the-art context-compression methods are typically used with a fixed compression budget, or with the rate selected offline and then applied at inference time. This static view ignores both workload variation and the live state of the edge device. On an edge SoC, compression is not free: the compressor itself runs on the same SoC and consumes latency and energy that can offset any generation savings. This paper proposes a vision for telemetry-informed adaptive compression in edge RAG, grounded in experimental evidence. We characterize the compression tradeoff on the NVIDIA Jetson AGX Thor using Llama and Qwen generators, Natural Questions and HotpotQA datasets, and LLMLingua-2 compression. Our measurements show that generation dominates the RAG budget for larger models, reaching roughly 90% of per-query latency and 91% of GPU energy for 7B-8B generators. Exploring the impact of the compression rate reveals an adaptive operating region: mild compression can miss energy opportunities, and overly aggressive compression can hurt inference quality. Intermediate compression can reduce GPU energy by up to 53.2%, and SoC energy by up to 48.2%, with negligible quality loss. We argue for runtime policies that dynamically manage compression, guided by workload features and edge telemetry.
@inproceedings{feric2026retrieved,title={From Retrieved Context to Runtime Control: Adaptive Compression for Edge-based RAG},author={Feric, Zlatan and Taherin, Amir and Wang, Yanzhi and Kaeli, David},booktitle={Proceedings of the ACM AI Leadership Summit},year={2026},}