Stage-level benchmarking of retrieval-augmented generation, and compression as a runtime control knob
A RAG pipeline is a chain of very different stages — retrieval, reranking, context compression, generation — but it is usually measured as a single black box. RAGMark (joint work with Zlatan Feric) benchmarks the pipeline the way a systems person would profile a program: attributing latency, GPU utilization, memory, power, and answer quality to each stage, while sweeping retrieval depths, model scales, reranking, compression, and vector-database configurations efficiently.
Key findings:
Autoregressive generation dominates naive pipelines, but context-reduction techniques shift the bottleneck across compute, memory bandwidth, and preprocessing stages.
Reranking and compression compound: reranking shrinks the compressor’s own workload, and together they cut prefill and KV-cache traversal costs — lowering energy by up to 66%.
Small upstream context reductions cascade through downstream latency, memory traffic, and energy.
The follow-on work turns compression into a runtime control knob for edge RAG: on the Jetson AGX Thor, intermediate compression rates reduce GPU energy by up to 53.2% with negligible quality loss — but the best rate shifts with workload and system state, motivating telemetry-informed adaptive control.
We present RAGMark, a modular benchmarking framework for advanced Retrieval-Augmented Generation (RAG) systems targeting small-scale multi-GPU environments. RAGMark evaluates diverse RAG components, including retrievers, vector databases, prompt-processing methods, and generator models, while collecting detailed per-stage metrics such as latency, GPU utilization, memory consumption, power usage, time to first token (TTFT), throughput, and answer quality. The framework is highly extensible, separating RAG stages, timing, and resource monitoring into modular components, and is designed to efficiently sweep large configuration spaces while minimizing repeated model and database initialization overhead. Using RAGMark, we characterize five RAG workloads on open-domain QA datasets across varying retrieval depths, model scales, reranking, compression methods, and vector database configurations. We show that while autoregressive generation dominates latency in naive pipelines, context-reduction techniques shift bottlenecks across compute, memory bandwidth, and preprocessing stages. Reranking and compression produce compounding benefits: reranking reduces compression workload itself, while both jointly reduce prefill and KV-cache traversal costs, lowering energy consumption by up to 66%. We further observe strong cross-stage interactions, where small upstream context reductions cascade through downstream latency, memory traffic, and energy consumption.
@inproceedings{feric2026ragmark,title={RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systems},author={Feric, Zlatan and Taherin, Amir and Ren, Bin and Wang, Yanzhi and Dy, Jennifer and Kaeli, David},booktitle={IEEE International Symposium on Workload Characterization (IISWC)},year={2026},}
Retrieval-augmented generation (RAG) improves language-model responses by grounding generation in external passages, which comes with overhead: retrieved context lengthens the prompt, increasing prefill work, KV-cache footprint, memory traffic, latency, and energy. Context compression offers a natural remedy by pruning retrieved text before generation. However, state-of-the-art context-compression methods are typically used with a fixed compression budget, or with the rate selected offline and then applied at inference time. This static view ignores both workload variation and the live state of the edge device. On an edge SoC, compression is not free: the compressor itself runs on the same SoC and consumes latency and energy that can offset any generation savings. This paper proposes a vision for telemetry-informed adaptive compression in edge RAG, grounded in experimental evidence. We characterize the compression tradeoff on the NVIDIA Jetson AGX Thor using Llama and Qwen generators, Natural Questions and HotpotQA datasets, and LLMLingua-2 compression. Our measurements show that generation dominates the RAG budget for larger models, reaching roughly 90% of per-query latency and 91% of GPU energy for 7B-8B generators. Exploring the impact of the compression rate reveals an adaptive operating region: mild compression can miss energy opportunities, and overly aggressive compression can hurt inference quality. Intermediate compression can reduce GPU energy by up to 53.2%, and SoC energy by up to 48.2%, with negligible quality loss. We argue for runtime policies that dynamically manage compression, guided by workload features and edge telemetry.
@inproceedings{feric2026retrieved,title={From Retrieved Context to Runtime Control: Adaptive Compression for Edge-based RAG},author={Feric, Zlatan and Taherin, Amir and Wang, Yanzhi and Kaeli, David},booktitle={Proceedings of the ACM AI Leadership Summit},year={2026},}