RAGMark (IISWC 2026)
2026
- IISWCZlatan Feric, Amir Taherin, Bin Ren, and 3 more authorsIn IEEE International Symposium on Workload Characterization (IISWC), 2026. To appear.
We present RAGMark, a modular benchmarking framework for advanced Retrieval-Augmented Generation (RAG) systems targeting small-scale multi-GPU environments. RAGMark evaluates diverse RAG components, including retrievers, vector databases, prompt-processing methods, and generator models, while collecting detailed per-stage metrics such as latency, GPU utilization, memory consumption, power usage, time to first token (TTFT), throughput, and answer quality. The framework is highly extensible, separating RAG stages, timing, and resource monitoring into modular components, and is designed to efficiently sweep large configuration spaces while minimizing repeated model and database initialization overhead. Using RAGMark, we characterize five RAG workloads on open-domain QA datasets across varying retrieval depths, model scales, reranking, compression methods, and vector database configurations. We show that while autoregressive generation dominates latency in naive pipelines, context-reduction techniques shift bottlenecks across compute, memory bandwidth, and preprocessing stages. Reranking and compression produce compounding benefits: reranking reduces compression workload itself, while both jointly reduce prefill and KV-cache traversal costs, lowering energy consumption by up to 66%. We further observe strong cross-stage interactions, where small upstream context reductions cascade through downstream latency, memory traffic, and energy consumption.
@inproceedings{feric2026ragmark, title = {RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systems}, author = {Feric, Zlatan and Taherin, Amir and Ren, Bin and Wang, Yanzhi and Dy, Jennifer and Kaeli, David}, booktitle = {IEEE International Symposium on Workload Characterization (IISWC)}, year = {2026}, }