Validating Distributed LLM Serving Benchmarks with NVIDIA srt-slurm, SLURM Recipes, Parameter Sweeps, and Pareto Analysis
AI
This tutorial explores NVIDIA's srt-slurm framework for validating distributed LLM serving benchmarks using SLURM recipes, parameter sweeps, and Pareto analysis in a Colab environment.
Intelligence Insights
Context + impact, normalized for TechCulture.
The Big Picture
The article presents a hands-on tutorial on NVIDIA's srt-slurm framework, which converts declarative YAML configurations into reproducible SLURM benchmark workflows for distributed LLM serving. Using Google Colab as a development workspace, the tutorial covers repository architecture, cluster configuration, dry-running built-in and custom recipes (including a disaggregated prefill-and-decode setup for DeepSeek-R1), generating parameter sweeps, and programmatic use of the Python API. It also simulates benchmark results to analyze throughput-versus-latency Pareto frontiers. The tutorial emphasizes that while Colab lacks a real SLURM environment, it serves as a practical space to validate and prepare production-grade recipes before deployment on actual GPU clusters. The workflow includes steps for preflight checks, job submission, monitoring, and reproducibility through identity blocks in recipes.
Why It Matters
This tutorial provides a practical framework for validating distributed LLM serving benchmarks before committing expensive GPU cluster resources, reducing the risk of misconfiguration and wasted compute. By enabling reproducible, parameterized experiments with tools like srt-slurm and Pareto analysis, it helps teams optimize throughput-latency trade-offs for large models like DeepSeek-R1, which is critical as disaggregated serving architectures become more common in production AI deployments.
Deepen your understanding
Use our AI to break down complex signals.
Select an AI action to generate more depth.
In this tutorial, we exploreNVIDIA’s srt-slurmframework and learn how we use srtctl to convert declarative YAML configurations into reproducible SLURM benchmark workflows for distributed LLM serving. We set up the project in Google Colab, inspect its internal architecture, define a cluster configuration, dry-run built-in and custom recipes, and model a disaggregated prefill-and-decode deployment for DeepSeek-R1. We also generate parameter sweeps, interact with the typed Python API, validate expanded configurations, and analyze simulated benchmark results through a throughput-versus-latency Pareto frontier. Although Colab does not provide a real SLURM environment, we use it as a practical development workspace to understand, validate, and prepare production-grade benchmark recipes before we submit them to an actual GPU cluster.
import os, sys, subprocess, textwrap, json, shutil, importlib
from pathlib import Path
def run(cmd, check=True, quiet=False):
"""Run a shell command, stream output."""
print(f"\n$ {cmd}")
r = subprocess.run(cmd, shell=True, text=True, capture_output=True)
out = (r.stdout or "") + (r.stderr or "")
if not quiet:
print(out[-6000:])
if check and r.returncode != 0:
raise RuntimeError(f"Command failed ({r.returncode}): {cmd}")
return out
def section(title):
print("\n" + "═"*78 + f"\n {title}\n" + "═"*78)
section("1. Install srt-slurm")
REPO = Path("/content/srt-slurm") if Path("/content").exists() else Path.cwd()/"srt-slurm"
if not REPO.exists():
run(f"git clone --depth 1 https://github.com/NVIDIA/srt-slurm.git {REPO}", quiet=True)
run(f"{sys.executable} -m pip install -q -e {REPO}", quiet=True)
sys.path.insert(0, str(REPO / "src"))
importlib.invalidate_caches()
os.chdir(REPO)
run("srtctl --help")
We prepare the Colab environment by importing the required modules and defining reusable helper functions for command execution and section formatting. We clone the NVIDIA srt-slurm repository, install it in editable mode, and expose its source directory to the active Python runtime. We then switch to the repository directory and verify that the srtctl command-line interface is installed correctly.
We inspect the repository structure to understand how srtctl organizes its command-line tools, schemas, backends, templates, recipes, and analysis components. We then create a local srtslurm.yaml file containing simulated cluster defaults, container aliases, GPU settings, and model paths. We use this configuration to resolve recipe references in Colab without requiring access to an actual SLURM cluster.
We dry-run the built-in mocker recipe to examine how srtctl validates configurations and generates SLURM submission artifacts without executing a real benchmark. We then define an advanced DeepSeek-R1 recipe that separates prefill and decode workloads across independent node and worker pools. We validate this disaggregated SGLang configuration through another dry run and inspect how the serving parameters are translated into job scripts.
section("6. Parameter sweep (grid search) — dry-run + expansion on disk")
run("srtctl dry-run -f examples/example-sweep.yaml", check=False)
sweep_dirs = sorted((REPO/"dry-runs").glob("example-sweep_sweep_*"))
if sweep_dirs:
latest = sweep_dirs[-1]
print("Per-job configs generated by the sweep expander:")
for p in sorted(latest.rglob("config.yaml")):
print(" ", p.relative_to(REPO))
section("7. Programmatic use of srtctl's Python API")
import yaml
from srtctl.core.config import load_config
from srtctl.core.sweep import generate_sweep_configs, expand_template
from srtctl.core.schema import BenchmarkType, Precision, GpuType
cfg = load_config("my-disagg.yaml")
print(f"Loaded : {cfg.name}")
print(f"Model : {cfg.model.path} ({cfg.model.precision}) on {cfg.resources.gpu_type}")
print(f"Layout : {cfg.resources.prefill_nodes}P + {cfg.resources.decode_nodes}D nodes, "
f"{cfg.resources.gpus_per_node} GPUs/node")
print(f"Bench : {cfg.benchmark.type} isl={cfg.benchmark.isl} osl={cfg.benchmark.osl} "
f"concurrencies={cfg.benchmark.concurrencies}")
print(f"Enums : benchmarks={[b.value for b in BenchmarkType]}")
print(f" precisions={[p.value for p in Precision]}, gpus={[g.value for g in GpuType]}")
raw_sweep = yaml.safe_load(Path("examples/example-sweep.yaml").read_text())
jobs = generate_sweep_configs(raw_sweep)
print(f"\nSweep expands to {len(jobs)} jobs:")
for job_cfg, params in jobs:
pf = job_cfg["backend"]["sglang_config"]["prefill"]
print(f" {params} → chunked-prefill-size={pf['chunked-prefill-size']}, "
f"max-total-tokens={pf['max-total-tokens']}")
print("\nTemplate substitution:",
expand_template({"flag": "{x}", "n": "{y}"}, {"x": 4096, "y": 2}))
We execute the example parameter sweep and inspect the individual job configurations created from its Cartesian search space. We load our custom recipe through the typed Python API and examine its model, precision, GPU topology, benchmark settings, and supported enumeration values. We also programmatically expand sweep templates and verify how each parameter combination affects the generated backend configuration.
section("8. Analysis: Pareto frontier from (simulated) benchmark results")
import numpy as np, matplotlib.pyplot as plt
rng = np.random.default_rng(0)
def simulate(variant, base_tps, base_itl):
rows = []
tps_gpu = base_tps * c / (c + 90) * rng.uniform(.97, 1.03)
itl = base_itl * (1 + c/220) * rng.uniform(.97, 1.03)
rows.append({"variant": variant, "concurrency": c,
"tok_s_gpu": tps_gpu, "itl_ms": itl})
return rows
results = simulate("chunked=4096", 260, 9.5) + simulate("chunked=8192", 300, 11.5)
print(json.dumps(results[:3], indent=2), "...")
plt.figure(figsize=(8, 5))
for variant in ("chunked=4096", "chunked=8192"):
pts = [(r["itl_ms"], r["tok_s_gpu"], r["concurrency"])
for r in results if r["variant"] == variant]
xs, ys, cs = zip(*pts)
plt.plot(xs, ys, "o-", label=variant)
for x, y, c in pts:
plt.annotate(str(c), (x, y), fontsize=7, xytext=(3, 3),
textcoords="offset points")
plt.xlabel("Inter-token latency (ms/token) → worse")
plt.ylabel("Throughput (tokens/s/GPU) → better")
plt.title("Pareto frontier: sweep variants (points labeled by concurrency)")
plt.legend(); plt.grid(alpha=.3); plt.tight_layout(); plt.show()
We simulate benchmark observations for two chunked-prefill variants across increasing concurrency levels. We calculate representative throughput per GPU and inter-token latency values to model the saturation and latency growth commonly observed in distributed serving systems. We then visualize these results in a Pareto-style plot to compare throughput and responsiveness across configurations.
section("9. Next steps on a real cluster")
print(textwrap.dedent("""\
make setup ARCH=aarch64|x86_64
srtctl preflight -f my-disagg.yaml
srtctl apply -f my-disagg.yaml
srtctl apply -f sweep.yaml
srtctl monitor
uv run streamlit run analysis/dashboard/app.py
srtctl diff runA runB
Logs land in outputs/{JOB_ID}/logs/; analysis/srtlog parses them
(NodeAnalyzer, RunLoader) for programmatic post-processing.
Reproducibility tip (srtctl also prints this in dry-run): add an
identity: block to your recipe — HF model repo + revision, container
image URI, and framework versions — so srtctl can verify the runtime
matches your declaration at job start.
"""))
print("✅ Tutorial complete.")
We outline the production workflow that we follow when transferring validated recipes from Colab to a real SLURM-based GPU cluster. We review the commands used for environment setup, preflight validation, job submission, sweep execution, monitoring, dashboard analysis, and experiment comparison. We also emphasize reproducibility by declaring model revisions, container identities, and framework versions inside each benchmark recipe.
In conclusion, we built a complete understanding of how srtctl structures, validates, expands, and prepares distributed LLM-serving benchmarks for execution on SLURM infrastructure. We moved from basic installation and repository inspection to advanced disaggregated serving configurations, Cartesian parameter sweeps, programmatic schema access, and benchmark-result analysis. We also saw how dry runs expose generated job artifacts and help us detect configuration problems before expensive cluster resources are allocated. With this workflow in place, we can confidently transfer our validated recipes to a real NVIDIA GPU cluster, run preflight checks, submit benchmark jobs, monitor execution, compare experiment fingerprints, and analyze performance trade-offs in a reproducible manner.
