Inside vLLM: Anatomy of a High-Throughput LLM Inference System

Inside vLLM: Anatomy of a High-Throughput LLM Inference System

Source: Aleksa Gordíc, August 29, 2025

This post walks through the core system components and advanced features that make up a modern high-throughput LLM inference system, using vLLM as the concrete reference implementation.

It is the first post in a series. It starts broad and then layers in detail following an inverse-pyramid approach, so that you can form an accurate high-level mental model of the complete system without drowning in minutiae. Later posts zoom into individual subsystems.

The material is structured into five parts:

1. LLM engine & engine core — the fundamentals of vLLM: scheduling, paged attention, continuous batching.
2. Advanced features — chunked prefill, prefix caching, guided and speculative decoding, disaggregated prefill/decode.
3. Scaling up — from single-GPU to multi-GPU execution.
4. Serving layer — the distributed, concurrent web scaffolding.
5. Benchmarks and auto-tuning — measuring latency and throughput.

Notes on scope. The analysis is based on commit 42172ad (August 9, 2025) and focuses on the V1 engine; V0 is now deprecated, although exploring it is still valuable for understanding how the project evolved, and many concepts carry over. Because class names and signatures may shift, the emphasis here is on core ideas rather than exact APIs. Target audience: anyone curious about how state-of-the-art LLM engines work, and anyone interested in contributing to vLLM, SGLang, or similar projects.

LLM Engine & Engine Core

The LLM engine is the fundamental building block of vLLM. On its own it already enables high-throughput inference, but only in an offline setting: you cannot serve it to customers over the web yet.

The running example is this offline inference snippet, adapted from basic.py:

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

def main():
    llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")

    outputs = llm.generate(prompts, sampling_params)

if __name__ == "__main__":
    main()

Two environment variables pin the configuration:

VLLM_USE_V1="1"                    # use engine V1
VLLM_ENABLE_V1_MULTIPROCESSING="0" # run in a single process

With those settings the engine is offline (no web or distributed scaffolding), synchronous (all execution happens in a single blocking process), single-GPU (no data, tensor, pipeline, or expert parallelism; DP/TP/PP/EP all equal 1), and it runs a standard transformer. Hybrid models such as Jamba require a more complex hybrid KV-cache memory allocator. From this baseline the post gradually builds up to an online, asynchronous, multi-GPU, multi-node inference system that still serves a standard transformer.

The example does exactly two things: instantiate an engine, and call generate on it to sample from the given prompts. Let us start with the constructor.

LLM Engine Constructor

The main components of the engine are:

• vLLM config — all of the knobs for configuring the model, cache, parallelism, and so on.
• Processor — turns raw inputs into EngineCoreRequest objects through validation, tokenization, and processing.
• Engine core client — in this example an InprocClient, which is essentially identical to EngineCore; later it grows into DPLBAsyncMPClient, the client that allows serving at scale.
• Output processor — converts raw EngineCoreOutputs into the RequestOutput the user sees.

The engine core itself is made up of several sub-components:

• Model executor — drives forward passes on the model. Here it is a UniProcExecutor with a single worker process on a single GPU; it later grows into MultiProcExecutor, which supports multiple GPUs.
• Structured output manager — used for guided decoding, covered below.
• Scheduler — decides which requests go into the next engine step. It further contains the policy setting (either FCFS, first come first served, or priority, where higher-priority requests are served first), the waiting and running queues, and the KV-cache manager, which is the heart of paged attention.

The KV-cache manager maintains a free_block_queue: a pool of available KV-cache blocks, often on the order of hundreds of thousands depending on VRAM size and block size. During paged attention these blocks serve as the indexing structure that maps tokens to their computed KV-cache blocks.

For a standard transformer layer (non-MLA), the per-block byte cost of the KV cache is computed as follows:

2 (key/value) * block_size (default=16) * num_kv_heads * head_size * dtype_num_bytes (e.g. 2 for bf16)

During model-executor construction a Worker object is created and three key procedures run. Later, with MultiProcExecutor, these same procedures run independently in each worker process across different GPUs.

1. Init device. Assign a CUDA device (for example cuda:0) to the worker and check that the model dtype is supported (for example bf16). Verify that enough VRAM is available given the requested gpu_memory_utilization (0.8 means 80% of total VRAM). Set up distributed settings (DP / TP / PP / EP). Instantiate a model_runner, which holds the sampler, the KV cache, and forward-pass buffers such as input_ids and positions. Instantiate an InputBatch object, which holds CPU-side forward-pass buffers, block tables for KV-cache indexing, and sampling metadata.

2. Load model. Instantiate the model architecture, load the weights, call model.eval() (PyTorch inference mode), and optionally call torch.compile() on the model.

3. Initialize KV cache. Get the per-layer KV-cache spec. Historically this was always FullAttentionSpec for a homogeneous transformer, but hybrid models (sliding window, transformer/SSM mixtures such as Jamba) made it more complex; see Jenga. Then run a dummy profiling forward pass and take a GPU memory snapshot to compute how many KV-cache blocks fit in the available VRAM; allocate, reshape, and bind the KV-cache tensors to attention layers; and prepare attention metadata (for example, set the backend to FlashAttention) that kernels consume during the forward pass. Unless --enforce-eager is passed, the engine does a dummy run for each warmup batch size and captures CUDA graphs. CUDA graphs record the whole sequence of GPU work into a DAG, so later forward passes replay pre-baked graphs, cutting kernel-launch overhead and improving latency.

Many low-level details are abstracted away here, but these are the pieces the rest of the post refers to repeatedly.

The generate Function

The first step is to validate requests and feed them into the engine. For each prompt vLLM:

• Creates a unique request ID and captures its arrival time.
• Calls an input preprocessor that tokenizes the prompt and returns a dictionary containing prompt, prompt_token_ids, and a type (text, tokens, embeds, and so on).
• Packs this information into an EngineCoreRequest, adding priority, sampling params, and other metadata.
• Passes the request into the engine core, which wraps it in a Request object, sets its status to WAITING, and adds it to the scheduler's waiting queue (appended for FCFS, heap-pushed for priority).

At this point the engine has been fed and execution can begin. In the synchronous example these initial prompts are the only ones processed: there is no mechanism to inject new requests mid-run. The asynchronous engine does support that, which is what continuous batching means — after each step, both new and old requests are considered. Because the forward pass flattens the batch into a single sequence and custom kernels handle it efficiently, continuous batching is fundamentally supported even in the synchronous engine.

Next, as long as there are requests to process, the engine repeatedly calls its step() function. Each step has three stages:

• Schedule — select which requests run in this step (decode, and/or (chunked) prefill).
• Forward pass — run the model and sample tokens.
• Postprocess — append sampled token IDs to each Request, detokenize, and check stop conditions. If a request is finished, clean up (for example return its KV-cache blocks to free_block_queue) and return the output early.

Stop conditions. A request stops when it exceeds its length limit (max_model_length or its own max_tokens); when the sampled token is the EOS ID (unless ignore_eos is enabled, which is useful for benchmarking when you want to force generation of a fixed number of output tokens); when the sampled token matches any of the stop_token_ids in the sampling parameters; or when a stop string appears in the output, in which case the output is truncated at the first stop string and the request is aborted in the engine. Note that stop_token_ids remain present in the output while stop strings do not.

In streaming mode intermediate tokens would be sent as they are generated; that is set aside for now.

Scheduler

An inference engine handles two main types of workload:

• Prefill requests — a forward pass over all prompt tokens. These are usually compute-bound (the threshold depends on hardware and prompt length). At the end, a single token is sampled from the probability distribution of the final token position.
• Decode requests — a forward pass over just the most recent token, since all earlier KV vectors are already cached. These are memory-bandwidth-bound, because the engine still has to load all LLM weights (and KV caches) to compute one token.

The benchmarking section below analyzes the roofline model of GPU performance, which explains these prefill and decode profiles in more detail. The V1 scheduler can mix both request types in the same step thanks to smarter design choices; the V0 engine could only process either prefill or decode at once.

The scheduler prioritizes decode requests, that is, those already in the running queue. For each such request it computes the number of new tokens to generate (not always 1, because of speculative decoding and async scheduling), calls the KV-cache manager's allocate_slots function, and updates the token budget by subtracting that count. After that it processes prefill requests from the waiting queue: it retrieves the number of computed blocks (0 if prefix caching is disabled), calls allocate_slots, pops the request from waiting and moves it to running with status RUNNING, and updates the token budget.

allocate_slots itself does three things:

• Computes the number of blocks — determines how many new KV-cache blocks (n) must be allocated. Each block stores 16 tokens by default, so a prefill request with 17 new tokens needs ceil(17/16) = 2 blocks.
• Checks availability — if the manager's pool has too few blocks, it exits early. Depending on whether the request is decode or prefill, the engine may attempt recompute preemption (swap preemption existed in V0) by evicting low-priority requests, calling kv_cache_manager.free to return their blocks to the pool, or it may skip scheduling and continue execution.
• Allocates blocks — through the KV-cache manager's coordinator it fetches the first n blocks from the pool (the free_block_queue doubly linked list) and stores them in req_to_blocks, the dictionary mapping each request_id to its list of KV-cache blocks.

Run Forward Pass

The engine calls the model executor's execute_model, which delegates to the Worker, which in turn delegates to the model runner. The main steps are:

• Update states — prune finished requests from input_batch and update forward-pass metadata, such as the KV-cache blocks per request that index into paged KV-cache memory.
• Prepare inputs — copy buffers from CPU to GPU, compute positions, build slot_mapping, and construct attention metadata.
• Forward pass — run the model with custom paged-attention kernels. All sequences are flattened and concatenated into one long "super sequence". Position indices and attention masks ensure each sequence attends only to its own tokens, which is what enables continuous batching without right-padding.
• Gather last-token states — extract the hidden state at each sequence's final position and compute logits.
• Sample — sample tokens from the computed logits as dictated by the sampling config (greedy, temperature, top-p, top-k, and so on).

The forward-pass step has two execution modes: eager mode, which runs the standard PyTorch forward pass when eager execution is enabled, and "captured" mode, which replays a pre-captured CUDA graph when eager is not enforced (those graphs were captured during engine construction, in the initialize-KV-cache procedure).

Advanced Features: Extending the Core Engine Logic

With the basic engine flow in place, the advanced features become much easier to place. Preemption, paged attention, and continuous batching have already been covered. What follows is chunked prefill, prefix caching, guided decoding through grammar-constrained finite-state machines, speculative decoding, and disaggregated prefill/decode.

Chunked prefill

Chunked prefill handles long prompts by splitting their prefill step into smaller chunks. Without it, a single very long request can monopolize one engine step and prevent other prefill requests from running, postponing them and increasing their latency.

Take a concrete example where each chunk contains n (=8) tokens, labeled with lowercase letters separated by hyphens. A long prompt P could look like x-y-z, where z is an incomplete chunk (say 2 tokens). Executing the full prefill for P then takes at least 3 engine steps (it can take more if it is not scheduled for execution in one of the steps), and only in the last chunked prefill step does the engine sample one new token.

The implementation is straightforward: cap the number of new tokens per step. If the requested number exceeds long_prefill_token_threshold, reset it to exactly that value. The underlying indexing logic described earlier takes care of the rest. In vLLM V1 you enable chunked prefill by setting long_prefill_token_threshold to a positive integer. Technically it can also happen irrespective of that setting: if the prompt length exceeds the token budget, the engine truncates it and runs a chunked prefill.

Prefix caching

To explain prefix caching, tweak the original code example slightly:

from vllm import LLM, SamplingParams

long_prefix = "<a piece of text that is encoded into more than block_size tokens>"

prompts = [
    "Hello, my name is",
    "The president of the United States is",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

def main():
    llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")

    outputs = llm.generate(long_prefix + prompts[0], sampling_params)
    outputs = llm.generate(long_prefix + prompts[1], sampling_params)

if __name__ == "__main__":
    main()

Prefix caching avoids recomputing tokens that multiple prompts share at the beginning — hence prefix. The crucial piece is long_prefix, defined as any prefix longer than a KV-cache block (16 tokens by default). To simplify, assume long_prefix has exactly length n x block_size with n ≥ 1, so it aligns perfectly with block boundaries; otherwise the engine would have to recompute long_prefix_len % block_size tokens, because incomplete blocks cannot be cached.

Without prefix caching, every new request carrying the same long_prefix recomputes all n x block_size tokens. With prefix caching those tokens are computed once, their KVs are stored in paged KV-cache memory, and later requests reuse them, so only the new prompt tokens need processing. This speeds up prefill requests; it does not help decode.

How does this work in vLLM? During the first generate call, in the scheduling stage, kv_cache_manager.get_computed_blocks invokes hash_request_tokens:

• The function splits long_prefix + prompts[0] into 16-token chunks.
• For each complete chunk it computes a hash, using either the built-in hash or SHA-256, which is slower but has fewer collisions. The hash combines the previous block's hash, the current tokens, and optional metadata.
• Optional metadata includes the multimodal hash, the LoRA ID, and the cache salt. A cache salt injected into the hash of the first block ensures that only requests carrying the same salt can reuse those blocks.
• Each result is stored as a BlockHash object containing both the hash and its token IDs, and the function returns a list of block hashes that is kept in self.req_to_block_hashes[request_id].

Next the engine calls find_longest_cache_hit to check whether any of these hashes already exist in cached_block_hash_to_block. On the first request, no hits are found.

The engine then calls allocate_slots, which calls coordinator.cache_blocks. That associates the new BlockHash entries with the allocated KV blocks and records them in cached_block_hash_to_block. Afterwards the forward pass populates the KVs in paged KV-cache memory for the blocks allocated above.

After many engine steps the request allocates more KV-cache blocks, but that does not matter for this example, because the prefix diverges immediately after long_prefix.

On a second generate call with the same prefix, the same steps repeat, but now find_longest_cache_hit finds matches for all n blocks (via linear search) and the engine reuses those KV blocks directly.

If the original request were still alive, the reference count for those blocks would increment, say to 2. In this example the first request has already completed, so the blocks were freed back to the pool and their reference counts reset to 0. Because they could still be retrieved from cached_block_hash_to_block, the KV-cache manager knows they are valid, so it simply removes them from free_block_queue again.

Advanced note. KV-cache blocks become invalid only when they are about to be reallocated from free_block_queue (which pops from the left) and the engine discovers that the block still has an associated hash and is present in cached_block_hash_to_block. At that moment it clears the block's hash and removes the entry from cached_block_hash_to_block, ensuring the block can no longer be reused through prefix caching, at least not for that old prefix.

That is the gist of prefix caching: do not recompute prefixes you have already seen, just reuse their KV cache. And if you understood this example you also understood how paged attention works. Prefix caching is enabled by default; disable it with enable_prefix_caching = False.

Guided decoding (FSM)

Guided decoding constrains the logits at each decoding step with a grammar-based finite-state machine, so only tokens allowed by the grammar can be sampled. It is a powerful setup: you can enforce anything from regular grammars (Chomsky type-3, such as arbitrary regex patterns) up to context-free grammars (type-2, which cover most programming languages).

The simplest possible example builds on the earlier code:

from vllm import LLM, SamplingParams
from vllm.sampling_params import GuidedDecodingParams

prompts = [
    "This sucks",
    "The weather is beautiful",
]

guided_decoding_params = GuidedDecodingParams(choice=["Positive", "Negative"])
sampling_params = SamplingParams(guided_decoding=guided_decoding_params)

def main():
    llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")

    outputs = llm.generate(prompts, sampling_params)

if __name__ == "__main__":
    main()

In this toy example (assume character-level tokenization), at prefill the FSM masks logits so only "P" or "N" are viable. If "P" is sampled, the FSM moves to the "Positive" branch; at the next step only "o" is allowed, and so on.

Inside vLLM this works as follows:

1. At LLM engine construction a StructuredOutputManager is created; it has access to the tokenizer and maintains a _grammar_bitmask tensor.
2. When a request is added, its status is set to WAITING_FOR_FSM and grammar_init selects the backend compiler, for example xgrammar. These backends are third-party code.
3. The grammar for that request is compiled asynchronously.
4. During scheduling, if the async compile has finished, the status switches to WAITING and the request_id is added to structured_output_request_ids; otherwise the request lands in skipped_waiting_requests to retry on the next engine step.
5. After the scheduling loop, still inside scheduling, if there are FSM requests the StructuredOutputManager asks the backend to prepare or update _grammar_bitmask.
6. After the forward pass produces logits, xgrammar's torch-compiled function expands the bitmask to vocab size (a 32x expansion ratio, because 32-bit integers are used) and masks disallowed logits to –∞.
7. After sampling the next token, the request's FSM advances via accept_tokens. Visually, the machine moves to the next state on the FSM diagram.

Step 6 deserves more detail. If vocab_size = 32, _grammar_bitmask is a single integer whose binary representation encodes which tokens are allowed ("1") versus disallowed ("0"). For example, "101…001" expands to the length-32 array [1, 0, 1, …, 0, 0, 1], and positions with 0 get their logits set to –∞. For larger vocabularies multiple 32-bit words are used and expanded or concatenated accordingly. The backend, xgrammar in this case, is responsible for producing these bit patterns from the current FSM state.

Note. Most of the complexity here is hidden in third-party libraries such as xgrammar.

Here is an even simpler example with vocab_size = 8 and 8-bit integers:

You enable all of this in vLLM by passing in the desired guided_decoding config.

Speculative decoding

In autoregressive generation each new token requires a forward pass of the large LM. That is expensive: every step reloads and applies all model weights just to compute a single token (assuming batch size 1; in general it is B).

Speculative decoding speeds this up by introducing a smaller draft LM. The draft proposes k tokens cheaply. The goal is never to sample from the smaller model — it is only there to guess candidate continuations, while the large model still decides what is valid. The steps are:

• Draft. Run the small model on the current context and propose k tokens.
• Verify. Run the large model once on context + k draft tokens. This produces probabilities for those k positions plus one extra, so there are k+1 candidates.
• Accept/reject. Going left to right over the k draft tokens: if the large model's probability for the draft token is at least the draft's probability, accept it; otherwise accept it with probability p_large(token)/p_draft(token). Stop at the first rejection, or accept all k draft tokens.
• If all k draft tokens are accepted, also sample the extra (k+1)-th token "for free" from the large model, since that distribution has already been computed.
• If there was a rejection, create a new rebalanced distribution at that position (p_large - p_draft, clamped at a minimum of 0, normalized to sum to 1) and sample the last token from it.

Why this works. Although the small model proposes candidates, the accept/reject rule guarantees that in expectation the sequence is distributed exactly as if it had been sampled token by token from the large model. Speculative decoding is therefore statistically equivalent to standard autoregressive decoding, but potentially much faster, since a single large-model pass can yield up to k+1 tokens.

Note. gpt-fast is a good place to see a simple implementation, and the original paper covers the math and the proof of equivalence to sampling from the full model.

vLLM V1 does not support the LLM-draft-model method. Instead it implements faster but less accurate proposal schemes: n-gram, EAGLE, and Medusa. One-liners on each:

• n-gram — take the last prompt_lookup_max tokens, find a prior match in the sequence, and if found propose the k tokens that followed that match; otherwise decrement the window and retry down to prompt_lookup_min. The current implementation returns the k tokens after the first match. It feels more natural to introduce a recency bias and reverse the search direction, that is, use the last match.
• EAGLE — perform "model surgery" on the large LM: keep the embeddings and the LM head, replace the transformer stack with a lightweight MLP, and fine-tune that as a cheap draft.
• Medusa — train auxiliary linear heads on top of the large model (on the embeddings before the LM head) to predict the next k tokens in parallel, and use those heads to propose tokens more efficiently than running a separate small LM.

Here is how to invoke speculative decoding in vLLM with ngram as the draft method:

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

speculative_config={
    "method": "ngram",
    "prompt_lookup_max": 5,
    "prompt_lookup_min": 3,
    "num_speculative_tokens": 3,
}

def main():
    llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0", speculative_config=speculative_config)

    outputs = llm.generate(prompts, sampling_params)

if __name__ == "__main__":
    main()

Inside vLLM, setup happens during engine construction: init device creates a drafter (the draft model, for example NgramProposer) and a rejection_sampler (parts of which are written in Triton), and load model loads the draft model weights (a no-op for n-gram).

Then, in the generate function, assuming a brand-new request:

1. Run the regular prefill step with the large model.
2. After the forward pass and standard sampling, call propose_draft_token_ids(k) to sample k draft tokens from the draft model.
3. Store these in request.spec_token_ids, updating the request metadata.
4. On the next engine step, when the request is in the running queue, add len(request.spec_token_ids) to the "new tokens" count so allocate_slots reserves enough KV blocks for the forward pass.
5. Copy spec_token_ids into input_batch.token_ids_cpu to form the (context + draft) tokens.
6. Compute metadata via _calc_spec_decode_metadata, which copies tokens from input_batch.token_ids_cpu and prepares logits, then run a large-model forward pass over the draft tokens.
7. Instead of regular sampling from logits, use the rejection_sampler to accept or reject left to right and produce output_token_ids.
8. Repeat steps 2 to 7 until a stop condition is met.

The best way to internalize this is to fire up a debugger and step through the codebase, but the two diagrams below give a taste of it.

Disaggregated prefill/decode

The motivation for disaggregated P/D has already been hinted at. Prefill and decode have very different performance profiles, compute-bound versus memory-bandwidth-bound, so separating their execution is a sensible design. It gives tighter control over latency, both TTFT (time to first token) and ITL (inter-token latency), which the benchmarking section below picks up again.

In practice you run N vLLM prefill instances and M vLLM decode instances, autoscaling them based on the live request mix. Prefill workers write KV to a dedicated KV-cache service and decode workers read from it. This isolates long, bursty prefill from steady, latency-sensitive decode.

For clarity the example below relies on SharedStorageConnector, a debugging connector implementation used to illustrate the mechanics. Connector is vLLM's abstraction for handling the exchange of KVs between instances; the connector interface is not yet stable, and near-term improvements are planned, some potentially breaking.

The setup launches two vLLM instances, GPU 0 for prefill and GPU 1 for decode, and transfers the KV cache between them:

import os
import time
from multiprocessing import Event, Process
import multiprocessing as mp

from vllm import LLM, SamplingParams
from vllm.config import KVTransferConfig

prompts = [
    "Hello, my name is",
    "The president of the United States is",
]

def run_prefill(prefill_done):
  os.environ["CUDA_VISIBLE_DEVICES"] = "0"

  sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=1)

  ktc=KVTransferConfig(
      kv_connector="SharedStorageConnector",
      kv_role="kv_both",
      kv_connector_extra_config={"shared_storage_path": "local_storage"},
  )

  llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0", kv_transfer_config=ktc)
  llm.generate(prompts, sampling_params)

  prefill_done.set()  # notify decode instance that KV cache is ready

  # Keep the prefill node running in case the decode node is not done;
  # otherwise the script might exit prematurely, causing incomplete decoding.
  try:
      while True:
          time.sleep(1)
  except KeyboardInterrupt:
      print("Script stopped by user.")

def run_decode(prefill_done):
  os.environ["CUDA_VISIBLE_DEVICES"] = "1"

  sampling_params = SamplingParams(temperature=0, top_p=0.95)

  ktc=KVTransferConfig(
      kv_connector="SharedStorageConnector",
      kv_role="kv_both",
      kv_connector_extra_config={"shared_storage_path": "local_storage"},
  )

  llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0", kv_transfer_config=ktc)

  prefill_done.wait()  # block waiting for KV cache from prefill instance

  # Internally it will first fetch the KV cache before starting the decoding loop
  outputs = llm.generate(prompts, sampling_params)

if __name__ == "__main__":
  prefill_done = Event()
  prefill_process = Process(target=run_prefill, args=(prefill_done,))
  decode_process = Process(target=run_decode, args=(prefill_done,))

  prefill_process.start()
  decode_process.start()

  decode_process.join()
  prefill_process.terminate()

Note. LMCache, the fastest production-ready connector (it uses NVIDIA's NIXL as the backend), was also explored, but it is still at the bleeding edge and some bugs were hit. Since much of its complexity lives in an external repo, SharedStorageConnector is the better choice for explanation.

The steps inside vLLM are:

• Instantiation. During engine construction, connectors are created in two places: inside the worker's init device procedure (under the init-worker-distributed-environment function) with role "worker", and inside the scheduler constructor with role "scheduler".
• Cache lookup. When the scheduler processes prefill requests from the waiting queue, after local prefix-cache checks, it calls the connector's get_num_new_matched_tokens. This checks for externally cached tokens in the KV-cache server. Prefill always sees 0 here; decode may have a cache hit. The result is added to the local count before calling allocate_slots.
• State update. The scheduler then calls connector.update_state_after_alloc, which records requests that had a cache hit (a no-op for prefill).
• Meta build. At the end of scheduling the scheduler calls meta = connector.build_connector_meta. Prefill adds all requests with is_store=True (to upload KV); decode adds requests with is_store=False (to fetch KV).
• Context manager. Before the forward pass the engine enters a KV-connector context manager. On enter, kv_connector.start_load_kv is called: for decode this loads KV from the external server and injects it into paged memory, for prefill it is a no-op. On exit, kv_connector.wait_for_save is called: for prefill this blocks until KV is uploaded to the external server, for decode it is a no-op.

Additional notes. For SharedStorageConnector the "external server" is just the local file system. Depending on configuration, KV transfers can also be done layer by layer, before or after each attention layer. Decode loads external KV only once, on the first step of its requests; afterwards it computes and stores locally.

From UniProcExecutor to MultiProcExecutor

With the core techniques in place, scaling up becomes the next question. Suppose the model weights no longer fit into a single GPU's VRAM. The first option is to shard the model across multiple GPUs on the same node using tensor parallelism, for example TP=8. If the model still does not fit, the next step is pipeline parallelism across nodes.

Notes. Intranode bandwidth is significantly higher than internode bandwidth, which is why tensor parallelism is generally preferred over pipeline parallelism (it is also true that PP communicates less data than TP). Expert parallelism is not covered here, since the focus is standard transformers rather than MoE, nor is sequence parallelism, because TP and PP are the most commonly used in practice.

At this stage multiple GPU processes (workers) and an orchestration layer to coordinate them are needed. That is exactly what MultiProcExecutor provides.

How this works in vLLM:

• MultiProcExecutor initializes an rpc_broadcast_mq message queue, implemented with shared memory under the hood.
• The constructor loops over world_size (TP=8 implies world_size=8) and spawns a daemon process for each rank via WorkerProc.make_worker_process.
• For each worker the parent first creates a reader and a writer pipe.
• The new process runs WorkerProc.worker_main, which instantiates a worker going through the same "init device", "load model", and KV-cache procedures as in UniProcExecutor.
• Each worker determines whether it is the driver (rank 0 in the TP group) or a regular worker. Every worker sets up two queues: rpc_broadcast_mq, shared with the parent, for receiving work, and worker_response_mq for sending responses back.
• During initialization each child sends its worker_response_mq handle to the parent through the pipe. Once all are received the parent unblocks, which completes coordination.
• Workers then enter a busy loop, blocking on rpc_broadcast_mq.dequeue. When a work item arrives they execute it, just like in UniProcExecutor but now with TP/PP-specific partitioned work, and send results back through worker_response_mq.enqueue.
• At runtime, when a request arrives, MultiProcExecutor enqueues it into rpc_broadcast_mq (non-blocking) for all child workers, then waits on the designated output rank's worker_response_mq.dequeue to collect the final result.

From the engine's perspective nothing has changed: all of this multiprocessing complexity is abstracted away behind a call to the model executor's execute_model. With UniProcExecutor, execute_model directly leads to calling execute_model on the worker. With MultiProcExecutor, it indirectly leads to calling execute_model on each worker through rpc_broadcast_mq. At this point you can run models as large as your resources allow, through the same engine interface.

The next step is to scale out: enable data parallelism (DP > 1) by replicating the model across nodes, add a lightweight DP coordination layer, introduce load balancing across replicas, and place one or more API servers in front to handle incoming traffic.

Distributed System Serving vLLM

There are many ways to set up serving infrastructure. To stay concrete, take one example: two H100 nodes, with four vLLM engines running across them. If the model requires TP=4, the nodes can be configured as follows.

On the first node, run the engine in headless mode (no API server):

vllm serve <model-name>
  --tensor-parallel-size 4
  --data-parallel-size 4
  --data-parallel-size-local 2
  --data-parallel-start-rank 0
  --data-parallel-address <master-ip>
  --data-parallel-rpc-port 13345
  --headless

Then run the same command on the other node with two tweaks: no --headless, and a different DP start rank.

vllm serve <model-name>
  --tensor-parallel-size 4
  --data-parallel-size 4
  --data-parallel-size-local 2
  --data-parallel-start-rank 2
  --data-parallel-address <master-ip>
  --data-parallel-rpc-port 13345

Note. This assumes networking is configured so that all nodes can reach the specified IP and port.

On the headless server node

On the headless node a CoreEngineProcManager launches two processes (one per --data-parallel-size-local), each running EngineCoreProc.run_engine_core. Each of those functions creates a DPEngineCoreProc (the engine core) and then enters its busy loop.

DPEngineCoreProc initializes its parent EngineCoreProc (a child of EngineCore), which:

• Creates an input_queue and an output_queue (queue.Queue).
• Performs an initial handshake with the frontend on the other node using a DEALER ZMQ socket (an async messaging library) and receives coordination address info.
• Initializes the DP group, for example using the NCCL backend.
• Initializes the EngineCore with MultiProcExecutor (TP=4 on 4 GPUs, as described earlier).
• Creates a ready_event (threading.Event).
• Starts an input daemon thread (threading.Thread) running process_input_sockets(..., ready_event), and similarly starts an output thread.
• Still in the main thread, waits on ready_event until all input threads across all four processes spanning the two nodes have completed the coordination handshake and finally executed ready_event.set().
• Once unblocked, sends a "ready" message to the frontend with metadata such as num_gpu_blocks available in paged KV-cache memory.
• The main, input, and output threads then enter their respective busy loops.

In short: four child processes (one per DP replica), each running a main, an input, and an output thread. They complete a coordination handshake with the DP coordinator and the frontend, then all three threads per process run in steady-state busy loops.

The current steady state looks like this:

• Input thread — blocks on the input socket until a request is routed from the API server; on receipt it decodes the payload, enqueues a work item via input_queue.put_nowait(...), and returns to blocking on the socket.
• Main thread — wakes on input_queue.get(...), feeds the request to the engine; MultiProcExecutor runs the forward pass and enqueues results to output_queue.
• Output thread — wakes on output_queue.get(...), sends the result back to the API server, then resumes blocking.

Three additional mechanics are worth naming:

• DP wave counter. The system tracks "waves": when all engines become idle they quiesce, and the counter increments when new work arrives. This is useful for coordination and metrics.
• Control messages. The API server can send more than inference requests, for example aborts and utility or control RPCs.
• Dummy steps for lockstep. If any DP replica has work, all replicas execute a forward step; replicas without requests perform a dummy step to participate in the required synchronization points, which avoids blocking the active replica.

Lockstep clarification. Lockstep is actually only required for MoE models, where the expert layers form an EP or TP group while attention layers are still DP. It is currently always done with DP, mainly because there is limited use for "built-in" non-MoE DP: you could just run multiple independent vLLM instances and load-balance between them in the normal way.

On the API server node

The API server node instantiates an AsyncLLM object, an asyncio wrapper around the LLM engine. Internally this creates a DPLBAsyncMPClient: a data-parallel, load-balancing, asynchronous, multiprocessing client.

Inside the parent class of MPClient, the launch_core_engines function runs and creates the ZMQ addresses used for the startup handshake (as seen on the headless node), spawns a DPCoordinator process, and creates a CoreEngineProcManager, the same as on the headless node.

Inside AsyncMPClient (a child of MPClient), vLLM creates an outputs_queue (asyncio.Queue), creates an asyncio task process_outputs_socket that communicates through the output socket with the output threads of all four DPEngineCoreProc instances and writes into outputs_queue, and then creates one more asyncio task, output_handler from AsyncLLM, which reads from that queue and finally sends information out to the create_completion function. Inside DPAsyncMPClient it also creates an asyncio task run_engine_stats_update_task that communicates with the DP coordinator.

The DP coordinator mediates between the frontend (API server) and the backend (engine cores). It periodically sends load-balancing info (queue sizes, waiting and running requests) to the frontend's run_engine_stats_update_task; handles SCALE_ELASTIC_EP commands from the frontend by dynamically changing the number of engines (this only works with the Ray backend); and sends START_DP_WAVE events to the backend when triggered by the frontend, reporting wave-state updates back.

To recap, the frontend (AsyncLLM) runs several asyncio tasks (concurrent, not parallel): a class of tasks handles input requests through the generate path, with each new client request spawning a new asyncio task; two tasks (process_outputs_socket, output_handler) process output messages from the underlying engines; and one task (run_engine_stats_update_task) maintains communication with the DP coordinator, sending wave triggers, polling load-balancing state, and handling dynamic scaling requests.

Finally the main server process creates a FastAPI app and mounts endpoints such as OpenAIServingCompletion and OpenAIServingChat, which expose /completion, /chat/completion, and others. The stack is then served via Uvicorn.

The full request lifecycle

Putting it all together, you send this from your terminal:

curl -X POST http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{
  "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
  "prompt": "The capital of France is",
  "max_tokens": 50,
  "temperature": 0.7
}'

What happens next:

1. The request hits OpenAIServingCompletion's create_completion route on the API server.
2. The function tokenizes the prompt asynchronously and prepares metadata (request ID, sampling params, timestamp, and so on).
3. It then calls AsyncLLM.generate, which follows the same flow as the synchronous engine, eventually invoking DPAsyncMPClient.add_request_async.
4. That in turn calls get_core_engine_for_request, which load-balances across engines based on the DP coordinator's state, picking the one with the minimal score: score = len(waiting) * 4 + len(running).
5. The ADD request is sent to the chosen engine's input_socket.

At that engine:

• Input thread — unblocks, decodes data from the input socket, and places a work item on the input_queue for the main thread.
• Main thread — unblocks on input_queue, adds the request to the engine, and repeatedly calls engine_core.step(), enqueueing intermediate results to output_queue until a stop condition is met. Reminder: step() calls the scheduler, the model executor (which can itself be a MultiProcExecutor), and so on — machinery already covered above.
• Output thread — unblocks on output_queue and sends results back through the output socket.
• Those results trigger the AsyncLLM output asyncio tasks (process_outputs_socket and output_handler), which propagate tokens back to FastAPI's create_completion route.
• FastAPI attaches metadata (finish reason, logprobs, usage info) and returns a JSONResponse via Uvicorn to your terminal.

And just like that the completion comes back, with the whole distributed machinery hidden behind a simple curl command.

Additional notes. When adding more API servers, load balancing is handled at the OS/socket level; from the application's perspective nothing significant changes, because the complexity is hidden. With Ray as a DP backend you can expose a URL endpoint (/scale_elastic_ep) that enables automatic scaling of the number of engine replicas up or down.

Benchmarks and Auto-Tuning: Latency vs Throughput

So far the analysis has followed the "gas particles", the internals of how requests flow through the engine and system. Now it is time to zoom out and look at the system as a whole, and ask: how do you measure the performance of an inference system?

At the highest level there are two competing metrics. Latency is the time from when a request is submitted until tokens are returned. Throughput is the number of tokens or requests per second the system can generate or process. Latency matters most for interactive applications, where users are waiting on responses. Throughput matters in offline workloads such as synthetic data generation for pre- and post-training runs, data cleaning and processing, and in general any type of offline batch inference job.

Before explaining why latency and throughput compete, here are the common inference metrics:

Metric	Definition
TTFT (time to first token)	Time from request submission until the first output token is received.
ITL (inter-token latency)	Time between two consecutive tokens, for example from token i-1 to token i.
TPOT (time per output token)	The average ITL across all output tokens in a request.
Latency / E2E (end-to-end latency)	Total time to process a request, that is TTFT plus the sum of all ITLs, or equivalently the time between submitting the request and receiving the last output token.
Throughput	Total tokens processed per second (input, output, or both), or alternatively requests per second.
Goodput	Throughput that meets service-level objectives (SLOs) such as max TTFT, TPOT, or E2E latency. For example, only tokens from requests meeting those SLOs are counted.

A simplified model explains the competing nature of these two metrics. The assumption is that weight I/O, not KV-cache I/O, dominates, meaning short sequences. The tradeoff becomes clear when looking at how batch size B affects a single decode step. As B falls toward 1, ITL drops: there is less work per step and the token is not "competing" with others. As B rises toward infinity, ITL grows because each step does more FLOPs, but throughput improves until peak performance is hit, because weight I/O is amortized across more tokens.

A roofline model helps here. Below a saturation batch B_sat, step time is dominated by HBM bandwidth (streaming weights layer by layer into on-chip memory), so step latency is nearly flat: computing 1 versus 10 tokens can take a similar amount of time. Beyond B_sat the kernels become compute-bound and step time grows roughly with B; each extra token adds to ITL.

Note. A more rigorous treatment has to account for kernel auto-tuning: as B grows, the runtime may switch to more efficient kernels for that shape, changing the achieved performance P_kernel. Step latency is t = FLOPs_step / P_kernel, where FLOPs_step is the work in the step. As P_kernel approaches P_peak, more compute per step leads directly to higher latency.

How to benchmark in vLLM

vLLM provides a vllm bench {serve,latency,throughput} CLI that wraps vllm/benchmarks/{server,latency,throughput}.py. Here is what each script does:

• latency — uses a short input (default 32 tokens) and samples 128 output tokens with a small batch (default 8). It runs several iterations and reports end-to-end latency for the batch.
• throughput — submits a fixed set of prompts (default: 1000 ShareGPT samples) all at once, also known as QPS=Inf mode, and reports input, output, and total tokens and requests per second across the run.
• serve — launches a vLLM server and simulates a real-world workload by sampling request inter-arrival times from a Poisson (or more generally Gamma) distribution. It sends requests over a time window, measures all the metrics discussed above, and can optionally enforce a server-side max concurrency through a semaphore, for example limiting the server to 64 concurrent requests.

Here is an example of running the latency script:

vllm bench latency
  --model <model-name>
  --input-tokens 32
  --output-tokens 128
  --batch-size 8

Benchmark configs used in CI live under .buildkite/nightly-benchmarks/tests. There is also an auto-tune script that drives the serve benchmark to find argument settings that meet target SLOs, for example "maximize throughput while keeping p99 E2E < 500 ms", and returns a suggested config.

Epilogue

The tour began with the basic engine core (UniProcExecutor), added advanced features such as speculative decoding and prefix caching, scaled up to MultiProcExecutor with TP/PP > 1, then scaled out by wrapping everything in the asynchronous engine and the distributed serving stack, and closed with how to measure system performance.

vLLM also includes specialized handling that this post skips, for example:

• Diverse hardware backends — TPUs, AWS Neuron (Trainium/Inferentia), and others.
• Architectures and techniques — MLA, MoE, encoder-decoder models (Whisper), pooling and embedding models, EPLB, m-RoPE, LoRA, ALiBi, attention-free variants, sliding-window attention, multimodal LMs, and state-space models (Mamba, Mamba-2, Jamba).
• TP / PP / SP parallelism variants.
• Hybrid KV-cache logic (Jenga), more complex sampling methods such as beam sampling, and more.
• Experimental — async scheduling.

The nice thing is that most of these are orthogonal to the main flow described above, so you can almost treat them like plugins (in practice there is some coupling, of course). Resolution definitely suffers at this altitude; the follow-up posts zoom into specific subsystems and get into the details.

The original post was written by Aleksa Gordíc, with experiments run on H100s provided by Hyperstack, and pre-release feedback from Nick Hill (core vLLM contributor, Red Hat), Mark Saroufim (PyTorch), Kyle Krannen (NVIDIA, Dynamo), and Ashish Vaswani.

References

1. vLLM
2. Attention Is All You Need
3. Efficient Memory Management for Large Language Model Serving with PagedAttention
4. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
5. Jenga: Effective Memory Management for Serving LLM with Heterogeneity
6. Orca: A Distributed Serving System for Transformer-Based Generative Models
7. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models
8. Accelerating Large Language Model Decoding with Speculative Sampling
9. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
10. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
11. LMCache

---

Source: Inside vLLM: Anatomy of a High-Throughput LLM Inference System, Aleksa Gordíc, August 29, 2025.