NInfer: A From-Scratch C++/CUDA Inference Engine for a Single RTX 5090

Written From Scratch for One RTX 5090

NInfer is a from-scratch C++/CUDA inference engine for five explicitly registered Qwen checkpoints, running on a single NVIDIA GeForce RTX 5090 (32 GiB). It serves text, image and video prompts through a local CLI or OpenAI- and Anthropic-compatible HTTP APIs. There is no PyTorch runtime, no upstream engine forked and trimmed: the kernels, the KV allocator, the scheduler, the tokenizer/chat-template frontend and the HTTP surface are all first-party code in one C++20 build tree.

The specialization is deliberate and stated plainly in the README: one GPU, one resident model, and an active-request capacity of one to eight that is fixed when the process starts. The build itself refuses anything else: it rejects CUDA architectures other than sm_120a, and there is no install target or packaged binary distribution, so you run NInfer straight out of the source build tree. Everything the engine declines to do is declined in exchange for throughput on that one card.

Five Registered Artifact Identities

An artifact identity fixes the exact model and weight profile at once. Each .ninfer file also embeds the tokenizer, chat template and media frontend resources its registered target needs, so the runtime never reaches back to Hugging Face at request time. The quick-start commands use Qwen3.8-27B NVFP4.

Model	Weights	Artifact
Qwen3.6-27B	groupwise-int	qwen3_6_27b.ninfer
Qwen3.6-27B	nvfp4	qwen3_6_27b_nvfp4.ninfer
Qwen3.8-27B	groupwise-int	qwen3_8_27b.ninfer
Qwen3.8-27B	nvfp4	qwen3_8_27b_nvfp4.ninfer
Qwen3.6-35B-A3B	groupwise-int	qwen3_6_35b_a3b.ninfer

The published artifacts derive from Qwen/Qwen3.6-27B, Qwen/Qwen3.8-27B and Qwen/Qwen3.6-35B-A3B (all Apache-2.0); the two NVFP4 variants additionally use fixed packed weights from rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm and unsloth/Qwen3.8-27B-NVFP4.

Concurrency-Scaled Decode: 1,146.9 tok/s on One Card

Saturated decode was measured with INT8 group-64 KV, CUDA Graphs, MTP3 and one 8,192-token generation per active request. Values are aggregate committed decode throughput plus MTP acceptance, taken from complete intervals whose actual decode batch equalled the configured concurrency.

Model profile	C=1 tok/s / accept	C=2	C=4	C=8	C8 / C1
Qwen3.6-27B groupwise-int	185.8 / 68.2%	247.0 / 69.0%	309.5 / 68.4%	535.0 / 68.3%	2.88×
Qwen3.6-27B nvfp4	202.4 / 69.3%	399.7 / 71.4%	699.7 / 69.3%	1,146.9 / 68.6%	5.67×
Qwen3.6-35B-A3B groupwise-int	593.0 / 67.2%	877.7 / 68.2%	1,166.0 / 69.8%	1,313.8 / 67.3%	2.22×
Qwen3.8-27B nvfp4	143.8 / 48.9%	267.6 / 48.1%	461.1 / 45.8%	766.6 / 46.0%	5.33×

The MoE artifact is the throughput outlier: Qwen3.6-35B-A3B already does 593.0 tok/s at C=1 because only about 3B parameters activate per token, and it reaches 1,313.8 tok/s at C=8. Notice the acceptance-rate gap, too: the Qwen3.6 profiles hold 67-71% MTP acceptance while Qwen3.8 sits at 46-49%, which is exactly why the newer model has the lower absolute decode rate despite being the flagship of the quick start.

Serial single-request serving used the same INT8 group-64 KV and CUDA Graphs with a 1,024-token prefill chunk and five fixed seeds after warm-up:

Model profile	7,680-token prefill	260,096-token prefill	Structured MTP3 decode
Qwen3.6-35B-A3B groupwise-int	15,544.3 tok/s	5,157.1 tok/s	770.9 tok/s
Qwen3.6-27B groupwise-int	3,218.1 tok/s	1,614.8 tok/s	193.0 tok/s
Qwen3.6-27B nvfp4	11,191.5 tok/s	2,510.6 tok/s	252.2 tok/s
Qwen3.8-27B groupwise-int	3,274.7 tok/s	1,609.7 tok/s	224.4 tok/s
Qwen3.8-27B nvfp4	8,340.4 tok/s	2,203.1 tok/s	219.8 tok/s

The Honest Finding: Smaller Weights Beat Faster Weights at C=8

A separate makespan campaign runs the complete speculative-decode corpus (three long-reasoning fixtures plus twelve code/story/translation/structured fixtures, five seeds each, 75 requests per point, same shuffled send order at every concurrency) and measures wall-clock time from worker release to the last HTTP response read. The result inverts the naive expectation that NVFP4 always wins:

• groupwise-int: makespan 4,622.59 s down to 2,211.20 s from C=1 to C=8, a 2.09× speedup, with average batch rising to 4.76. C=8 is the best measured point.
• nvfp4: 4,670.27 s down to 1,647.74 s at C=4 (2.83×), but C=8 regresses to 2,164.90 s with average batch only 2.36. C=4 is the best measured point.

The cause is memory, not kernels. The groupwise-int weight arena is 16.672 GiB against 19.729 GiB for NVFP4, so at C=8 the lighter profile leaves room for 313,984 tokens of Device KV instead of 187,712, nearly twice the cache and hence nearly twice the achievable batch. --kv-capacity auto resolved to 131,072 / 262,144 / 341,952 / 313,984 tokens at C=1/2/4/8 for groupwise-int. All 300 requests in each campaign completed with no request, CUDA or out-of-memory failure, and C=8 peaked at four waiting requests without a single spill, owner degradation/eviction or search-exhaustion event.

240k-Token Contexts and Prefix Checkpoints That Survive Memory Pressure

A reusable prefix checkpoint holds the KV plus the complete continuation state for an exact prompt frontier. A Device-resident checkpoint resumes directly; under pressure the planner weighs Device retention, pinned Host State/KV and eviction against immediate restore work and later reuse cost, while active requests always keep their completion reservations. The recommended long-context server profile makes the tiers explicit:

./build/apps/ninfer-serve models/qwen3_8_27b_nvfp4.ninfer \
  --max-context 240000 \
  --kv-capacity 240000 \
  --max-concurrency 2 \
  --kv-dtype fp8 \
  --device-state-slots 2 \
  --host-state-slots 8 \
  --host-kv-mib 8192 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft \
  --preserve-thinking

Each request gets a 240,000-token logical ceiling; a shared 240,000-token Device KV pool serves admitted requests, so either request may use the whole pool when running alone. Beyond the two active StateImages the process keeps two Device checkpoint slots, eight pinned Host State slots and 8 GiB of pinned Host KV. --max-context is the per-sequence limit, --kv-capacity sizes the shared Main Text KV pool used by active requests and retained prefixes, and auto resolves the largest legal capacity at startup from the memory left after weights while keeping 1 GiB of sizing headroom. Explicit capacities stay fixed for the process lifetime. The tools/bench/ttft harness covers hot reuse, Host resume, eviction, shared prefixes, scheduling boundaries and multimodal load over the public HTTP route.

Speculative Residency Is a Startup Decision

GPU residency is frozen when the Engine starts, and the flags that choose it are not lazy loading: an Engine started without Vision rejects media and can never enable it later:

• --spec mtp with draft windows of one to five positions; the published numbers use MTP3 (--draft-tokens 3 --lm-head-draft).
• --spec dflash on the 35B-A3B target with draft windows of one to fifteen; measured at block length eight (--draft-tokens 7) and at the full native block of fifteen. MTP and DFlash cannot be enabled together.
• --lm-head-draft loads the optimized proposal head; a backend with the full proposal head omits it.
• --vision loads Vision weights and expands the single Program workspace for encode/handoff; off by default, which also omits the Vision-specific unified-workspace extent.

DFlash may be combined with Vision, but it accelerates generated-text decode after multimodal prefill, not the Vision encode itself. Omitting --spec drops MTP/DFlash weights, state and the optimized proposal head entirely, which is how the smallest resident profile is obtained. The one-request CLI additionally runs in root-only context mode, so it never reserves an extra Device checkpoint StateImage or captures a continuation no later request could consume.

Five KV Storage Formats, One Chunked Prefill Path

All registered model IDs support BF16, INT8, FP8, NVFP4 and K8V4 KV storage, chunked prefill with a configurable positive chunk in multiples of 128 (default 1,024), exact-batch CUDA Graph decode with startup-bounded batched decode, offline causal-perplexity scoring, private and shared exact-prefix reuse with Device/Host State and KV retention, and model-aware sampling defaults with explicit sampler overrides. Every published measurement uses INT8 group-64 KV.

A Small but Strict Serving Surface

Method and path	Behavior
GET /health	Engine readiness; 503 after an Engine-wide failure, unauthenticated, unaffected by queue saturation
GET /v1/models, /v1/models/{id}	configured OpenAI alias and effective max_model_len
POST /v1/chat/completions	OpenAI-style chat generation, streaming and non-streaming
POST /v1/responses	OpenAI Responses Core generation, local state, typed Items, SSE
POST /v1/responses/input_tokens	prompt-token count without generation
GET/DELETE /v1/responses/{id}, /input_items	retrieve, delete, or list a locally stored Response's normalized input Items
POST /v1/messages	Anthropic-style message generation
POST /v1/messages/count_tokens	checkpoint-native expanded input-token count

The transport details are unusually specified for a personal project: every OpenAI-compatible response carries a unique x-request-id (including streaming and error responses), all three generation SSE endpoints emit a : keep-alive comment after five seconds without a protocol event, and accepted Linux connections use TCP keepalive with a 15-second TCP_USER_TIMEOUT, so a dead or unacknowledging peer is normally cancelled within about 20 seconds, including while the request is queued or prefilling.

Capability handling is strict rather than permissive. Options whose observable behaviour the Engine cannot provide are rejected when they request it: JSON constrained output, nonzero logit_bias, requested log probabilities, audio/file input or audio output, strict:true, required or named tool choice, parallel_tool_calls:false with enabled tools, explicit low/high image detail, web search, moderation, low/high verbosity, stored Chat Completions and non-empty legacy functions. Known constrained-decoding aliases (grammar, structured_outputs, guided_json, guided_regex, guided_choice, guided_grammar) receive the same explicit rejection instead of being silently ignored as unknown hints. Semantically neutral fields (all-zero logit_bias, logprobs:false, top_logprobs:0, verbosity:"medium", prediction, metadata and service-tier or prompt-cache hints) are accepted without changing execution, and unknown top-level fields are ignored. Each rejection names the affected field and the guarantee NInfer cannot provide.

Tool calls are parsed and returned to the client; NInfer never executes them. Thinking is on by default, --reasoning-effort low|medium|xhigh selects an effort only when the embedded chat template exposes one, and --thinking-budget N bounds model-origin thinking tokens: if </think> has not appeared at that exact boundary, the Engine appends Qwen's canonical early-close guidance plus the closing tag to the resident sequence without sampling, publishes it through the reasoning stream, then resumes ordinary generation. A natural close, stop condition, cancellation or total limit at the boundary takes priority and suppresses the insertion.

Quality Measured Through the Serving Route

Capability scores were produced through NInfer's own OpenAI-compatible serving route with thinking enabled and MTP3, using EvalScope 1.9.0 with 0-shot rule scoring and one sample per problem, so the numbers characterize the shipping path rather than a separate evaluation harness.

Model profile	AIME 2025	AIME 2026	GPQA-Diamond	ERQA	RealWorldQA
Qwen3.6-27B groupwise-int	86.67%	93.33%	86.87%	-	-
Qwen3.6-27B NVFP4	93.33%	93.33%	84.34%	-	-
Qwen3.6-35B-A3B groupwise-int	90.00%	90.00%	85.35%	-	-
Qwen3.8-27B groupwise-int	96.67%	96.67%	87.37%	66.25%	82.22%
Qwen3.8-27B NVFP4	96.67%	96.67%	90.40%	66.25%	83.53%

Sampling differs by generation: the Qwen3.6 rows used temperature 0.6 with presence penalty 1.0, the Qwen3.8 rows temperature 1.0 with presence penalty 0.0. Multimodal evaluation used --vision and an 81,920-token context limit; text evaluation used 262,144 tokens except Qwen3.8-27B NVFP4, which used 252,928 to fit the RTX 5090 after weights. Each score is one sample per problem, and the model cards carry the correct/total counts and evaluation notes.

What It Explicitly Refuses to Be

• One RTX 5090 and one resident model per Engine: no multi-GPU, no distributed serving, no weight offload.
• A startup-fixed capacity of one to eight active requests with bounded FIFO ingress; no request preemption, no priority/QoS, no active-request swapping.
• One shared startup-fixed KV pool across active requests and retained prefixes.
• No runtime model discovery and no unregistered-checkpoint fallback.
• Parsed tool calls go back to the client; tools are never executed in-process.
• The in-tree C++ headers are not distributed as an installed SDK.

Build, Run, Containerize

Requirements: 64-bit Linux, an RTX 5090, CUDA Toolkit 13.1+, CMake 3.28+, a C++20 host compiler, Ninja, pkg-config, FFmpeg development libraries (libavformat ≥ 60, libavcodec ≥ 60, libavutil ≥ 58, libswscale ≥ 7) and libcurl ≥ 7.85. Tests, benchmarks and maintainer tools are excluded from the default build.

git clone https://github.com/Neroued/ninfer.git && cd ninfer
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

hf download neroued/Qwen3.8-27B-nvfp4-NInfer \
  qwen3_8_27b_nvfp4.ninfer --local-dir models

./build/apps/ninfer models/qwen3_8_27b_nvfp4.ninfer \
  --prompt "Explain prefill and decode, then give a concise conclusion." \
  --max-context 32768 --max-new 8192 \
  --kv-dtype fp8 --spec mtp --draft-tokens 3 --lm-head-draft

Answer content goes to stdout; startup diagnostics and the CLI's reasoning, timing, throughput, memory and speculative-decoding report go to stderr as unprefixed product output, so > answer.txt 2> run.log cleanly separates them. A Docker image is provided for hosts with the NVIDIA Container Toolkit (docker build --tag ninfer:local .), and the same server profile runs unchanged inside it with the model directory mounted read-only. Documentation covers the CLI, HTTP serving, performance methodology, perplexity evaluation and the resource-scheduling/context-cache algorithm; the repo also ships committed CLI examples for text, image, video, mixed-media, thinking, long-decode and long-context inputs. NInfer is Apache-2.0 and a personal project of its maintainer, with voluntary support via Ko-fi and no promised services, features or roadmap input.