ThinCtx: Qwen3.5 with a small context memory

Qwen3.5 0.8B, 2B and 4B, converted so that the GPU memory their context needs stays small and grows 15 to 20 times more slowly than the original's cache: for the 4B, — GB instead of the base model's — GB at —, with accuracy on par with the base model (on average — % against the base model's — % on BABILong, RULER and BFCL).

Qwen3.5-0.8B

GPU memory for 128k tokens of context— original — RULER at 64k tokens, %— original —

Qwen3.5-2B

GPU memory for 128k tokens of context— original — RULER at 64k tokens, %— original —

Qwen3.5-4B

GPU memory for 128k tokens of context— original — RULER at 64k tokens, %— original —

What this is

A language model keeps a working memory of everything it has read so far. With every token of a conversation, a document or an agent's tool output, that memory grows on the GPU, until long contexts no longer fit on an ordinary graphics card or only one user fits on it at a time.

We take existing open models and convert them so that the part of this memory that has to sit on the GPU stays small and grows 15 to 20 times more slowly than the original's cache as the context gets longer. The conversion starts from the original weights; it is not a new model. Every number below compares the converted model with the unconverted original on the same items.

Much smaller context memory. The GPU part is small and grows 15 to 20 times more slowly than the original's cache, so long contexts and several parallel sessions fit on one consumer card.
Slower than the original in vLLM, except at very long contexts. ThinCtx reads prompts 2.4 to 4.5 times more slowly than the original running in vLLM, and generates 1.2 to 2.9 times more slowly up to 128k tokens; from 512k tokens on its generation is level or ahead (0.8B, 2B; at 1M tokens ahead), and the 4B runs 512k tokens on a 24 GB card where the original does not fit (see Results per model).
The original's quality. Function calling, long documents, needle search and multi-step questions score as the original does, and on some tests a little better.
Built for agents. Rules given early in the system prompt or in a user message are still followed far later, and tool calls whose definitions are far back stay correct.
Languages kept. Czech and German stay at the original's level, also on long documents.
Cheap memory for the rest. The full context is kept in ordinary system memory (RAM), which is larger and far cheaper than GPU memory; with --compress_context or the -small releases it is stored compressed, about 2.5 times smaller, at the same quality.

How the conversion works is not described on this page. The 2B and 4B were measured on one 16 GB or 24 GB consumer GPU per run, with greedy decoding and thinking off unless stated. For each size, the converted column shows the released model of that size with its released settings.

Results per model

For each size: the original Qwen3.5, the ThinCtx conversion with its 16-bit weights, the same conversion run with --compress_context (thinctx 1.0.2: the context in system memory stored compressed), and ThinCtx-small (quantized weights and the compressed context). The chart and the table show the same four, side by side.

Memory: thinctx 1.0.2 on one RTX 3090 at 131,072 (128k) and 541,584 (540k) tokens; GPU = weights + the context's part on the GPU; system RAM = the stored context and its index; the original's cache is computed from the model's shape. Speed: one conversation on one RTX 3090; the original in vLLM 0.31 (batch 1, CUDA graphs); ThinCtx and -small with thinctx 1.0.1 (the 16-bit 4B: 0.3.1), --compress_context with 1.0.2; quality beyond 256k tokens was not measured. Quality: every row on the same items; the --compress_context and -small values come from separate runs, whose own ThinCtx reference differs from the ThinCtx column by up to 2 points. BFCL: the full non-live set, and the first 50 items per category for the -small comparison. –: not measured; measuring: the run is in progress.

How it compares

The usual ways to make long contexts affordable, and what each costs. Most leave the model as it is and manage its cache; trained-in sparse attention needs a new model. This conversion adapts an existing model instead.

Approach	Examples	What it changes	GPU memory for the context	Cost

Measured on Qwen3.5-2B

Two common approaches run on the original 2B weights, against ThinCtx made from the same model, on the same items. Scores in %; rules: effect kept far back (1.00 = as the original); divergence: KL from the original's next-token distribution, lower is better.

System	BABILong 64k	RULER 16k	RULER 64k	Distant tool calls, accuracy	Distant tool calls, divergence	Rules kept, worst case	Text divergence

"measuring": the run is in progress. "–": not measured for this size. Original and converted were run on the same items with the same settings.