RTX 2050 4GB: Qwen3.6-35B-A3B-NVFP4 runs at ~21 tok/s with single-layer CPU MoE prefill buffer

[yakuto4]

This is my first time submitting a technical report on GitHub.
I am not an expert on FreeToken internals. I tested this with the assistance of AI (ChatGPT), which helped me diagnose the errors and modify the configuration/source code. I am reporting the result because it may be useful for low-VRAM GPU support.
My goal was to see whether Qwen3.6-35B-A3B-NVFP4 could run practically on a GPU with only 4 GB VRAM.
Environment

• GPU: NVIDIA GeForce RTX 2050, 4 GB VRAM
• OS: Windows 11 + WSL2 / Ubuntu 24.04
• FreeToken: current main branch, version 0.1.2
• Model: nvidia/Qwen3.6-35B-A3B-NVFP4
• CPU MoE threads: 8
  With the default CPU MoE configuration, FreeToken allocates a two-layer prefill buffer:
  override("moe_cache_size", 2 * num_experts)
  override("moe_prefill_overlap", True)

On my 4 GB GPU this eventually failed with CUDA device not ready errors during GPU allocations.
I changed only these two source lines in python/freetoken/engine/engine.py:

• override("moe_cache_size", 2 * num_experts)
• override("moe_prefill_overlap", True)

• override("moe_cache_size", num_experts)
• override("moe_prefill_overlap", False)

For this model, num_experts = 256, so this reduces the GPU MoE cache from 512 slots to 256 slots and disables prefill overlap.
I also used:
export FREETOKEN_PIN_BUDGET_GB=10
export FREETOKEN_MAMBA_SSM_DTYPE=float16

ft serve
--model-path nvidia/Qwen3.6-35B-A3B-NVFP4
--max-seq-len-override 1024
--num-tokens 512
--memory-ratio 0.98
--attention-backend triton
--moe-backend cpu
--expert-load serial
--moe-cpu-threads 8
--max-running-requests 1
--host 127.0.0.1
--port 1919
The server successfully initialized and accepted OpenAI-compatible /v1/chat/completions requests.
Example measured decode throughput:
Decode batch ... gen throughput (token/s): 20.70
Decode batch ... gen throughput (token/s): 21.33
So the steady-state decode throughput was approximately 21 tokens/sec on an RTX 2050 with only 4 GB VRAM.
For comparison, in my previous testing with a 12 GB GPU, the same general FreeToken/Qwen3.6-35B-A3B-NVFP4 setup was around 30 tokens/sec.
There are still limitations. The expert banks could not all be OS-locked and FreeToken reported that they were left pageable. The configuration is also deliberately using a very small 512-token KV cache. Therefore, I am not suggesting that this is an optimal general-purpose configuration.
However, I thought the result was interesting because reducing the CPU-MoE prefill GPU cache from two expert layers to one allowed this 35B-A3B NVFP4 model to run and generate successfully on a 4 GB GPU.
Perhaps a low-VRAM option to use a single-layer prefill buffer with prefill overlap disabled could be useful upstream.
Again, this report was prepared with AI assistance, so please let me know if I have misunderstood any FreeToken internals or if additional logs/tests would be useful.

[yakuto4]

Additional test results

I continued testing the maximum KV/context capacity on the same RTX 2050 4GB setup, without changing the other settings.

KV / max sequence tokens	Result	Steady decode throughput
512	Success	20.70–21.33 tok/s
1024	Success	20.95–21.29 tok/s
2048	Success	19.54–19.84 tok/s
4096	Success	21.52–21.85 tok/s
8192	Failed	—

4096 tokens still worked normally, with no obvious decode performance degradation in this short-prompt test.

At 8192 tokens, FreeToken successfully allocated the KV cache:

Allocating 8192 tokens for KV cache, K + V = 0.16 GiB

=============================================

Additional test results: RTX 2050 4GB / Qwen3.6-35B-A3B-NVFP4

I continued testing the modified CPU MoE configuration on an RTX 2050 4GB.

Please note that I am not an expert on FreeToken, CUDA, or PyTorch internals. I am performing these tests with assistance from AI (ChatGPT), including investigation of the errors and preparation of this report.

Environment

• GPU: NVIDIA GeForce RTX 2050
• VRAM: 4 GB
• FreeToken: 0.1.2 (current source tree)
• Model: nvidia/Qwen3.6-35B-A3B-NVFP4
• MoE backend: cpu
• Attention backend: triton
• Max running requests: 1
• Max sequence length: 4096
• CPU MoE threads: 8
• WSL2 / Ubuntu 24.04

Environment variables used:

export FREETOKEN_PIN_BUDGET_GB=10
export FREETOKEN_MAMBA_SSM_DTYPE=float16

Local modification

I changed the CPU MoE configuration in:

python/freetoken/engine/engine.py

Original:

override("moe_cache_size", 2 * num_experts)
override("moe_prefill_overlap", True)

Modified:

override("moe_cache_size", num_experts)
override("moe_prefill_overlap", False)

The intention was to reduce the GPU-side prefill buffer from two expert layers to one expert layer and disable prefill overlap.

With the original configuration, the 4 GB RTX 2050 could not fit the required GPU allocations. With this modification, the model was able to start.

Server startup

The server successfully starts with:

• RTX 2050 4 GB
• Qwen3.6-35B-A3B-NVFP4
• CPU MoE backend
• maximum sequence length configured to 4096

This does not mean that a 4096-token prompt can actually be processed. The tests below show that the practical prefill limit is currently much lower because additional temporary GPU memory is required during prefill.

424-token test

A real prompt containing 424 input tokens completed successfully.

FreeToken reported:

Prefill batch, #new-seq: 1, #new-token: 424,
input throughput (token/s): 19.48

During decode:

gen throughput (token/s): 21.81
gen throughput (token/s): 21.44

The request completed with:

HTTP 200 OK

485-token test

I then increased the real prompt size.

A prompt containing 485 input tokens also completed successfully.

FreeToken reported:

Prefill batch, #new-seq: 1, #new-token: 485,
#cached-token: 0,
token usage: 0.12,
#mamba-slot: 4/8,
mamba usage: 0.50,
#running-req: 1,
#queue-req: 0,
input throughput (token/s): 15.07

Decode performance was:

Decode batch, #token: 526,
gen throughput (token/s): 1.18

Decode batch, #token: 566,
gen throughput (token/s): 21.32

Decode batch, #token: 606,
gen throughput (token/s): 20.36

The request completed successfully:

HTTP 200 OK

The first decode throughput measurement appears to include startup/transition overhead. Once steady-state decode begins, generation is approximately 20-21 tokens/sec.

Longer prompt tests

I then tested longer prompts.

Prompts around 600 tokens and above failed during GatedDeltaNet / FLA prefill.

One of the failures occurred here:

File "freetoken/models/qwen3_5_moe/gdn.py", line 206, in forward
    result = gdn_prefill_chunk_fla(

File "freetoken/kernel/fla/chunk.py", line 112, in forward
    g, o, A, w, h, v_new = chunk_gated_delta_rule_fwd(

File "freetoken/kernel/fla/chunk_delta_h.py", line 326, in chunk_gated_delta_rule_fwd_h
    v_new = torch.empty_like(u) if save_new_value else None

RuntimeError: CUDA driver error: device not ready

Other longer-prompt attempts failed at nearby points in the same GDN/FLA prefill path, including tensor allocation/contiguous operations.

Current observed results

Input tokens	Result	Prefill throughput	Decode throughput
424	Success	19.48 tok/s	~21.4-21.8 tok/s
485	Success	15.07 tok/s	~20.4-21.3 tok/s
~600	Failure	GDN/FLA prefill crash	-
~750	Failure	GDN/FLA prefill crash	-
~1000	Failure	GDN/FLA prefill crash	-

At the moment, the practical prefill boundary on this RTX 2050 4 GB configuration appears to be somewhere between approximately 485 and 600 input tokens.

What seems interesting

The most interesting result to me is that Qwen3.6-35B-A3B-NVFP4 is usable at all on an RTX 2050 with only 4 GB of VRAM.

Once generation reaches steady-state decode, performance remains around 20-21 tokens/sec.

The limitation I am seeing does not appear to be steady-state decode performance. Instead, the current limit appears during GDN/FLA prefill when the prompt becomes longer and temporary GPU allocations increase.

There is also an important distinction between configured context capacity and practical prompt capacity:

• The server can start with a maximum sequence length of 4096.
• KV cache allocation for that configuration can succeed.
• Short and medium prompts work.
• A real 485-token prompt works.
• Longer prefill currently causes a CUDA device not ready failure.

So simply reporting that "4096 context starts successfully" would be misleading. The practical usable input length is currently limited by prefill memory requirements.

Possible area for investigation

If the temporary GPU workspace/memory requirement of the GatedDeltaNet / FLA prefill path can be reduced, chunked, or otherwise constrained, it may be possible to substantially increase usable prompt length on 4 GB GPUs.

The local change reducing the CPU MoE GPU buffer from:

2 * num_experts

to:

num_experts

and disabling:

moe_prefill_overlap

was enough to move this model from not fitting to actually running and generating at approximately 20-21 tok/s on the RTX 2050 4 GB.

I do not know whether using only one expert-layer buffer is correct for every execution path, so I am reporting this primarily as an experimental result rather than proposing it as a complete fix.

Again, I am not experienced with contributing to FreeToken and this investigation/report was prepared with AI assistance. Please correct any incorrect assumptions or terminology.

I hope these results are useful, especially for investigating low-VRAM operation and GDN/FLA prefill memory usage.