FreeToken Roadmap (2026)

[jason-fxz]

FreeToken RoadMap - 2026

New Model Support

• GLM-5.3-Flash @andy-yang-1
• Qwen3.8-Flash-Next @jason-fxz

Hardware Support

• macOS: a native Metal engine on Apple Silicon Macs.
• AMD: support for AMD GPUs via ROCm.
• DGX Spark: aarch64 wheels, sm_121 kernels, and a unified-memory mode.

Multimodal

• Support Image input for vision-language models
  • Qwen3.5 / Qwen3.6 / Qwen3.8 family @jason-fxz
  • Gemma-4 family

Multi-GPU

• DGX Spark: multi-GPU optimisation for linked DGX Spark units.
• Tensor parallelism (TP): serve one model across multiple GPUs on one machine.

Quantization

• GGUF: support GGUF checkpoints and their quantization types across model architectures.

Speculative Decoding

• Speculative decoding, including MTP / DFlash / Dspark.

---

Any contribution is welcome. Join the Developer Slack if you want to get involved.

---

TODO: This is still WIP.

[dusty44]

Any thoughts about enabling experts to stream from NVME drives? with the cached experts optimised in three tiers vram, system ram then streamed from disk? I've built it in a dwarfstar derived engine and it can work shockingly well.

[Aladincz]

Is there any chance that this technology could be usable and give some benefits to old macs running small LLM on CPU? Would it be technically applicable?

[Headmaster218]

Interesting timing — I’ve independently been working on a fairly similar MoE serving problem on AMD RDNA3/Vulkan, mainly around expert caching/paging, heterogeneous CPU–GPU execution and large-model offloading.
It’s still very much WIP and I’m cleaning up the benchmarks/docs, but the current code is here: [infr]
Running on a single RX 7900 XTX + 64GB DDR4. Happy to compare notes on the AMD side.
Successfully running with
Qwen3.6-35B-A3B with decode at 150 ~ 40tk/s, prefill at 3000 ~ 400tk/s,
Qwen3.5-122B-A10B with decode at ~23tk/s
Ling 3.0 flash with decode at 36tk/s,
Deepseek V4 Flash at ~4tk/s

[orangelightening]

Looking forward to the qwen3.8 27b model support. I am using the qwen3.6 35b in the freetoken engine and using deepseek harness to serve it via the openai compatible endpoint. I have an rx3060 and 128gb of ram on an old xeon server and this is the best performing local ai system I have owned. Absolutely wonderful.

[Headmaster218]

Different from MoE, 27B is a kind of dense model, that doesnt suit this kind of cache architecture. But, Qwen team said they will public a new middle size MoE model this week, probably Qwen3.8-122B-A10B, (so thats why now im optimizing Qwen3.5-122B-A10B ;) ), and probably it will be more powerful than Qwen3.8 27B, so just, wait for it.

[erikfenriz]

Would be so awesome to see it on macOS!