Support pre-Volta GPUs (sm_61 / Pascal) - #19
Conversation
Two device-side constructs in the JIT kernels are sm_70+ and make nvcc/ptxas reject the whole translation unit on Pascal (sm_6x), so the offload gather, the index kernels and the KV store never build there: - `__grid_constant__` on kernel parameters (compute_70+) - the `L1::no_allocate` modifier on `ld.global` (sm_70+) Both are codegen hints with no semantic content: the parameter is passed identically without the annotation, and the load returns the same bytes without the cache hint. Gate each behind `__CUDA_ARCH__ < 700` so every sm_70+ device pass and the host pass emit exactly what they did before. `__nanosleep` in the same header was already guarded this way. Also stop a failed CUDA call in the pinned-tensor extension from poisoning the process: the runtime keeps a failure in the per-thread last-error slot, so the next unrelated `C10_CUDA_CHECK` reported it instead of its own result. This was visible as an unrelated `torch.empty` on CUDA raising "invalid argument" after `host_device_ptr` rejected unregistered memory. `cudaHostGetDevicePointer` on *unregistered* memory is unspecified -- newer arches let UVA degenerate it to identity, Pascal validates registration and returns cudaErrorInvalidValue. Widen the test to accept either, and keep asserting the invariant that actually matters: it must never return a different nonzero alias. The in-contract case stays covered by test_host_bank_pin_registers_and_translates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KQy3DziJN9peJ7nA8L59ns
`_fast_tanh` emits `tanh.approx.f32` as inline PTX. That instruction is sm_75+,
and ptxas rejects the whole module without it ("Feature 'tanh' requires .target
sm_75 or higher"), so every GELU_TANH activation failed to compile on Pascal.
Gate it on a `constexpr_function` reading the compilation target, the same idiom
e4m3_compat.py uses to branch fp8e4nv below sm_89. Being compile-time, this adds
no kernel parameter and leaves the cache key untouched: sm_75+ still takes the
single-instruction path, folds the branch away, and emits the PTX it did before.
Older cards compute the same tanh through libdevice, which this file already
depends on for `libdevice.erf`.
Adds tests/kernels/test_activation.py: the four *_and_mul kernels against torch
references, plus a saturation case at +-1e4 that would catch a fallback that
overflows to nan at the tails. gelu_tanh had no direct coverage before -- it was
only reached transitively through the MoE tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KQy3DziJN9peJ7nA8L59ns
`moe_align.py` ranks its scatter with `tl.atomic_add`. Triton lowers every atomic to a scoped *and* ordered PTX encoding -- `atom.global.gpu.<sem>.add` -- and both the `.gpu` scope and every memory order (`.relaxed` included, the weakest it can emit) arrived with sm_70. ptxas therefore rejects the entire module on Pascal, and no `sem=`/`scope=` argument avoids it. Verified by probing all four sems and all three scopes on an sm_61 device: every one fails to assemble. Rather than emulate atomics, route around them. `moe_impl.py` already carries a second implementation -- the staged `moe_align_block_size_triton`, a counts/cumsum/binary-search chain across five launches with no atomic anywhere. It produces the same three buffers, so pre-sm_70 dispatches there. Checked against a reference of the documented contract on sm_61 across decode and prefill shapes (1..1024 rows, 8..257 experts, block 16..64): identical padding, every token placed exactly once inside its own expert's region. sm_70+ is untouched -- same branch, same kernel, same launch as before. The staged path costs five launches instead of one, which is the right trade against not running. Not attempted: emulating the atomic with unqualified inline PTX (`atom.global.add.u32` is sm_20+). It assembles and gives correct ranks, but `tl.inline_asm_elementwise` is contracted for *pure* elementwise ops -- when the operand tensor is narrower than the thread block, the layout replicates elements across threads and a side-effecting instruction executes once per replica. A 128-thread block over an 8-wide tensor adds 16x. That silently miscounts at exactly the decode widths that matter (numel = batch x topk), so it is not a safe basis for expert dispatch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KQy3DziJN9peJ7nA8L59ns
… budget
`_select_extend_tile` already shrank the prefill tile when a device's opt-in shared
memory could not hold it, but two gaps let it overflow anyway:
- head_dim <= 128 returned (128, 64) before any budget check, so the most common
head sizes never consulted the device at all;
- the ladder stopped one rung short of what a 48KB budget needs (pre-Volta gets
48KB per block with no opt-in at all).
Carry the ladder down to (16, 16) -- the floor, since `tl.dot` needs N >= 16 -- and
budget-check every rung including the largest.
The byte estimate is also only a lower bound: what triton allocates depends on how it
schedules the pipeline, and the split kernel wants about twice the q/k/v tile bytes.
A head_dim 64 model passed the estimate and still needed 64KB. Rather than carry a
per-kernel fudge factor, `_select_extend_tiles` now returns the descending ladder and
`_launch_first_fitting_tile` walks it, letting the compiler have the final say:
OutOfResources is raised during launch setup, after the compile and before any GPU
work, so retrying smaller is side-effect free and triton caches each compile.
The split-k decode kernel had no sizing at all: BLOCK_N=32 with num_stages=2
hardcoded, whose pipelined k/v tiles want 128KB at head_dim 512 -- fine on A100/H100,
over budget on consumer Ampere/Ada, never mind Pascal. Add `_select_decode_tile`,
stepping 32/2 -> 32/1 -> 16/1.
Both selectors only ever shrink a config that does not fit, and the existing
tile-selection test still pins the exact choice for every datacenter and consumer
budget it covered before, so any device that fits the current default keeps it.
`smem_optin == 0` (unavailable) keeps the prior choice. Devices that previously
raised OutOfResources here may now run.
head_dim 512 extend has no fitting configuration on a 48KB device rather than a
merely slower one -- 16x16 q/k/v tiles alone want 48KB. Its test now skips with that
reason, gated on the device's actual budget so it still runs everywhere it can.
On sm_61: tests/kernels/test_triton_attention.py goes from 9 failed to 35 passed,
2 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KQy3DziJN9peJ7nA8L59ns
The triton top-p/top-k threshold search reduces through `tl.atomic_add` and `tl.atomic_max`. Triton lowers every atomic to a scoped, ordered PTX encoding, and both the scope and every memory order it can emit arrived with sm_70, so ptxas rejects those kernels on Pascal. No unit test covered them, so this surfaced only when serving: `top_k` and `top_p` are set by default for most models, and the first real request killed the scheduler with a raw ptxas dump. Sorting needs no atomics. `kernel/torch_sampling.py` reproduces the same selection with torch ops -- top-k by the k-th largest value, top-p by the shortest descending prefix that reaches p, both keeping the elements the kernel path keeps, including ties and the crossing element. `softmax` has no atomics and is re-exported from the triton module unchanged, so only the threshold search changes. The draw uses cumsum + searchsorted rather than torch.multinomial: it stays capturable in a CUDA graph and never syncs to the host, matching what the kernel path guarantees. Seeding mirrors `triton.sampling._gen_u`, including its plain-torch.rand path while a stream is capturing. Selected in `sample_impl` only when flashinfer is absent *and* the device is pre-sm_70, so every currently supported GPU keeps the kernel path untouched. Full-vocabulary sorting is slower than the bracketed histogram search it replaces. On a card that cannot run the kernels at all, sampling is not the bottleneck. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KQy3DziJN9peJ7nA8L59ns
|
I can confirm that this PR works on my Tesla P40 which is a Pascal Based GPU with 24GB VRAM on my computer with 128GB RAM ft shell --server 192.168.0.100:5001
Thinking... [ (Spaces are added only for readability; the 100‑digit fractional part is the str Why 100 digits? [done] gpt-oss-120b (medium) | ↓77 ↑497 4.7 tok/s | cache 1255 lru 27.2% | kv 57 |
|
This PR also fixes a Turing (sm_75) crash, which the title undersells — worth noting since
Verified on a Tesla T4, running upstream's own decode parametrization
That middle row is the surprising one: the PR named "fix(cuda): support Turing GPUs" does not Boundary on One detail that may be useful for the tile ladder: on this device Practical impact: this is not an exotic geometry. Reproduces only on devices with ≤64 KiB opt-in shared memory (sm_75 and below) — Environment: FreeToken |
|
I tested this on a GTX 1060 6GB, Pascal Hardware was a GTX 1060 6GB with about 5.1 GiB free VRAM after the Windows desktop, i9-9900K, 32 GB RAM, Docker Desktop with WSL2. The container was running driver 582.28, CUDA 12.6, Python 3.12.14, torch 2.11.0+cu126, and Triton 3.6.0. I tested the merge of Model was Kernel tests
One failure is:
kernel needs 96 KiB of shared memory per block. Pascal only has 48 KiB, so it cannot fit on
ServingEnd-to-end serving works. Auto-selection picked: CUDA graph capture also succeeded at batch sizes 1, 2, and 4. The main thing I found is that
CPU-MoE is about 2.32x faster than PCIe gather The problem is that Measured difference:
Offload made the first request look basically hung. Hybrid made the same setup usable. VRAM after loading was: That leaves about 0.4 GiB of headroom. Host RAM usage for the container was about 11.4 GiB. With hybrid and a warm Triton cache, repeat TTFT landed around 1.3 to 2.9 seconds. fp16I also tried forcing: That does not boot It OOMs during weight materialization: Auto-selected bf16 loads successfully with the same amount of free VRAM. I reproduced the fp16 OOM twice. PR #26One separate issue I hit while testing #26. The group-scoped uv source does not appear to control torch's normal Running: resolved torch from PyPI instead of the cu126 index, which pulled the cu130 build. The nvcc/torch toolchain check then failed. For testing I worked around it by pointing torch's source entry directly at the cu126 index. ReproductionDocker base: The
Setup was:
For Pascal I would persist once, then either let auto use the saved profile or explicitly serve with: All numbers above are from this one machine at temperature 0. Repeated runs produced byte-identical token counts. |
|
I have 6x Tesla P4 (sm_61) in a single box, and an external P100 (sm_60) so I'll also test this PR as soon as I get chance (too excited for Qwen 3.8 Flash MoE at the moment, so that's taking priority). |
5 participants
FreeToken cannot currently be installed or run on a pre-Volta GPU. This branch gets it serving on a GTX 1080 Ti (sm_61, Pascal). It does not change what any supported GPU does.
Result
Serving
Qwen/Qwen2.5-0.5B-Instructon a GTX 1080 Ti, with default sampling:Test suite on that card: 66 failed and 1270 passed becomes 33 failed and 1327 passed, with 29 skipped.
What was blocking
Five separate issues. Each is a construct that NVIDIA introduced after Pascal.
__grid_constant__(compute_70 and up) andld.global.L1::no_allocate(sm_70 and up) in the JIT kernels. nvcc and ptxas reject the whole translation unit, so the offload gather, the index kernels and the KV store never build.__CUDA_ARCH__ < 700, as__nanosleepin the same header already is.tanh.approx.f32(sm_75 and up) in_fast_tanh.constexpr_functiongate and a libdevice fallback, following whate4m3_compat.pyalready does.tl.atomic_addinmoe_align. Triton lowers every atomic to a scoped and ordered PTX encoding, all of which need sm_70.moe_align_block_size_triton.tl.atomic_addandtl.atomic_maxin the top-k and top-p threshold search.Both codegen hints in the first row are semantically empty. The parameter is passed identically without the annotation, and the load returns the same bytes without the cache hint.
Testing
I cannot re-test this on
mainmyself. The machine I have access to holds two sm_89 cards and no pre-Volta GPU, so I can only confirm that the gates compile and that the sm_89 paths still behave. The GTX 1080 Ti figures above stand as originally measured.Anyone with a Pascal or Maxwell card who can re-run the suite would be worth more to this PR than anything further I can do.
Note that a pre-Turing card also needs #26, because the pinned CUDA 13 torch build carries no cubin below sm_75.