Please help support older hardware (1080/2080 series)

[gdevenyi]

The current install uses a pytorch version which doesn't support my hardware.

(freetoken) [gdevenyi@monster freetoken]$ ft bench
usage: ft bench <subcommand> [args]

Subcommands:
  bw   Benchmark CPU vs PCIe bandwidth and pick the MoE backend (hybrid/offload)

Use "ft bench <subcommand> --help" for subcommand-specific options.
(freetoken) [gdevenyi@monster freetoken]$ ft bench bw
/home/gdevenyi/projects/freetoken/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:371: UserWarning: Found GPU0 NVIDIA GeForce GTX 1080 Ti which is of compute capability (CC) 6.1.
The following list shows the CCs this version of PyTorch was built for and the hardware CCs it supports:
- 7.5 which supports hardware CC >=7.5,<8.0
- 8.0 which supports hardware CC >=8.0,<9.0 except {8.7}
- 8.6 which supports hardware CC >=8.6,<9.0 except {8.7}
- 9.0 which supports hardware CC >=9.0,<10.0
- 10.0 which supports hardware CC >=10.0,<11.0 except {10.1}
- 12.0 which supports hardware CC >=12.0,<13.0
Please follow the instructions at https://pytorch.org/get-started/locally/ to install a PyTorch release that supports one of these CUDA versions: 12.6
  _warn_unsupported_code(d, device_cc, code_ccs)
/home/gdevenyi/projects/freetoken/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:489: UserWarning:
NVIDIA GeForce GTX 1080 Ti with CUDA capability sm_61 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_75 sm_80 sm_86 sm_90 sm_100 sm_120.
If you want to use the NVIDIA GeForce GTX 1080 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  queued_call()
[2026-08-21|22:22:56] INFO     benchbw: measuring hardware ceilings on cuda:0 ...
[2026-08-21|22:22:58] INFO     benchbw: dtype bf16 real kernels ...
[2026-08-21|22:22:58] WARNING  benchbw: PCIe gather bench failed for dtype:bf16/bf16: CUDA error: no kernel image is available for execution on the device
Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[2026-08-21|22:22:59] INFO     benchbw: dtype nvfp4 real kernels ...
[2026-08-21|22:22:59] WARNING  benchbw: PCIe gather bench failed for dtype:nvfp4/nvfp4: CUDA error: no kernel image is available for execution on the device
Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[2026-08-21|22:23:00] INFO     benchbw: dtype fp8 real kernels ...
[2026-08-21|22:23:00] WARNING  benchbw: PCIe gather bench failed for dtype:fp8/fp8_block: CUDA error: no kernel image is available for execution on the device
Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[2026-08-21|22:23:00] INFO     benchbw: dtype mxfp4 real kernels ...
[2026-08-21|22:23:00] WARNING  benchbw: PCIe gather bench failed for dtype:mxfp4/mxfp4_triton: CUDA error: no kernel image is available for execution on the device
Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[2026-08-21|22:23:01] INFO     benchbw: dtype ds_fp4 real kernels ...
[2026-08-21|22:23:01] WARNING  benchbw: PCIe gather bench failed for dtype:ds_fp4/ds_fp4: CUDA error: no kernel image is available for execution on the device
Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


  host monster   gpu cuda:0 (NVIDIA GeForce GTX 1080 Ti)   cpu 20c/20t
  ceilings: CPU STREAM read 64.0  |  PCIe linear H2D 12.4  D2H 13.2  GB/s   (threshold 2.0x)

  per-dtype (tuning — what the runtime backend pick matches on)
    format      expert       CPU-MoE   PCIe-gather  CPU/PCIe  backend
    bf16       9.00 MB     23.1 GB/s           n/a         —  offload
           └─ pcie gather unavailable (CUDA error: no kernel image is available for execution on the device
Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
)
    nvfp4      7.61 MB     18.3 GB/s           n/a         —  offload
           └─ pcie gather unavailable (CUDA error: no kernel image is available for execution on the device
Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
)
    fp8            n/a           n/a           n/a         —  offload
           └─ pcie gather unavailable (CUDA error: no kernel image is available for execution on the device
Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
) | CPU MoE has no fp8_block weight path; hybrid unavailable
    mxfp4     12.62 MB     15.3 GB/s           n/a         —  offload
           └─ pcie gather unavailable (CUDA error: no kernel image is available for execution on the device
Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
)
    ds_fp4    12.75 MB     18.3 GB/s           n/a         —  offload
           └─ pcie gather unavailable (CUDA error: no kernel image is available for execution on the device
Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
)

  saved: /home/gdevenyi/.cache/freetoken/benchbw.json
``

[gdevenyi]

Downgrading using:

uv pip install -U torch --index-url https://download.pytorch.org/whl/cu126

Results in:

Traceback (most recent call last):
  File "/home/gdevenyi/projects/freetoken/.venv/lib/python3.11/site-packages/freetoken/kernel/pinned.py", line 19, in _load_pinned_extension
    return importlib.import_module("freetoken.kernel._pinned_tensor")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gdevenyi/.local/share/uv/python/cpython-3.11.15-linux-x86_64-gnu/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 676, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 573, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1233, in create_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: libcudart.so.13: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/gdevenyi/projects/freetoken/.venv/bin/ft", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/gdevenyi/projects/freetoken/.venv/lib/python3.11/site-packages/freetoken/cli.py", line 126, in main
    return runner(args[1:])
           ^^^^^^^^^^^^^^^^
  File "/home/gdevenyi/projects/freetoken/.venv/lib/python3.11/site-packages/freetoken/cli.py", line 87, in _run_bench
    return main(argv[1:], prog="ft bench bw")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gdevenyi/projects/freetoken/.venv/lib/python3.11/site-packages/freetoken/moe/benchbw.py", line 953, in main
    result = run_benchbw(
             ^^^^^^^^^^^^
  File "/home/gdevenyi/projects/freetoken/.venv/lib/python3.11/site-packages/freetoken/moe/benchbw.py", line 721, in run_benchbw
    pcie = measure_pcie_bw(device, nbytes=pcie_bytes, iters=pcie_iters)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gdevenyi/projects/freetoken/.venv/lib/python3.11/site-packages/freetoken/moe/benchbw.py", line 249, in measure_pcie_bw
    src = alloc_pinned_tensor(n, dtype=torch.bfloat16)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gdevenyi/projects/freetoken/.venv/lib/python3.11/site-packages/freetoken/kernel/pinned.py", line 45, in alloc_pinned_tensor
    return _load_pinned_extension().alloc_pinned_tensor(list(shape), dtype)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gdevenyi/projects/freetoken/.venv/lib/python3.11/site-packages/freetoken/kernel/pinned.py", line 21, in _load_pinned_extension
    raise ImportError(
ImportError: freetoken.kernel._pinned_tensor is not installed. Reinstall FreeToken so the pinned tensor CUDA extension is built at install time.

And trying to install from scratch:

(freetoken) [gdevenyi@monster freetoken]$ uv pip install "freetoken[accel]" --index-url https://download.pytorch.org/whl/cu126
  × No solution found when resolving dependencies:
  ╰─▶ Because there are no versions of freetoken[accel] and you require freetoken[accel], we can conclude that your requirements are unsatisfiable.

[andy-yang-1]

RTX 10 and 20 are on the roadmap. Stay tuned!

[rog77]

RTX 10 and 20 are on the roadmap. Stay tuned!

I don't suppose V100 is on the road map too? Thanks :-)

[apcameron]

See #19
It works on my Tesla P40 Pascal Based GPU