Large-model HF download shows no progress and looks stuck; server stays 503 'loading'

[ackalker]

Symptom

Running ft serve --model <HF-id> with a frontier MoE checkpoint (hundreds of GB) appears to stall downloading after fetching a few small files, and any inference request during that time returns 503 "model is still loading" / "server unavailable".

Root cause

The checkpoint download in python/freetoken/utils/hf.py disables its progress bar entirely:

def download_hf_weight(model_path: str) -> str:
    if os.path.isdir(model_path):
        return model_path
    try:
        return snapshot_download(
            model_path,
            allow_patterns=["*.safetensors"],
            tqdm_class=DisabledTqdm,   # <-- forces disable=True
        )
    except Exception as e:
        raise ValueError(...)

DisabledTqdm sets disable=True, and huggingface_hub's _create_progress_bar honors it, so all download progress output is suppressed. The download itself works — I measured ~70 MB/s (11 GB of a 16 GB 4-shard model in ~150 s; a fresh 4.5 GB model downloaded cleanly in <50 s). But with zero feedback, a legitimately long download of a 100s-GB checkpoint looks exactly like a stall.

The 503 is a direct, expected consequence: the download runs inside the primary scheduler worker (_run_scheduler -> Scheduler(args) in server/launch.py), which only sends its "ready" ack after weights are loaded. Until then the frontend maintenance_state stays "loading", and every API adapter returns 503.

Why it reads as "stalls after small files"

Small files (config.json, tokenizer) download fast and their bars are not disabled, so you see them complete — then the large safetensors download proceeds silently and appears stuck.

No recorded rationale

DisabledTqdm and its snapshot_download usage were introduced verbatim in the initial open-source release (3af9d90, feat: initial open-source release) with no explanatory comment, and the only later edit to hf.py (7a1853d) touched config loading, not the download. The multi-rank weight-loading bars in python/freetoken/utils/progress.py are gated on tensor_parallel_size > 1 ("so multi-rank logs stay clean"); the HF download bar is not, so single-GPU serves lose all feedback.

Suggested fix

• Only suppress the bar when it is actually noisy (e.g. tensor_parallel_size > 1); otherwise show a byte-level bar so large downloads are observable.
• Add resume_download=True (and/or prune stale *.incomplete blobs) so an interrupted download resumes instead of restarting or hanging.
• Optionally forward the download via the existing ("progress", ...) ack channel so the frontend can show real progress instead of a static "loading" 503.