Labels: bug, distill, priority:high
File: src/tether/finetune/backends/snapflow_backend.py:292 (single-pass loop), :390, :414
Description. The loop is for step, batch in enumerate(loader, 1): ... if step >= cfg.num_steps: break with no outer epoch/cycle wrapper and drop_last=True. If the dataloader yields fewer than num_steps batches, training ends when the dataset is exhausted, then still saves a checkpoint and reports status="ok".
Why it matters. tether distill --steps 10000 on a 600-batch dataset trains 600 steps, writes a checkpoint, and returns status="ok", training_steps_completed=600 — the user believes they got 10k with no warning. Silent quality loss on the paid continuous-learning product.
Tasks.
Dedup. PR #247 seeds randomness; #16 fixes gradient checkpointing; #19 multi-GPU DDP. None address the missing epoch loop. Distinct.
Labels:
bug,distill,priority:highFile:
src/tether/finetune/backends/snapflow_backend.py:292(single-pass loop),:390,:414Description. The loop is
for step, batch in enumerate(loader, 1): ... if step >= cfg.num_steps: breakwith no outer epoch/cyclewrapper anddrop_last=True. If the dataloader yields fewer thannum_stepsbatches, training ends when the dataset is exhausted, then still saves a checkpoint and reportsstatus="ok".Why it matters.
tether distill --steps 10000on a 600-batch dataset trains 600 steps, writes a checkpoint, and returnsstatus="ok", training_steps_completed=600— the user believes they got 10k with no warning. Silent quality loss on the paid continuous-learning product.Tasks.
itertools.cycle(loader); break only onstep >= num_steps.training_steps_completed < num_steps.Dedup. PR #247 seeds randomness; #16 fixes gradient checkpointing; #19 multi-GPU DDP. None address the missing epoch loop. Distinct.