Question about training loss

[zjr2000]

Hi,

I’m reproducing continued pre-training with LLaMA Factory, initializing from qwen3-base. The loss starts around 30 in the first steps, which is much higher than with standard AR training. When I initialize from your released checkpoints, the loss starts around 8, which also seems relatively high.

From your experience, is this expected behavior, or does it suggest a configuration issue on my side?

Thanks!

[zjr2000]

Here is my config. The loss appears normal when I use your example config for sft.

### model
model_name_or_path: ../model/SDAR-2B-Chat
train_from_scratch: false
trust_remote_code: true

### method
stage: pt
do_train: true
finetuning_type: full
deepspeed: examples/deepspeed/ds_z0_config.json
# gradient_checkpointing: true

### dataset
dataset: slimpajama-6B
template: qwen3
block_length: 4
cutoff_len: 8193 # should be context length + 1
truncate_mode: drop 
overwrite_cache: true
tokenized_path: ../cached_tokenized_dataset/slimpajama-6B
max_samples: 1000
preprocessing_num_workers: 8
dataloader_num_workers: 4
packing: true
# neat_packing only support in sft
# neat_packing: true
# streaming: true
# max_steps: 1000

### output
output_dir: ../output/sdar_2b_qwen3_init_cpt_slimpajama
logging_steps: 1
save_steps: 2000
save_total_limit: 2
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: tensorboard  # choices: [none, wandb, tensorboard, swanlab, mlflow]

### train
run_name: sdar_2b_qwen3_init_cpt_slimpajama
include_effective_tokens_per_second: true
per_device_train_batch_size: 1
gradient_accumulation_steps: 4
max_steps: 1000
learning_rate: 2.0e-5
# num_train_epochs: 10.0
lr_scheduler_type: constant_with_warmup
warmup_ratio: 0.01
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null

### eval
# val_size: 0.1
# per_device_eval_batch_size: 1
# eval_strategy: steps
# eval_steps: 500

[zjr2000]

Hi,

I found that the loss scales with gradient_accumulation_steps, similar to huggingface/transformers#40564 and hiyouga/LlamaFactory#7743. When I set accepts_loss_kwargs to False, the behavior goes back to normal. Here is the code I changed:

class SDARModel(SDARPreTrainedModel):
    accepts_loss_kwargs = False
    ...

class SDARForCausalLM(SDARPreTrainedModel, GenerationMixin):
    _tied_weights_keys = ["lm_head.weight"]
    _tp_plan = {"lm_head": "colwise_rep"}
    _pp_plan = {"lm_head": (["hidden_states"], ["logits"])}
    accepts_loss_kwargs = False
    ...

I don’t think loss_kwargs are actually used in the model, so maybe you could merge this change into your code.

[chengshuang18]

Please refer to this issue regarding the problem of relatively high loss:

#26 (comment).