yakazimir/smollm_1_7B_tulu3

HuggingFaceTB/SmolLM2-1.7B

This model is a fine-tuned version of HuggingFaceTB/SmolLM2-1.7B on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.7823

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 4
• total_train_batch_size: 256
• total_eval_batch_size: 64
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss
1.3348	0.0551	200	1.2704
1.0411	0.1101	400	1.0435
1.0483	0.1652	600	0.9694
0.8801	0.2202	800	0.9227
0.8996	0.2753	1000	0.8888
0.8682	0.3303	1200	0.8648
0.8757	0.3854	1400	0.8468
0.8441	0.4404	1600	0.8311
0.8197	0.4955	1800	0.8206
0.7807	0.5505	2000	0.8090
0.7757	0.6056	2200	0.8015
0.7818	0.6607	2400	0.7957
0.8235	0.7157	2600	0.7915
0.7854	0.7708	2800	0.7883
0.7958	0.8258	3000	0.7863
0.8192	0.8809	3200	0.7829
0.765	0.9359	3400	0.7824
0.7939	0.9910	3600	0.7824

Framework versions

• Transformers 4.48.3
• Pytorch 2.2.2+cu121
• Datasets 2.18.0
• Tokenizers 0.21.0