Model save

README.md

@@ -0,0 +1,161 @@
+---
+library_name: transformers
+license: apache-2.0
+base_model: Qwen/Qwen3-0.6B
+tags:
+- axolotl
+- generated_from_trainer
+datasets:
+- Rexhaif/wmt23-pairs-sft
+model-index:
+- name: Qwen3-0.6B-MTEval-SFT
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
+<details><summary>See axolotl config</summary>
+
+axolotl version: `0.9.2`
+```yaml
+base_model: Qwen/Qwen3-0.6B
+# Automatically upload checkpoint and final model to HF
+hub_model_id: Rexhaif/Qwen3-0.6B-MTEval-SFT
+hub_private_repo: false
+
+load_in_8bit: false
+load_in_4bit: false
+strict: false
+
+chat_template: tokenizer_default
+datasets:
+  - path: Rexhaif/wmt23-pairs-sft
+    split: "train"
+    type: chat_template
+    field_messages: messages
+    roles_to_train: ["assistant"]
+
+shuffle_merged_datasets: true
+
+skip_prepare_dataset: false
+dataset_prepared_path: ./data/wmt23-pairs-sft
+output_dir: /hnvme/workspace/v106be28-outputs/sft-0.6b
+
+dataloader_prefetch_factor: 32
+dataloader_num_workers: 2
+dataloader_pin_memory: true
+
+gc_steps: 1
+
+sequence_len: 512
+sample_packing: false
+eval_sample_packing: false
+pad_to_sequence_len: false
+
+wandb_project: llm-reasoning-mt-eval
+wandb_entity:
+wandb_name: qw3-0.6b-sft
+
+plugins:
+  - axolotl.integrations.liger.LigerPlugin
+liger_rope: true
+liger_rms_norm: true
+liger_glu_activation: true
+liger_layer_norm: true
+liger_fused_linear_cross_entropy: true
+gradient_accumulation_steps: 1
+micro_batch_size: 64  # should match num_generations / num_gpus
+
+optimizer: adamw_torch_fused
+lr_scheduler: cosine
+learning_rate: 5.0e-5
+cosine_min_lr_ratio: 1.0e-7
+max_grad_norm: 1.0
+weight_decay: 0.1
+
+bf16: true
+tf32: true
+
+flash_attention: true
+flash_attn_fuse_qkv: true
+flash_attn_fuse_mlp: true
+auto_resume_from_checkpoints: true
+
+n_epochs: 3
+logging_steps: 10
+warmup_ratio: 0.1
+evals_per_epoch: 10
+saves_per_epoch: 10
+save_total_limit: 1
+#max_steps: 5000
+seed: 42
+val_set_size: 0.01
+
+gradient_checkpointing: false
+gradient_checkpointing_kwargs:
+  use_reentrant: false
+
+```
+
+</details><br>
+
+# Qwen3-0.6B-MTEval-SFT
+
+This model is a fine-tuned version of [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) on the Rexhaif/wmt23-pairs-sft dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.0486
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-05
+- train_batch_size: 64
+- eval_batch_size: 64
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 4
+- total_train_batch_size: 256
+- total_eval_batch_size: 256
+- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 101
+- num_epochs: 1.0
+
+### Training results
+
+| Training Loss | Epoch  | Step | Validation Loss |
+|:-------------:|:------:|:----:|:---------------:|
+| No log        | 0.0010 | 1    | 7.5881          |
+| 0.2427        | 0.1003 | 102  | 0.2504          |
+| 0.2062        | 0.2006 | 204  | 0.1936          |
+| 0.1631        | 0.3009 | 306  | 0.1606          |
+| 0.1315        | 0.4012 | 408  | 0.1243          |
+| 0.0999        | 0.5015 | 510  | 0.1098          |
+| 0.0871        | 0.6018 | 612  | 0.0871          |
+| 0.0611        | 0.7021 | 714  | 0.0702          |
+| 0.0586        | 0.8024 | 816  | 0.0564          |
+| 0.0478        | 0.9027 | 918  | 0.0486          |
+
+
+### Framework versions
+
+- Transformers 4.51.3
+- Pytorch 2.6.0+cu124
+- Datasets 3.5.1
+- Tokenizers 0.21.1

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "bos_token_id": 151643,
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "pad_token_id": 151643,
+  "temperature": 0.6,
+  "top_k": 20,
+  "top_p": 0.95,
+  "transformers_version": "4.51.3"
+}