laurent-maille/Valuoty-industry-plc-7B-V0.8Adap

See axolotl config

axolotl version: 0.8.0.dev0

# === MODÈLE ===
#base_model: grimjim/mistralai-Mistral-7B-v0.3
base_model: mistralai/Mistral-7B-Instruct-v0.3
model_type: MistralForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true

# === LORA CONFIGURATION ===
adapter: lora
lora_r: 32
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]
lora_modules_to_save:
  - lm_head

# === DATASET ===
datasets:
  - path: laurent-maille/pcl-test-S27
    type: chat_template
    field_messages: messages 
chat_template: chatml

test_size: 0.2  #  Split automatique 80/20
shuffle: true

dataset_prepared_path: ./prepared/plc_sharegpt
output_dir: ./outputs/valuoty-indus-plc-7BV2

# === DATA LOADING ===
dataloader_num_workers: 4
dataloader_pin_memory: true



# === SEQUENCES ===
sequence_len: 4096
sample_packing: true 
pad_to_sequence_len: true
train_on_inputs: false
group_by_length: true # batching par longueur pour meilleur débit

# === QUANTIZATION ===
load_in_4bit: true
bnb_4bit_quant_type: nf4
bnb_4bit_use_double_quant: true
bnb_4bit_compute_dtype: bfloat16
bf16: true

# === OPTIMIZER ===
optimizer: paged_adamw_8bit
learning_rate: 1e-4
weight_decay: 0.0
max_grad_norm: 0.3
lr_scheduler: cosine
warmup_ratio: 0.05
seed: 42

# === TRAINING PARAMETERS ===
epochs: 6
micro_batch_size: 6
gradient_accumulation_steps: 16
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
resize_token_embeddings_to_multiple_of: 8
mask_user_tokens: true


# === ATTENTION ===
flash_attention_2: true

# === SAVING & EVALUATION ===
save_safetensors: true
save_strategy: steps
save_steps: 20
save_total_limit: 4
evaluation_strategy: steps
per_device_eval_batch_size: 5
eval_steps: 10
logging_steps: 5

# === EARLY STOPPING  ===
load_best_model_at_end: true
metric_for_best_model: eval_loss  
greater_is_better: false
#early_stopping_patience: 3

# === LOGGING ===
wandb_project: Valuoty-plc
wandb_run_name: V0.8_a40_run_01
wandb_watch: gradients

outputs/valuoty-indus-plc-7BV2

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.3 on the laurent-maille/pcl-test-S27 dataset.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 6
• eval_batch_size: 6
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 96
• optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 2
• num_epochs: 1.0

Training results

Framework versions

• PEFT 0.14.0
• Transformers 4.49.0
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0