---
base_model: BramVanroy/GEITje-7B-ultra
datasets:
- BramVanroy/ultra_feedback_dutch
library_name: peft
tags:
- alignment-handbook
- trl
- dpo
- generated_from_trainer
model-index:
- name: FinGEITje-7B-dpo
  results: []
language:
- nl
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/snoels/huggingface/runs/yng7mdb0)
# FinGEITje-7B-dpo

This model is a fine-tuned version of [/mnt/trained_models/fingeitje](https://huggingface.co//mnt/trained_models/fingeitje) on the BramVanroy/ultra_feedback_dutch dataset.
It achieves the following results on the evaluation set:
- Loss: 0.0279
- Rewards/chosen: -3.8986
- Rewards/rejected: -15.9713
- Rewards/accuracies: 0.9836
- Rewards/margins: 12.0727
- Logps/rejected: -1952.6360
- Logps/chosen: -789.0983
- Logits/rejected: -1.7369
- Logits/chosen: -1.8936

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 16
- total_train_batch_size: 64
- total_eval_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 1

### Training results

| Training Loss | Epoch  | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 0.1029        | 0.1327 | 100  | 0.1099          | -1.8067        | -5.3683          | 0.9679             | 3.5616          | -892.3373      | -579.9115    | -2.4775         | -2.3705       |
| 0.042         | 0.2654 | 200  | 0.0430          | -3.5129        | -10.6778         | 0.9828             | 7.1649          | -1423.2883     | -750.5289    | -1.9744         | -1.9895       |
| 0.0278        | 0.3981 | 300  | 0.0344          | -3.7335        | -13.5153         | 0.9828             | 9.7818          | -1707.0360     | -772.5893    | -1.7454         | -1.8191       |
| 0.0223        | 0.5308 | 400  | 0.0308          | -3.6554        | -13.7712         | 0.9858             | 10.1158         | -1732.6289     | -764.7831    | -1.8020         | -1.9184       |
| 0.0378        | 0.6635 | 500  | 0.0297          | -4.0018        | -16.3285         | 0.9851             | 12.3266         | -1988.3542     | -799.4221    | -1.6924         | -1.8650       |
| 0.0352        | 0.7962 | 600  | 0.0278          | -3.8104        | -15.6430         | 0.9836             | 11.8327         | -1919.8119     | -780.2752    | -1.7437         | -1.8978       |
| 0.0238        | 0.9289 | 700  | 0.0279          | -3.8974        | -15.9642         | 0.9828             | 12.0668         | -1951.9310     | -788.9780    | -1.7371         | -1.8937       |


### Framework versions

- PEFT 0.11.1
- Transformers 4.42.4
- Pytorch 2.3.1
- Datasets 2.20.0
- Tokenizers 0.19.1