See axolotl config

axolotl version: 0.4.1

adapter: qlora
auto_resume_from_checkpoints: true
base_model: fxmarty/really-tiny-falcon-testing
bf16: auto
chat_template: llama3
dataset_prepared_path: null
dataset_processes: 6
datasets:
- data_files:
  - e87224c8eb065bc2_train_data.json
  ds_type: json
  format: custom
  path: /workspace/input_data/e87224c8eb065bc2_train_data.json
  type:
    field_input: intent
    field_instruction: instruction
    field_output: response
    format: '{instruction} {input}'
    no_input_format: '{instruction}'
    system_format: '{system}'
    system_prompt: ''
debug: null
deepspeed: null
early_stopping_patience: 3
eval_max_new_tokens: 128
eval_steps: 2000
eval_table_size: null
evals_per_epoch: null
flash_attention: false
fp16: false
fsdp: null
fsdp_config: null
gradient_accumulation_steps: 1
gradient_checkpointing: false
group_by_length: false
hub_model_id: error577/f4e4984a-1046-4a12-80b7-8c5f44f7a124
hub_repo: null
hub_strategy: checkpoint
hub_token: null
learning_rate: 0.0002
load_in_4bit: true
load_in_8bit: false
local_rank: null
logging_steps: 1
lora_alpha: 16
lora_dropout: 0.1
lora_fan_in_fan_out: null
lora_model_dir: null
lora_r: 8
lora_target_linear: true
lr_scheduler: cosine
max_grad_norm: 1.0
max_steps: null
micro_batch_size: 8
mlflow_experiment_name: /tmp/e87224c8eb065bc2_train_data.json
model_type: AutoModelForCausalLM
num_epochs: 3
optimizer: adamw_bnb_8bit
output_dir: miner_id_24
pad_to_sequence_len: true
resume_from_checkpoint: null
s2_attention: null
sample_packing: false
save_steps: 2000
sequence_len: 512
strict: false
tf32: false
tokenizer_type: AutoTokenizer
train_on_inputs: false
trust_remote_code: true
val_set_size: 0.005
wandb_entity: null
wandb_mode: online
wandb_name: 1cd1f24c-cd92-4120-99af-dc7e5e1a2626
wandb_project: Gradients-On-Demand
wandb_run: your_name
wandb_runid: 1cd1f24c-cd92-4120-99af-dc7e5e1a2626
warmup_steps: 30
weight_decay: 0.0
xformers_attention: null

f4e4984a-1046-4a12-80b7-8c5f44f7a124

This model is a fine-tuned version of fxmarty/really-tiny-falcon-testing on the None dataset. It achieves the following results on the evaluation set:

• Loss: 10.9646

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 30
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
11.0862	0.0000	1	11.0843
10.9987	0.0538	2000	10.9976
10.9932	0.1076	4000	10.9906
10.9885	0.1614	6000	10.9861
10.995	0.2152	8000	10.9831
10.9707	0.2690	10000	10.9803
10.9651	0.3228	12000	10.9776
10.9695	0.3766	14000	10.9759
10.9875	0.4304	16000	10.9745
10.9761	0.4842	18000	10.9730
10.966	0.5380	20000	10.9719
10.9683	0.5918	22000	10.9714
10.9792	0.6456	24000	10.9708
10.9706	0.6994	26000	10.9704
10.9697	0.7532	28000	10.9696
10.9873	0.8070	30000	10.9694
10.97	0.8608	32000	10.9688
10.9856	0.9146	34000	10.9686
10.9675	0.9684	36000	10.9682
10.9662	1.0222	38000	10.9679
10.9676	1.0760	40000	10.9676
10.9814	1.1298	42000	10.9676
10.9753	1.1836	44000	10.9673
10.971	1.2374	46000	10.9671
10.9726	1.2912	48000	10.9669
10.9625	1.3450	50000	10.9668
10.964	1.3988	52000	10.9665
10.973	1.4526	54000	10.9663
10.9662	1.5063	56000	10.9661
10.9552	1.5601	58000	10.9661
10.9673	1.6139	60000	10.9659
10.9705	1.6677	62000	10.9657
10.9763	1.7215	64000	10.9658
10.9759	1.7753	66000	10.9655
10.9692	1.8291	68000	10.9655
10.9806	1.8829	70000	10.9655
10.9717	1.9367	72000	10.9652
10.9796	1.9905	74000	10.9652
10.9789	2.0443	76000	10.9651
10.979	2.0981	78000	10.9651
10.9706	2.1519	80000	10.9650
10.9829	2.2057	82000	10.9649
10.9865	2.2595	84000	10.9649
10.9722	2.3133	86000	10.9648
10.975	2.3671	88000	10.9648
10.9728	2.4209	90000	10.9647
10.9681	2.4747	92000	10.9647
10.9694	2.5285	94000	10.9647
10.9733	2.5823	96000	10.9646
10.9622	2.6361	98000	10.9647
10.9594	2.6899	100000	10.9646
10.9764	2.7437	102000	10.9646
10.9628	2.7975	104000	10.9646
10.9667	2.8513	106000	10.9646

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1