|
|
--- |
|
|
language: |
|
|
- en |
|
|
tags: |
|
|
- automatic-speech-recognition |
|
|
- speech-to-text |
|
|
- whisper |
|
|
- peft |
|
|
- lora |
|
|
- seq2seq |
|
|
base_model: openai/whisper-small |
|
|
pipeline_tag: automatic-speech-recognition |
|
|
--- |
|
|
|
|
|
This model is a fine-tuned version of openai/whisper-small on the UA-Speech dataset. |
|
|
|
|
|
## Model description |
|
|
This model fine-tunes Whisper-small for English transcription on dysarthric speech. Training used LoRA (PEFT) on attention and feed-forward modules, and the adapter was merged into the base model weights for deployment (no PEFT required at inference time). This model is used to show generalization via constrained capacity. |
|
|
|
|
|
## Intended uses & limitations |
|
|
This model is intended for automatic speech recognition (ASR) on English speech, with an emphasis on robustness to atypical/dysarthric speech patterns resembling UA-Speech-style data. Performance may degrade on out-of-domain audio, heavy noise, non-English speech, or audio sampled far from 16 kHz. For best results, provide mono 16 kHz audio. |
|
|
|
|
|
## Audio preprocessing |
|
|
- Audio loaded with `soundfile.read(file_path)` |
|
|
- If stereo/multi-channel, converted to mono by averaging channels |
|
|
- Features extracted with `WhisperProcessor.feature_extractor(..., sampling_rate=16000)` |
|
|
|
|
|
## Training procedure |
|
|
|
|
|
### Training hyperparameters |
|
|
The following hyperparameters were used during training: |
|
|
|
|
|
learning_rate: 3e-4 |
|
|
train_batch_size: 16 |
|
|
seed: 42 |
|
|
optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 |
|
|
lr_scheduler_type: linear |
|
|
num_epochs: 8 |
|
|
mixed_precision_training: Native AMP |
|
|
|
|
|
## LoRA / PEFT configuration |
|
|
- task_type: SEQ_2_SEQ_LM |
|
|
- r: 64 |
|
|
- lora_alpha: 128 |
|
|
- lora_dropout: 0.1 |
|
|
- target_modules: ["q_proj", "v_proj", "fc1", "fc2"] |
|
|
- modules_to_save: None |
|
|
|
|
|
## Model config modifications |
|
|
- model.config.forced_decoder_ids = None |
|
|
- model.config.suppress_tokens = [] |
|
|
|
|
|
## Training results |
|
|
WER/CER were computed offline after training. |
|
|
|
|
|
## Framework versions |
|
|
Transformers 4.56.2 |
|
|
Pytorch 2.8.0+cu128 |
|
|
Datasets 4.4.1 |
|
|
Tokenizers 0.22.1 |
|
|
|
