Add files using upload-large-folder tool

README.md

@@ -0,0 +1,158 @@
+---
+license: mit
+language:
+- en
+- ja
+- nl
+- fr
+- de
+- it
+- pl
+- pt
+- es
+tags:
+- speech
+- audio
+- tokenizer
+datasets:
+- sarulab-speech/mls_sidon
+- mythicinfinity/Libriheavy-HQ
+- nvidia/hifitts-2
+pipeline_tag: audio-to-audio
+---
+
+# MioCodec: High-Fidelity 44.1kHz Neural Audio Codec for Efficient Spoken Language Modeling
+
+[![GitHub](https://img.shields.io/badge/Code-GitHub-black)](https://github.com/Aratako/MioCodec)
+
+**MioCodec-25Hz** is a high-fidelity neural audio codec designed for efficient spoken language modeling. Based on the [Kanade-Tokenizer](https://github.com/frothywater/kanade-tokenizer) implementation, MioCodec extends the capabilities to 44.1 kHz sampling rates, providing superior audio quality while maintaining a very low token rate.
+
+## 🌟 Overview
+
+MioCodec decomposes speech into two distinct components:
+
+1.  **Content Tokens:** Discrete representations that primarily capture linguistic information and phonetic content ("what" is being said) at a low frame rate (25 Hz).
+2.  **Global Embeddings:** A continuous vector representing broad acoustic characteristics ("how")—including speaker identity, recording environment, and microphone traits.
+
+By disentangling these elements, MioCodec is ideal for **Spoken Language Modeling**.
+
+### Key features
+
+* **High-Resolution:** Supports **44.1 kHz** audio (compared to the standard 24 kHz in Kanade).
+* **Ultra-Low Bitrate:** Achieves high-fidelity reconstruction at only **341 bps**.
+* **End-to-End Optimization:** Unlike original two-stage approaches, the codec and vocoder are jointly fine-tuned to minimize waveform artifacts and jitter.
+
+## 📊 Model Comparison
+
+| Model | Token Rate | Vocab Size | Bit Rate | Sample Rate | SSL Encoder | Vocoder | Parameters |
+| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
+| **MioCodec-25Hz** | **25 Hz** | **12,800** | **341 bps** | **44.1 kHz** | **WavLM-base+** | **[MioVocoder](https://huggingface.co/Aratako/MioVocoder)** (Jointly Tuned) | **118M** |
+| kanade-25hz | 25 Hz | 12,800 | 341 bps | 24 kHz | WavLM-base+ | Vocos 24kHz | 118M |
+| kanade-12.5hz | 12.5 Hz | 12,800 | 171 bps | 24 kHz | WavLM-base+ | Vocos 24kHz | 120M |
+
+## 🚀 Quick Start
+
+### Installation
+
+```bash
+# Install via pip
+pip install git+https://github.com/Aratako/MioCodec
+
+# Or using uv
+uv add git+https://github.com/Aratako/MioCodec
+
+```
+
+### Basic Inference
+
+Basic usage for encoding and decoding audio:
+
+```python
+from miocodec import MioCodec, load_audio
+import soundfile as sf
+
+# 1. Load model
+model = MioCodec.from_pretrained("Aratako/MioCodec-25Hz").eval().cuda()
+
+# 2. Load audio
+waveform = load_audio("input.wav", sample_rate=model.config.sample_rate).cuda()
+
+# 3. Encode Audio
+features = model.encode(waveform)
+
+# 4. Decode to Waveform
+resynth = model.decode(features=features)
+
+# 5. Save
+sf.write("output.wav", resynth.cpu().numpy(), samplerate=model.config.sample_rate)
+```
+
+### Voice Conversion (Zero-shot)
+
+MioCodec allows you to swap speaker identities by combining the content tokens of a source with the global embedding of a reference.
+
+```python
+source = load_audio("source_content.wav", sample_rate=model.config.sample_rate).cuda()
+reference = load_audio("target_speaker.wav", sample_rate=model.config.sample_rate).cuda()
+
+# Perform conversion
+vc_wave = model.voice_conversion(source, reference)
+sf.write("converted.wav", vc_wave.cpu().numpy(), samplerate=model.config.sample_rate)
+```
+
+## 🏗️ Training Methodology
+
+To achieve high-fidelity 44.1kHz reconstruction, MioCodec was trained in three phases. Phase 1 and 2 strictly follow the [original Kanade paper](https://openreview.net/pdf?id=dNUcKJEPTh) to establish feature alignment and spectral sharpness, while Phase 3 introduces a novel end-to-end waveform refinement stage.
+
+### Phase 1: Feature Alignment
+This phase corresponds to the "Main Training Phase" described in the original paper. The model is trained to minimize both **Mel-spectrogram loss** and **SSL feature reconstruction loss** (using WavLM-base+). The vocoder is not utilized; the loss is computed directly on the predicted mel-spectrograms.
+
+### Phase 2: Adversarial Alignment
+Following the "GAN Post-Training" phase of the original paper, we introduce adversarial training to sharpen the spectrograms. In this stage, the content branch is frozen, and only the decoder and global branch are updated. The model is trained using **Mel-spectrogram loss** combined with **GAN losses** (Adversarial + Feature Matching) applied in the mel domain.
+
+### Phase 3: End-to-End Waveform Refinement
+To address residual artifacts such as jitter or tremor often found in mel-only training, we introduced a third phase that shifts the domain to raw waveforms. 
+
+In this phase, Vocoder is unfrozen, allowing the Codec Decoder and Vocoder to be fine-tuned jointly in an end-to-end manner. Similar to Phase 2, the content branch remains frozen. The training objective minimizes waveform artifacts using objectives adapted from [XCodec2](https://github.com/zhenye234/X-Codec-2.0) and [Inworld TTS-1](https://arxiv.org/html/2507.21138v1), with specific parameters tuned for 44.1 kHz:
+
+* **Multi-Resolution Mel Spectrogram Loss:** Using window lengths of `[32, 64, 128, 256, 512, 1024, 2048, 4096]`.
+* **Multi-Period Discriminator (MPD):** Using periods of `[2, 3, 5, 7, 11, 17, 23, 37]`.
+* **Multi-Scale STFT Discriminator (MS-STFTD):** Using FFT sizes of `[216, 348, 568, 920, 1494, 2414, 3908, 6328]`.
+* **RMS Loss:** Adopted from Inworld TTS-1 to stabilize energy and volume.
+
+## 📚 Training Data
+
+The training datasets are listed below:
+
+| Language | Approx. Hours | Dataset | Used in Phases |
+| :--- | :--- | :--- | :---: |
+| **Japanese** | ~15,000h | Various public HF datasets | 1, 2, 3 |
+| **English** | ~500h | [Libriheavy-HQ](https://huggingface.co/datasets/mythicinfinity/Libriheavy-HQ) | 1, 2, 3 |
+| **English** | ~4000h | [MLS-Sidon](https://huggingface.co/datasets/sarulab-speech/mls_sidon) | 1, 2 |
+| **English** | ~9000h | [HiFiTTS-2](https://huggingface.co/datasets/nvidia/hifitts-2) | 3 |
+| **German** | ~1,950h | [MLS-Sidon](https://huggingface.co/datasets/sarulab-speech/mls_sidon) | 1, 2 |
+| **Dutch** | ~1,550h | [MLS-Sidon](https://huggingface.co/datasets/sarulab-speech/mls_sidon) | 1, 2 |
+| **French** | ~1,050h | [MLS-Sidon](https://huggingface.co/datasets/sarulab-speech/mls_sidon) | 1, 2 |
+| **Spanish** | ~900h | [MLS-Sidon](https://huggingface.co/datasets/sarulab-speech/mls_sidon) | 1, 2 |
+| **Italian** | ~240h | [MLS-Sidon](https://huggingface.co/datasets/sarulab-speech/mls_sidon) | 1, 2 |
+| **Portuguese** | ~160h | [MLS-Sidon](https://huggingface.co/datasets/sarulab-speech/mls_sidon) | 1, 2 |
+| **Polish** | ~100h | [MLS-Sidon](https://huggingface.co/datasets/sarulab-speech/mls_sidon) | 1, 2 |
+
+## 📜 Acknowledgements
+
+* **Codec Architecture:** Based on the brilliant work of [kanade-tokenizer](https://github.com/frothywater/kanade-tokenizer).
+* **Vocoder Base:** Weights and codebase derived from [AliasingFreeNeuralAudioSynthesis](https://github.com/sizigi/AliasingFreeNeuralAudioSynthesis).
+* **Training Techniques:** Phase 3 training objectives were heavily inspired by [XCodec2](https://github.com/zhenye234/X-Codec-2.0) and [Inworld TTS-1](https://arxiv.org/html/2507.21138v1).
+
+## 🖊️ Citation
+
+```bibtex
+@misc{miocodec-25hz,
+  author = {Chihiro Arata},
+  title = {MioCodec: High-Fidelity 44.1kHz Neural Audio Codec},
+  year = {2026},
+  publisher = {Hugging Face},
+  journal = {Hugging Face repository},
+  howpublished = {\url{https://huggingface.co/Aratako/MioCodec-25Hz}}
+}
+```

config.yaml

@@ -0,0 +1,99 @@
+model:
+  class_path: miocodec.model.MioCodecModel
+  init_args:
+    config:
+      # SSL Feature settings
+      local_ssl_layers: [6, 9]
+      global_ssl_layers: [1, 2]
+      normalize_ssl_features: true
+
+      # Down/up-sampling settings
+      downsample_factor: 2
+      mel_upsample_factor: 4
+      use_conv_downsample: true
+      mel_interpolation_mode: linear
+
+      # Audio settings (match Pupu-Vocoder mel specs)
+      sample_rate: 44100
+      n_fft: 2048
+      hop_length: 512
+      n_mels: 128
+      padding: same
+      mel_backend: pupu
+      mel_fmin: 0.0
+      mel_fmax: 22050.0
+      mel_win_length: 2048
+
+    ssl_feature_extractor:
+      class_path: miocodec.module.ssl_extractor.SSLFeatureExtractor
+      init_args:
+        model_name: wavlm_base_plus
+        output_layer: 9 # Use at most 9 layers
+        sample_rate: 44100 # Consistent to the target sample rate for reconstruction
+
+    local_encoder:
+      class_path: miocodec.module.transformer.Transformer
+      init_args:
+        dim: 768
+        n_layers: 6
+        n_heads: 12
+        window_size: 125
+        use_rope: true
+        rope_theta: 10000.0
+        max_seq_len: 512
+        use_flash_attention: true
+
+    local_quantizer:
+      class_path: miocodec.module.fsq.FiniteScalarQuantizer
+      init_args:
+        input_dim: 768 # Must match local encoder output dimension
+        output_dim: 768 # Must match feature decoder input dimension
+        levels: [8, 8, 8, 5, 5] # 12800
+
+    feature_decoder: null
+
+    global_encoder:
+      class_path: miocodec.module.global_encoder.GlobalEncoder
+      init_args:
+        input_channels: 768 # WavLM base plus feature dimension
+        output_channels: 128
+        num_layers: 4
+        dim: 384
+        intermediate_dim: 1152
+
+    mel_prenet:
+      class_path: miocodec.module.transformer.Transformer
+      init_args:
+        dim: 768
+        output_dim: 512
+        n_layers: 6
+        n_heads: 12
+        window_size: 65
+        use_rope: true
+        rope_theta: 10000.0
+        max_seq_len: 512
+        use_flash_attention: true
+
+    mel_decoder:
+      class_path: miocodec.module.transformer.Transformer
+      init_args:
+        dim: 512
+        output_dim: 128 # Number of mel frequency bins
+        n_layers: 6
+        n_heads: 8
+        window_size: 65
+        use_rope: true
+        rope_theta: 10000.0
+        max_seq_len: 512
+        adanorm_condition_dim: 128 # Must match global encoder output dimension
+        use_adaln_zero: true # Use AdaLNZero for conditioning
+        use_flash_attention: true
+
+    mel_postnet:
+      class_path: miocodec.module.postnet.PostNet
+      init_args:
+        input_channels: 128 # Number of mel frequency bins
+        channels: 256
+        kernel_size: 7
+        num_layers: 4
+        use_layer_norm: true

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e523449750ad0c9f66c08a5ece0ea60291bb8e615abcc0c46577f56a1c2beb62
+size 532367664

vocoder_config.json

@@ -0,0 +1,58 @@
+{
+  "base_config": "egs/exp_config_pupuvocoder_base.json",
+  "model_type": "PupuVocoder",
+  "model": {
+    "generator": "pupuvocoder",
+    "pupuvocoder": {
+      "resblock": "1",
+      "upsample_rates": [
+        8,
+        8,
+        2,
+        2,
+        2
+      ],
+      "upsample_kernel_sizes": [
+        16,
+        16,
+        4,
+        4,
+        4
+      ],
+      "upsample_initial_channel": 512,
+      "resblock_kernel_sizes": [
+        3,
+        7,
+        11
+      ],
+      "resblock_dilation_sizes": [
+        [
+          1,
+          3,
+          5
+        ],
+        [
+          1,
+          3,
+          5
+        ],
+        [
+          1,
+          3,
+          5
+        ]
+      ]
+    }
+  },
+  "train": {
+    "criterions": [
+        "feature",
+        "discriminator",
+        "generator",
+        "multimel"
+    ]
+  },
+  "inference": {
+    "batch_size": 1
+  }
+}