musicbert-large

Model Description

MusicBERT large is a 24-layer BERT-style masked language model trained on REMI+BPE symbolic music sequences extracted from the GigaMIDI corpus. It is tailored for symbolic music understanding, fill-mask style infilling, and as a backbone for downstream generative tasks.

• Checkpoint: 120000 steps
• Hidden size: 1024
• Parameters: ~330M
• Validation loss: 1.19900381565094

Training Configuration

• Objective: Masked language modeling with span-aware masking
• Dataset: GigaMIDI (REMI tokens → BPE, vocab size 50000)
• Sequence length: 1024
• Max events per MIDI: 2048

Inference Example

Using with MIDI files

import torch
from transformers import BertForMaskedLM
from miditok import MusicTokenizer

# Load model and tokenizer
model = BertForMaskedLM.from_pretrained("manoskary/musicbert-large")
tokenizer = MusicTokenizer.from_pretrained("manoskary/miditok-REMI")

# Convert MIDI to BPE tokens (MIDI → REMI → BPE pipeline)
midi_path = "path/to/your/file.mid"
tok_seq = tokenizer(midi_path)
bpe_ids = tok_seq.ids

# Mask some tokens for prediction
import random
mask_token_id = 3  # MASK_None token
input_ids = bpe_ids.copy()
mask_positions = random.sample(range(1, len(input_ids)-1), k=5)
for pos in mask_positions:
    input_ids[pos] = mask_token_id

# Run inference
input_tensor = torch.tensor([input_ids])
with torch.no_grad():
    outputs = model(input_tensor)
    predictions = outputs.logits[0, mask_positions, :].argmax(dim=-1)

print("Predicted token IDs:", predictions.tolist())

Limitations and Risks

• Model is trained purely on symbolic data; it does not produce audio directly.
• The GigaMIDI dataset is biased towards Western tonal music.
• Long-form structure beyond 1024 tokens requires chunking or iterative decoding.
• Generated continuations may need post-processing to ensure musical coherence.

Citation

If you use this checkpoint, please cite the original MusicBERT introduction and the GigaMIDI dataset.