binary-tokenizer-001-4k

A cross-platform BPE tokenizer for binary executables and machine code. Trained on 13 GB of diverse binaries spanning Linux, Windows, macOS, and Android platforms.

🔗 Model: mjbommar/binary-tokenizer-001-4k 📊 Dataset: mjbommar/binary-30k-tokenized 📄 Paper: Binary BPE: Cross-Platform Tokenization for Binary Analysis (arXiv preprint coming soon)

Overview

• Vocabulary Size: 4,096 tokens (2^12)
• Token Composition: 256 base bytes + 3,833 learned merges + 7 special tokens
• Average Token Length: 3.000 bytes
• 3-byte Instructions: 20.6% of vocabulary (841 tokens)
• Compression Ratio: ~2.0 bytes/token on typical binaries

---

Training Configuration

Training Corpus:

• Source: mjbommar/binary-30k-tokenized
• Size: ~13 GB
• Files: 30,738 binary files
• Platforms: Linux (ELF), Windows (PE), macOS (Mach-O), Android (APK)
• Architectures: x86-64, x86, ARM64, ARM, MIPS, RISC-V

Training Parameters:

• Vocabulary size: 4,096 (including 7 special tokens)
• Min frequency: 10
• Chunk size: 8,192 bytes
• Allowed lengths: DEFAULT (1-16 bytes)
• Training duration: ~1h 46min

---

Vocabulary Statistics

Composition:

• Base bytes (0-255): 256 tokens
• Learned merges: 3,833 tokens
• Special tokens: 7 tokens (<|start|>, <|end|>, <|pad|>, <|unk|>, <|cls|>, <|sep|>, <|mask|>)
• Total: 4,096 tokens

Quality Metrics:

• All tokens reachable: ✓ Yes
• Valid merges: 3,833 / 3,833
• Power-of-2 size: ✓ Yes (2^12)

---

Token Length Distribution

Length	Count	Percentage	Description
1 byte	256	6.3%	Base bytes
2 bytes	1,974	48.3%	Byte pairs
3 bytes	841	20.6%	Complete x86-64 instructions
4 bytes	649	15.9%	Instructions with operands
5 bytes	95	2.3%	Complex patterns
6 bytes	86	2.1%	Complex patterns
7 bytes	40	1.0%	Complex patterns
8 bytes	59	1.4%	Complex patterns
9+ bytes	89	2.2%	Long patterns

Average Token Length: 3.000 bytes

---

Byte Content Analysis

Content Categories:

• Contains NULL byte (0x00): 1,094 tokens (26.7%)
• ASCII printable (0x20-0x7E): 896 tokens (21.9%)
• All ASCII (<0x80): 1,879 tokens (45.9%)
• High bytes (≥0x80): 2,210 tokens (54.0%)

Most Common Bytes in Tokens:

• 0x00 (NULL): 2,468 occurrences - Padding and alignment
• 0xFF: 404 occurrences - Sentinel values
• 0x48 (REX.W): 340 occurrences - x86-64 REX prefix
• 0x8B (MOV): 233 occurrences - x86-64 MOV opcode
• 0xCC (INT3): 170 occurrences - Debug breakpoint padding

---

Sequence Coverage

N-byte Sequence Diversity:

Length	Learned Tokens	Possible Sequences	Coverage
1-byte	256	256	100.00%
2-byte	1,974	65,536	3.01%
3-byte	841	16,777,216	0.005%
4-byte	649	4,294,967,296	0.000015%

---

Files

• tokenizer-4096.json - Trained tokenizer model (286 KB)
• analysis_results.json - Detailed analysis statistics
• training.log - Training output log
• training_stats.txt - Training summary

---

Usage

Load from HuggingFace Hub:

from tokenizers import Tokenizer

# Load directly from HuggingFace
tokenizer = Tokenizer.from_pretrained("mjbommar/binary-tokenizer-001-4k")

Load from local file:

# With bbpe CLI
bbpe encode --tokenizer tokenizer-4096.json /path/to/binary
bbpe info tokenizer-4096.json

Complete Python Example:

from tokenizers import Tokenizer

# Load from HuggingFace or local file
tokenizer = Tokenizer.from_pretrained("mjbommar/binary-tokenizer-001-4k")
# OR: tokenizer = Tokenizer.from_file("tokenizer-4096.json")

# Read binary file and decode as latin-1 (preserves all byte values 0-255)
with open("/usr/bin/ls", "rb") as f:
    data = f.read()
    data_str = data.decode("latin-1")

# Encode the binary data
encoding = tokenizer.encode(data_str)
print(f"File size: {len(data)} bytes")
print(f"Total tokens: {len(encoding.ids)}")
print(f"Compression: {len(data) / len(encoding.ids):.3f} bytes/token")

# First 10 tokens
for i, (token_id, token) in enumerate(zip(encoding.ids[:10], encoding.tokens[:10])):
    token_bytes = token.encode("latin-1")
    print(f"  Token {i}: ID={token_id:5d} hex={token_bytes.hex():20s} ({len(token_bytes)} bytes)")

# Decode tokens back to bytes
decoded_str = tokenizer.decode(encoding.ids)
decoded_bytes = decoded_str.encode("latin-1")
assert decoded_bytes == data  # Perfect reconstruction

Example output for /usr/bin/ls (142,312 bytes):

File size: 142312 bytes
Total tokens: 71272
Compression: 1.997 bytes/token

First 10 tokens:
  Token 0: ID=  127 hex=7f                   (1 bytes)
  Token 1: ID= 3732 hex=454c                 (2 bytes)
  Token 2: ID=   70 hex=46                   (1 bytes)
  Token 3: ID=    2 hex=02                   (1 bytes)
  Token 4: ID=  392 hex=0101                 (2 bytes)
  Token 5: ID=  662 hex=000000000000000000   (9 bytes)
  Token 6: ID=  265 hex=0300                 (2 bytes)
  Token 7: ID= 1369 hex=3e00                 (2 bytes)
  Token 8: ID=  279 hex=01000000             (4 bytes)
  Token 9: ID=   48 hex=30                   (1 bytes)

Decoded: 7f454c4602010100000000000000000003003e000100000030...
(ELF header: 7f 45 4c 46 = ELF magic bytes)

---

Citation

If you use this tokenizer in your research, please cite:

@article{bommarito2025binarybpe,
  title={Binary BPE: Cross-Platform Tokenization for Binary Analysis},
  author={Bommarito II, Michael J.},
  journal={arXiv preprint},
  year={2025},
  note={Preprint coming soon}
}

Author: Michael J. Bommarito II (michael.bommarito@gmail.com)

---

Generated: November 12, 2025 Training Script: train_tokenizers.sh Analysis Script: analyze_tokenizer.py