| | --- |
| | base_model: answerdotai/ModernBERT-base |
| | language: |
| | - en |
| | library_name: transformers |
| | pipeline_tag: text-classification |
| | tags: |
| | - reasoning |
| | - complexity |
| | - education |
| | - regression |
| | - fineweb-edu |
| | --- |
| | |
| | # Reasoning Complexity Classifier |
| |
|
| | A ModernBERT-base model fine-tuned to predict the **reasoning complexity** of educational text on a continuous 1–4 scale. Trained on FineWeb-Edu documents labeled by GPT-5-nano via the OpenAI Batch API (~$20 in credits). |
| |
|
| | ## Model Description |
| |
|
| | This is a regression model (`num_labels=1`, `problem_type="regression"`) that outputs a continuous score. The score can be rounded to the nearest integer to obtain a discrete complexity level. Level 5 (Formal/Abstract reasoning) was excluded from training due to data scarcity; the model's effective range is **1.0–4.0**. |
| |
|
| | ### Complexity Levels |
| |
|
| | | Level | Name | Description | Example | |
| | |-------|------|-------------|---------| |
| | | 1 | Factual/Declarative | States facts with no reasoning | "The Pacific Ocean covers ~165 million km²." | |
| | | 2 | Single-step reasoning | One inference or comparison | "Because boiling point decreases at altitude, water boils faster in Denver than Miami." | |
| | | 3 | Multi-step reasoning | 2–4 chained logical steps | "Demand rose while supply held fixed → prices rose → consumer spending fell → GDP slowed." | |
| | | 4 | Complex reasoning | 5+ steps, conditionals, competing factors | Medical differential diagnosis with branching conditions and exclusion criteria. | |
| |
|
| | ## Training Details |
| |
|
| | ### Data |
| |
|
| | - **Source**: [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) — a curated subset of Common Crawl filtered for educational content. |
| | - **Labeling**: ~100,000 documents reservoir-sampled from ~6,000 records per subject category, then labeled with GPT-5-nano via the OpenAI Batch API using structured output (integer 1–5). |
| | - **Splits**: 80% train / 10% validation / 10% test (stratified by integer complexity level). |
| | - **Preprocessing**: Texts truncated to 8,000 characters before labeling; tokenized to 512 tokens during training with dynamic padding. |
| | - **Level 5 exclusion**: Rows labeled as level 5 were excluded from the training set. |
| |
|
| | ### Hyperparameters |
| |
|
| | | Parameter | Value | |
| | |-----------|-------| |
| | | Base model | `answerdotai/ModernBERT-base` | |
| | | Epochs | 3 | |
| | | Batch size | 32 | |
| | | Learning rate | 2e-5 | |
| | | Weight decay | 0.01 | |
| | | Warmup ratio | 0.1 | |
| | | Max token length | 512 | |
| | | Optimizer | AdamW | |
| | | Scheduler | Linear with warmup | |
| | | AMP | bf16 (CUDA) | |
| | | Loss | MSE | |
| |
|
| | ### Training History |
| |
|
| | | Epoch | Train Loss | Val MAE | Val Acc (rounded) | Val Spearman r | |
| | |-------|-----------|---------|-------------------|----------------| |
| | | 1 | 0.6002 | 0.5190 | 56.98% | 0.7533 | |
| | | **2** | **0.3631** | **0.5040** | **58.43%** | **0.7597** | |
| | | 3 | 0.2040 | 0.5114 | 58.19% | 0.7485 | |
| |
|
| | The best checkpoint (by validation MAE) was saved at **epoch 2**. |
| |
|
| | ## Evaluation Results |
| |
|
| | Evaluated on a held-out test set: |
| |
|
| | | Metric | Value | |
| | |--------|-------| |
| | | MSE | 0.4388 | |
| | | MAE | 0.5063 | |
| | | Rounded accuracy | 58.6% | |
| | | Spearman r | 0.7527 | |
| |
|
| | **Interpretation**: The model achieves a Spearman correlation of ~0.75 with gold labels, indicating strong ordinal ranking ability. The MAE of ~0.51 means predictions are on average within half a level of the true score when treated as a continuous signal. |
| |
|
| | ### Output Interpretation |
| |
|
| | | Raw score | Meaning | |
| | |-----------|---------| |
| | | ~1.0 | Factual/Declarative | |
| | | ~2.0 | Single-step reasoning | |
| | | ~3.0 | Multi-step reasoning | |
| | | ~4.0 | Complex reasoning | |
| |
|
| | Clip and round the raw float output to `[1, 4]` for a discrete level. |
| |
|
| | ## Architecture |
| |
|
| | Based on `answerdotai/ModernBERT-base`: |
| |
|
| | - **Layers**: 22 transformer layers (alternating full and sliding attention) |
| | - **Hidden size**: 768 |
| | - **Attention heads**: 12 |
| | - **Intermediate size**: 1,152 |
| | - **Max position embeddings**: 8,192 |
| | - **Classifier pooling**: mean |
| | - **Classifier activation**: GELU |
| |
|
| | ## Limitations |
| |
|
| | - Labels are silver-standard (GPT-5-nano), not human-annotated; label noise may affect the ~1.5% of ambiguous texts. |
| | - Texts are truncated to 512 tokens; very long documents are judged on their first ~512 tokens only. |
| | - Trained primarily on English educational web text; performance may degrade on other domains or languages. |
| |
|
| | ## Intended Use |
| |
|
| | Designed for data curation pipelines that need to filter or balance training corpora by reasoning complexity — for example, constructing curriculum-ordered datasets for language model training. |
| |
|
