| | --- |
| | license: apache-2.0 |
| | language: |
| | - multilingual |
| | datasets: |
| | - agentlans/chat-annotated |
| | base_model: |
| | - ibm-granite/granite-embedding-107m-multilingual |
| | pipeline_tag: text-classification |
| | --- |
| | # granite-embedding-107m-multilingual-chat-difficulty |
| |
|
| | A fine-tuned model that estimates the difficulty of multilingual, multi-turn human–AI conversations based on reasoning complexity. |
| |
|
| | - **Input:** A condensed conversation in the format `<|user|>prompt<|assistant|>reply...` |
| | - **Output:** A normalized difficulty score (lower scores indicate easier conversations) |
| |
|
| | Based on [ibm-granite/granite-embedding-107m-multilingual](https://huggingface.co/ibm-granite/granite-embedding-107m-multilingual). |
| |
|
| | **Evaluation results:** |
| | - **Loss:** 0.5663 |
| | - **MSE:** 0.5663 |
| | - **Tokens processed:** 51,173,120 |
| |
|
| | ## Model description |
| |
|
| | This model maps multi-turn chat logs to a continuous difficulty representation, enabling comparison across languages and reasoning styles. |
| |
|
| | Use cases include: |
| | - Categorizing multilingual chat transcripts by reasoning depth. |
| | - Supporting dataset curation or curriculum design. |
| | - Serving as a difficulty scoring component in evaluation pipelines. |
| |
|
| | ## Intended uses and limitations |
| |
|
| | **Use cases** |
| | - Estimating reasoning difficulty in multilingual conversations. |
| | - Comparing dialogue complexity across datasets. |
| | - Benchmarking conversational reasoning. |
| |
|
| | **Limitations** |
| | - Not suitable for assessing factual accuracy, coherence, or sentiment. |
| | - May not generalize well to highly domain-specific data. |
| | - Produces relative difficulty scores, not absolute intelligence measures. |
| |
|
| | ## Training procedure |
| |
|
| | ### Hyperparameters |
| |
|
| | | Parameter | Value | |
| | |------------|-------| |
| | | learning_rate | 5e-5 | |
| | | train_batch_size | 8 | |
| | | eval_batch_size | 8 | |
| | | seed | 42 | |
| | | optimizer | AdamW (fused), betas=(0.9, 0.999), epsilon=1e-8 | |
| | | lr_scheduler_type | linear | |
| | | num_epochs | 5.0 | |
| |
|
| | ### Results |
| |
|
| | | Metric | Value | |
| | |---------|-------| |
| | | Training loss | 0.5663 | |
| | | MSE | 0.5663 | |
| | | Tokens processed | 51,173,120 | |
| |
|
| | ## Framework versions |
| |
|
| | - **Transformers:** 5.0.0.dev0 |
| | - **PyTorch:** 2.9.1+cu128 |
| | - **Datasets:** 4.4.1 |
| | - **Tokenizers:** 0.22.1 |
| |
|
| | ## See also |
| |
|
| | [agentlans/bge-small-en-v1.5-prompt-difficulty](https://huggingface.co/agentlans/bge-small-en-v1.5-prompt-difficulty) for single-turn English conversations and prompts |
