--- library_name: transformers license: apache-2.0 base_model: distilbert-base-uncased tags: - generated_from_trainer model-index: - name: schedulebot-nlu-engine results: [] datasets: - andreaceto/hasd language: - en --- # Schedulebot-nlu-engine ## Model Description This model is a multi-task Natural Language Understanding (NLU) engine designed specifically for an appointment scheduling chatbot. It is fine-tuned from a **`distilbert-base-uncased`** backbone and is capable of performing two tasks simultaneously: - **Intent Classification**: Identifying the user's primary goal (e.g., `schedule`, `cancel`). - **Named Entity Recognition (NER)**: Extracting custom, domain-specific entities (e.g., `appointment_type`). This model stands out due to its custom classification heads, which use a more complex architecture to improve performance on nuanced tasks. ## Model Architecture The model uses a standard `distilbert-base-uncased` model as its core feature extractor. Two custom classification "heads" are placed on top of this base to perform the downstream tasks. - **Base Model**: `distilbert-base-uncased` - **Classifier Heads**: each head is a Multi-Layer Perceptron (MLP) with the following structure to allow for more complex feature interpretation: 1. A Linear layer projecting the transformer's output dimension (768) to an intermediate size (384). 2. A GELU activation function. 3. A Dropout layer with a rate of 0.3 for regularization. 4. A final Linear layer projecting the intermediate size to the number of output labels for the specific task (intent or NER). ## Intended Use This model is intended to be the core NLU component of a conversational AI system for managing appointments. For instructions on how to use the model check the [dedicated file](./how_to_use.md). ## Training Data The model was trained on the **HASD (Hybrid Appointment Scheduling Dataset)**, a custom dataset built specifically for this task. - **Source**: The dataset is a hybrid of real-world conversational examples from `clinc/clinc_oos` (for simple intents) and synthetically generated, template-based examples for complex scheduling intents. - **Balancing**: To combat class imbalance, intents sourced from `clinc/clinc_oos` were **down-sampled** to a maximum of **150 examples** each. - **Augmentation**: To increase data diversity for complex intents (`schedule`, `reschedule`, etc.), **Contextual Word Replacement** was used. A `distilbert-base-uncased` model augmented the templates by replacing non-placeholder words with contextually relevant synonyms. The dataset is available [here](https://huggingface.co/datasets/andreaceto/hasd). ### Intents The model is trained to recognize the following intents: `schedule`, `reschedule`, `cancel`, `query_avail`, `greeting`, `positive_reply`, `negative_reply`, `bye`, `oos` (out-of-scope). ### Entities The model is trained to recognize the following custom named entities: `practitioner_name`, `appointment_type`, `appointment_id`. ## Training Procedure The model was trained using a two-stage fine-tuning strategy to ensure stability and performance. ### Stage 1: Training the Classifier Heads - The `distilbert-base-uncased` base model was entirely **frozen**. - Only the randomly initialized MLP heads for intent and NER classification were trained. **Setup**: ```python # Define a data collator to handle padding for token classification data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer) # Define Training Arguments training_args = TrainingArguments( output_dir="path/to/output_dir", overwrite_output_dir=True, num_train_epochs=200, # Training epochs per_device_train_batch_size=32, per_device_eval_batch_size=32, learning_rate=1e-4, # Learning Rate weight_decay=1e-5, # AdamW weight decay logging_dir="path/to/logging_dir", logging_strategy="epoch", eval_strategy="epoch", save_strategy="epoch", load_best_model_at_end=True, metric_for_best_model="eval_loss", # Focus on validation loss as the key metric # --- Hub Arguments --- push_to_hub=True, hub_model_id=hub_model_id, hub_strategy="end", hub_token=hf_token, report_to="tensorboard" # Tensorboard to monitor training ) # Create the Trainer trainer = Trainer( model=model, args=training_args, train_dataset=processed_datasets["train"], eval_dataset=processed_datasets["validation"], processing_class=tokenizer, data_collator=data_collator, compute_metrics=compute_metrics, # Custom function (check how_to_use.md) callbacks=[EarlyStoppingCallback(early_stopping_patience=10)] ) ``` ### Stage 2: Fine-Tuning - The DistilBERT backbone was entirely **unfrozen**. - Using a very low LR allows the model to adapt even better to the new data while preserving the powerful, general-purpose knowledge. **Setup**: ```python # Define Training Arguments training_args = TrainingArguments( output_dir="path/to/output_dir", overwrite_output_dir=True, num_train_epochs=50, # Fine-tuning epochs per_device_train_batch_size=32, per_device_eval_batch_size=32, learning_rate=1e-6, # Learning Rate weight_decay=1e-3, # AdamW weight decay logging_dir="path/to/logging_dir", logging_strategy="epoch", eval_strategy="epoch", save_strategy="epoch", load_best_model_at_end=True, metric_for_best_model="eval_loss", # Focus on NER F1 as the key metric # --- Hub Arguments --- push_to_hub=True, hub_model_id=hub_model_id, hub_strategy="end", hub_token=hf_token, report_to="tensorboard" # Tensorboard to monitor training ) # Create the Trainer trainer = Trainer( model=model, args=training_args, train_dataset=processed_datasets["train"], eval_dataset=processed_datasets["validation"], processing_class=tokenizer, data_collator=data_collator, compute_metrics=compute_metrics, # Custom function (check how_to_use.md) callbacks=[EarlyStoppingCallback(early_stopping_patience=5)] ) ``` ## Evaluation The model was evaluated on a held-out test set, and its performance was measured for both tasks. ### Intent Classification Performance | Intent | Precision | Recall | F1-Score | Support | | --- | --- | --- | --- | --- | | bye | 0.9500 | 0.8261 | 0.8837 | 23 | | cancel | 0.9211 | 0.8434 | 0.8805 | 83 | | greeting | 0.9545 | 0.9545 | 0.9545 | 22 | |negative_reply | 0.9091 | 0.9091 | 0.9091 | 22 | | oos | 1.0000 | 0.8696 | 0.9302 | 23 | |positive_reply | 0.7407 | 0.9091 | 0.8163 | 22 | | query_avail | 0.9620 | 0.9383 | 0.9500 | 81 | | reschedule | 0.8506 | 0.8916 | 0.8706 | 83 | | schedule | 0.8488 | 0.9125 | 0.8795 | 80 | | --- | --- | --- | --- | ---- | | **Accuracy** | | | **0.8952** | 439 | | **Macro Avg** | **0.9041** | **0.8949** | **0.8972** | 439 | | **Weighted Avg** | **0.8998** | **0.8952** | **0.8960** | 439 | ### NER (Token Classification) Performance | Entity | Precision | Recall | F1-Score | Support | | --- | --- | --- | --- | --- | | B-appointment_id | 1.0000 | 1.0000 | 1.0000 | 61 | | B-appointment_type | 0.8646 | 0.7477 | 0.8019 | 111 | | B-practitioner_name | 0.9161 | 0.9467 | 0.9311 | 150 | | I-appointment_id | 0.9667 | 0.9667 | 0.9667 | 210 | | I-appointment_type | 0.8182 | 0.7368 | 0.7754 | 171 | | I-practitioner_name | 0.9540 | 0.8941 | 0.9231 | 255 | | O | 0.9782 | 0.9892 | 0.9837 | 3813 | | --- | --- | --- | --- | ---- | | **Accuracy** | | | 0.9673 | 4771 | | **Macro Avg** | 0.9283 | 0.8973 | 0.9117 | 4771 | | **Weighted Avg** | 0.9664 | 0.9673 | 0.9666 | 4771 | The model achieves near-perfect results on the NER task and excellent results on the intent classification task for this specific dataset. ## Limitations and Bias - The model's performance is highly dependent on the quality and scope of the **HASD dataset**. It may not generalize well to phrasing or appointment types significantly different from what it was trained on. - The dataset was primarily generated from templates, which may not capture the full diversity of real human language. - The model inherits any biases present in the `distilbert-base-uncased` model and the `clinc/clinc_oos` dataset.