SentenceTransformer based on google/embeddinggemma-300m
This is a sentence-transformers model finetuned from google/embeddinggemma-300m on the mortgage-qa-dataset dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
Model Details
Model Description
- Model Type: Sentence Transformer
- Base model: google/embeddinggemma-300m
- Maximum Sequence Length: 2048 tokens
- Output Dimensionality: 768 dimensions
- Similarity Function: Cosine Similarity
- Training Dataset:
- Language: en
Model Sources
- Documentation: Sentence Transformers Documentation
- Repository: Sentence Transformers on GitHub
- Hugging Face: Sentence Transformers on Hugging Face
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 2048, 'do_lower_case': False, 'architecture': 'Gemma3TextModel'})
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
(3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
(4): Normalize()
)
Usage
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the ๐ค Hub
model = SentenceTransformer("sugiv/embeddinggemma-300m-mortgage")
# Run inference
queries = [
"When is a borrower eligible for a streamline refinance?",
]
documents = [
'A borrower is eligible for a streamline refinance if they have made at least six consecutive on-time payments on their current mortgage.',
'fha_handbook_4000_1',
'fha_handbook_4000_1_chunk_007',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.8276, -0.0791, -0.0792]])
Evaluation
Metrics
Information Retrieval
- Datasets:
mortgage-evalandmortgage-test - Evaluated with
InformationRetrievalEvaluator
| Metric | mortgage-eval | mortgage-test |
|---|---|---|
| cosine_accuracy@1 | 0.3442 | 0.2781 |
| cosine_accuracy@3 | 0.6469 | 0.5621 |
| cosine_accuracy@5 | 0.7864 | 0.7101 |
| cosine_accuracy@10 | 0.9318 | 0.8728 |
| cosine_precision@1 | 0.3442 | 0.2781 |
| cosine_precision@3 | 0.2156 | 0.1874 |
| cosine_precision@5 | 0.1573 | 0.142 |
| cosine_precision@10 | 0.0932 | 0.0873 |
| cosine_recall@1 | 0.3442 | 0.2781 |
| cosine_recall@3 | 0.6469 | 0.5621 |
| cosine_recall@5 | 0.7864 | 0.7101 |
| cosine_recall@10 | 0.9318 | 0.8728 |
| cosine_ndcg@10 | 0.6209 | 0.5531 |
| cosine_mrr@10 | 0.5233 | 0.453 |
| cosine_map@100 | 0.5285 | 0.4622 |
Training Details
Training Dataset
mortgage-qa-dataset
- Dataset: mortgage-qa-dataset at de29792
- Size: 2,699 training samples
- Columns:
question,answer,source_document, andsource_chunk - Approximate statistics based on the first 1000 samples:
question answer source_document source_chunk type string string string string details - min: 9 tokens
- mean: 16.15 tokens
- max: 27 tokens
- min: 14 tokens
- mean: 31.67 tokens
- max: 62 tokens
- min: 8 tokens
- mean: 9.99 tokens
- max: 14 tokens
- min: 14 tokens
- mean: 15.99 tokens
- max: 20 tokens
- Samples:
question answer source_document source_chunk When is a full appraisal required for a mortgage application?A full appraisal is required for most transactions, but it can be waived for certain streamlined refinance programs if the Loan-to-Value LTV ratio is 90 or less.fha_handbook_4000_1fha_handbook_4000_1_chunk_005When getting a mortgage, who orders the title insurance for the lender?While often coordinated by the settlement agent, the lender typically requires and is the ultimate recipient of the lenders title insurance policy to protect their financial interest.va_chapter4_underwritingva_chapter4_underwriting_chunk_012What components of a loan application does an underwriter assess?Underwriters analyze the four Cs of credit: Capacity income and DTI, Capital assets and reserves, Collateral property value, and Credit credit history and score.va_chapter5_processingva_chapter5_processing_chunk_005 - Loss:
CachedMultipleNegativesRankingLosswith these parameters:{ "scale": 20.0, "similarity_fct": "cos_sim", "mini_batch_size": 32, "gather_across_devices": false }
Evaluation Dataset
mortgage-qa-dataset
- Dataset: mortgage-qa-dataset at de29792
- Size: 337 evaluation samples
- Columns:
question,answer,source_document, andsource_chunk - Approximate statistics based on the first 337 samples:
question answer source_document source_chunk type string string string string details - min: 9 tokens
- mean: 16.44 tokens
- max: 28 tokens
- min: 17 tokens
- mean: 32.28 tokens
- max: 62 tokens
- min: 8 tokens
- mean: 10.14 tokens
- max: 14 tokens
- min: 14 tokens
- mean: 16.14 tokens
- max: 20 tokens
- Samples:
question answer source_document source_chunk What financial metrics are crucial for an AUS to issue an approval?Key AUS factors include credit score, loan-to-value ratio, debt-to-income ratio, and the overall strength and stability of the borrowers financial profile.va_chapter4_underwritingva_chapter4_underwriting_chunk_017Can you explain how an LTV ratio is figured out?The LTV ratio is calculated by dividing the mortgage loan amount by the appraised value or purchase price of the property, whichever is lower.fannie_mae_servicing_guidefannie_mae_servicing_guide_chunk_002How do lenders verify a borrowers employment history?Lenders verify employment by contacting employers directly and typically require a two-year history, which may be confirmed via recent pay stubs and W-2 forms.freddie_mac_guidefreddie_mac_guide_chunk_002 - Loss:
CachedMultipleNegativesRankingLosswith these parameters:{ "scale": 20.0, "similarity_fct": "cos_sim", "mini_batch_size": 32, "gather_across_devices": false }
Training Hyperparameters
Non-Default Hyperparameters
eval_strategy: stepsper_device_train_batch_size: 16per_device_eval_batch_size: 16learning_rate: 3e-06num_train_epochs: 4warmup_steps: 100fp16: Trueload_best_model_at_end: Truebatch_sampler: no_duplicates
All Hyperparameters
Click to expand
overwrite_output_dir: Falsedo_predict: Falseeval_strategy: stepsprediction_loss_only: Trueper_device_train_batch_size: 16per_device_eval_batch_size: 16per_gpu_train_batch_size: Noneper_gpu_eval_batch_size: Nonegradient_accumulation_steps: 1eval_accumulation_steps: Nonetorch_empty_cache_steps: Nonelearning_rate: 3e-06weight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08max_grad_norm: 1.0num_train_epochs: 4max_steps: -1lr_scheduler_type: linearlr_scheduler_kwargs: {}warmup_ratio: 0.0warmup_steps: 100log_level: passivelog_level_replica: warninglog_on_each_node: Truelogging_nan_inf_filter: Truesave_safetensors: Truesave_on_each_node: Falsesave_only_model: Falserestore_callback_states_from_checkpoint: Falseno_cuda: Falseuse_cpu: Falseuse_mps_device: Falseseed: 42data_seed: Nonejit_mode_eval: Falseuse_ipex: Falsebf16: Falsefp16: Truefp16_opt_level: O1half_precision_backend: autobf16_full_eval: Falsefp16_full_eval: Falsetf32: Nonelocal_rank: 0ddp_backend: Nonetpu_num_cores: Nonetpu_metrics_debug: Falsedebug: []dataloader_drop_last: Falsedataloader_num_workers: 0dataloader_prefetch_factor: Nonepast_index: -1disable_tqdm: Falseremove_unused_columns: Truelabel_names: Noneload_best_model_at_end: Trueignore_data_skip: Falsefsdp: []fsdp_min_num_params: 0fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}fsdp_transformer_layer_cls_to_wrap: Noneaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}parallelism_config: Nonedeepspeed: Nonelabel_smoothing_factor: 0.0optim: adamw_torch_fusedoptim_args: Noneadafactor: Falsegroup_by_length: Falselength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falsedataloader_pin_memory: Truedataloader_persistent_workers: Falseskip_memory_metrics: Trueuse_legacy_prediction_loop: Falsepush_to_hub: Falseresume_from_checkpoint: Nonehub_model_id: Nonehub_strategy: every_savehub_private_repo: Nonehub_always_push: Falsehub_revision: Nonegradient_checkpointing: Falsegradient_checkpointing_kwargs: Noneinclude_inputs_for_metrics: Falseinclude_for_metrics: []eval_do_concat_batches: Truefp16_backend: autopush_to_hub_model_id: Nonepush_to_hub_organization: Nonemp_parameters:auto_find_batch_size: Falsefull_determinism: Falsetorchdynamo: Noneray_scope: lastddp_timeout: 1800torch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Noneinclude_tokens_per_second: Falseinclude_num_input_tokens_seen: Falseneftune_noise_alpha: Noneoptim_target_modules: Nonebatch_eval_metrics: Falseeval_on_start: Falseuse_liger_kernel: Falseliger_kernel_config: Noneeval_use_gather_object: Falseaverage_tokens_across_devices: Falseprompts: Nonebatch_sampler: no_duplicatesmulti_dataset_batch_sampler: proportionalrouter_mapping: {}learning_rate_mapping: {}
Training Logs
| Epoch | Step | Training Loss | Validation Loss | mortgage-eval_cosine_ndcg@10 | mortgage-test_cosine_ndcg@10 |
|---|---|---|---|---|---|
| -1 | -1 | - | - | 0.5803 | - |
| 0.1479 | 25 | 0.1574 | - | - | - |
| 0.2959 | 50 | 0.1053 | 0.0722 | 0.5993 | - |
| 0.4438 | 75 | 0.0969 | - | - | - |
| 0.5917 | 100 | 0.0765 | 0.0773 | 0.6085 | - |
| 0.7396 | 125 | 0.079 | - | - | - |
| 0.8876 | 150 | 0.0802 | 0.0858 | 0.6056 | - |
| 1.0355 | 175 | 0.021 | - | - | - |
| 1.1834 | 200 | 0.0728 | 0.0549 | 0.6093 | - |
| 1.3314 | 225 | 0.0857 | - | - | - |
| 1.4793 | 250 | 0.071 | 0.0659 | 0.6145 | - |
| 1.6272 | 275 | 0.0633 | - | - | - |
| 1.7751 | 300 | 0.1844 | 0.0687 | 0.6209 | - |
| 1.9231 | 325 | 0.0545 | - | - | - |
| 2.0710 | 350 | 0.0474 | 0.0646 | 0.6025 | - |
| 2.2189 | 375 | 0.0702 | - | - | - |
| 2.3669 | 400 | 0.0831 | 0.0699 | 0.6026 | - |
| 2.5148 | 425 | 0.0635 | - | - | - |
| 2.6627 | 450 | 0.103 | 0.0674 | 0.6031 | - |
| 2.8107 | 475 | 0.097 | - | - | - |
| 2.9586 | 500 | 0.077 | 0.0686 | 0.6032 | - |
| 3.1065 | 525 | 0.0713 | - | - | - |
| 3.2544 | 550 | 0.1617 | 0.0668 | 0.6087 | - |
| 3.4024 | 575 | 0.1084 | - | - | - |
| 3.5503 | 600 | 0.0791 | 0.0658 | 0.6038 | - |
| 3.6982 | 625 | 0.0477 | - | - | - |
| 3.8462 | 650 | 0.0956 | 0.0659 | 0.6073 | - |
| 3.9941 | 675 | 0.0587 | - | - | - |
| -1 | -1 | - | - | 0.6209 | 0.5531 |
- The bold row denotes the saved checkpoint.
Framework Versions
- Python: 3.11.11
- Sentence Transformers: 5.1.0
- Transformers: 4.57.0.dev0
- PyTorch: 2.8.0.dev20250319+cu128
- Accelerate: 1.10.1
- Datasets: 4.0.0
- Tokenizers: 0.22.0
Citation
BibTeX
Sentence Transformers
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
CachedMultipleNegativesRankingLoss
@misc{gao2021scaling,
title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
year={2021},
eprint={2101.06983},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
- Downloads last month
- 3
Model tree for sugiv/embeddinggemma-300m-mortgage
Base model
google/embeddinggemma-300mEvaluation results
- Cosine Accuracy@1 on mortgage evalself-reported0.344
- Cosine Accuracy@3 on mortgage evalself-reported0.647
- Cosine Accuracy@5 on mortgage evalself-reported0.786
- Cosine Accuracy@10 on mortgage evalself-reported0.932
- Cosine Precision@1 on mortgage evalself-reported0.344
- Cosine Precision@3 on mortgage evalself-reported0.216
- Cosine Precision@5 on mortgage evalself-reported0.157
- Cosine Precision@10 on mortgage evalself-reported0.093
- Cosine Recall@1 on mortgage evalself-reported0.344
- Cosine Recall@3 on mortgage evalself-reported0.647