metadata
language: en
license: apache-2.0
library_name: transformers
base_model: bert-base-uncased
model_name: cross-encoder-bert-base-Hinge
source: https://github.com/xpmir/cross-encoders
paper: http://arxiv.org/abs/2603.03010
tags:
- cross-encoder
- sequence-classification
- tensorboard
datasets:
- msmarco
pipeline_tag: text-classification
cross-encoder-bert-base-Hinge
This model is a cross-encoder based on bert-base-uncased. It was trained on Ms-Marco using loss hingeLoss as part of a reproducibility paper for training cross encoders: "Reproducing and Comparing Distillation Techniques for Cross-Encoders", see the paper for more details.
Contents
Model Description
This model is intended for re-ranking the top results returned by a retrieval system (like BM25, Bi-Encoders or SPLADE).
- Training Data: MS MARCO Passage
- Language: English
- Loss hingeLoss
Training can be easily reproduced using the assiciated repository. The exact training configuration used for this model is also detailed in config.yaml.
Usage
Quick Start:
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained("xpmir/cross-encoder-bert-base-Hinge")
features = tokenizer("What is experimaestro ?", "Experimaestro is a powerful framework for ML experiments management...", padding=True, truncation=True, return_tensors="pt")
model.eval()
with torch.no_grad():
scores = model(**features).logits
print(scores)
Evaluations
We provide evaluations of this cross-encoder re-ranking the top 1000 documents retrieved by naver/splade-v3-distilbert.
| dataset | RR@10 | nDCG@10 |
|---|---|---|
| msmarco_dev | 38.05 | 44.38 |
| trec2019 | 97.09 | 73.12 |
| trec2020 | 93.52 | 70.83 |
| fever | 79.83 | 79.96 |
| arguana | 22.79 | 33.87 |
| climate_fever | 33.40 | 24.97 |
| dbpedia | 73.46 | 43.13 |
| fiqa | 42.20 | 34.97 |
| hotpotqa | 88.41 | 72.40 |
| nfcorpus | 54.88 | 33.45 |
| nq | 51.22 | 56.13 |
| quora | 77.95 | 79.84 |
| scidocs | 26.64 | 15.26 |
| scifact | 66.32 | 69.18 |
| touche | 59.68 | 32.27 |
| trec_covid | 89.62 | 63.75 |
| robust04 | 71.38 | 46.18 |
| lotte_writing | 64.17 | 55.35 |
| lotte_recreation | 59.69 | 54.46 |
| lotte_science | 42.39 | 35.07 |
| lotte_technology | 51.80 | 42.35 |
| lotte_lifestyle | 69.76 | 60.15 |
| Mean In Domain | 76.22 | 62.78 |
| BEIR 13 | 58.95 | 49.17 |
| LoTTE (OOD) | 59.86 | 48.93 |