truefoundry/setfit-mxbai-embed-xsmall-v1-test

SetFit with mixedbread-ai/mxbai-embed-xsmall-v1

This is a SetFit model that can be used for Text Classification. This SetFit model uses mixedbread-ai/mxbai-embed-xsmall-v1 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Sentence Transformer body: mixedbread-ai/mxbai-embed-xsmall-v1
• Classification head: a LogisticRegression instance
• Maximum Sequence Length: 512 tokens
• Number of Classes: 31 classes

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
external_rx_transfer	'these Pharmacy to call Walgreens so they can transfer a RINVOQ TAB and EQ HYDROCORTISONE MAXIMU M STRENGTH CRE prescription' 'I do not accept alternative to SILACE SYP and ISENTRESS POW. If you don’t have it tell me so I may use alternate Pharmacy for this prescription. Let me know asap.' 'I would like you to transfer LORAZEPAM VIA and ENFAMIL NEUROPRO INFANT LIQ prescription to a walmart pharmacy that does have it please'
rx_cancel	'cancel all autofill of MIDODRINE HCL TAB and BASIS EXTRA DRY BAR?' 'trying to cancel a prescription for SLEEP OVERS SMALL/MEDIUM MIS and FLANDERS BUTTOCKS OIN?' 'cancellation for a MELATONIN LIQ and ALIVE MENS GUMMY MULTIVITAMIN CHW?'
store_faq	'Are medications available at the store in 15106?' 'Are money order services offered at the store in 22180?' 'How do I change the insurance on file for LIQUID HOPE PEPTIDE HIGH PROTEIN LIQ and CYRED EQ TAB at 89113?'
rx_refill	'will you fill my ZILBRYSQ INJ and FAST-HEAT HEATING PAD PAD?' 'I need to have SKIN REPAIR LOT and CEFUROXIME SODIUM INJ refilled' 'I need the prescription for Basaglar 100 unit filled today.my pharmacy was out of stock.'
rx_action_notes_status	"there's a mistake in my prescription to be put on hold for the PSEUDOEPHEDRINE HCL TAB and EFAVIRENZ TAB?" 'Ok this is a new LANCETS ULTRA THIN 30G MIS and ASPIRIN POW prescription I don’t understand why it is not allowed to be filled?' 'Why are the SPIRIVA RESPIMAT SPR and FT HYDROGEN PEROXIDE SOL on hold?'
cancel_vaccine_appointment	'How do I cancel my Rabies vaccine appointment at the prescription drug plan?' 'Can I cancel my Rabies vaccine appointment at the VA health plan?' 'How do I confirm the cancellation of my RSV-(Respiratory Syncytial Virus) vaccine appointment?'
drug_usage	'when can I take my KNEE SUPPORT/CRISS-CROSS/ELASTIC/SMALL MIS and HM DRY EYE RELIEF SOL?' 'need to ask a question about directions to take a KNEE SUPPORT/NEOPRENE/OVAL PAD/LARGE MIS and EQ DAYTIME COLD & FLU MULTI-SYMPTOM RELIEF LIQ?' 'in antibiotic mixed with a SM CALCIUM/VITAMIN D3 TAB and DOLISHALE TAB that I take every day that they want me to cut in half questionable?'
drug_insurance_coverage	'check insurance coverage for PEN NEEDLES 31GX8MM (5/16) MIS and EQL COTTON BALLS SUPER JUMBO MIS?' 'get pharmacist my insurance to cover for the LUMINEUX WHITENING TOOTHPASTE PST and BD INSULIN SYRINGE/0.5ML/29G X 12.7MM SYN?' 'I need my insurance information on file for TGT BLOOD PRESSURE MONITOR/AUTOMATIC MIS and GILTUSS BUCALSEP SPR?'
get_vaccine_appointment	'Can I get a RSV-(Respiratory Syncytial Virus) vaccine appointment after hours?' 'can you schedule a Dtap DT vaccination appointment on Monday' 'Whooping Cough-(Adacel) vaccine prescription schedule'
available_vaccine	'what is the current RSV-(Respiratory Syncytial Virus) vaccine being used' 'What about the availability of Typhoid ?' 'confirming Tdap available'
rephrase	'Could you run that by me again? one more time' "Sorry I didn't catch that. a little louder" 'you told me'
store_timezone	'Can you tell me if the store in REDDING is in the MST time zone?' 'Can you provide the time zone for the store at 189 SOUTH TURNPIKE ROAD?' 'Can you provide the time zone for the store at 760 MILES RD.?'
hold_on	"I would like to be put on hold until 10 minutes? and I'll wait on hold until someone can assist me." 'One second I need to check my email.' 'hold on just a minute'
healthcare	'hello i am nurse Is there a patient counseling sheet available for WRIST SPLINT/NEOPRENE/LEFT/MEDIUM MIS?' 'Doctor I need to transfer patient prescription for ORAL ANALGESIC MAXIMUM STRENGTH GEL to your pharmacy.' "i am Healthcare Provider from Dr's office Is NESTABS TAB safe for pregnant women?"
drug_dosage_instructions	'MURI-LUBE OIL and D3 5000 CAP? I need some instructions on the medication' 'dosage of a LIDOCAINE HCL POW and CLOBETASOL PROPIONATE CRE?' 'How much ROGAINE MENS AER and CONTOUR NEXT EZ BLOOD GLUCOSE MONITORING SYSTEM KIT should I take daily?'
drug_interactions	'find out about a drug interaction with OMRON 7 SERIES BLOOD PRESSURE MONITOR/UPPER ARM/BLUETOOTH MIS and FABRIC BANDAGES/ASSORTED MIS?' "I want to understand if there's an interaction between 2 medications of XYWAV SOL and GERI-TUSSIN LIQ that I took today?" 'I need to know about specific TRANSPORE SURGICAL TAPE 1X10YDS TAP and AYR NASAL MIST ALLERGY & SINUS HYPERTONIC SALINE SPR drug interactions?'
store_hours	'will that stores open on Yom Kippur - October 12?' 'I was wanting to know what the store hours are for this weekend' "is the front door open on Valentine's Day - February 14"
store_directions	'how to find this store from my home address 3700 BRADFORDVILLE RD' 'Can you give me directions to the store at 06117?' 'is this store near to FRESNO city'
drug_information	'I want to find out about ROZLYTREK CAP and ADULT BLOOD PRESSURE CUFF/LARGE KIT?' 'What are the uses of NON-ASPIRIN PAIN RELIEF EXTRA STRENGTH TAB and CRANBERRY CONCENTRATE TRIPLE STRENGTH ULTRA/VITAMIN C CAP?' 'questions about my medicine'
rx_expedite	'I would like a call back from the pharmacy to pick up my ULTRATHON INSECT REPELLENT 8 AER and AIMSCO LUBRICATED MIS possibly 2 days early due to a medical procedure happening on Monday?' 'expedite refill of MEDIHONEY CALCIUM ALGINATE DRESSING/2X2 PAD and ADVOCATE ALCOHOL PREP PADS PAD?' "I need an auto refill sooner than the date it's currently scheduled for MONOJECT LIFESHIELD BLUNTCANNULA/LUER LOCK SYR/3ML/18G X 1 MIS and EQ COMPLETE CHEWABLE MULTIVITAMIN CHILDRENS CHW ?"
drug_side_effects	'Are there any common side effects for SSS 10%-5% CRE and FINGERSTIX LANCETS MIS?' 'is my prescription for SANDOSTATIN LAR DEPOT KIT and DIGESTIVE ADVANTAGE CAP cancerous?' 'my WAL-ZYR ALL DAY ALLERGY CHILDRENS SOL and GNP VITAMIN B-1 TAB making me sick?'
rx_status	'check prescription status on my NAROPIN INJ and AUGMENTIN TAB. When will it be ready?' 'check prescription status on my SENNA-TIME S TAB and DROPLET PEN NEEDLES 32GX8MM PEN. When will it be ready?' 'what was the other medication refill'
rx_drug_price	'how much is it going to cost for MICROPORE SURGICAL TAPE 1X1-1/2YDS TAP and DROPLET INSULIN SYRINGE U-100/1ML/31G X 5/16 SYN?' "what's the cost of PREFERRED PLUS UNIFINE PENTIPS 29G X 12MM MIS and EQ COUGH DM SUS and PREFERRED PLUS UNIFINE PENTIPS 29G X 12MM MIS and EQ COUGH DM SUS?" 'cost of QC CHILDRENS CHEWABLE COMPLETE CHW and AZELASTINE HCL SPR and a transfer?'
internal_rx_transfer	"Please transfer the prescription for RITALIN LA CAP and COMFORT TOUCH PEN NEEDLES/32G X 4MM MISwhich is on hold because it's on back order but when it comes in I would like to have it delivered please" 'transfer or receive a new SUR-FIT NATURA DRAINABLE POUCH/10/2-SIDE FLOCKING 1-3/4 MIS and CILOXAN SOL prescription' 'need to request a transfer for VITABEX CAP and AUROVELA 24 FE TAB'
create_vaccine_appointment	'What documents do I need for my Meningitis vaccine appointment?' 'Can I schedule a Flu vaccine appointment for next week?' 'I would like to make an appointment for the Flu shot and the Flu shot'
drug_availability	'SANDOSTATIN LAR DEPOT KIT and AVANDIA TAB in stock?' 'do you have this QC BORDER ISLAND GAUZE PAD 2X2 PAD and ARGYLE STERILE SALINE SOL?' 'what is available for MIRENA IUD and BUDESONIDE CAP?'
store_connect	'let me talk to the front store technician in 74012.' 'I need to check if a product is available at the store in 432 S. MAIN ST..' 'Can you connect me to the front store at 615-621 BELMONT AVENUE to check my unpicked item?'
thank	'Starbucks our best' 'no okay thank you' "no that'll be all"
rx_delivery	'deliver VYVANSE CHW and IMIPRAMINE HYDROCHLORIDE TAB prescription?' 'um delivery of TWYNEO CRE and GLUCOCARD VITAL BLOOD GLUCOSE MONITORING SYSTEM KIT for?' 'RA ENEMA ENE and BABY WIPES MIS delivery to my apartment in silver Spring.Phyllis?'
live_pharmacist	'yes operator please' 'give me a call' 'I need to talk to someone about refill prescriptions'
store_address	'Is there a pharmacy in AVONDALE tto pickup UNIFINE ULTRA PEN NEEDLE/31GX8MM MIS and GNP STERILE LANCETS 30G MIS? Please provide the address?' 'Can you provide the location of the pharmacy in 98374 zipcode ?' 'What is the store address for the pharmacy in NORTHBOROUGH?'

Evaluation

Metrics

Label	Accuracy
all	0.7935

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("setfit_model_id")
# Run inference
preds = model("I'm happy to wait on hold.  thank you.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	1	14.1935	56

Label	Training Sample Count
available_vaccine	8
cancel_vaccine_appointment	8
create_vaccine_appointment	8
drug_availability	8
drug_dosage_instructions	8
drug_information	8
drug_insurance_coverage	8
drug_interactions	8
drug_side_effects	8
drug_usage	8
external_rx_transfer	8
get_vaccine_appointment	8
healthcare	8
hold_on	8
internal_rx_transfer	8
live_pharmacist	8
rephrase	8
rx_action_notes_status	8
rx_cancel	8
rx_delivery	8
rx_drug_price	8
rx_expedite	8
rx_refill	8
rx_status	8
store_address	8
store_connect	8
store_directions	8
store_faq	8
store_hours	8
store_timezone	8
thank	8

Training Hyperparameters

• batch_size: (32, 32)
• num_epochs: (1, 1)
• max_steps: -1
• sampling_strategy: oversampling
• body_learning_rate: (2e-05, 1e-05)
• head_learning_rate: 0.01
• loss: CosineSimilarityLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: False
• use_amp: False
• warmup_proportion: 0.1
• l2_weight: 0.01
• seed: 42
• eval_max_steps: -1
• load_best_model_at_end: True

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0005	1	0.1621	-
0.0269	50	0.1732	-
0.0538	100	0.1495	-
0.0806	150	0.1129	-
0.1075	200	0.0773	-
0.1344	250	0.062	-
0.1613	300	0.0452	-
0.1882	350	0.0435	-
0.2151	400	0.0354	-
0.2419	450	0.0283	-
0.2688	500	0.0284	-
0.2957	550	0.0236	-
0.3226	600	0.0218	-
0.3495	650	0.0182	-
0.3763	700	0.019	-
0.4032	750	0.0159	-
0.4301	800	0.0161	-
0.4570	850	0.0124	-
0.4839	900	0.0133	-
0.5108	950	0.0154	-
0.5376	1000	0.0137	-
0.5645	1050	0.0125	-
0.5914	1100	0.0164	-
0.6183	1150	0.0125	-
0.6452	1200	0.0092	-
0.6720	1250	0.0129	-
0.6989	1300	0.0109	-
0.7258	1350	0.0115	-
0.7527	1400	0.0109	-
0.7796	1450	0.0106	-
0.8065	1500	0.0111	-
0.8333	1550	0.0092	-
0.8602	1600	0.0073	-
0.8871	1650	0.0081	-
0.9140	1700	0.0089	-
0.9409	1750	0.01	-
0.9677	1800	0.0084	-
0.9946	1850	0.008	-
1.0	1860	-	0.0890

Framework Versions

• Python: 3.12.0
• SetFit: 1.1.1
• Sentence Transformers: 3.4.1
• Transformers: 4.49.0
• PyTorch: 2.4.1
• Datasets: 3.4.1
• Tokenizers: 0.21.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}