rahulseetharaman/reranker-msmarco-v1.1-bert-uncased_L-12_H-128_A-2-listnet

CrossEncoder based on bansalaman18/bert-uncased_L-12_H-128_A-2

This is a Cross Encoder model finetuned from bansalaman18/bert-uncased_L-12_H-128_A-2 on the ms_marco dataset using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: bansalaman18/bert-uncased_L-12_H-128_A-2
• Maximum Sequence Length: 512 tokens
• Number of Output Labels: 1 label
• Training Dataset:
  • ms_marco
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("rahulseetharaman/reranker-msmarco-v1.1-bert-uncased_L-12_H-128_A-2-listnet")
# Get scores for pairs of texts
pairs = [
    ['is narcissistic personality disorder genetic', 'http://samvak.tripod.com/index.html. no it is not genetic ,it is a learned disorder. The narcissistic behavioral or personality disorder is a learned trait, and is not passed through genetic. However, some psychologists would argue that narcissism is a genetic trait due to an imbalance of seortonin within the brain. '],
    ['is narcissistic personality disorder genetic', 'Report Abuse. The narcissistic personality disorder is probably not genetic in the same way as the shape of your thumb. Rather, it is probably the vulnerability to the disorder that is inherited. The narcissistic behavioral or personality disorder is a learned trait, and is not passed through genetic. However, some psychologists would argue that narcissism is a genetic trait due to an imbalance of seortonin within the brain'],
    ['is narcissistic personality disorder genetic', 'Causes of Narcissistic Personality Disorder. The causes of NPD aren’t well understood. Inherited genetic defects are thought to be responsible for up to 50 percent of cases. Contributing environmental factors may include childhood abuse or neglect, excessive pampering, or unrealistic expectations from parents. Narcissistic personality disorder (NPD) is a mental condition in which sufferers have an inflated opinion of themselves and an intense need for the admiration and attention of others.'],
    ['is narcissistic personality disorder genetic', 'Abstract. Genetic epidemiologic studies indicate that all ten personality disorders (PDs) classified on the DSM-IV axis II are modestly to moderately heritable. Shared environmental and nonadditive genetic factors are of minor or no importance. No sex differences have been identified'],
    ['is narcissistic personality disorder genetic', 'Narcissistic personality disorder, also known as NPD, is a personality disorder in which the individual has a distorted self image, unstable and intense emotions, is overly preoccupied with vanity, prestige, power and personal adequacy, lacks empathy, and has an exaggerated sense of superiority. Below are the most common signs and symptoms found in people with narcissistic personality disorder: 1  An insatiable appetite for the attention of other people. 2  Generally prone to extreme feelings of jealousy. 3  Behave is if they deserve special treatment. 4  Commonly exaggerate their achievements, talents and importance.'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'is narcissistic personality disorder genetic',
    [
        'http://samvak.tripod.com/index.html. no it is not genetic ,it is a learned disorder. The narcissistic behavioral or personality disorder is a learned trait, and is not passed through genetic. However, some psychologists would argue that narcissism is a genetic trait due to an imbalance of seortonin within the brain. ',
        'Report Abuse. The narcissistic personality disorder is probably not genetic in the same way as the shape of your thumb. Rather, it is probably the vulnerability to the disorder that is inherited. The narcissistic behavioral or personality disorder is a learned trait, and is not passed through genetic. However, some psychologists would argue that narcissism is a genetic trait due to an imbalance of seortonin within the brain',
        'Causes of Narcissistic Personality Disorder. The causes of NPD aren’t well understood. Inherited genetic defects are thought to be responsible for up to 50 percent of cases. Contributing environmental factors may include childhood abuse or neglect, excessive pampering, or unrealistic expectations from parents. Narcissistic personality disorder (NPD) is a mental condition in which sufferers have an inflated opinion of themselves and an intense need for the admiration and attention of others.',
        'Abstract. Genetic epidemiologic studies indicate that all ten personality disorders (PDs) classified on the DSM-IV axis II are modestly to moderately heritable. Shared environmental and nonadditive genetic factors are of minor or no importance. No sex differences have been identified',
        'Narcissistic personality disorder, also known as NPD, is a personality disorder in which the individual has a distorted self image, unstable and intense emotions, is overly preoccupied with vanity, prestige, power and personal adequacy, lacks empathy, and has an exaggerated sense of superiority. Below are the most common signs and symptoms found in people with narcissistic personality disorder: 1  An insatiable appetite for the attention of other people. 2  Generally prone to extreme feelings of jealousy. 3  Behave is if they deserve special treatment. 4  Commonly exaggerate their achievements, talents and importance.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Reranking

• Datasets: NanoMSMARCO_R100, NanoNFCorpus_R100 and NanoNQ_R100
• Evaluated with CrossEncoderRerankingEvaluator with these parameters:

  {
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Metric	NanoMSMARCO_R100	NanoNFCorpus_R100	NanoNQ_R100
map	0.0614 (-0.4282)	0.2758 (+0.0148)	0.0457 (-0.3739)
mrr@10	0.0357 (-0.4418)	0.4025 (-0.0974)	0.0219 (-0.4048)
ndcg@10	0.0547 (-0.4858)	0.2663 (-0.0587)	0.0390 (-0.4617)

Cross Encoder Nano BEIR

• Dataset: NanoBEIR_R100_mean
• Evaluated with CrossEncoderNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "rerank_k": 100,
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Metric	Value
map	0.1276 (-0.2624)
mrr@10	0.1534 (-0.3147)
ndcg@10	0.1200 (-0.3354)

Training Details

Training Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 78,704 training samples
• Columns: query, docs, and labels
• Approximate statistics based on the first 1000 samples:

  	query	docs	labels
  type	string	list	list
  details	min: 9 characters mean: 34.32 characters max: 100 characters	min: 3 elements mean: 6.50 elements max: 10 elements	min: 3 elements mean: 6.50 elements max: 10 elements

• Samples:

  query	docs	labels
  what is light soy sauce classified as	['Types of Chinese Soy Sauce. There are two types of Chinese soy sauce: dark and light. Chinese dark soy sauce is very deep in color with high viscosity, and it contains additives such as caramel coloring and monosodium glutamate (MSG). Chinese light soy sauce has a lighter color with lower viscosity. In general, this soy sauce contains MSG and other additives used to control its strong saltiness.', 'This pale amber-colored soy sauce, even lighter than light color soy sauce, originated in the Hekinan district of Aichi prefecture. Its flavor is bland, but with a strong sweetness, and it has a unique fragrance. It is used in cooking dishes such as soups and chawanmushi egg custard, using its fragrance and light color. ', 'Soy sauce (also called soya sauce) is a condiment made from a fermented paste of boiled soybeans, roasted grain, brine, and Aspergillus oryzae or Aspergillus sojae molds. After fermentation, the paste is pressed, producing a liquid, which is the soy sauce, and a solid b...	[1, 0, 0, 0, 0, ...]
  normal respiration rate for adults	['Strength of the pulse. The normal pulse for healthy adults ranges from 60 to 100 beats per minute. The pulse rate may fluctuate and increase with exercise, illness, injury, and emotions. Females ages 12 and older, in general, tend to have faster heart rates than do males. The rate is usually measured when a person is at rest and simply involves counting the number of breaths for one minute by counting how many times the chest rises. Respiration rates may increase with fever, illness, and with other medical conditions.', 'Human respiration rate is measured when a person is at rest and involves counting the number of breaths for one minute by counting how many times the chest rises. An optical breath rate sensor can be used for monitoring patients during a magnetic resonance imaging scan. A normal respiratory rate is termed eupnea, an increased respiratory rate is termed tachypnea and a lower-than-normal respiratory rate is termed bradypnea. Breathing (which in organisms with lungs is ...	[1, 0, 0, 0, 0, ...]
  what is chlamydia harmful	['Chlamydia is a curable bacterial infection that can be transmitted through genital, oral, or anal sex. It can also be passed from a woman to her baby during delivery. ', 'Chlamydia can cause a condition called nongonococcal urethritis (NGU) -- an infection of the urethra (the tube by which men and women pass urine), epididymitis -- an infection of the epididymis (the tube that carries sperm away from the testes), or proctitis -- an inflammation of the rectum. If left untreated, chlamydia infection can cause pelvic inflammatory disease which can lead to damage of the fallopian tubes (the tubes connecting the ovaries to the uterus) or even cause infertility (the inability to have children).', 'Pregnant women with an untreated chlamydia infection are at a greater risk of developing complications such as a miscarriage, premature birth or stillbirth. The first thing you should do is discuss your condition with your GP to find out which antibiotic is best for you. ', "Chlamydia is a bac...	[1, 0, 0, 0, 0, ...]

• Loss: ListNetLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "mini_batch_size": 16
  }
  

Evaluation Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 1,000 evaluation samples
• Columns: query, docs, and labels
• Approximate statistics based on the first 1000 samples:

  	query	docs	labels
  type	string	list	list
  details	min: 11 characters mean: 34.1 characters max: 93 characters	min: 2 elements mean: 6.00 elements max: 10 elements	min: 2 elements mean: 6.00 elements max: 10 elements

• Samples:

  query	docs	labels
  is narcissistic personality disorder genetic	['http://samvak.tripod.com/index.html. no it is not genetic ,it is a learned disorder. The narcissistic behavioral or personality disorder is a learned trait, and is not passed through genetic. However, some psychologists would argue that narcissism is a genetic trait due to an imbalance of seortonin within the brain. ', 'Report Abuse. The narcissistic personality disorder is probably not genetic in the same way as the shape of your thumb. Rather, it is probably the vulnerability to the disorder that is inherited. The narcissistic behavioral or personality disorder is a learned trait, and is not passed through genetic. However, some psychologists would argue that narcissism is a genetic trait due to an imbalance of seortonin within the brain', 'Causes of Narcissistic Personality Disorder. The causes of NPD aren’t well understood. Inherited genetic defects are thought to be responsible for up to 50 percent of cases. Contributing environmental factors may include childhood abuse or negle...	[1, 0, 0, 0, 0, ...]
  cost of home dialysis machine	['1 A basic dialysis machine can be purchased in the regions of $15,000; however, this does not include any necessary supplies such as a drip. 2 For example, a NxStage kidney dialysis machine purchased with supplies can cost upwards of $45,000. 3 This does not include the costs of running the machine.', 'Tweet. Dialysis is expensive….around $30,000 per year. If you have to dialyze, you have two choices as to where you get your treatment: in-center or at home. Both options are covered by Medicare, Medicaid and many private insurances.', 'Tips to know: 1 Before purchasing a home dialysis machine, ensure that it will be cost effective. 2 If the cost of your health insurance covering the use of the dialysis machine will be more expensive than purchasing one for yourself, then it may be a worthwhile investment.', 'In contrast, with home dialysis, his center has to pay a nurse $25 to $50 an hour to work with a patient one-on-one daily for at least three weeks. For patients, the energy co...	[1, 0, 0, 0, 0, ...]
  what is capoeira angola	['Capoeira Angola, an ancient martial art of African origin, is one of the many cultural weapons used to break the chains of enslavement in Brazil. Played close to the ground, Capoeira Angola combines fluid, dance-like movements with kicks, head butts, tripping sweeps and the appearance of playfulness or vulnerability.', "The root of capoeira angola is African, West Africa. I've seen many artform that I see similarities between capoeira angola and capoeira in this artform in Africa. Capoeira haguenau is developed by Master Biba in the early '30's, was not allowed to be taught until 1932.", 'What is Capoeira Angola? Capoeira is an art form that weaves together fighting, dance, music, ritual and philosophy into a unique game called the jogo de Capoeira (play of Capoeira). While there is much dispute over the exact origins of Capoeira, it is known that it was created by African slaves in Brazil.', 'The intricate movements of Capoeira Angola weave tradition, history, spirituality, and phil...	[1, 0, 0, 0, 0, ...]

• Loss: ListNetLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "mini_batch_size": 16
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• seed: 12
• bf16: True
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 12
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	NanoMSMARCO_R100_ndcg@10	NanoNFCorpus_R100_ndcg@10	NanoNQ_R100_ndcg@10	NanoBEIR_R100_mean_ndcg@10
-1	-1	-	-	0.0138 (-0.5267)	0.2714 (-0.0536)	0.0429 (-0.4577)	0.1094 (-0.3460)
0.0002	1	2.0647	-	-	-	-	-
0.0508	250	2.0899	-	-	-	-	-
0.1016	500	2.0878	2.0881	0.0271 (-0.5133)	0.2780 (-0.0471)	0.0261 (-0.4746)	0.1104 (-0.3450)
0.1525	750	2.0914	-	-	-	-	-
0.2033	1000	2.0908	2.0878	0.0545 (-0.4860)	0.2643 (-0.0607)	0.0390 (-0.4616)	0.1193 (-0.3361)
0.2541	1250	2.0919	-	-	-	-	-
0.3049	1500	2.0872	2.0874	0.0581 (-0.4823)	0.2651 (-0.0600)	0.0225 (-0.4782)	0.1152 (-0.3402)
0.3558	1750	2.0889	-	-	-	-	-
0.4066	2000	2.0902	2.0871	0.0484 (-0.4921)	0.2633 (-0.0618)	0.0471 (-0.4535)	0.1196 (-0.3358)
0.4574	2250	2.0857	-	-	-	-	-
0.5082	2500	2.0864	2.0868	0.0614 (-0.4791)	0.2701 (-0.0550)	0.0262 (-0.4745)	0.1192 (-0.3362)
0.5591	2750	2.0913	-	-	-	-	-
0.6099	3000	2.0828	2.0868	0.0599 (-0.4805)	0.2656 (-0.0595)	0.0345 (-0.4662)	0.1200 (-0.3354)
0.6607	3250	2.0857	-	-	-	-	-
0.7115	3500	2.0878	2.0866	0.0662 (-0.4742)	0.2462 (-0.0788)	0.0344 (-0.4663)	0.1156 (-0.3398)
0.7624	3750	2.0856	-	-	-	-	-
0.8132	4000	2.0837	2.0866	0.0609 (-0.4795)	0.2533 (-0.0717)	0.0343 (-0.4663)	0.1162 (-0.3392)
0.8640	4250	2.0876	-	-	-	-	-
0.9148	4500	2.0871	2.0865	0.0547 (-0.4858)	0.2663 (-0.0587)	0.0390 (-0.4617)	0.1200 (-0.3354)
0.9656	4750	2.0865	-	-	-	-	-
-1	-1	-	-	0.0547 (-0.4858)	0.2663 (-0.0587)	0.0390 (-0.4617)	0.1200 (-0.3354)

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.18
• Sentence Transformers: 5.0.0
• Transformers: 4.56.0.dev0
• PyTorch: 2.7.1+cu126
• Accelerate: 1.9.0
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

ListNetLoss

@inproceedings{cao2007learning,
    title={Learning to Rank: From Pairwise Approach to Listwise Approach},
    author={Cao, Zhe and Qin, Tao and Liu, Tie-Yan and Tsai, Ming-Feng and Li, Hang},
    booktitle={Proceedings of the 24th international conference on Machine learning},
    pages={129--136},
    year={2007}
}