adamNLP/modernbert-embed-base-legal-matryoshka-2

ModernBERT Embed base Legal Matryoshka

This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("adamNLP/modernbert-embed-base-legal-matryoshka-2")
# Run inference
sentences = [
    'Oglesby, 920 F.2d at 68 (“There is no requirement that an agency search every record system.”); \nMarks v. U.S. Dep’t of Justice, 578 F.2d 261, 263 (9th Cir. 1978) (holding that “the FOIA does \nnot mandate that [an agency] comply” with a request that would require “an all-encompassing \nsearch of the records of every field office”).  Therefore, the Court will grant summary judgment',
    'What case is cited to state there is no requirement for an agency to search every record system?',
    '¿Qué no pudo persuadir la parte apelante al tribunal?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 768
  }
  

Metric	Value
cosine_accuracy@1	0.5703
cosine_accuracy@3	0.6151
cosine_accuracy@5	0.6924
cosine_accuracy@10	0.7573
cosine_precision@1	0.5703
cosine_precision@3	0.5384
cosine_precision@5	0.408
cosine_precision@10	0.2368
cosine_recall@1	0.2029
cosine_recall@3	0.5292
cosine_recall@5	0.6489
cosine_recall@10	0.7482
cosine_ndcg@10	0.6653
cosine_mrr@10	0.6131
cosine_map@100	0.6524

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 512
  }
  

Metric	Value
cosine_accuracy@1	0.5425
cosine_accuracy@3	0.592
cosine_accuracy@5	0.6677
cosine_accuracy@10	0.7419
cosine_precision@1	0.5425
cosine_precision@3	0.5167
cosine_precision@5	0.3947
cosine_precision@10	0.2303
cosine_recall@1	0.1902
cosine_recall@3	0.5044
cosine_recall@5	0.6248
cosine_recall@10	0.7285
cosine_ndcg@10	0.6409
cosine_mrr@10	0.5875
cosine_map@100	0.629

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 256
  }
  

Metric	Value
cosine_accuracy@1	0.5209
cosine_accuracy@3	0.5611
cosine_accuracy@5	0.6383
cosine_accuracy@10	0.7141
cosine_precision@1	0.5209
cosine_precision@3	0.4951
cosine_precision@5	0.3796
cosine_precision@10	0.2236
cosine_recall@1	0.1802
cosine_recall@3	0.4798
cosine_recall@5	0.5976
cosine_recall@10	0.7023
cosine_ndcg@10	0.6156
cosine_mrr@10	0.563
cosine_map@100	0.6041

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 128
  }
  

Metric	Value
cosine_accuracy@1	0.4683
cosine_accuracy@3	0.5162
cosine_accuracy@5	0.5842
cosine_accuracy@10	0.6631
cosine_precision@1	0.4683
cosine_precision@3	0.4503
cosine_precision@5	0.3462
cosine_precision@10	0.208
cosine_recall@1	0.1624
cosine_recall@3	0.4366
cosine_recall@5	0.5453
cosine_recall@10	0.6521
cosine_ndcg@10	0.5657
cosine_mrr@10	0.5122
cosine_map@100	0.5528

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 64
  }
  

Metric	Value
cosine_accuracy@1	0.3601
cosine_accuracy@3	0.3941
cosine_accuracy@5	0.4621
cosine_accuracy@10	0.527
cosine_precision@1	0.3601
cosine_precision@3	0.3462
cosine_precision@5	0.2723
cosine_precision@10	0.164
cosine_recall@1	0.1236
cosine_recall@3	0.3331
cosine_recall@5	0.4283
cosine_recall@10	0.5126
cosine_ndcg@10	0.4407
cosine_mrr@10	0.3971
cosine_map@100	0.4367

Training Details

Training Dataset

json

• Dataset: json
• Size: 5,822 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 15 tokens mean: 96.89 tokens max: 160 tokens	min: 8 tokens mean: 16.72 tokens max: 41 tokens

• Samples:

  positive	anchor
  As with the deliberative-process privilege, the submissions of the DIA and ODNI regarding the invocation of the attorney-client privilege mirror many of the same deficiencies contained in the CIA’s submissions. Beginning with the DIA, that agency’s Vaughn index repeats the exact same boilerplate language in each entry—often using identical boilerplate	Which two agencies are specifically mentioned in relation to the invocation of the attorney-client privilege?
  3. Counts One, Five and Six in No. 11-445: December 1, 2009 FOIA Requests to the CIA, DIA, and ODNI ....................................................................................................................... 8 B. 2010 FOIA Requests .......................................................................................................... 10	Which counts are associated with case number 11-445?
  Going forward, GSA may proceed with its plan to apply 41 U.S.C. § 3306(c)(3) to the Polaris Solicitations but must do so in the manner Congress intended: by issuing IDIQ contracts that will feature time-and-materials and labor-hour task orders. Based on what the parties have informed this Court about the goals and requirements of the Polaris Program, it is apparent that	Who is required to proceed in the manner Congress intended?

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: True
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
0.8791	10	90.6612	-	-	-	-	-
1.0	12	-	0.6238	0.5996	0.5730	0.5156	0.3939
1.7033	20	40.4006	-	-	-	-	-
2.0	24	-	0.6623	0.6391	0.6102	0.5505	0.4218
2.5275	30	29.9759	-	-	-	-	-
3.0	36	-	0.6645	0.6383	0.6135	0.5639	0.4366
3.3516	40	26.9547	-	-	-	-	-
3.7033	44	-	0.6653	0.6409	0.6156	0.5657	0.4407

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 4.0.2
• Transformers: 4.51.0
• PyTorch: 2.6.0+cu124
• Accelerate: 0.34.2
• Datasets: 3.5.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}