splendor1811/BGE-mapping-tool_epoch3

BGE-mapping-tool

This is a sentence-transformers model finetuned from splendor1811/BGE-base-banking-ONE-v0106 on the json dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: splendor1811/BGE-base-banking-ONE-v0106
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: vi
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("splendor1811/BGE-mapping-tool_epoch3")
# Run inference
sentences = [
    'VIET ANH DUONG CO., LTD',
    'CTY TNHH VIET ANH DUONG',
    'ROCK GRANITES',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.9738, -0.0072],
#         [ 0.9738,  1.0000,  0.0011],
#         [-0.0072,  0.0011,  1.0000]])

Evaluation

Metrics

Information Retrieval

• Dataset: dim_1024
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 1024
  }
  

Metric	Value
cosine_accuracy@1	0.8442
cosine_accuracy@3	0.9169
cosine_accuracy@5	0.954
cosine_accuracy@10	0.9837
cosine_precision@1	0.8442
cosine_precision@3	0.821
cosine_precision@5	0.8172
cosine_precision@10	0.7957
cosine_recall@1	0.0407
cosine_recall@3	0.1184
cosine_recall@5	0.1963
cosine_recall@10	0.3819
cosine_ndcg@10	0.8056
cosine_mrr@10	0.8894
cosine_map@100	0.8042

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 768
  }
  

Metric	Value
cosine_accuracy@1	0.8501
cosine_accuracy@3	0.9125
cosine_accuracy@5	0.9525
cosine_accuracy@10	0.9837
cosine_precision@1	0.8501
cosine_precision@3	0.82
cosine_precision@5	0.816
cosine_precision@10	0.7958
cosine_recall@1	0.0409
cosine_recall@3	0.1183
cosine_recall@5	0.1959
cosine_recall@10	0.382
cosine_ndcg@10	0.806
cosine_mrr@10	0.8915
cosine_map@100	0.8046

Training Details

Training Dataset

json

• Dataset: json
• Size: 6,066 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 4 tokens mean: 15.22 tokens max: 35 tokens	min: 5 tokens mean: 15.46 tokens max: 48 tokens

• Samples:

  anchor	positive
  ZHONGSHAN CHINHAO MOLD HARDWARE COMPANY LIMITED	ZHONGSHAN CHINHAO MOLD HARDWARE CO., LTD.
  Changshu Longte International Trade Co., Ltd	CHANGSHU LONGTE INTERNATIONAL TRADE CO., LTD
  ACT CHEMICAL AND THERAPEUTIC LABORATORIES SARL	MEDICAL SUPPLIES PHARMACEUTICALS & EQUIPMENT

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          768
      ],
      "matryoshka_weights": [
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• gradient_accumulation_steps: 8
• learning_rate: 2e-05
• num_train_epochs: 5
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: False
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 8
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	dim_1024_cosine_ndcg@10	dim_768_cosine_ndcg@10
0.1054	10	0.4225	-	-
0.2108	20	0.2415	-	-
0.3162	30	0.1252	-	-
0.4216	40	0.0765	-	-
0.5270	50	0.0573	-	-
0.6324	60	0.0345	-	-
0.7378	70	0.0448	-	-
0.8432	80	0.0355	-	-
0.9486	90	0.0996	-	-
1.0	95	-	0.7994	0.7994
1.0527	100	0.0194	-	-
1.1581	110	0.0418	-	-
1.2635	120	0.0339	-	-
1.3689	130	0.0427	-	-
1.4743	140	0.0339	-	-
1.5797	150	0.0333	-	-
1.6851	160	0.0396	-	-
1.7905	170	0.0877	-	-
1.8959	180	0.0608	-	-
2.0	190	0.0352	0.8037	0.8031
2.1054	200	0.023	-	-
2.2108	210	0.0638	-	-
2.3162	220	0.0401	-	-
2.4216	230	0.0274	-	-
2.5270	240	0.0405	-	-
2.6324	250	0.0305	-	-
2.7378	260	0.0414	-	-
2.8432	270	0.0178	-	-
2.9486	280	0.0535	-	-
3.0	285	-	0.8008	0.8012
3.0527	290	0.0629	-	-
3.1581	300	0.0283	-	-
3.2635	310	0.0567	-	-
3.3689	320	0.0167	-	-
3.4743	330	0.0349	-	-
3.5797	340	0.053	-	-
3.6851	350	0.0517	-	-
3.7905	360	0.0603	-	-
3.8959	370	0.0323	-	-
4.0	380	0.0229	0.8042	0.8055
4.1054	390	0.0476	-	-
4.2108	400	0.06	-	-
4.3162	410	0.0412	-	-
4.4216	420	0.0553	-	-
4.5270	430	0.0446	-	-
4.6324	440	0.016	-	-
4.7378	450	0.0302	-	-
4.8432	460	0.0223	-	-
4.9486	470	0.0649	-	-
5.0	475	-	0.8056	0.806

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 5.1.0
• Transformers: 4.56.1
• PyTorch: 2.6.0+cu124
• Accelerate: 1.8.1
• Datasets: 3.6.0
• Tokenizers: 0.22.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}