hcy5561/distilroberta-base-sentence-transformer-snli

SentenceTransformer based on google-bert/bert-base-uncased

This is a sentence-transformers model finetuned from google-bert/bert-base-uncased on the stanfordnlp/snli dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google-bert/bert-base-uncased
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Training Dataset:
  • stanfordnlp/snli
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("hcy5561/distilroberta-base-sentence-transformer-snli")
# Run inference
sentences = [
    'A pilot dressed in a dark-colored sweater is sitting in the cock-pit of a plane with his hands crossed.',
    'A pilot is sitting in his plain with his hands crossed',
    'The boys are playing outside on a log.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

stanfordnlp/snli

• Dataset: stanfordnlp/snli at cdb5c3d
• Size: 494,430 training samples
• Columns: premise, hypothesis, and label
• Approximate statistics based on the first 1000 samples:

  	premise	hypothesis	label
  type	string	string	int
  details	min: 6 tokens mean: 16.24 tokens max: 50 tokens	min: 4 tokens mean: 10.55 tokens max: 26 tokens	0: ~31.10% 1: ~33.40% 2: ~35.50%

• Samples:

  premise	hypothesis	label
  Two men, one in yellow, are on a wooden boat.	Two men swimming in water	2
  Two people sleep on a couch.	Two people are asleep.	0
  a little boy is learning to swim with the help of a float board.	The boy is crawling.	2

• Loss: SoftmaxLoss

Evaluation Dataset

stanfordnlp/snli

• Dataset: stanfordnlp/snli at cdb5c3d
• Size: 27,468 evaluation samples
• Columns: premise, hypothesis, and label
• Approximate statistics based on the first 1000 samples:

  	premise	hypothesis	label
  type	string	string	int
  details	min: 6 tokens mean: 16.66 tokens max: 44 tokens	min: 4 tokens mean: 10.48 tokens max: 31 tokens	0: ~36.10% 1: ~31.80% 2: ~32.10%

• Samples:

  premise	hypothesis	label
  A taxi cab driver looks stressed out in his car.	a taxi driver is stressed	0
  Two men do trick in a park.	The men only sat on the bench in the park, doing nothing.	2
  Two woman walking, the blond is looking at the camera wearing sunglasses making an oh face.	One lady makes a shocked face at the camera as the photographer tells the women they are lost.	1

• Loss: SoftmaxLoss

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• num_train_epochs: 4
• warmup_ratio: 0.1
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	loss
0.1294	1000	0.9208	0.7448
0.2589	2000	0.7095	0.6462
0.3883	3000	0.6415	0.6199
0.5177	4000	0.6125	0.5940
0.6472	5000	0.5935	0.5672
0.7766	6000	0.5748	0.5550
0.9060	7000	0.5654	0.5506
1.0355	8000	0.5524	0.5376
1.1649	9000	0.5386	0.5319
1.2943	10000	0.5192	0.5361
1.4238	11000	0.4863	0.5304
1.5532	12000	0.4687	0.5278
1.6826	13000	0.4586	0.5305
1.8121	14000	0.4474	0.5222
1.9415	15000	0.4447	0.5237
2.0709	16000	0.434	0.5172
2.2004	17000	0.4243	0.5235
2.3298	18000	0.398	0.5224
2.4592	19000	0.3747	0.5344
2.5887	20000	0.3669	0.5301
2.7181	21000	0.3583	0.5406
2.8475	22000	0.3496	0.5354
2.9770	23000	0.3527	0.5324
3.1064	24000	0.3419	0.5299
3.2358	25000	0.3358	0.5456
3.3653	26000	0.3096	0.5562
3.4947	27000	0.2964	0.5644
3.6241	28000	0.2998	0.5565
3.7536	29000	0.2906	0.5590
3.8830	30000	0.2923	0.5564

Framework Versions

• Python: 3.10.6
• Sentence Transformers: 3.0.1
• Transformers: 4.39.3
• PyTorch: 2.2.2+cu118
• Accelerate: 0.28.0
• Datasets: 2.20.0
• Tokenizers: 0.15.2

Citation

BibTeX

Sentence Transformers and SoftmaxLoss

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}