initial commit

README.md

@@ -1,3 +1,184 @@
----
-license: mit
----
+---
+language:
+- fr
+- en
+- de
+- es
+- ru
+- it
+- zh
+- sv
+- pt 
+- pl
+- ar
+- nl
+- ca
+- vi
+- ja 
+- hu
+- he
+- id 
+- no
+- fa
+- ko
+- tr
+- fi
+- ro
+- el
+- hy
+- da
+- eu
+- ms
+- sl
+- az
+- bn
+- cy
+- hi
+- ta
+- ur
+- th
+- ka 
+- te
+- af
+- sq
+- lv
+- ml
+- kn
+- tl
+- is 
+- sw
+- jv 
+- my
+- mn
+- km
+- am
+
+
+license: apache-2.0
+datasets:
+- wikipedia
+---
+
+# Multilingual ModernBERT Base Cased 128k
+
+Pretrained multilingual language model using a masked language modeling (MLM) objective.
+
+## Model description
+
+mALBERT is a transformers model pretrained on 16Go of French Wikipedia in a self-supervised fashion. This means it
+was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
+publicly available data) with an automatic process to generate inputs and labels from those texts.
+
+This model has the following configuration:
+
+- 12 repeating layers
+- 768 embedding dimension
+- 768 hidden dimension
+- 12 attention heads
+- 11M parameters
+- 128k of vocabulary size
+
+## Intended uses & limitations
+
+You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
+be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=malbert-base-cased-128k) to look for
+fine-tuned versions on a task that interests you.
+
+Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
+to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
+generation you should look at model like GPT2.
+
+### How to use
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import AlbertTokenizer, AlbertModel
+tokenizer = AlbertTokenizer.from_pretrained('cservan/multilingual-albert-base-cased-128k')
+model = AlbertModel.from_pretrained("cservan/multilingual-albert-base-cased-128k")
+text = "Remplacez-moi par le texte en français que vous souhaitez."
+encoded_input = tokenizer(text, return_tensors='pt')
+output = model(**encoded_input)
+```
+
+and in TensorFlow:
+
+```python
+from transformers import AlbertTokenizer, TFAlbertModel
+tokenizer = AlbertTokenizer.from_pretrained('cservan/multilingual-albert-base-cased-128k')
+model = TFAlbertModel.from_pretrained("cservan/multilingual-albert-base-cased-128k")
+text = "Remplacez-moi par le texte en français que vous souhaitez."
+encoded_input = tokenizer(text, return_tensors='tf')
+output = model(encoded_input)
+```
+
+
+## Training data
+
+The mALBERT model was pretrained on 13go of [Multiligual Wikipedia](https://scouv.lisn.upsaclay.fr/#malbert) (excluding lists, tables and
+headers).
+
+## Training procedure
+
+### Preprocessing
+
+The texts are lowercased and tokenized using SentencePiece and a vocabulary size of 128,000. The inputs of the model are
+then of the form:
+
+```
+[CLS] Sentence A [SEP] Sentence B [SEP]
+```
+
+### Training
+
+The mALBERT procedure follows the BERT setup.
+
+The details of the masking procedure for each sentence are the following:
+- 15% of the tokens are masked.
+- In 80% of the cases, the masked tokens are replaced by `[MASK]`.
+- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
+- In the 10% remaining cases, the masked tokens are left as is.
+
+### Tools
+
+The tools used to pre-train the model are available [here](https://gitlab.lisn.upsaclay.fr/nlp/deep-learning/UER-py)
+
+
+## Evaluation results
+
+When fine-tuned on downstream tasks, the ALBERT models achieve the following results:
+
+Slot-filling:
+
+|Models ⧹ Tasks |  MMNLU |  MultiATIS++ |  CoNLL2003 |  MultiCoNER |  SNIPS |  MEDIA |
+|---------------|--------------|--------------|--------------|--------------|--------------|--------------|
+|EnALBERT |  N/A |  N/A |  89.67 (0.34) |  42.36 (0.22) |  95.95 (0.13) |  N/A |
+|FrALBERT |  N/A |  N/A |  N/A |  N/A |  N/A |  81.76 (0.59)
+|mALBERT-128k |  65.81 (0.11) |  89.14 (0.15) |  88.27 (0.24) |  46.01 (0.18) |  91.60 (0.31) |  83.15 (0.38) |
+|mALBERT-64k  |  65.29 (0.14) |  88.88 (0.14) |  86.44 (0.37) |  44.70 (0.27) |  90.84 (0.47) |  82.30 (0.19) |
+|mALBERT-32k  |  64.83 (0.22) |  88.60 (0.27) |  84.96 (0.41) |  44.13 (0.39) |  89.89 (0.68) |  82.04 (0.28) |
+
+Classification task:
+
+|Models ⧹ Tasks | MMNLU | MultiATIS++ | SNIPS | SST2 |
+|---------------|--------------|--------------|--------------|--------------|
+|mALBERT-128k | 72.35 (0.09) | 90.58 (0.98) | 96.84 (0.49) | 34.66 (1.46) |
+|mALBERT-64k  | 71.26 (0.11) | 90.97 (0.70) | 96.53 (0.44) | 34.64 (1.02) |
+|mALBERT-32k  | 70.76 (0.11) | 90.55 (0.98) | 96.49 (0.45) | 34.18 (1.64) |
+
+### BibTeX entry and citation info
+
+```bibtex
+@inproceedings{servan2024mALBERT,
+  author    = {Christophe Servan and
+               Sahar Ghannay and
+               Sophie Rosset},
+  booktitle = {the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
+  title     = {{mALBERT: Is a Compact Multilingual BERT Model Still Worth It?}},
+  year      = {2024},
+  address   = {Torino, Italy},
+  month     = may,
+}
+```
+
+Link to the paper: [PDF](https://hal.science/hal-04520797)

config.json

@@ -0,0 +1,45 @@
+{
+  "architectures": [
+    "ModernBertForMaskedLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 2,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "embedding_dropout": 0.0,
+  "eos_token_id": 3,
+  "global_attn_every_n_layers": 3,
+  "global_rope_theta": 160000.0,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 768,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 1152,
+  "layer_norm_eps": 1e-05,
+  "local_attention": 128,
+  "local_rope_theta": 10000.0,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 6,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "repad_logits_with_grad": false,
+  "sep_token_id": 3,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.55.0",
+  "vocab_size": 129008
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3baa69d2ecf3f1367f7f99db19ba495b0dac2a434d6b038977b507894f27a50e
+size 639940718

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}