Upload folder using huggingface_hub

README.md

@@ -1,7 +1,7 @@
 ---
 language:
-- zh
 - en
+- zh
 tags:
 - translation
 license: cc-by-4.0
@@ -21,30 +21,41 @@ model-index:
     metrics:
     - name: CHRF
       type: chrf
-      value: 34.53
+      value: 58.10
+    - name: COMET
+      type: comet
+      value: 58.10
 ---
 
 
 # `quickmt-en-zh` Neural Machine Translation Model 
 
-# Usage
+`quickmt-en-zh` is a reasonably fast and reasonably accurate neural machine translation model for translation from `en` into `zh`.
+
+
+## Model Information
+
+* Trained using [`eole`](https://github.com/eole-nlp/eole)
+* 200M parameter transformer 'big' with 8 encoder layers and 2 decoder layers
+* Separate source and target Sentencepiece tokenizers
+* Exported for fast inference to [CTranslate2](https://github.com/OpenNMT/CTranslate2) format
+* Training data: https://huggingface.co/datasets/quickmt/quickmt-train.zh-en/tree/main
+
+See the `eole` model configuration in this repository for further details. 
+
 
-## Install `quickmt`
+## Usage with `quickmt`
+
+First, install `quickmt` and download the model
 
 ```bash
 git clone https://github.com/quickmt/quickmt.git
 pip install ./quickmt/
-```
-
-## Download model
 
-```bash
 quickmt-model-download quickmt/quickmt-en-zh ./quickmt-en-zh
 ```
 
-## Use model
-
-Inference with `quickmt`:
+Next use the model in python:
 
 ```python
 from quickmt import Translator
@@ -53,48 +64,27 @@ from quickmt import Translator
 t = Translator("./quickmt-en-zh/", device="auto")
 
 # Translate - set beam size to 5 for higher quality (but slower speed)
-t(["The roe deer (Capreolus capreolus), also known as the roe, western roe deer,[3][4] or European roe,[3] is a species of deer."], beam_size=1)
+t(["The Boot Monument is an American Revolutionary War memorial located in Saratoga National Historical Park in the state of New York."], beam_size=1)
 
 # Get alternative translations by sampling
 # You can pass any cTranslate2 `translate_batch` arguments
-t(["The roe deer (Capreolus capreolus), also known as the roe, western roe deer,[3][4] or European roe,[3] is a species of deer."], sampling_temperature=1.2, beam_size=1, sampling_topk=50, sampling_topp=0.9)
+t(["The Boot Monument is an American Revolutionary War memorial located in Saratoga National Historical Park in the state of New York."], sampling_temperature=1.2, beam_size=1, sampling_topk=50, sampling_topp=0.9)
 ```
 
-The model is in `ctranslate2` format, and the tokenizers are `sentencepiece`, so you can use the model files directly if you want. It would be fairly easy to get them to work with e.g. [LibreTranslate](https://libretranslate.com/) which also uses `ctranslate2` and `sentencepiece`.
-
-# Model Information
-
-* Trained using [`eole`](https://github.com/eole-nlp/eole)
-    - Trained for 82k steps with an effective batch size of 49152, which took less than 1 day on a single RTX 4090 on [vast.ai](https://cloud.vast.ai)
-* Exported for fast inference to []CTranslate2](https://github.com/OpenNMT/CTranslate2) format
-* Training data: https://huggingface.co/datasets/quickmt/quickmt-train.zh-en/tree/main
-* Seperate source and target Sentencepiece tokenizers (size 32k)
-* Transformer "Big"
-    - 241,870,080 parameters 
-    - 8 encoder layers and 2 decoder layers
-    - Gated-silu activations
-    - Trained and saved in bfloat16 
+The model is in `ctranslate2` format, and the tokenizers are `sentencepiece`, so you can use `ctranslate2` directly instead of through `quickmt`. It is also possible  to get this model to work with e.g. [LibreTranslate](https://libretranslate.com/) which also uses `ctranslate2` and `sentencepiece`.
 
-See `eole-config.yaml` for more detail.
 
 ## Metrics
 
-CHRF2 calculated with [sacrebleu](https://github.com/mjpost/sacrebleu) on the Flores200 `devtest` test set ("eng_Latn"->"zho_Hans").
-
-"GPU Time" is the time to translate the flores-devtest corpus using a batch size of 32 on a GTX 1080 GPU. "CPU Time" is the time to translate the following input with a single CPU core:
-
-> James Joyce (2 February 1882 – 13 January 1941) was an Irish novelist, poet and literary critic who contributed to the modernist avant-garde movement and is regarded as one of the most influential and important writers of the 20th century. 
-
-| Model                             | chrf2 | comet22    | CPU Time (s) | GPU Time (s)   |
-| --------------------------------  | ----- | --------   | -------------|-------------   |
-| quickmt/quickmt-zh-en             | 34.53 | 0.8512     | 1.91         | 3.92           |  
-| Helsinki-NLP/opus-mt-zh-en        | 29.20 | 0.8236     | 1.50         | 10.10          | 
-| facebook/m2m100_418M              | 26.63 | 0.7376     | 10.2         | 49.02          | 
-| facebook/nllb-200-distilled-600M  | 24.68 | 0.7840     | 13.2         | 55.92          | 
-
-`quickmt-en-zh` is the highest quality and is the fastest on GPU (and not far behind on CPU).
-
-Helsinki-NLP/opus-mt-en-zh is one of the most downloaded machine translation models on HuggingFace, and this model is considerably more accurate *and* similar in speed.
-
+`chrf2` is calculated with [sacrebleu](https://github.com/mjpost/sacrebleu) on the [Flores200 `devtest` test set](https://huggingface.co/datasets/facebook/flores) ("eng_Latn"->"zho_Hans"). `comet22` with the [`comet`](https://github.com/Unbabel/COMET) library and the [default model](https://huggingface.co/Unbabel/wmt22-comet-da). "Time (s)" is the time in seconds to translate (using `ctranslate2`) the flores-devtest dataset (1012 sentences) on an RTX 4070s GPU with batch size 32. 
 
+| Model                            | chrf2 | comet22 | Time (s) |
+| -------------------------------- | ----- | ------- | -------- |
+| quickmt/quickmt-en-zh            | 35.22 | 85.39  |  0.96    |
+| Helsinki-NLP/opus-mt-en-zh       | 29.20 | 82.36  |  3.41    |
+| facebook/m2m100_418M             | 25.86 | 73.76  | 16.71    |
+| facebook/m2m100_1.2B             | 28.94 | 78.38  | 31.09    |
+| facebook/nllb-200-distilled-600M | 24.52 | 78.41  | 19.01    |
+| facebook/nllb-200-distilled-1.3B | 26.79 | 79.87  | 32.03    |
 
+`quickmt-en-zh` is the fastest *and* highest quality.

eole-config.yaml

@@ -1,5 +1,5 @@
 ## IO
-save_data: en_zh/data_spm
+save_data: en-zh/data_spm
 overwrite: True
 seed: 1234
 report_every: 100
@@ -18,8 +18,8 @@ n_sample: 0
 
 data:
     corpus_1:
-        path_tgt: hf://quickmt/quickmt-train-zh-en/zh
         path_src: hf://quickmt/quickmt-train-zh-en/en
+        path_tgt: hf://quickmt/quickmt-train-zh-en/zh
         path_sco: hf://quickmt/quickmt-train-zh-en/sco
     valid:
         path_src: en-zh/dev.eng
@@ -31,15 +31,15 @@ transforms_configs:
     src_subword_model: "en-zh/src.spm.model"
     tgt_subword_model: "en-zh/tgt.spm.model"
   filtertoolong:
-    src_seq_length: 512
-    tgt_seq_length: 512
+    src_seq_length: 256
+    tgt_seq_length: 256
 
 training:
     # Run configuration
     model_path: en-zh/model
     keep_checkpoint: 4
     save_checkpoint_steps: 2000
-    train_steps: 200000
+    train_steps: 100000
     valid_steps: 2000
     
     # Train on a single GPU
@@ -48,22 +48,22 @@ training:
 
     # Batching
     batch_type: "tokens"
-    batch_size: 8192
-    valid_batch_size: 8192
+    batch_size: 16384
+    valid_batch_size: 16384
     batch_size_multiple: 8
-    accum_count: [6]
+    accum_count: [8]
     accum_steps: [0]
 
     # Optimizer & Compute
-    compute_dtype: "bfloat16"
+    compute_dtype: "bf16"
     optim: "pagedadamw8bit"
-    learning_rate: 1.0
+    learning_rate: 2.0
     warmup_steps: 10000
     decay_method: "noam"
     adam_beta2: 0.998
 
     # Data loading
-    bucket_size: 262144
+    bucket_size: 128000
     num_workers: 8
     prefetch_factor: 100
     
@@ -71,7 +71,7 @@ training:
     dropout_steps: [0]
     dropout: [0.1]
     attention_dropout: [0.1]
-    max_grad_norm: 0
+    max_grad_norm: 2
     label_smoothing: 0.1
     average_decay: 0.0001
     param_init_method: xavier_uniform
@@ -83,7 +83,7 @@ model:
     share_embeddings: false
     share_decoder_embeddings: true
     add_ffnbias: true
-    mlp_activation_fn: gated-silu
+    mlp_activation_fn: gelu
     add_estimator: false
     add_qkvbias: false
     norm_eps: 1e-6

eole_model/config.json

@@ -0,0 +1,149 @@
+{
+  "valid_metrics": [
+    "BLEU"
+  ],
+  "report_every": 100,
+  "src_vocab": "zh-en-benchmark/tgt.eole.vocab",
+  "src_vocab_size": 32000,
+  "tensorboard": true,
+  "seed": 1234,
+  "transforms": [
+    "sentencepiece",
+    "filtertoolong"
+  ],
+  "share_vocab": false,
+  "tensorboard_log_dir_dated": "tensorboard/Feb-15_15-35-05",
+  "overwrite": true,
+  "tensorboard_log_dir": "tensorboard",
+  "tgt_vocab_size": 32000,
+  "tgt_vocab": "zh-en-benchmark/src.eole.vocab",
+  "save_data": "zh_en/data_spm",
+  "vocab_size_multiple": 8,
+  "n_sample": 0,
+  "training": {
+    "compute_dtype": "torch.bfloat16",
+    "learning_rate": 2.0,
+    "param_init_method": "xavier_uniform",
+    "gpu_ranks": [
+      0
+    ],
+    "normalization": "tokens",
+    "bucket_size": 128000,
+    "warmup_steps": 10000,
+    "attention_dropout": [
+      0.1
+    ],
+    "num_workers": 0,
+    "label_smoothing": 0.1,
+    "model_path": "zh-en-benchmark/model",
+    "valid_batch_size": 16384,
+    "accum_count": [
+      8
+    ],
+    "max_grad_norm": 2.0,
+    "batch_size": 16384,
+    "accum_steps": [
+      0
+    ],
+    "save_checkpoint_steps": 2000,
+    "prefetch_factor": 100,
+    "valid_steps": 2000,
+    "optim": "pagedadamw8bit",
+    "world_size": 1,
+    "decay_method": "noam",
+    "dropout": [
+      0.1
+    ],
+    "batch_type": "tokens",
+    "dropout_steps": [
+      0
+    ],
+    "batch_size_multiple": 8,
+    "keep_checkpoint": 4,
+    "adam_beta2": 0.998,
+    "average_decay": 0.0001,
+    "train_steps": 100000
+  },
+  "model": {
+    "position_encoding_type": "SinusoidalInterleaved",
+    "share_decoder_embeddings": true,
+    "mlp_activation_fn": "gelu",
+    "norm_eps": 1e-06,
+    "add_ffnbias": true,
+    "add_estimator": false,
+    "architecture": "transformer",
+    "share_embeddings": false,
+    "layer_norm": "standard",
+    "transformer_ff": 4096,
+    "add_qkvbias": false,
+    "hidden_size": 1024,
+    "heads": 16,
+    "decoder": {
+      "tgt_word_vec_size": 1024,
+      "decoder_type": "transformer",
+      "position_encoding_type": "SinusoidalInterleaved",
+      "layer_norm": "standard",
+      "transformer_ff": 4096,
+      "layers": 2,
+      "mlp_activation_fn": "gelu",
+      "n_positions": null,
+      "add_qkvbias": false,
+      "hidden_size": 1024,
+      "norm_eps": 1e-06,
+      "add_ffnbias": true,
+      "heads": 16
+    },
+    "encoder": {
+      "encoder_type": "transformer",
+      "src_word_vec_size": 1024,
+      "position_encoding_type": "SinusoidalInterleaved",
+      "layer_norm": "standard",
+      "transformer_ff": 4096,
+      "layers": 8,
+      "mlp_activation_fn": "gelu",
+      "n_positions": null,
+      "add_qkvbias": false,
+      "hidden_size": 1024,
+      "norm_eps": 1e-06,
+      "add_ffnbias": true,
+      "heads": 16
+    },
+    "embeddings": {
+      "src_word_vec_size": 1024,
+      "position_encoding_type": "SinusoidalInterleaved",
+      "word_vec_size": 1024,
+      "tgt_word_vec_size": 1024
+    }
+  },
+  "transforms_configs": {
+    "sentencepiece": {
+      "src_subword_model": "${MODEL_PATH}/tgt.spm.model",
+      "tgt_subword_model": "${MODEL_PATH}/src.spm.model"
+    },
+    "filtertoolong": {
+      "src_seq_length": 256,
+      "tgt_seq_length": 256
+    }
+  },
+  "data": {
+    "corpus_1": {
+      "path_align": null,
+      "transforms": [
+        "sentencepiece",
+        "filtertoolong"
+      ],
+      "path_sco": "hf://quickmt/quickmt-train-zh-en/sco",
+      "path_tgt": "hf://quickmt/quickmt-train-zh-en/zh",
+      "path_src": "hf://quickmt/quickmt-train-zh-en/en"
+    },
+    "valid": {
+      "path_tgt": "zh-en-benchmark/dev.zho",
+      "transforms": [
+        "sentencepiece",
+        "filtertoolong"
+      ],
+      "path_src": "zh-en-benchmark/dev.eng",
+      "path_align": null
+    }
+  }
+}

eole_model/model.00.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8048ce4ad988fd291e807de6c3bce62e2d87b7054b186a9d0e7ca829b1a2ff7b
+size 820042008

eole_model/src.spm.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23d03d562fc3f8fe57e497dac0ece4827c254675a80c103fc4bb4040638ceb67
+size 733978

eole_model/tgt.spm.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c373f1d78753313b0dbc337058bf8450e1fdd6fe662a49e0941affce44ec14c5
+size 800955

model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:346e81879b33a777f74eeac9ed1e1c17fcb7b5baa943cea1a1114adb10fd5190
-size 493941910
+oid sha256:433e7c41b63223f2d4600c740ce2a00a1bbe6dba1190a5fc10e8cc245e2ef387
+size 409972810