--- language: - multilingual license: apache-2.0 library_name: sentence-transformers tags: - echo-dsrn - sentence-transformers - intent-classification - multilingual - mteb - massive - recurrent-neural-network - rnn datasets: - AmazonScience/massive metrics: - accuracy - f1 pipeline_tag: sentence-similarity base_model: ethicalabs/Echo-DSRN-v0.1.3-Embed-Exp model-index: - name: Echo-DSRN-v0.1.3-Embed-Intent results: - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (af) config: af split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7382 - type: f1 value: 0.6650 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (am) config: am split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.6850 - type: f1 value: 0.6138 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ar) config: ar split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.6693 - type: f1 value: 0.6039 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (az) config: az split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7425 - type: f1 value: 0.6834 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (bn) config: bn split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7039 - type: f1 value: 0.6334 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (cy) config: cy split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7247 - type: f1 value: 0.6593 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (da) config: da split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7446 - type: f1 value: 0.6821 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (de) config: de split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7289 - type: f1 value: 0.6544 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (el) config: el split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7332 - type: f1 value: 0.6782 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (en) config: en split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7833 - type: f1 value: 0.7194 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (es) config: es split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7533 - type: f1 value: 0.6995 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (fa) config: fa split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7203 - type: f1 value: 0.6534 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (fi) config: fi split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7180 - type: f1 value: 0.6632 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (fr) config: fr split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7589 - type: f1 value: 0.6970 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (he) config: he split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7145 - type: f1 value: 0.6559 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (hi) config: hi split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7156 - type: f1 value: 0.6480 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (hu) config: hu split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7029 - type: f1 value: 0.6392 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (hy) config: hy split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7108 - type: f1 value: 0.6480 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (id) config: id split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7702 - type: f1 value: 0.7067 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (is) config: is split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7181 - type: f1 value: 0.6498 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (it) config: it split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7612 - type: f1 value: 0.7096 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ja) config: ja split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7529 - type: f1 value: 0.6931 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (jv) config: jv split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7291 - type: f1 value: 0.6641 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ka) config: ka split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.6521 - type: f1 value: 0.6093 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (km) config: km split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.6298 - type: f1 value: 0.5652 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (kn) config: kn split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.6987 - type: f1 value: 0.6391 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ko) config: ko split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7381 - type: f1 value: 0.6787 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (lv) config: lv split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7387 - type: f1 value: 0.6859 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ml) config: ml split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7169 - type: f1 value: 0.6545 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (mn) config: mn split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7278 - type: f1 value: 0.6731 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ms) config: ms split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7504 - type: f1 value: 0.6865 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (my) config: my split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.6875 - type: f1 value: 0.6190 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (nb) config: nb split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7485 - type: f1 value: 0.6781 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (nl) config: nl split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7455 - type: f1 value: 0.6728 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (pl) config: pl split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7334 - type: f1 value: 0.6830 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (pt) config: pt split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7564 - type: f1 value: 0.7057 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ro) config: ro split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7316 - type: f1 value: 0.6566 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ru) config: ru split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7098 - type: f1 value: 0.6470 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sl) config: sl split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7133 - type: f1 value: 0.6588 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sq) config: sq split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7454 - type: f1 value: 0.6919 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sv) config: sv split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7434 - type: f1 value: 0.6823 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sw) config: sw split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7120 - type: f1 value: 0.6580 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ta) config: ta split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.6931 - type: f1 value: 0.6297 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (te) config: te split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.6907 - type: f1 value: 0.6350 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (th) config: th split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.6911 - type: f1 value: 0.6455 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (tl) config: tl split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7428 - type: f1 value: 0.6783 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (tr) config: tr split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7422 - type: f1 value: 0.6663 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ur) config: ur split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7168 - type: f1 value: 0.6475 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (vi) config: vi split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7228 - type: f1 value: 0.6603 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (zh-CN) config: zh-CN split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7492 - type: f1 value: 0.6969 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (zh-TW) config: zh-TW split: test revision: 4672e20407010da34463acc759c162ca9734bca6 metrics: - type: accuracy value: 0.7277 - type: f1 value: 0.6713 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (af) config: af split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8009 - type: f1 value: 0.7982 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (am) config: am split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7509 - type: f1 value: 0.7393 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ar) config: ar split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7428 - type: f1 value: 0.7357 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (az) config: az split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8033 - type: f1 value: 0.8041 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (bn) config: bn split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7721 - type: f1 value: 0.7639 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (cy) config: cy split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7853 - type: f1 value: 0.7785 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (da) config: da split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8077 - type: f1 value: 0.8013 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (de) config: de split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8018 - type: f1 value: 0.7941 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (el) config: el split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7835 - type: f1 value: 0.7770 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (en) config: en split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8430 - type: f1 value: 0.8408 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (es) config: es split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8248 - type: f1 value: 0.8183 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (fa) config: fa split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7814 - type: f1 value: 0.7725 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (fi) config: fi split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7806 - type: f1 value: 0.7757 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (fr) config: fr split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8230 - type: f1 value: 0.8166 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (he) config: he split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7835 - type: f1 value: 0.7710 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (hi) config: hi split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7836 - type: f1 value: 0.7744 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (hu) config: hu split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7850 - type: f1 value: 0.7753 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (hy) config: hy split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7794 - type: f1 value: 0.7699 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (id) config: id split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8196 - type: f1 value: 0.8089 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (is) config: is split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7904 - type: f1 value: 0.7783 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (it) config: it split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8171 - type: f1 value: 0.8139 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ja) config: ja split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8128 - type: f1 value: 0.8133 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (jv) config: jv split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7978 - type: f1 value: 0.7854 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ka) config: ka split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7319 - type: f1 value: 0.7286 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (km) config: km split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7162 - type: f1 value: 0.7005 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (kn) config: kn split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7601 - type: f1 value: 0.7511 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ko) config: ko split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7959 - type: f1 value: 0.7902 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (lv) config: lv split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7992 - type: f1 value: 0.7913 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ml) config: ml split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7766 - type: f1 value: 0.7687 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (mn) config: mn split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7902 - type: f1 value: 0.7763 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ms) config: ms split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8154 - type: f1 value: 0.8058 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (my) config: my split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7730 - type: f1 value: 0.7613 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (nb) config: nb split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8150 - type: f1 value: 0.8145 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (nl) config: nl split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8109 - type: f1 value: 0.8084 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (pl) config: pl split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7947 - type: f1 value: 0.7873 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (pt) config: pt split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8019 - type: f1 value: 0.7966 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ro) config: ro split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7941 - type: f1 value: 0.7911 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ru) config: ru split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7842 - type: f1 value: 0.7767 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sl) config: sl split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7753 - type: f1 value: 0.7691 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sq) config: sq split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8038 - type: f1 value: 0.8001 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sv) config: sv split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8007 - type: f1 value: 0.7897 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sw) config: sw split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7903 - type: f1 value: 0.7784 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ta) config: ta split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7683 - type: f1 value: 0.7562 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (te) config: te split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7527 - type: f1 value: 0.7471 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (th) config: th split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7821 - type: f1 value: 0.7721 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (tl) config: tl split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8073 - type: f1 value: 0.8030 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (tr) config: tr split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8076 - type: f1 value: 0.8014 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ur) config: ur split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7754 - type: f1 value: 0.7701 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (vi) config: vi split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.7844 - type: f1 value: 0.7779 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (zh-CN) config: zh-CN split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8100 - type: f1 value: 0.8042 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (zh-TW) config: zh-TW split: test revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 metrics: - type: accuracy value: 0.8017 - type: f1 value: 0.7963 --- # Echo-DSRN-v0.1.3-Embed-Intent [![GitHub](https://img.shields.io/badge/GitHub-ethicalabs.ai-black.svg)](https://github.com/ethicalabs-ai/Echo-DSRN/) [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![Python](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/) [![Model Collection](https://img.shields.io/badge/Echo--DSRN-HuggingFace-yellow.svg)](https://huggingface.co/collections/ethicalabs/echo-dsrn) [![Hybrid Collection](https://img.shields.io/badge/Echo--Hybrid-HuggingFace-red.svg)](https://huggingface.co/collections/ethicalabs/echo-dsrn-hybrid) [![Working Paper](https://img.shields.io/badge/Working--Paper-Echo_DSRN-green.svg)](https://github.com/ethicalabs-ai/Echo-DSRN/blob/main/PAPER.md) 98M-parameter multilingual intent classification embedding model based on the Echo-DSRN architecture (Dual-State Recurrent Neural Network) ◦ Recurrent Hybrid. Fine-tuned from [`ethicalabs/Echo-DSRN-v0.1.3-Embed-Exp`](https://huggingface.co/ethicalabs/Echo-DSRN-v0.1.3-Embed-Exp) on Amazon MASSIVE across all 51 languages using `MultipleNegativesRankingLoss` (MNRL). ## Model specs | Property | Value | |---|---| | Architecture | Echo-DSRN (Recurrent Neural Network) | | Parameters | 98,266,629 (~98M) | | Layers | 8 DSRN blocks | | Hidden dim | 512 | | Attention heads | 4 | | Vocab size | 32,017 tokens | | Precision | fp32 | | Base model | [ethicalabs/Echo-DSRN-v0.1.3-Embed-Exp](https://huggingface.co/ethicalabs/Echo-DSRN-v0.1.3-Embed-Exp) | | GPU | AMD Radeon AI Pro R9700 (ROCm 7.2) | ## MTEB Results ### MassiveIntentClassification (60 intents, 51 languages) | Metric | Score | |--------|-------| | Accuracy (mean) | **72.42%** | | Accuracy (min) | 62.98% | | Accuracy (max) | 78.33% | | F1 (mean) | **66.27%** | | F1 (min) | 56.52% | | F1 (max) | 71.94% | ### MassiveScenarioClassification (17 scenarios, 51 languages) | Metric | Score | |--------|-------| | Accuracy (mean) | **79.00%** | | Accuracy (min) | 71.62% | | Accuracy (max) | 84.30% | | F1 (mean) | **78.28%** | | F1 (min) | 70.05% | | F1 (max) | 84.08% | *Evaluated via MTEB v2.12.30 logistic regression protocol on frozen embeddings. Per-language scores available in the model-index metadata.* ## Training - **Base model:** `ethicalabs/Echo-DSRN-v0.1.3-Embed-Exp` (STS-pretrained, 0.753 avg Spearman on MTEB STS) - **Dataset:** Amazon MASSIVE, all 51 locales (~1M training utterances) - **Loss:** `MultipleNegativesRankingLoss` with intent-grouped positive pairs - **Pooling:** `mean_c_all` (2048-dim recurrent slow state) - **Convergence:** Early stopping at epoch 1.2; linear accuracy gain (+2 pts/1k steps), no grokking plateau - **Random baseline:** ~1.7% (60-class 1-NN) ## Example Usage ```python from sentence_transformers import SentenceTransformer model = SentenceTransformer("ethicalabs/Echo-DSRN-v0.1.3-Embed-Intent", trust_remote_code=True, device="cpu") sentences = [ "Can I order a pizza?", "I am so hungry. what about pizza?", "I like spaghetti." ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) ``` ### Console output ``` torch.Size([3, 3]) >>> print(similarities) tensor([[1.0000, 0.9333, 0.7381], [0.9333, 1.0000, 0.8411], [0.7381, 0.8411, 1.0000]]) ``` ### Intent Vector Space Behavior Here is what the model is actually doing under the hood for each pair: 1. **`Sim(0, 1) = 0.9333` — Matching Actionable Intent** * **Sentence 0:** `"Can I order a pizza?"` * **Sentence 1:** `"I am so hungry. what about pizza?"` * **Analysis:** Despite using completely different phrasing and syntax (one is a direct question, the other is a multi-sentence conversational prompt), the model maps them to nearly the same spot in vector space. The recurrent slow state identifies the underlying action (`order_food`) and topic (`pizza`), yielding a massive 0.9333 correlation. 2. **`Sim(0, 2) = 0.7381` — Action vs. Statement Separation** * **Sentence 0:** `"Can I order a pizza?"` * **Sentence 2:** `"I like spaghetti."` * **Analysis:** Notice the significant drop down to 0.7381. Even though both sentences live in the general domain of Italian food, the model correctly separates an actionable transactional request (`"Can I order..."`) from a static statement of personal preference (`"I like..."`). This is where the fine-tuning on the MASSIVE dataset with `MultipleNegativesRankingLoss` shines: it prevents the model from relying purely on lexical topic overlap. 3. **`Sim(1, 2) = 0.8411` — Conversational Context** * **Sentence 1:** `"I am so hungry. what about pizza?"` * **Sentence 2:** `"I like spaghetti."` * **Analysis:** This pair scores higher (0.8411) than (0, 2). Because Sentence 1 expresses a state/desire (`"I am so hungry"`), its semantic profile sits naturally between an explicit ordering command and a preference statement. ## Citation ```bibtex @software{echo_dsrn_embed_intent, author = {Massimo Roberto Scamarcia}, title = {Echo-DSRN-v0.1.3-Embed-Intent: Multilingual Intent Classification Embeddings}, year = {2026}, url = {https://huggingface.co/Echo-DSRN-v0.1.3-Embed-Intent} } ``` --- ## Note on tokenizer padding The benchmark results on this card were measured with left padding (`padding_side: left`), and this model version reproduces them under that convention. A right-padded training version is planned: right padding keeps padded-batch embeddings consistent with single-request embeddings (leading pad tokens do not pollute the recurrent state), so future checkpoints will be batch-composition independent.