mrs83's picture
docs: note that a right-padded training version is planned
297700e verified
|
Raw History Blame Contribute Delete
39 kB
metadata
language:
  - multilingual
license: apache-2.0
library_name: sentence-transformers
tags:
  - echo-dsrn
  - sentence-transformers
  - intent-classification
  - multilingual
  - mteb
  - massive
  - recurrent-neural-network
  - rnn
datasets:
  - AmazonScience/massive
metrics:
  - accuracy
  - f1
pipeline_tag: sentence-similarity
base_model: ethicalabs/Echo-DSRN-v0.1.3-Embed-Exp
model-index:
  - name: Echo-DSRN-v0.1.3-Embed-Intent
    results:
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (af)
          config: af
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7382
          - type: f1
            value: 0.665
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (am)
          config: am
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.685
          - type: f1
            value: 0.6138
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (ar)
          config: ar
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.6693
          - type: f1
            value: 0.6039
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (az)
          config: az
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7425
          - type: f1
            value: 0.6834
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (bn)
          config: bn
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7039
          - type: f1
            value: 0.6334
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (cy)
          config: cy
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7247
          - type: f1
            value: 0.6593
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (da)
          config: da
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7446
          - type: f1
            value: 0.6821
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (de)
          config: de
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7289
          - type: f1
            value: 0.6544
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (el)
          config: el
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7332
          - type: f1
            value: 0.6782
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (en)
          config: en
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7833
          - type: f1
            value: 0.7194
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (es)
          config: es
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7533
          - type: f1
            value: 0.6995
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (fa)
          config: fa
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7203
          - type: f1
            value: 0.6534
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (fi)
          config: fi
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.718
          - type: f1
            value: 0.6632
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (fr)
          config: fr
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7589
          - type: f1
            value: 0.697
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (he)
          config: he
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7145
          - type: f1
            value: 0.6559
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (hi)
          config: hi
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7156
          - type: f1
            value: 0.648
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (hu)
          config: hu
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7029
          - type: f1
            value: 0.6392
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (hy)
          config: hy
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7108
          - type: f1
            value: 0.648
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (id)
          config: id
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7702
          - type: f1
            value: 0.7067
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (is)
          config: is
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7181
          - type: f1
            value: 0.6498
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (it)
          config: it
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7612
          - type: f1
            value: 0.7096
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (ja)
          config: ja
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7529
          - type: f1
            value: 0.6931
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (jv)
          config: jv
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7291
          - type: f1
            value: 0.6641
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (ka)
          config: ka
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.6521
          - type: f1
            value: 0.6093
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (km)
          config: km
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.6298
          - type: f1
            value: 0.5652
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (kn)
          config: kn
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.6987
          - type: f1
            value: 0.6391
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (ko)
          config: ko
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7381
          - type: f1
            value: 0.6787
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (lv)
          config: lv
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7387
          - type: f1
            value: 0.6859
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (ml)
          config: ml
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7169
          - type: f1
            value: 0.6545
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (mn)
          config: mn
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7278
          - type: f1
            value: 0.6731
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (ms)
          config: ms
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7504
          - type: f1
            value: 0.6865
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (my)
          config: my
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.6875
          - type: f1
            value: 0.619
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (nb)
          config: nb
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7485
          - type: f1
            value: 0.6781
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (nl)
          config: nl
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7455
          - type: f1
            value: 0.6728
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (pl)
          config: pl
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7334
          - type: f1
            value: 0.683
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (pt)
          config: pt
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7564
          - type: f1
            value: 0.7057
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (ro)
          config: ro
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7316
          - type: f1
            value: 0.6566
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (ru)
          config: ru
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7098
          - type: f1
            value: 0.647
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (sl)
          config: sl
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7133
          - type: f1
            value: 0.6588
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (sq)
          config: sq
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7454
          - type: f1
            value: 0.6919
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (sv)
          config: sv
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7434
          - type: f1
            value: 0.6823
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (sw)
          config: sw
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.712
          - type: f1
            value: 0.658
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (ta)
          config: ta
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.6931
          - type: f1
            value: 0.6297
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (te)
          config: te
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.6907
          - type: f1
            value: 0.635
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (th)
          config: th
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.6911
          - type: f1
            value: 0.6455
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (tl)
          config: tl
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7428
          - type: f1
            value: 0.6783
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (tr)
          config: tr
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7422
          - type: f1
            value: 0.6663
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (ur)
          config: ur
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7168
          - type: f1
            value: 0.6475
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (vi)
          config: vi
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7228
          - type: f1
            value: 0.6603
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (zh-CN)
          config: zh-CN
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7492
          - type: f1
            value: 0.6969
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (zh-TW)
          config: zh-TW
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 0.7277
          - type: f1
            value: 0.6713
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (af)
          config: af
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.8009
          - type: f1
            value: 0.7982
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (am)
          config: am
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7509
          - type: f1
            value: 0.7393
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (ar)
          config: ar
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7428
          - type: f1
            value: 0.7357
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (az)
          config: az
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.8033
          - type: f1
            value: 0.8041
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (bn)
          config: bn
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7721
          - type: f1
            value: 0.7639
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (cy)
          config: cy
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7853
          - type: f1
            value: 0.7785
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (da)
          config: da
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.8077
          - type: f1
            value: 0.8013
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (de)
          config: de
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.8018
          - type: f1
            value: 0.7941
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (el)
          config: el
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7835
          - type: f1
            value: 0.777
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (en)
          config: en
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.843
          - type: f1
            value: 0.8408
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (es)
          config: es
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.8248
          - type: f1
            value: 0.8183
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (fa)
          config: fa
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7814
          - type: f1
            value: 0.7725
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (fi)
          config: fi
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7806
          - type: f1
            value: 0.7757
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (fr)
          config: fr
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.823
          - type: f1
            value: 0.8166
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (he)
          config: he
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7835
          - type: f1
            value: 0.771
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (hi)
          config: hi
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7836
          - type: f1
            value: 0.7744
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (hu)
          config: hu
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.785
          - type: f1
            value: 0.7753
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (hy)
          config: hy
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7794
          - type: f1
            value: 0.7699
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (id)
          config: id
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.8196
          - type: f1
            value: 0.8089
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (is)
          config: is
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7904
          - type: f1
            value: 0.7783
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (it)
          config: it
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.8171
          - type: f1
            value: 0.8139
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (ja)
          config: ja
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.8128
          - type: f1
            value: 0.8133
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (jv)
          config: jv
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7978
          - type: f1
            value: 0.7854
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (ka)
          config: ka
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7319
          - type: f1
            value: 0.7286
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (km)
          config: km
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7162
          - type: f1
            value: 0.7005
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (kn)
          config: kn
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7601
          - type: f1
            value: 0.7511
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (ko)
          config: ko
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7959
          - type: f1
            value: 0.7902
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (lv)
          config: lv
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7992
          - type: f1
            value: 0.7913
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (ml)
          config: ml
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7766
          - type: f1
            value: 0.7687
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (mn)
          config: mn
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7902
          - type: f1
            value: 0.7763
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (ms)
          config: ms
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.8154
          - type: f1
            value: 0.8058
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (my)
          config: my
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.773
          - type: f1
            value: 0.7613
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (nb)
          config: nb
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.815
          - type: f1
            value: 0.8145
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (nl)
          config: nl
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.8109
          - type: f1
            value: 0.8084
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (pl)
          config: pl
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7947
          - type: f1
            value: 0.7873
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (pt)
          config: pt
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.8019
          - type: f1
            value: 0.7966
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (ro)
          config: ro
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7941
          - type: f1
            value: 0.7911
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (ru)
          config: ru
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7842
          - type: f1
            value: 0.7767
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (sl)
          config: sl
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7753
          - type: f1
            value: 0.7691
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (sq)
          config: sq
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.8038
          - type: f1
            value: 0.8001
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (sv)
          config: sv
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.8007
          - type: f1
            value: 0.7897
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (sw)
          config: sw
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7903
          - type: f1
            value: 0.7784
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (ta)
          config: ta
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7683
          - type: f1
            value: 0.7562
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (te)
          config: te
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7527
          - type: f1
            value: 0.7471
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (th)
          config: th
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7821
          - type: f1
            value: 0.7721
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (tl)
          config: tl
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.8073
          - type: f1
            value: 0.803
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (tr)
          config: tr
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.8076
          - type: f1
            value: 0.8014
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (ur)
          config: ur
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7754
          - type: f1
            value: 0.7701
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (vi)
          config: vi
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.7844
          - type: f1
            value: 0.7779
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (zh-CN)
          config: zh-CN
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.81
          - type: f1
            value: 0.8042
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (zh-TW)
          config: zh-TW
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 0.8017
          - type: f1
            value: 0.7963

Echo-DSRN-v0.1.3-Embed-Intent

GitHub License Python Model Collection Hybrid Collection Working Paper

98M-parameter multilingual intent classification embedding model based on the Echo-DSRN architecture (Dual-State Recurrent Neural Network) ◦ Recurrent Hybrid.

Fine-tuned from ethicalabs/Echo-DSRN-v0.1.3-Embed-Exp on Amazon MASSIVE across all 51 languages using MultipleNegativesRankingLoss (MNRL).

Model specs

Property Value
Architecture Echo-DSRN (Recurrent Neural Network)
Parameters 98,266,629 (~98M)
Layers 8 DSRN blocks
Hidden dim 512
Attention heads 4
Vocab size 32,017 tokens
Precision fp32
Base model ethicalabs/Echo-DSRN-v0.1.3-Embed-Exp
GPU AMD Radeon AI Pro R9700 (ROCm 7.2)

MTEB Results

MassiveIntentClassification (60 intents, 51 languages)

Metric Score
Accuracy (mean) 72.42%
Accuracy (min) 62.98%
Accuracy (max) 78.33%
F1 (mean) 66.27%
F1 (min) 56.52%
F1 (max) 71.94%

MassiveScenarioClassification (17 scenarios, 51 languages)

Metric Score
Accuracy (mean) 79.00%
Accuracy (min) 71.62%
Accuracy (max) 84.30%
F1 (mean) 78.28%
F1 (min) 70.05%
F1 (max) 84.08%

Evaluated via MTEB v2.12.30 logistic regression protocol on frozen embeddings. Per-language scores available in the model-index metadata.

Training

  • Base model: ethicalabs/Echo-DSRN-v0.1.3-Embed-Exp (STS-pretrained, 0.753 avg Spearman on MTEB STS)
  • Dataset: Amazon MASSIVE, all 51 locales (~1M training utterances)
  • Loss: MultipleNegativesRankingLoss with intent-grouped positive pairs
  • Pooling: mean_c_all (2048-dim recurrent slow state)
  • Convergence: Early stopping at epoch 1.2; linear accuracy gain (+2 pts/1k steps), no grokking plateau
  • Random baseline: ~1.7% (60-class 1-NN)

Example Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("ethicalabs/Echo-DSRN-v0.1.3-Embed-Intent", trust_remote_code=True, device="cpu")

sentences = [
  "Can I order a pizza?",
  "I am so hungry. what about pizza?",
  "I like spaghetti."
]
embeddings = model.encode(sentences)

similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)

Console output

torch.Size([3, 3])
>>> print(similarities)
tensor([[1.0000, 0.9333, 0.7381],
        [0.9333, 1.0000, 0.8411],
        [0.7381, 0.8411, 1.0000]])

Intent Vector Space Behavior

Here is what the model is actually doing under the hood for each pair:

  1. Sim(0, 1) = 0.9333 — Matching Actionable Intent

    • Sentence 0: "Can I order a pizza?"
    • Sentence 1: "I am so hungry. what about pizza?"
    • Analysis: Despite using completely different phrasing and syntax (one is a direct question, the other is a multi-sentence conversational prompt), the model maps them to nearly the same spot in vector space. The recurrent slow state identifies the underlying action (order_food) and topic (pizza), yielding a massive 0.9333 correlation.
  2. Sim(0, 2) = 0.7381 — Action vs. Statement Separation

    • Sentence 0: "Can I order a pizza?"
    • Sentence 2: "I like spaghetti."
    • Analysis: Notice the significant drop down to 0.7381. Even though both sentences live in the general domain of Italian food, the model correctly separates an actionable transactional request ("Can I order...") from a static statement of personal preference ("I like..."). This is where the fine-tuning on the MASSIVE dataset with MultipleNegativesRankingLoss shines: it prevents the model from relying purely on lexical topic overlap.
  3. Sim(1, 2) = 0.8411 — Conversational Context

    • Sentence 1: "I am so hungry. what about pizza?"
    • Sentence 2: "I like spaghetti."
    • Analysis: This pair scores higher (0.8411) than (0, 2). Because Sentence 1 expresses a state/desire ("I am so hungry"), its semantic profile sits naturally between an explicit ordering command and a preference statement.

Citation

@software{echo_dsrn_embed_intent,
  author = {Massimo Roberto Scamarcia},
  title = {Echo-DSRN-v0.1.3-Embed-Intent: Multilingual Intent Classification Embeddings},
  year = {2026},
  url = {https://huggingface.co/Echo-DSRN-v0.1.3-Embed-Intent}
}

Note on tokenizer padding

The benchmark results on this card were measured with left padding (padding_side: left), and this model version reproduces them under that convention. A right-padded training version is planned: right padding keeps padded-batch embeddings consistent with single-request embeddings (leading pad tokens do not pollute the recurrent state), so future checkpoints will be batch-composition independent.