Upload folder using huggingface_hub

README.md

@@ -0,0 +1,158 @@
+---
+task: token-classification
+tags:
+- biomedical
+- bionlp
+license: mit
+base_model: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext
+---
+
+# bioner_medmentions_st21pv_finegrain
+
+This is a named entity recognition model fine-tuned from the [microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext](https://huggingface.co/microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext) model. It predicts spans with 91 possible labels. The labels are **Acquired Abnormality, Amino Acid Sequence, Amino Acid, Peptide, or Protein, Amphibian, Anatomical Abnormality, Anatomical Structure, Animal, Antibiotic, Bacterium, Biologic Function, Biologically Active Substance, Biomedical Occupation or Discipline, Biomedical or Dental Material, Bird, Body Location or Region, Body Part, Organ, or Organ Component, Body Space or Junction, Body Substance, Body System, Cell, Cell Component, Cell Function, Cell or Molecular Dysfunction, Chemical, Chemical Viewed Functionally, Chemical Viewed Structurally, Classification, Clinical Attribute, Congenital Abnormality, Diagnostic Procedure, Disease or Syndrome, Drug Delivery Device, Element, Ion, or Isotope, Embryonic Structure, Enzyme, Eukaryote, Experimental Model of Disease, Finding, Fish, Food, Fully Formed Anatomical Structure, Fungus, Gene or Genome, Genetic Function, Geographic Area, Hazardous or Poisonous Substance, Health Care Activity, Health Care Related Organization, Hormone, Human, Idea or Concept, Immunologic Factor, Indicator, Reagent, or Diagnostic Aid, Injury or Poisoning, Inorganic Chemical, Intellectual Product, Laboratory Procedure, Laboratory or Test Result, Mammal, Medical Device, Mental Process, Mental or Behavioral Dysfunction, Molecular Biology Research Technique, Molecular Function, Molecular Sequence, Neoplastic Process, Nucleic Acid, Nucleoside, or Nucleotide, Nucleotide Sequence, Organ or Tissue Function, Organic Chemical, Organism Function, Organization, Pathologic Function, Pharmacologic Substance, Physiologic Function, Plant, Population Group, Professional Society, Professional or Occupational Group, Receptor, Regulation or Law, Reptile, Research Activity, Self-help or Relief Organization, Sign or Symptom, Spatial Concept, Therapeutic or Preventive Procedure, Tissue, Vertebrate, Virus and Vitamin**.
+
+The code used for training this model can be found at https://github.com/Glasgow-AI4BioMed/bioner along with links to other biomedical NER models trained on well-known biomedical corpora. The source dataset information is below.
+
+## Example Usage
+
+The code below will load up the model and apply it to the provided text. It uses a simple aggregation strategy to post-process the individual tokens into larger multi-token entities where needed.
+
+```python
+from transformers import pipeline
+
+# Load the model as part of an NER pipeline
+ner_pipeline = pipeline("token-classification", 
+                        model="Glasgow-AI4BioMed/bioner_medmentions_st21pv_finegrain",
+                        aggregation_strategy="max")
+
+# Apply it to some text
+ner_pipeline("EGFR T790M mutations have been known to affect treatment outcomes for NSCLC patients receiving erlotinib.")
+
+# Output:
+# [ {"entity_group": "Gene or Genome", "score": 0.96229, "word": "egfr", "start": 0, "end": 4},
+#   {"entity_group": "Genetic Function", "score": 0.91988, "word": "t790m mutations", "start": 5, "end": 20},
+#   {"entity_group": "Neoplastic Process", "score": 0.99883, "word": "nsclc", "start": 51, "end": 56},
+#   {"entity_group": "Pharmacologic Substance", "score": 0.99931, "word": "erlotinib", "start": 76, "end": 85} ]
+```
+
+## Dataset Info
+
+**Source:** The ST21pv version of MedMentions was downloaded from: https://github.com/chanzuckerberg/MedMentions/tree/master/st21pv
+
+The dataset should be cited with: Mohan, Sunil, and Donghui Li. "MedMentions: A Large Biomedical Corpus Annotated with UMLS Concepts." Automated Knowledge Base Construction (AKBC), 2019, https://openreview.net/forum?id=SylxCx5pTQ. DOI: [10.24432/C5G59C](https://doi.org/10.24432/C5G59C)
+
+An overview of semantic types can be found at: https://www.nlm.nih.gov/research/umls/META3_current_semantic_types.html
+
+**Preprocessing:** The training, validation and test splits were maintained from the original dataset. Concept identifiers (CUIs) were used to map each annotation to its associated UMLS entry to recover semantic types (from the MRSTY.RRF UMLS file). Semantic types provided in MedMentions were not used. Annotations were mapped to specific *semantic types* names using the Semantic Groups file available at: https://www.nlm.nih.gov/research/umls/knowledge_sources/semantic_network/index.html. This contrasts with the finegrained version that mapped annotations to *semantic groups*. The preprocessing script for this dataset is [prepare_medmentions.py](https://github.com/Glasgow-AI4BioMed/bioner/blob/main/prepare_medmentions.py.py) with the --finegrain flag.
+
+## Performance
+
+The span-level performance on the test split for the different labels are shown in the tables below. The full performance results are available in the model repo in Markdown format for viewing and JSON format for easier loading. These include the performance at token level (with individual B- and I- labels as the token classifier uses IOB2 token labelling).
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| Acquired Abnormality | 0.273 | 0.240 | 0.255 | 50 |
+| Amino Acid Sequence | 0.303 | 0.357 | 0.328 | 84 |
+| Amino Acid, Peptide, or Protein | 0.258 | 0.277 | 0.267 | 166 |
+| Anatomical Abnormality | 0.190 | 0.145 | 0.164 | 76 |
+| Anatomical Structure | 0.160 | 0.308 | 0.211 | 13 |
+| Animal | 0.605 | 0.742 | 0.667 | 93 |
+| Antibiotic | 0.835 | 0.784 | 0.808 | 148 |
+| Bacterium | 0.752 | 0.732 | 0.742 | 448 |
+| Biologic Function | 0.341 | 0.369 | 0.355 | 157 |
+| Biologically Active Substance | 0.586 | 0.640 | 0.612 | 2080 |
+| Biomedical Occupation or Discipline | 0.437 | 0.444 | 0.441 | 196 |
+| Biomedical or Dental Material | 0.369 | 0.452 | 0.406 | 197 |
+| Bird | 0.810 | 0.819 | 0.814 | 83 |
+| Body Location or Region | 0.386 | 0.461 | 0.420 | 232 |
+| Body Part, Organ, or Organ Component | 0.565 | 0.604 | 0.584 | 1092 |
+| Body Space or Junction | 0.272 | 0.311 | 0.290 | 90 |
+| Body Substance | 0.542 | 0.731 | 0.622 | 212 |
+| Body System | 0.554 | 0.511 | 0.532 | 90 |
+| Cell | 0.680 | 0.737 | 0.708 | 924 |
+| Cell Component | 0.589 | 0.637 | 0.612 | 311 |
+| Cell Function | 0.484 | 0.599 | 0.535 | 499 |
+| Cell or Molecular Dysfunction | 0.607 | 0.657 | 0.631 | 99 |
+| Chemical | 0.348 | 0.333 | 0.340 | 72 |
+| Chemical Viewed Functionally | 0.286 | 0.432 | 0.344 | 37 |
+| Chemical Viewed Structurally | 0.443 | 0.427 | 0.435 | 82 |
+| Classification | 0.520 | 0.544 | 0.532 | 309 |
+| Clinical Attribute | 0.596 | 0.625 | 0.610 | 323 |
+| Congenital Abnormality | 0.438 | 0.443 | 0.440 | 79 |
+| Diagnostic Procedure | 0.672 | 0.648 | 0.660 | 735 |
+| Disease or Syndrome | 0.757 | 0.774 | 0.766 | 2199 |
+| Element, Ion, or Isotope | 0.713 | 0.657 | 0.684 | 385 |
+| Embryonic Structure | 0.587 | 0.509 | 0.545 | 53 |
+| Enzyme | 0.766 | 0.761 | 0.763 | 681 |
+| Eukaryote | 0.745 | 0.793 | 0.768 | 397 |
+| Experimental Model of Disease | 0.286 | 0.356 | 0.317 | 45 |
+| Finding | 0.391 | 0.388 | 0.389 | 2759 |
+| Fish | 1.000 | 0.947 | 0.973 | 19 |
+| Food | 0.556 | 0.455 | 0.501 | 336 |
+| Fully Formed Anatomical Structure | 0.000 | 0.000 | 0.000 | 1 |
+| Fungus | 0.819 | 0.798 | 0.809 | 119 |
+| Gene or Genome | 0.551 | 0.539 | 0.545 | 912 |
+| Genetic Function | 0.598 | 0.646 | 0.621 | 652 |
+| Geographic Area | 0.673 | 0.712 | 0.692 | 598 |
+| Hazardous or Poisonous Substance | 0.513 | 0.522 | 0.518 | 293 |
+| Health Care Activity | 0.487 | 0.458 | 0.472 | 1061 |
+| Health Care Related Organization | 0.531 | 0.642 | 0.581 | 296 |
+| Hormone | 0.806 | 0.746 | 0.775 | 189 |
+| Human | 0.799 | 0.880 | 0.837 | 158 |
+| Idea or Concept | 0.000 | 0.000 | 0.000 | 1 |
+| Immunologic Factor | 0.674 | 0.606 | 0.638 | 434 |
+| Indicator, Reagent, or Diagnostic Aid | 0.427 | 0.451 | 0.439 | 182 |
+| Injury or Poisoning | 0.617 | 0.703 | 0.657 | 357 |
+| Inorganic Chemical | 0.611 | 0.680 | 0.643 | 256 |
+| Intellectual Product | 0.495 | 0.485 | 0.490 | 2075 |
+| Laboratory Procedure | 0.445 | 0.452 | 0.448 | 908 |
+| Laboratory or Test Result | 0.183 | 0.196 | 0.190 | 112 |
+| Mammal | 0.778 | 0.838 | 0.807 | 456 |
+| Medical Device | 0.434 | 0.437 | 0.435 | 355 |
+| Mental Process | 0.546 | 0.546 | 0.546 | 740 |
+| Mental or Behavioral Dysfunction | 0.710 | 0.774 | 0.741 | 518 |
+| Molecular Biology Research Technique | 0.500 | 0.539 | 0.519 | 206 |
+| Molecular Function | 0.504 | 0.555 | 0.528 | 719 |
+| Molecular Sequence | 0.417 | 0.556 | 0.476 | 9 |
+| Neoplastic Process | 0.761 | 0.745 | 0.753 | 918 |
+| Nucleic Acid, Nucleoside, or Nucleotide | 0.331 | 0.450 | 0.381 | 109 |
+| Nucleotide Sequence | 0.320 | 0.491 | 0.387 | 110 |
+| Organ or Tissue Function | 0.482 | 0.425 | 0.452 | 247 |
+| Organic Chemical | 0.396 | 0.464 | 0.427 | 511 |
+| Organism Function | 0.462 | 0.518 | 0.488 | 471 |
+| Organization | 0.270 | 0.442 | 0.335 | 77 |
+| Pathologic Function | 0.541 | 0.541 | 0.541 | 669 |
+| Pharmacologic Substance | 0.577 | 0.623 | 0.599 | 1258 |
+| Physiologic Function | 0.283 | 0.286 | 0.284 | 182 |
+| Plant | 0.603 | 0.618 | 0.610 | 403 |
+| Population Group | 0.710 | 0.711 | 0.711 | 1263 |
+| Professional Society | 0.000 | 0.000 | 0.000 | 7 |
+| Professional or Occupational Group | 0.599 | 0.725 | 0.656 | 360 |
+| Receptor | 0.614 | 0.686 | 0.648 | 271 |
+| Regulation or Law | 0.182 | 0.125 | 0.148 | 16 |
+| Reptile | 1.000 | 0.318 | 0.483 | 22 |
+| Research Activity | 0.559 | 0.540 | 0.549 | 1653 |
+| Self-help or Relief Organization | 0.000 | 0.000 | 0.000 | 2 |
+| Sign or Symptom | 0.629 | 0.647 | 0.638 | 340 |
+| Spatial Concept | 0.474 | 0.483 | 0.479 | 1282 |
+| Therapeutic or Preventive Procedure | 0.609 | 0.617 | 0.613 | 2036 |
+| Tissue | 0.562 | 0.525 | 0.543 | 259 |
+| Vertebrate | 0.000 | 0.000 | 0.000 | 1 |
+| Virus | 0.678 | 0.831 | 0.747 | 172 |
+| Vitamin | 0.706 | 0.511 | 0.593 | 47 |
+| macro avg | 0.507 | 0.525 | 0.512 | 40144 |
+| weighted avg | 0.571 | 0.589 | 0.578 | 40144 |
+
+
+## Hyperparameters
+
+Hyperparameter tuning was done with [optuna](https://optuna.org/) and the [hyperparameter_search](https://huggingface.co/docs/transformers/en/hpo_train) functionality. 100 trials were run. Early stopping was applied during training. The best performing model was selected using the macro F1 performance on the validation set. The selected hyperparameters are in the table below.
+
+| Hyperparameter | Value |
+|----------------|-------|
+| epochs | 25.0 |
+| learning_rate | 7.86794379743531e-05 |
+| per_device_train_batch_size | 16 |
+| weight_decay | 0.06816454557507429 |
+| warmup_ratio | 0.07903396276412193 |
+

best_hyperparameters.json

@@ -0,0 +1,7 @@
+{
+  "epochs": 25.0,
+  "learning_rate": 7.86794379743531e-05,
+  "per_device_train_batch_size": 16,
+  "weight_decay": 0.06816454557507429,
+  "warmup_ratio": 0.07903396276412193
+}

config.json

@@ -0,0 +1,210 @@
+{
+  "_name_or_path": "microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext",
+  "architectures": [
+    "BertForTokenClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "O",
+    "1": "B-Acquired Abnormality",
+    "2": "I-Acquired Abnormality",
+    "3": "B-Amino Acid Sequence",
+    "4": "I-Amino Acid Sequence",
+    "5": "B-Amino Acid, Peptide, or Protein",
+    "6": "I-Amino Acid, Peptide, or Protein",
+    "7": "B-Amphibian",
+    "8": "I-Amphibian",
+    "9": "B-Anatomical Abnormality",
+    "10": "I-Anatomical Abnormality",
+    "11": "B-Anatomical Structure",
+    "12": "I-Anatomical Structure",
+    "13": "B-Animal",
+    "14": "I-Animal",
+    "15": "B-Antibiotic",
+    "16": "I-Antibiotic",
+    "17": "B-Bacterium",
+    "18": "I-Bacterium",
+    "19": "B-Biologic Function",
+    "20": "I-Biologic Function",
+    "21": "B-Biologically Active Substance",
+    "22": "I-Biologically Active Substance",
+    "23": "B-Biomedical Occupation or Discipline",
+    "24": "I-Biomedical Occupation or Discipline",
+    "25": "B-Biomedical or Dental Material",
+    "26": "I-Biomedical or Dental Material",
+    "27": "B-Bird",
+    "28": "I-Bird",
+    "29": "B-Body Location or Region",
+    "30": "I-Body Location or Region",
+    "31": "B-Body Part, Organ, or Organ Component",
+    "32": "I-Body Part, Organ, or Organ Component",
+    "33": "B-Body Space or Junction",
+    "34": "I-Body Space or Junction",
+    "35": "B-Body Substance",
+    "36": "I-Body Substance",
+    "37": "B-Body System",
+    "38": "I-Body System",
+    "39": "B-Cell",
+    "40": "I-Cell",
+    "41": "B-Cell Component",
+    "42": "I-Cell Component",
+    "43": "B-Cell Function",
+    "44": "I-Cell Function",
+    "45": "B-Cell or Molecular Dysfunction",
+    "46": "I-Cell or Molecular Dysfunction",
+    "47": "B-Chemical",
+    "48": "I-Chemical",
+    "49": "B-Chemical Viewed Functionally",
+    "50": "I-Chemical Viewed Functionally",
+    "51": "B-Chemical Viewed Structurally",
+    "52": "I-Chemical Viewed Structurally",
+    "53": "B-Classification",
+    "54": "I-Classification",
+    "55": "B-Clinical Attribute",
+    "56": "I-Clinical Attribute",
+    "57": "B-Congenital Abnormality",
+    "58": "I-Congenital Abnormality",
+    "59": "B-Diagnostic Procedure",
+    "60": "I-Diagnostic Procedure",
+    "61": "B-Disease or Syndrome",
+    "62": "I-Disease or Syndrome",
+    "63": "B-Drug Delivery Device",
+    "64": "I-Drug Delivery Device",
+    "65": "B-Element, Ion, or Isotope",
+    "66": "I-Element, Ion, or Isotope",
+    "67": "B-Embryonic Structure",
+    "68": "I-Embryonic Structure",
+    "69": "B-Enzyme",
+    "70": "I-Enzyme",
+    "71": "B-Eukaryote",
+    "72": "I-Eukaryote",
+    "73": "B-Experimental Model of Disease",
+    "74": "I-Experimental Model of Disease",
+    "75": "B-Finding",
+    "76": "I-Finding",
+    "77": "B-Fish",
+    "78": "I-Fish",
+    "79": "B-Food",
+    "80": "I-Food",
+    "81": "B-Fully Formed Anatomical Structure",
+    "82": "I-Fully Formed Anatomical Structure",
+    "83": "B-Fungus",
+    "84": "I-Fungus",
+    "85": "B-Gene or Genome",
+    "86": "I-Gene or Genome",
+    "87": "B-Genetic Function",
+    "88": "I-Genetic Function",
+    "89": "B-Geographic Area",
+    "90": "I-Geographic Area",
+    "91": "B-Hazardous or Poisonous Substance",
+    "92": "I-Hazardous or Poisonous Substance",
+    "93": "B-Health Care Activity",
+    "94": "I-Health Care Activity",
+    "95": "B-Health Care Related Organization",
+    "96": "I-Health Care Related Organization",
+    "97": "B-Hormone",
+    "98": "I-Hormone",
+    "99": "B-Human",
+    "100": "I-Human",
+    "101": "B-Idea or Concept",
+    "102": "I-Idea or Concept",
+    "103": "B-Immunologic Factor",
+    "104": "I-Immunologic Factor",
+    "105": "B-Indicator, Reagent, or Diagnostic Aid",
+    "106": "I-Indicator, Reagent, or Diagnostic Aid",
+    "107": "B-Injury or Poisoning",
+    "108": "I-Injury or Poisoning",
+    "109": "B-Inorganic Chemical",
+    "110": "I-Inorganic Chemical",
+    "111": "B-Intellectual Product",
+    "112": "I-Intellectual Product",
+    "113": "B-Laboratory Procedure",
+    "114": "I-Laboratory Procedure",
+    "115": "B-Laboratory or Test Result",
+    "116": "I-Laboratory or Test Result",
+    "117": "B-Mammal",
+    "118": "I-Mammal",
+    "119": "B-Medical Device",
+    "120": "I-Medical Device",
+    "121": "B-Mental Process",
+    "122": "I-Mental Process",
+    "123": "B-Mental or Behavioral Dysfunction",
+    "124": "I-Mental or Behavioral Dysfunction",
+    "125": "B-Molecular Biology Research Technique",
+    "126": "I-Molecular Biology Research Technique",
+    "127": "B-Molecular Function",
+    "128": "I-Molecular Function",
+    "129": "B-Molecular Sequence",
+    "130": "I-Molecular Sequence",
+    "131": "B-Neoplastic Process",
+    "132": "I-Neoplastic Process",
+    "133": "B-Nucleic Acid, Nucleoside, or Nucleotide",
+    "134": "I-Nucleic Acid, Nucleoside, or Nucleotide",
+    "135": "B-Nucleotide Sequence",
+    "136": "I-Nucleotide Sequence",
+    "137": "B-Organ or Tissue Function",
+    "138": "I-Organ or Tissue Function",
+    "139": "B-Organic Chemical",
+    "140": "I-Organic Chemical",
+    "141": "B-Organism Function",
+    "142": "I-Organism Function",
+    "143": "B-Organization",
+    "144": "I-Organization",
+    "145": "B-Pathologic Function",
+    "146": "I-Pathologic Function",
+    "147": "B-Pharmacologic Substance",
+    "148": "I-Pharmacologic Substance",
+    "149": "B-Physiologic Function",
+    "150": "I-Physiologic Function",
+    "151": "B-Plant",
+    "152": "I-Plant",
+    "153": "B-Population Group",
+    "154": "I-Population Group",
+    "155": "B-Professional Society",
+    "156": "I-Professional Society",
+    "157": "B-Professional or Occupational Group",
+    "158": "I-Professional or Occupational Group",
+    "159": "B-Receptor",
+    "160": "I-Receptor",
+    "161": "B-Regulation or Law",
+    "162": "I-Regulation or Law",
+    "163": "B-Reptile",
+    "164": "I-Reptile",
+    "165": "B-Research Activity",
+    "166": "I-Research Activity",
+    "167": "B-Self-help or Relief Organization",
+    "168": "I-Self-help or Relief Organization",
+    "169": "B-Sign or Symptom",
+    "170": "I-Sign or Symptom",
+    "171": "B-Spatial Concept",
+    "172": "I-Spatial Concept",
+    "173": "B-Therapeutic or Preventive Procedure",
+    "174": "I-Therapeutic or Preventive Procedure",
+    "175": "B-Tissue",
+    "176": "I-Tissue",
+    "177": "B-Vertebrate",
+    "178": "I-Vertebrate",
+    "179": "B-Virus",
+    "180": "I-Virus",
+    "181": "B-Vitamin",
+    "182": "I-Vitamin"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.48.0.dev0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8699f32b06c3650dedba2d05fab150b749339f2eab1c7ae58189892f0ffd5462
+size 436152844

performance_report.md

@@ -0,0 +1,868 @@
+# Performance on Training Set
+
+## Span Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| Acquired Abnormality | 1.000 | 1.000 | 1.000 | 105 |
+| Amino Acid Sequence | 0.995 | 0.986 | 0.990 | 207 |
+| Amino Acid, Peptide, or Protein | 0.994 | 0.971 | 0.982 | 659 |
+| Amphibian | 0.952 | 1.000 | 0.976 | 20 |
+| Anatomical Abnormality | 0.993 | 0.951 | 0.971 | 143 |
+| Anatomical Structure | 0.952 | 0.952 | 0.952 | 83 |
+| Animal | 1.000 | 0.996 | 0.998 | 238 |
+| Antibiotic | 0.997 | 0.997 | 0.997 | 364 |
+| Bacterium | 0.988 | 0.964 | 0.976 | 1125 |
+| Biologic Function | 0.992 | 0.989 | 0.990 | 622 |
+| Biologically Active Substance | 0.993 | 0.987 | 0.990 | 5343 |
+| Biomedical Occupation or Discipline | 0.996 | 0.985 | 0.990 | 522 |
+| Biomedical or Dental Material | 0.995 | 0.994 | 0.995 | 845 |
+| Bird | 0.978 | 0.938 | 0.958 | 242 |
+| Body Location or Region | 0.993 | 0.986 | 0.989 | 702 |
+| Body Part, Organ, or Organ Component | 0.996 | 0.991 | 0.993 | 3887 |
+| Body Space or Junction | 0.997 | 0.989 | 0.993 | 356 |
+| Body Substance | 0.996 | 0.994 | 0.995 | 774 |
+| Body System | 0.977 | 0.973 | 0.975 | 300 |
+| Cell | 0.993 | 0.992 | 0.993 | 3312 |
+| Cell Component | 0.998 | 0.998 | 0.998 | 929 |
+| Cell Function | 0.996 | 0.994 | 0.995 | 1941 |
+| Cell or Molecular Dysfunction | 0.994 | 0.997 | 0.996 | 350 |
+| Chemical | 0.994 | 0.987 | 0.990 | 156 |
+| Chemical Viewed Functionally | 0.994 | 0.994 | 0.994 | 176 |
+| Chemical Viewed Structurally | 0.993 | 0.986 | 0.989 | 284 |
+| Classification | 0.995 | 0.988 | 0.991 | 738 |
+| Clinical Attribute | 0.991 | 0.980 | 0.986 | 1046 |
+| Congenital Abnormality | 0.967 | 0.989 | 0.978 | 177 |
+| Diagnostic Procedure | 0.995 | 0.987 | 0.991 | 2426 |
+| Disease or Syndrome | 0.998 | 0.988 | 0.993 | 6911 |
+| Drug Delivery Device | 1.000 | 0.846 | 0.917 | 13 |
+| Element, Ion, or Isotope | 0.998 | 0.982 | 0.990 | 934 |
+| Embryonic Structure | 1.000 | 1.000 | 1.000 | 174 |
+| Enzyme | 0.991 | 0.982 | 0.986 | 2192 |
+| Eukaryote | 0.979 | 0.953 | 0.966 | 989 |
+| Experimental Model of Disease | 1.000 | 1.000 | 1.000 | 189 |
+| Finding | 0.992 | 0.986 | 0.989 | 8218 |
+| Fish | 0.974 | 0.955 | 0.964 | 155 |
+| Food | 0.995 | 0.972 | 0.983 | 851 |
+| Fully Formed Anatomical Structure | 0.000 | 0.000 | 0.000 | 1 |
+| Fungus | 0.979 | 0.946 | 0.963 | 353 |
+| Gene or Genome | 0.993 | 0.988 | 0.991 | 2776 |
+| Genetic Function | 0.993 | 0.997 | 0.995 | 2000 |
+| Geographic Area | 0.995 | 0.991 | 0.993 | 1712 |
+| Hazardous or Poisonous Substance | 0.993 | 0.981 | 0.987 | 702 |
+| Health Care Activity | 0.994 | 0.986 | 0.990 | 2858 |
+| Health Care Related Organization | 0.995 | 0.988 | 0.992 | 1041 |
+| Hormone | 1.000 | 0.992 | 0.996 | 504 |
+| Human | 1.000 | 0.998 | 0.999 | 437 |
+| Immunologic Factor | 0.996 | 0.992 | 0.994 | 1722 |
+| Indicator, Reagent, or Diagnostic Aid | 0.991 | 0.986 | 0.988 | 649 |
+| Injury or Poisoning | 0.992 | 0.992 | 0.992 | 1062 |
+| Inorganic Chemical | 0.995 | 0.996 | 0.995 | 765 |
+| Intellectual Product | 0.996 | 0.990 | 0.993 | 5358 |
+| Laboratory Procedure | 0.987 | 0.980 | 0.984 | 2761 |
+| Laboratory or Test Result | 0.986 | 0.982 | 0.984 | 509 |
+| Mammal | 0.996 | 0.987 | 0.992 | 1350 |
+| Medical Device | 0.994 | 0.989 | 0.991 | 1157 |
+| Mental Process | 0.994 | 0.990 | 0.992 | 1431 |
+| Mental or Behavioral Dysfunction | 0.997 | 0.997 | 0.997 | 1099 |
+| Molecular Biology Research Technique | 0.991 | 0.986 | 0.989 | 654 |
+| Molecular Function | 0.994 | 0.994 | 0.994 | 2007 |
+| Molecular Sequence | 0.974 | 0.974 | 0.974 | 39 |
+| Neoplastic Process | 0.996 | 0.987 | 0.991 | 2924 |
+| Nucleic Acid, Nucleoside, or Nucleotide | 0.998 | 0.992 | 0.995 | 508 |
+| Nucleotide Sequence | 0.986 | 0.975 | 0.980 | 360 |
+| Organ or Tissue Function | 0.992 | 0.988 | 0.990 | 904 |
+| Organic Chemical | 0.994 | 0.988 | 0.991 | 1411 |
+| Organism Function | 0.994 | 0.987 | 0.990 | 1639 |
+| Organization | 0.983 | 0.992 | 0.988 | 240 |
+| Pathologic Function | 0.999 | 0.991 | 0.995 | 2325 |
+| Pharmacologic Substance | 0.997 | 0.986 | 0.991 | 4127 |
+| Physiologic Function | 0.992 | 0.971 | 0.981 | 758 |
+| Plant | 0.982 | 0.969 | 0.975 | 1113 |
+| Population Group | 0.998 | 0.990 | 0.994 | 3580 |
+| Professional Society | 1.000 | 0.957 | 0.978 | 23 |
+| Professional or Occupational Group | 0.998 | 0.991 | 0.995 | 1053 |
+| Receptor | 0.993 | 0.990 | 0.992 | 829 |
+| Regulation or Law | 1.000 | 0.961 | 0.980 | 51 |
+| Reptile | 0.941 | 0.889 | 0.914 | 18 |
+| Research Activity | 0.996 | 0.989 | 0.992 | 4787 |
+| Self-help or Relief Organization | 0.000 | 0.000 | 0.000 | 4 |
+| Sign or Symptom | 0.997 | 0.997 | 0.997 | 1131 |
+| Spatial Concept | 0.995 | 0.989 | 0.992 | 4126 |
+| Therapeutic or Preventive Procedure | 0.994 | 0.986 | 0.990 | 6659 |
+| Tissue | 0.991 | 0.983 | 0.987 | 991 |
+| Vertebrate | 0.923 | 0.923 | 0.923 | 13 |
+| Virus | 0.991 | 0.980 | 0.986 | 702 |
+| Vitamin | 1.000 | 0.954 | 0.977 | 262 |
+| macro avg | 0.969 | 0.959 | 0.964 | 122183 |
+| weighted avg | 0.994 | 0.987 | 0.990 | 122183 |
+
+## Token Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| O | 1.000 | 1.000 | 1.000 | 557052 |
+| B-Acquired Abnormality | 1.000 | 1.000 | 1.000 | 105 |
+| I-Acquired Abnormality | 1.000 | 1.000 | 1.000 | 138 |
+| B-Amino Acid Sequence | 1.000 | 1.000 | 1.000 | 206 |
+| I-Amino Acid Sequence | 1.000 | 1.000 | 1.000 | 345 |
+| B-Amino Acid, Peptide, or Protein | 0.997 | 0.991 | 0.994 | 649 |
+| I-Amino Acid, Peptide, or Protein | 1.000 | 0.997 | 0.998 | 968 |
+| B-Amphibian | 1.000 | 1.000 | 1.000 | 20 |
+| I-Amphibian | 0.906 | 1.000 | 0.951 | 29 |
+| B-Anatomical Abnormality | 1.000 | 0.951 | 0.975 | 143 |
+| I-Anatomical Abnormality | 1.000 | 0.985 | 0.992 | 201 |
+| B-Anatomical Structure | 0.976 | 0.976 | 0.976 | 83 |
+| I-Anatomical Structure | 1.000 | 0.972 | 0.986 | 71 |
+| B-Animal | 1.000 | 0.996 | 0.998 | 238 |
+| I-Animal | 1.000 | 1.000 | 1.000 | 95 |
+| B-Antibiotic | 0.997 | 1.000 | 0.999 | 363 |
+| I-Antibiotic | 1.000 | 1.000 | 1.000 | 294 |
+| B-Bacterium | 1.000 | 1.000 | 1.000 | 1111 |
+| I-Bacterium | 1.000 | 1.000 | 1.000 | 2449 |
+| B-Biologic Function | 0.995 | 0.997 | 0.996 | 619 |
+| I-Biologic Function | 0.997 | 0.997 | 0.997 | 305 |
+| B-Biologically Active Substance | 0.999 | 0.998 | 0.998 | 5337 |
+| I-Biologically Active Substance | 0.999 | 1.000 | 0.999 | 6779 |
+| B-Biomedical Occupation or Discipline | 1.000 | 0.990 | 0.995 | 518 |
+| I-Biomedical Occupation or Discipline | 0.990 | 1.000 | 0.995 | 393 |
+| B-Biomedical or Dental Material | 0.995 | 0.998 | 0.996 | 842 |
+| I-Biomedical or Dental Material | 1.000 | 1.000 | 1.000 | 1080 |
+| B-Bird | 0.987 | 1.000 | 0.994 | 232 |
+| I-Bird | 0.997 | 1.000 | 0.999 | 338 |
+| B-Body Location or Region | 0.999 | 0.997 | 0.998 | 697 |
+| I-Body Location or Region | 0.997 | 1.000 | 0.998 | 643 |
+| B-Body Part, Organ, or Organ Component | 0.999 | 0.999 | 0.999 | 3868 |
+| I-Body Part, Organ, or Organ Component | 1.000 | 1.000 | 1.000 | 3015 |
+| B-Body Space or Junction | 0.997 | 0.997 | 0.997 | 353 |
+| I-Body Space or Junction | 1.000 | 1.000 | 1.000 | 425 |
+| B-Body Substance | 0.996 | 0.999 | 0.997 | 770 |
+| I-Body Substance | 1.000 | 1.000 | 1.000 | 378 |
+| B-Body System | 0.997 | 0.997 | 0.997 | 298 |
+| I-Body System | 0.993 | 0.997 | 0.995 | 297 |
+| B-Cell | 0.998 | 0.999 | 0.999 | 3301 |
+| I-Cell | 0.999 | 1.000 | 0.999 | 3622 |
+| B-Cell Component | 0.999 | 0.999 | 0.999 | 929 |
+| I-Cell Component | 1.000 | 0.999 | 0.999 | 825 |
+| B-Cell Function | 0.996 | 0.996 | 0.996 | 1937 |
+| I-Cell Function | 0.999 | 0.999 | 0.999 | 1716 |
+| B-Cell or Molecular Dysfunction | 0.997 | 1.000 | 0.999 | 350 |
+| I-Cell or Molecular Dysfunction | 1.000 | 0.998 | 0.999 | 423 |
+| B-Chemical | 0.994 | 1.000 | 0.997 | 154 |
+| I-Chemical | 1.000 | 1.000 | 1.000 | 22 |
+| B-Chemical Viewed Functionally | 0.994 | 0.994 | 0.994 | 176 |
+| I-Chemical Viewed Functionally | 1.000 | 1.000 | 1.000 | 98 |
+| B-Chemical Viewed Structurally | 0.996 | 1.000 | 0.998 | 281 |
+| I-Chemical Viewed Structurally | 0.996 | 1.000 | 0.998 | 282 |
+| B-Classification | 0.997 | 0.995 | 0.996 | 736 |
+| I-Classification | 0.998 | 1.000 | 0.999 | 447 |
+| B-Clinical Attribute | 0.996 | 0.999 | 0.998 | 1030 |
+| I-Clinical Attribute | 0.995 | 0.999 | 0.997 | 854 |
+| B-Congenital Abnormality | 0.967 | 1.000 | 0.983 | 177 |
+| I-Congenital Abnormality | 0.988 | 1.000 | 0.994 | 331 |
+| B-Diagnostic Procedure | 0.999 | 0.997 | 0.998 | 2412 |
+| I-Diagnostic Procedure | 0.998 | 1.000 | 0.999 | 3191 |
+| B-Disease or Syndrome | 0.999 | 0.999 | 0.999 | 6843 |
+| I-Disease or Syndrome | 0.999 | 1.000 | 1.000 | 7295 |
+| B-Drug Delivery Device | 1.000 | 1.000 | 1.000 | 11 |
+| I-Drug Delivery Device | 1.000 | 1.000 | 1.000 | 3 |
+| B-Element, Ion, or Isotope | 1.000 | 0.995 | 0.997 | 924 |
+| I-Element, Ion, or Isotope | 0.999 | 1.000 | 0.999 | 922 |
+| B-Embryonic Structure | 1.000 | 1.000 | 1.000 | 174 |
+| I-Embryonic Structure | 1.000 | 1.000 | 1.000 | 73 |
+| B-Enzyme | 0.998 | 1.000 | 0.999 | 2182 |
+| I-Enzyme | 1.000 | 1.000 | 1.000 | 3095 |
+| B-Eukaryote | 1.000 | 0.998 | 0.999 | 985 |
+| I-Eukaryote | 1.000 | 1.000 | 1.000 | 2332 |
+| B-Experimental Model of Disease | 1.000 | 1.000 | 1.000 | 189 |
+| I-Experimental Model of Disease | 1.000 | 1.000 | 1.000 | 213 |
+| B-Finding | 0.997 | 0.998 | 0.997 | 8158 |
+| I-Finding | 0.996 | 0.999 | 0.998 | 7036 |
+| B-Fish | 0.994 | 1.000 | 0.997 | 154 |
+| I-Fish | 1.000 | 1.000 | 1.000 | 364 |
+| B-Food | 0.999 | 0.998 | 0.998 | 832 |
+| I-Food | 0.997 | 0.999 | 0.998 | 705 |
+| B-Fully Formed Anatomical Structure | 0.000 | 0.000 | 0.000 | 1 |
+| I-Fully Formed Anatomical Structure | 0.000 | 0.000 | 0.000 | 0 |
+| B-Fungus | 1.000 | 1.000 | 1.000 | 348 |
+| I-Fungus | 1.000 | 1.000 | 1.000 | 748 |
+| B-Gene or Genome | 0.998 | 0.999 | 0.998 | 2768 |
+| I-Gene or Genome | 1.000 | 1.000 | 1.000 | 4238 |
+| B-Genetic Function | 0.995 | 0.999 | 0.997 | 1999 |
+| I-Genetic Function | 0.999 | 1.000 | 1.000 | 1245 |
+| B-Geographic Area | 0.998 | 1.000 | 0.999 | 1702 |
+| I-Geographic Area | 0.996 | 0.998 | 0.997 | 1370 |
+| B-Hazardous or Poisonous Substance | 0.997 | 0.997 | 0.997 | 697 |
+| I-Hazardous or Poisonous Substance | 0.999 | 1.000 | 0.999 | 882 |
+| B-Health Care Activity | 0.997 | 0.997 | 0.997 | 2833 |
+| I-Health Care Activity | 0.998 | 0.999 | 0.999 | 2092 |
+| B-Health Care Related Organization | 0.998 | 0.999 | 0.999 | 1031 |
+| I-Health Care Related Organization | 0.999 | 1.000 | 0.999 | 1447 |
+| B-Hormone | 1.000 | 1.000 | 1.000 | 500 |
+| I-Hormone | 1.000 | 1.000 | 1.000 | 454 |
+| B-Human | 1.000 | 1.000 | 1.000 | 436 |
+| I-Human | 1.000 | 1.000 | 1.000 | 75 |
+| B-Idea or Concept | 0.000 | 0.000 | 0.000 | 0 |
+| I-Idea or Concept | 0.000 | 0.000 | 0.000 | 0 |
+| B-Immunologic Factor | 0.997 | 0.998 | 0.997 | 1714 |
+| I-Immunologic Factor | 1.000 | 1.000 | 1.000 | 2675 |
+| B-Indicator, Reagent, or Diagnostic Aid | 0.995 | 0.998 | 0.997 | 644 |
+| I-Indicator, Reagent, or Diagnostic Aid | 1.000 | 1.000 | 1.000 | 952 |
+| B-Injury or Poisoning | 0.994 | 1.000 | 0.997 | 1056 |
+| I-Injury or Poisoning | 0.998 | 1.000 | 0.999 | 1028 |
+| B-Inorganic Chemical | 0.996 | 1.000 | 0.998 | 763 |
+| I-Inorganic Chemical | 0.998 | 1.000 | 0.999 | 588 |
+| B-Intellectual Product | 0.998 | 0.997 | 0.998 | 5336 |
+| I-Intellectual Product | 0.999 | 0.999 | 0.999 | 6776 |
+| B-Laboratory Procedure | 0.996 | 0.993 | 0.995 | 2745 |
+| I-Laboratory Procedure | 0.999 | 1.000 | 0.999 | 4182 |
+| B-Laboratory or Test Result | 0.994 | 0.992 | 0.993 | 505 |
+| I-Laboratory or Test Result | 0.997 | 0.999 | 0.998 | 684 |
+| B-Mammal | 0.999 | 0.999 | 0.999 | 1338 |
+| I-Mammal | 0.999 | 0.999 | 0.999 | 1075 |
+| B-Medical Device | 1.000 | 0.997 | 0.998 | 1152 |
+| I-Medical Device | 0.999 | 1.000 | 0.999 | 1599 |
+| B-Mental Process | 0.994 | 0.996 | 0.995 | 1423 |
+| I-Mental Process | 1.000 | 0.997 | 0.999 | 749 |
+| B-Mental or Behavioral Dysfunction | 0.997 | 1.000 | 0.999 | 1096 |
+| I-Mental or Behavioral Dysfunction | 1.000 | 1.000 | 1.000 | 867 |
+| B-Molecular Biology Research Technique | 0.994 | 0.992 | 0.993 | 651 |
+| I-Molecular Biology Research Technique | 0.998 | 0.998 | 0.998 | 1121 |
+| B-Molecular Function | 0.997 | 0.997 | 0.997 | 2004 |
+| I-Molecular Function | 0.998 | 1.000 | 0.999 | 2015 |
+| B-Molecular Sequence | 0.974 | 0.974 | 0.974 | 39 |
+| I-Molecular Sequence | 1.000 | 1.000 | 1.000 | 42 |
+| B-Neoplastic Process | 0.999 | 1.000 | 0.999 | 2900 |
+| I-Neoplastic Process | 0.999 | 0.999 | 0.999 | 2938 |
+| B-Nucleic Acid, Nucleoside, or Nucleotide | 1.000 | 0.992 | 0.996 | 508 |
+| I-Nucleic Acid, Nucleoside, or Nucleotide | 0.997 | 1.000 | 0.998 | 609 |
+| B-Nucleotide Sequence | 0.997 | 0.994 | 0.996 | 360 |
+| I-Nucleotide Sequence | 0.998 | 0.998 | 0.998 | 641 |
+| B-Organ or Tissue Function | 0.996 | 0.993 | 0.994 | 901 |
+| I-Organ or Tissue Function | 0.996 | 0.995 | 0.996 | 845 |
+| B-Organic Chemical | 0.999 | 0.999 | 0.999 | 1404 |
+| I-Organic Chemical | 0.999 | 1.000 | 1.000 | 2838 |
+| B-Organism Function | 0.995 | 0.996 | 0.996 | 1626 |
+| I-Organism Function | 0.999 | 0.996 | 0.997 | 944 |
+| B-Organization | 0.983 | 1.000 | 0.992 | 238 |
+| I-Organization | 0.996 | 1.000 | 0.998 | 260 |
+| B-Pathologic Function | 1.000 | 0.998 | 0.999 | 2309 |
+| I-Pathologic Function | 0.999 | 1.000 | 0.999 | 1677 |
+| B-Pharmacologic Substance | 0.999 | 1.000 | 0.999 | 4080 |
+| I-Pharmacologic Substance | 0.999 | 1.000 | 1.000 | 6760 |
+| B-Physiologic Function | 0.997 | 0.989 | 0.993 | 746 |
+| I-Physiologic Function | 0.996 | 1.000 | 0.998 | 507 |
+| B-Plant | 0.995 | 0.998 | 0.996 | 1104 |
+| I-Plant | 0.998 | 0.999 | 0.999 | 1821 |
+| B-Population Group | 0.999 | 0.999 | 0.999 | 3547 |
+| I-Population Group | 0.997 | 0.999 | 0.998 | 1310 |
+| B-Professional Society | 1.000 | 0.957 | 0.978 | 23 |
+| I-Professional Society | 1.000 | 1.000 | 1.000 | 55 |
+| B-Professional or Occupational Group | 0.998 | 0.999 | 0.999 | 1045 |
+| I-Professional or Occupational Group | 1.000 | 1.000 | 1.000 | 700 |
+| B-Receptor | 0.994 | 0.998 | 0.996 | 824 |
+| I-Receptor | 0.998 | 0.998 | 0.998 | 1291 |
+| B-Regulation or Law | 1.000 | 0.961 | 0.980 | 51 |
+| I-Regulation or Law | 1.000 | 1.000 | 1.000 | 54 |
+| B-Reptile | 1.000 | 1.000 | 1.000 | 18 |
+| I-Reptile | 1.000 | 1.000 | 1.000 | 23 |
+| B-Research Activity | 0.997 | 0.994 | 0.995 | 4768 |
+| I-Research Activity | 0.999 | 0.999 | 0.999 | 3780 |
+| B-Self-help or Relief Organization | 0.000 | 0.000 | 0.000 | 4 |
+| I-Self-help or Relief Organization | 0.000 | 0.000 | 0.000 | 4 |
+| B-Sign or Symptom | 0.996 | 0.999 | 0.998 | 1129 |
+| I-Sign or Symptom | 0.999 | 0.998 | 0.998 | 807 |
+| B-Spatial Concept | 0.997 | 0.997 | 0.997 | 4103 |
+| I-Spatial Concept | 0.996 | 0.999 | 0.997 | 1473 |
+| B-Therapeutic or Preventive Procedure | 0.998 | 0.998 | 0.998 | 6613 |
+| I-Therapeutic or Preventive Procedure | 1.000 | 1.000 | 1.000 | 6523 |
+| B-Tissue | 0.996 | 0.994 | 0.995 | 982 |
+| I-Tissue | 0.995 | 0.998 | 0.997 | 597 |
+| B-Vertebrate | 1.000 | 1.000 | 1.000 | 13 |
+| I-Vertebrate | 0.000 | 0.000 | 0.000 | 3 |
+| B-Virus | 0.999 | 1.000 | 0.999 | 696 |
+| I-Virus | 1.000 | 1.000 | 1.000 | 1027 |
+| B-Vitamin | 1.000 | 0.992 | 0.996 | 252 |
+| I-Vitamin | 0.998 | 1.000 | 0.999 | 447 |
+| macro avg | 0.959 | 0.959 | 0.959 | 805394 |
+| weighted avg | 0.999 | 0.999 | 0.999 | 805394 |
+
+
+# Performance on Validation Set
+
+## Span Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| Acquired Abnormality | 0.109 | 0.182 | 0.136 | 33 |
+| Amino Acid Sequence | 0.373 | 0.328 | 0.349 | 58 |
+| Amino Acid, Peptide, or Protein | 0.310 | 0.278 | 0.293 | 187 |
+| Amphibian | 1.000 | 0.143 | 0.250 | 7 |
+| Anatomical Abnormality | 0.034 | 0.037 | 0.035 | 54 |
+| Anatomical Structure | 0.625 | 0.263 | 0.370 | 38 |
+| Animal | 0.615 | 0.747 | 0.675 | 75 |
+| Antibiotic | 0.825 | 0.810 | 0.817 | 210 |
+| Bacterium | 0.709 | 0.755 | 0.731 | 470 |
+| Biologic Function | 0.389 | 0.479 | 0.429 | 142 |
+| Biologically Active Substance | 0.583 | 0.641 | 0.611 | 1882 |
+| Biomedical Occupation or Discipline | 0.584 | 0.475 | 0.524 | 198 |
+| Biomedical or Dental Material | 0.395 | 0.428 | 0.411 | 304 |
+| Bird | 0.846 | 0.759 | 0.800 | 116 |
+| Body Location or Region | 0.288 | 0.393 | 0.332 | 163 |
+| Body Part, Organ, or Organ Component | 0.639 | 0.665 | 0.652 | 1259 |
+| Body Space or Junction | 0.442 | 0.476 | 0.458 | 145 |
+| Body Substance | 0.641 | 0.725 | 0.681 | 269 |
+| Body System | 0.611 | 0.629 | 0.620 | 105 |
+| Cell | 0.713 | 0.733 | 0.723 | 1205 |
+| Cell Component | 0.609 | 0.630 | 0.619 | 338 |
+| Cell Function | 0.524 | 0.606 | 0.562 | 619 |
+| Cell or Molecular Dysfunction | 0.635 | 0.595 | 0.614 | 111 |
+| Chemical | 0.452 | 0.459 | 0.455 | 61 |
+| Chemical Viewed Functionally | 0.362 | 0.382 | 0.372 | 55 |
+| Chemical Viewed Structurally | 0.561 | 0.517 | 0.538 | 116 |
+| Classification | 0.532 | 0.520 | 0.526 | 304 |
+| Clinical Attribute | 0.545 | 0.522 | 0.534 | 404 |
+| Congenital Abnormality | 0.481 | 0.694 | 0.568 | 36 |
+| Diagnostic Procedure | 0.582 | 0.564 | 0.573 | 684 |
+| Disease or Syndrome | 0.765 | 0.762 | 0.764 | 2365 |
+| Drug Delivery Device | 0.000 | 0.000 | 0.000 | 6 |
+| Element, Ion, or Isotope | 0.626 | 0.682 | 0.653 | 280 |
+| Embryonic Structure | 0.698 | 0.769 | 0.732 | 39 |
+| Enzyme | 0.647 | 0.728 | 0.685 | 569 |
+| Eukaryote | 0.656 | 0.685 | 0.670 | 409 |
+| Experimental Model of Disease | 0.377 | 0.547 | 0.446 | 53 |
+| Finding | 0.355 | 0.348 | 0.351 | 2670 |
+| Fish | 0.857 | 0.884 | 0.870 | 95 |
+| Food | 0.585 | 0.641 | 0.611 | 259 |
+| Fungus | 0.729 | 0.647 | 0.685 | 133 |
+| Gene or Genome | 0.611 | 0.611 | 0.611 | 769 |
+| Genetic Function | 0.644 | 0.680 | 0.662 | 519 |
+| Geographic Area | 0.749 | 0.736 | 0.743 | 678 |
+| Hazardous or Poisonous Substance | 0.598 | 0.620 | 0.609 | 221 |
+| Health Care Activity | 0.450 | 0.446 | 0.448 | 981 |
+| Health Care Related Organization | 0.532 | 0.559 | 0.545 | 367 |
+| Hormone | 0.661 | 0.796 | 0.723 | 157 |
+| Human | 0.766 | 0.883 | 0.820 | 137 |
+| Immunologic Factor | 0.694 | 0.654 | 0.674 | 535 |
+| Indicator, Reagent, or Diagnostic Aid | 0.342 | 0.385 | 0.363 | 231 |
+| Injury or Poisoning | 0.621 | 0.625 | 0.623 | 435 |
+| Inorganic Chemical | 0.688 | 0.688 | 0.688 | 285 |
+| Intellectual Product | 0.480 | 0.465 | 0.472 | 1665 |
+| Laboratory Procedure | 0.438 | 0.494 | 0.464 | 852 |
+| Laboratory or Test Result | 0.144 | 0.189 | 0.163 | 132 |
+| Mammal | 0.796 | 0.815 | 0.806 | 465 |
+| Medical Device | 0.553 | 0.569 | 0.561 | 487 |
+| Mental Process | 0.527 | 0.492 | 0.509 | 590 |
+| Mental or Behavioral Dysfunction | 0.668 | 0.700 | 0.684 | 530 |
+| Molecular Biology Research Technique | 0.510 | 0.667 | 0.578 | 195 |
+| Molecular Function | 0.438 | 0.547 | 0.487 | 539 |
+| Molecular Sequence | 0.071 | 0.125 | 0.091 | 8 |
+| Neoplastic Process | 0.737 | 0.749 | 0.743 | 806 |
+| Nucleic Acid, Nucleoside, or Nucleotide | 0.379 | 0.482 | 0.424 | 139 |
+| Nucleotide Sequence | 0.404 | 0.423 | 0.414 | 130 |
+| Organ or Tissue Function | 0.419 | 0.428 | 0.423 | 306 |
+| Organic Chemical | 0.499 | 0.476 | 0.487 | 540 |
+| Organism Function | 0.445 | 0.522 | 0.480 | 510 |
+| Organization | 0.259 | 0.367 | 0.304 | 79 |
+| Pathologic Function | 0.564 | 0.579 | 0.572 | 877 |
+| Pharmacologic Substance | 0.634 | 0.682 | 0.657 | 1296 |
+| Physiologic Function | 0.391 | 0.299 | 0.339 | 264 |
+| Plant | 0.688 | 0.691 | 0.690 | 405 |
+| Population Group | 0.661 | 0.620 | 0.640 | 1303 |
+| Professional Society | 1.000 | 0.200 | 0.333 | 5 |
+| Professional or Occupational Group | 0.643 | 0.734 | 0.685 | 365 |
+| Receptor | 0.688 | 0.623 | 0.654 | 361 |
+| Regulation or Law | 0.167 | 0.222 | 0.190 | 18 |
+| Reptile | 1.000 | 0.222 | 0.364 | 18 |
+| Research Activity | 0.590 | 0.525 | 0.556 | 1666 |
+| Self-help or Relief Organization | 0.000 | 0.000 | 0.000 | 2 |
+| Sign or Symptom | 0.581 | 0.634 | 0.607 | 350 |
+| Spatial Concept | 0.431 | 0.439 | 0.435 | 1378 |
+| Therapeutic or Preventive Procedure | 0.576 | 0.607 | 0.591 | 2131 |
+| Tissue | 0.583 | 0.625 | 0.603 | 320 |
+| Vertebrate | 1.000 | 0.500 | 0.667 | 6 |
+| Virus | 0.690 | 0.777 | 0.731 | 224 |
+| Vitamin | 0.860 | 0.607 | 0.712 | 61 |
+| macro avg | 0.553 | 0.538 | 0.532 | 40864 |
+| weighted avg | 0.575 | 0.588 | 0.580 | 40864 |
+
+## Token Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| O | 0.938 | 0.939 | 0.939 | 187057 |
+| B-Acquired Abnormality | 0.185 | 0.303 | 0.230 | 33 |
+| I-Acquired Abnormality | 0.200 | 0.297 | 0.239 | 37 |
+| B-Amino Acid Sequence | 0.522 | 0.414 | 0.462 | 58 |
+| I-Amino Acid Sequence | 0.316 | 0.294 | 0.305 | 102 |
+| B-Amino Acid, Peptide, or Protein | 0.346 | 0.292 | 0.317 | 185 |
+| I-Amino Acid, Peptide, or Protein | 0.212 | 0.182 | 0.196 | 291 |
+| B-Amphibian | 1.000 | 0.143 | 0.250 | 7 |
+| I-Amphibian | 1.000 | 0.028 | 0.054 | 36 |
+| B-Anatomical Abnormality | 0.041 | 0.038 | 0.039 | 53 |
+| I-Anatomical Abnormality | 0.029 | 0.053 | 0.037 | 38 |
+| B-Anatomical Structure | 0.625 | 0.263 | 0.370 | 38 |
+| I-Anatomical Structure | 0.600 | 0.200 | 0.300 | 30 |
+| B-Animal | 0.622 | 0.778 | 0.691 | 72 |
+| I-Animal | 0.400 | 0.435 | 0.417 | 23 |
+| B-Antibiotic | 0.856 | 0.832 | 0.844 | 208 |
+| I-Antibiotic | 0.799 | 0.634 | 0.707 | 194 |
+| B-Bacterium | 0.786 | 0.818 | 0.802 | 468 |
+| I-Bacterium | 0.854 | 0.843 | 0.848 | 1112 |
+| B-Biologic Function | 0.410 | 0.479 | 0.442 | 142 |
+| I-Biologic Function | 0.328 | 0.367 | 0.346 | 60 |
+| B-Biologically Active Substance | 0.633 | 0.674 | 0.653 | 1862 |
+| I-Biologically Active Substance | 0.532 | 0.614 | 0.570 | 2209 |
+| B-Biomedical Occupation or Discipline | 0.606 | 0.477 | 0.534 | 197 |
+| I-Biomedical Occupation or Discipline | 0.615 | 0.370 | 0.462 | 173 |
+| B-Biomedical or Dental Material | 0.438 | 0.486 | 0.461 | 292 |
+| I-Biomedical or Dental Material | 0.427 | 0.262 | 0.325 | 447 |
+| B-Bird | 0.875 | 0.798 | 0.835 | 114 |
+| I-Bird | 0.938 | 0.822 | 0.876 | 219 |
+| B-Body Location or Region | 0.323 | 0.429 | 0.368 | 163 |
+| I-Body Location or Region | 0.333 | 0.417 | 0.370 | 144 |
+| B-Body Part, Organ, or Organ Component | 0.690 | 0.712 | 0.701 | 1252 |
+| I-Body Part, Organ, or Organ Component | 0.644 | 0.677 | 0.660 | 969 |
+| B-Body Space or Junction | 0.482 | 0.472 | 0.477 | 142 |
+| I-Body Space or Junction | 0.508 | 0.682 | 0.583 | 176 |
+| B-Body Substance | 0.647 | 0.710 | 0.677 | 269 |
+| I-Body Substance | 0.464 | 0.593 | 0.520 | 118 |
+| B-Body System | 0.642 | 0.648 | 0.645 | 105 |
+| I-Body System | 0.711 | 0.667 | 0.688 | 81 |
+| B-Cell | 0.758 | 0.761 | 0.760 | 1200 |
+| I-Cell | 0.727 | 0.855 | 0.786 | 1263 |
+| B-Cell Component | 0.687 | 0.663 | 0.675 | 338 |
+| I-Cell Component | 0.544 | 0.511 | 0.527 | 266 |
+| B-Cell Function | 0.590 | 0.663 | 0.624 | 614 |
+| I-Cell Function | 0.510 | 0.518 | 0.514 | 533 |
+| B-Cell or Molecular Dysfunction | 0.687 | 0.613 | 0.648 | 111 |
+| I-Cell or Molecular Dysfunction | 0.524 | 0.581 | 0.551 | 93 |
+| B-Chemical | 0.483 | 0.483 | 0.483 | 60 |
+| I-Chemical | 0.500 | 0.083 | 0.143 | 24 |
+| B-Chemical Viewed Functionally | 0.362 | 0.382 | 0.372 | 55 |
+| I-Chemical Viewed Functionally | 0.476 | 0.400 | 0.435 | 50 |
+| B-Chemical Viewed Structurally | 0.614 | 0.534 | 0.571 | 116 |
+| I-Chemical Viewed Structurally | 0.171 | 0.128 | 0.146 | 47 |
+| B-Classification | 0.577 | 0.560 | 0.569 | 300 |
+| I-Classification | 0.460 | 0.388 | 0.421 | 147 |
+| B-Clinical Attribute | 0.607 | 0.579 | 0.593 | 401 |
+| I-Clinical Attribute | 0.470 | 0.345 | 0.398 | 275 |
+| B-Congenital Abnormality | 0.481 | 0.694 | 0.568 | 36 |
+| I-Congenital Abnormality | 0.562 | 0.581 | 0.571 | 62 |
+| B-Diagnostic Procedure | 0.697 | 0.635 | 0.665 | 680 |
+| I-Diagnostic Procedure | 0.638 | 0.648 | 0.643 | 1000 |
+| B-Disease or Syndrome | 0.790 | 0.788 | 0.789 | 2335 |
+| I-Disease or Syndrome | 0.754 | 0.779 | 0.766 | 2454 |
+| B-Drug Delivery Device | 0.000 | 0.000 | 0.000 | 6 |
+| I-Drug Delivery Device | 0.000 | 0.000 | 0.000 | 3 |
+| B-Element, Ion, or Isotope | 0.691 | 0.716 | 0.703 | 278 |
+| I-Element, Ion, or Isotope | 0.608 | 0.734 | 0.665 | 192 |
+| B-Embryonic Structure | 0.698 | 0.833 | 0.759 | 36 |
+| I-Embryonic Structure | 0.889 | 0.889 | 0.889 | 9 |
+| B-Enzyme | 0.693 | 0.767 | 0.728 | 566 |
+| I-Enzyme | 0.704 | 0.758 | 0.730 | 770 |
+| B-Eukaryote | 0.782 | 0.830 | 0.805 | 406 |
+| I-Eukaryote | 0.828 | 0.878 | 0.852 | 950 |
+| B-Experimental Model of Disease | 0.431 | 0.596 | 0.500 | 52 |
+| I-Experimental Model of Disease | 0.360 | 0.300 | 0.327 | 60 |
+| B-Finding | 0.410 | 0.383 | 0.396 | 2640 |
+| I-Finding | 0.343 | 0.278 | 0.307 | 2209 |
+| B-Fish | 0.825 | 0.904 | 0.863 | 94 |
+| I-Fish | 0.788 | 0.732 | 0.759 | 142 |
+| B-Food | 0.663 | 0.715 | 0.688 | 256 |
+| I-Food | 0.625 | 0.689 | 0.655 | 244 |
+| B-Fully Formed Anatomical Structure | 0.000 | 0.000 | 0.000 | 0 |
+| I-Fully Formed Anatomical Structure | 0.000 | 0.000 | 0.000 | 0 |
+| B-Fungus | 0.842 | 0.722 | 0.777 | 133 |
+| I-Fungus | 0.881 | 0.776 | 0.825 | 210 |
+| B-Gene or Genome | 0.682 | 0.677 | 0.679 | 764 |
+| I-Gene or Genome | 0.734 | 0.658 | 0.694 | 1317 |
+| B-Genetic Function | 0.695 | 0.706 | 0.701 | 517 |
+| I-Genetic Function | 0.585 | 0.478 | 0.526 | 316 |
+| B-Geographic Area | 0.815 | 0.789 | 0.802 | 674 |
+| I-Geographic Area | 0.703 | 0.790 | 0.744 | 542 |
+| B-Hazardous or Poisonous Substance | 0.661 | 0.658 | 0.659 | 219 |
+| I-Hazardous or Poisonous Substance | 0.590 | 0.630 | 0.610 | 311 |
+| B-Health Care Activity | 0.506 | 0.496 | 0.501 | 970 |
+| I-Health Care Activity | 0.523 | 0.451 | 0.484 | 730 |
+| B-Health Care Related Organization | 0.589 | 0.610 | 0.599 | 359 |
+| I-Health Care Related Organization | 0.649 | 0.647 | 0.648 | 519 |
+| B-Hormone | 0.691 | 0.828 | 0.754 | 157 |
+| I-Hormone | 0.474 | 0.688 | 0.561 | 80 |
+| B-Human | 0.778 | 0.898 | 0.834 | 137 |
+| I-Human | 0.419 | 0.464 | 0.441 | 28 |
+| B-Idea or Concept | 0.000 | 0.000 | 0.000 | 0 |
+| I-Idea or Concept | 0.000 | 0.000 | 0.000 | 0 |
+| B-Immunologic Factor | 0.766 | 0.680 | 0.721 | 535 |
+| I-Immunologic Factor | 0.688 | 0.680 | 0.684 | 753 |
+| B-Indicator, Reagent, or Diagnostic Aid | 0.450 | 0.476 | 0.463 | 227 |
+| I-Indicator, Reagent, or Diagnostic Aid | 0.512 | 0.533 | 0.522 | 450 |
+| B-Injury or Poisoning | 0.654 | 0.614 | 0.633 | 433 |
+| I-Injury or Poisoning | 0.543 | 0.698 | 0.611 | 315 |
+| B-Inorganic Chemical | 0.704 | 0.711 | 0.708 | 284 |
+| I-Inorganic Chemical | 0.718 | 0.685 | 0.701 | 257 |
+| B-Intellectual Product | 0.539 | 0.506 | 0.522 | 1653 |
+| I-Intellectual Product | 0.557 | 0.486 | 0.519 | 2089 |
+| B-Laboratory Procedure | 0.516 | 0.544 | 0.530 | 847 |
+| I-Laboratory Procedure | 0.574 | 0.600 | 0.587 | 1237 |
+| B-Laboratory or Test Result | 0.217 | 0.250 | 0.232 | 132 |
+| I-Laboratory or Test Result | 0.199 | 0.224 | 0.211 | 152 |
+| B-Mammal | 0.831 | 0.838 | 0.834 | 457 |
+| I-Mammal | 0.840 | 0.849 | 0.844 | 345 |
+| B-Medical Device | 0.604 | 0.603 | 0.604 | 486 |
+| I-Medical Device | 0.605 | 0.604 | 0.605 | 685 |
+| B-Mental Process | 0.567 | 0.523 | 0.544 | 583 |
+| I-Mental Process | 0.609 | 0.533 | 0.568 | 304 |
+| B-Mental or Behavioral Dysfunction | 0.711 | 0.735 | 0.723 | 529 |
+| I-Mental or Behavioral Dysfunction | 0.700 | 0.670 | 0.685 | 397 |
+| B-Molecular Biology Research Technique | 0.576 | 0.718 | 0.639 | 195 |
+| I-Molecular Biology Research Technique | 0.712 | 0.724 | 0.718 | 409 |
+| B-Molecular Function | 0.501 | 0.596 | 0.544 | 539 |
+| I-Molecular Function | 0.449 | 0.483 | 0.466 | 532 |
+| B-Molecular Sequence | 0.143 | 0.250 | 0.182 | 8 |
+| I-Molecular Sequence | 0.222 | 0.286 | 0.250 | 7 |
+| B-Neoplastic Process | 0.778 | 0.793 | 0.786 | 794 |
+| I-Neoplastic Process | 0.774 | 0.788 | 0.781 | 905 |
+| B-Nucleic Acid, Nucleoside, or Nucleotide | 0.453 | 0.600 | 0.517 | 130 |
+| I-Nucleic Acid, Nucleoside, or Nucleotide | 0.392 | 0.396 | 0.394 | 197 |
+| B-Nucleotide Sequence | 0.446 | 0.425 | 0.435 | 127 |
+| I-Nucleotide Sequence | 0.316 | 0.524 | 0.394 | 103 |
+| B-Organ or Tissue Function | 0.459 | 0.447 | 0.453 | 304 |
+| I-Organ or Tissue Function | 0.478 | 0.481 | 0.480 | 270 |
+| B-Organic Chemical | 0.537 | 0.484 | 0.509 | 539 |
+| I-Organic Chemical | 0.450 | 0.363 | 0.402 | 1105 |
+| B-Organism Function | 0.466 | 0.540 | 0.500 | 506 |
+| I-Organism Function | 0.389 | 0.379 | 0.384 | 272 |
+| B-Organization | 0.287 | 0.403 | 0.335 | 77 |
+| I-Organization | 0.250 | 0.330 | 0.284 | 91 |
+| B-Pathologic Function | 0.598 | 0.595 | 0.597 | 872 |
+| I-Pathologic Function | 0.467 | 0.487 | 0.477 | 532 |
+| B-Pharmacologic Substance | 0.670 | 0.718 | 0.693 | 1274 |
+| I-Pharmacologic Substance | 0.602 | 0.666 | 0.633 | 1991 |
+| B-Physiologic Function | 0.437 | 0.317 | 0.367 | 262 |
+| I-Physiologic Function | 0.271 | 0.172 | 0.211 | 186 |
+| B-Plant | 0.781 | 0.779 | 0.780 | 403 |
+| I-Plant | 0.824 | 0.845 | 0.834 | 676 |
+| B-Population Group | 0.712 | 0.659 | 0.684 | 1289 |
+| I-Population Group | 0.489 | 0.467 | 0.478 | 486 |
+| B-Professional Society | 0.500 | 0.200 | 0.286 | 5 |
+| I-Professional Society | 1.000 | 0.250 | 0.400 | 8 |
+| B-Professional or Occupational Group | 0.683 | 0.786 | 0.731 | 359 |
+| I-Professional or Occupational Group | 0.631 | 0.613 | 0.622 | 302 |
+| B-Receptor | 0.755 | 0.657 | 0.702 | 361 |
+| I-Receptor | 0.686 | 0.647 | 0.666 | 541 |
+| B-Regulation or Law | 0.286 | 0.333 | 0.308 | 18 |
+| I-Regulation or Law | 0.640 | 0.627 | 0.634 | 51 |
+| B-Reptile | 1.000 | 0.222 | 0.364 | 18 |
+| I-Reptile | 1.000 | 0.115 | 0.207 | 52 |
+| B-Research Activity | 0.639 | 0.559 | 0.597 | 1650 |
+| I-Research Activity | 0.605 | 0.542 | 0.572 | 1274 |
+| B-Self-help or Relief Organization | 0.000 | 0.000 | 0.000 | 2 |
+| I-Self-help or Relief Organization | 0.000 | 0.000 | 0.000 | 4 |
+| B-Sign or Symptom | 0.616 | 0.646 | 0.630 | 350 |
+| I-Sign or Symptom | 0.504 | 0.513 | 0.509 | 232 |
+| B-Spatial Concept | 0.485 | 0.481 | 0.483 | 1371 |
+| I-Spatial Concept | 0.434 | 0.389 | 0.410 | 570 |
+| B-Therapeutic or Preventive Procedure | 0.618 | 0.637 | 0.628 | 2112 |
+| I-Therapeutic or Preventive Procedure | 0.611 | 0.623 | 0.617 | 1917 |
+| B-Tissue | 0.620 | 0.649 | 0.634 | 319 |
+| I-Tissue | 0.571 | 0.563 | 0.567 | 215 |
+| B-Vertebrate | 1.000 | 0.500 | 0.667 | 6 |
+| I-Vertebrate | 0.000 | 0.000 | 0.000 | 2 |
+| B-Virus | 0.760 | 0.856 | 0.805 | 222 |
+| I-Virus | 0.699 | 0.843 | 0.764 | 267 |
+| B-Vitamin | 0.886 | 0.639 | 0.743 | 61 |
+| I-Vitamin | 0.889 | 0.681 | 0.771 | 94 |
+| macro avg | 0.560 | 0.533 | 0.533 | 269146 |
+| weighted avg | 0.839 | 0.840 | 0.839 | 269146 |
+
+
+# Performance on Testing Set
+
+## Span Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| Acquired Abnormality | 0.273 | 0.240 | 0.255 | 50 |
+| Amino Acid Sequence | 0.303 | 0.357 | 0.328 | 84 |
+| Amino Acid, Peptide, or Protein | 0.258 | 0.277 | 0.267 | 166 |
+| Anatomical Abnormality | 0.190 | 0.145 | 0.164 | 76 |
+| Anatomical Structure | 0.160 | 0.308 | 0.211 | 13 |
+| Animal | 0.605 | 0.742 | 0.667 | 93 |
+| Antibiotic | 0.835 | 0.784 | 0.808 | 148 |
+| Bacterium | 0.752 | 0.732 | 0.742 | 448 |
+| Biologic Function | 0.341 | 0.369 | 0.355 | 157 |
+| Biologically Active Substance | 0.586 | 0.640 | 0.612 | 2080 |
+| Biomedical Occupation or Discipline | 0.437 | 0.444 | 0.441 | 196 |
+| Biomedical or Dental Material | 0.369 | 0.452 | 0.406 | 197 |
+| Bird | 0.810 | 0.819 | 0.814 | 83 |
+| Body Location or Region | 0.386 | 0.461 | 0.420 | 232 |
+| Body Part, Organ, or Organ Component | 0.565 | 0.604 | 0.584 | 1092 |
+| Body Space or Junction | 0.272 | 0.311 | 0.290 | 90 |
+| Body Substance | 0.542 | 0.731 | 0.622 | 212 |
+| Body System | 0.554 | 0.511 | 0.532 | 90 |
+| Cell | 0.680 | 0.737 | 0.708 | 924 |
+| Cell Component | 0.589 | 0.637 | 0.612 | 311 |
+| Cell Function | 0.484 | 0.599 | 0.535 | 499 |
+| Cell or Molecular Dysfunction | 0.607 | 0.657 | 0.631 | 99 |
+| Chemical | 0.348 | 0.333 | 0.340 | 72 |
+| Chemical Viewed Functionally | 0.286 | 0.432 | 0.344 | 37 |
+| Chemical Viewed Structurally | 0.443 | 0.427 | 0.435 | 82 |
+| Classification | 0.520 | 0.544 | 0.532 | 309 |
+| Clinical Attribute | 0.596 | 0.625 | 0.610 | 323 |
+| Congenital Abnormality | 0.438 | 0.443 | 0.440 | 79 |
+| Diagnostic Procedure | 0.672 | 0.648 | 0.660 | 735 |
+| Disease or Syndrome | 0.757 | 0.774 | 0.766 | 2199 |
+| Element, Ion, or Isotope | 0.713 | 0.657 | 0.684 | 385 |
+| Embryonic Structure | 0.587 | 0.509 | 0.545 | 53 |
+| Enzyme | 0.766 | 0.761 | 0.763 | 681 |
+| Eukaryote | 0.745 | 0.793 | 0.768 | 397 |
+| Experimental Model of Disease | 0.286 | 0.356 | 0.317 | 45 |
+| Finding | 0.391 | 0.388 | 0.389 | 2759 |
+| Fish | 1.000 | 0.947 | 0.973 | 19 |
+| Food | 0.556 | 0.455 | 0.501 | 336 |
+| Fully Formed Anatomical Structure | 0.000 | 0.000 | 0.000 | 1 |
+| Fungus | 0.819 | 0.798 | 0.809 | 119 |
+| Gene or Genome | 0.551 | 0.539 | 0.545 | 912 |
+| Genetic Function | 0.598 | 0.646 | 0.621 | 652 |
+| Geographic Area | 0.673 | 0.712 | 0.692 | 598 |
+| Hazardous or Poisonous Substance | 0.513 | 0.522 | 0.518 | 293 |
+| Health Care Activity | 0.487 | 0.458 | 0.472 | 1061 |
+| Health Care Related Organization | 0.531 | 0.642 | 0.581 | 296 |
+| Hormone | 0.806 | 0.746 | 0.775 | 189 |
+| Human | 0.799 | 0.880 | 0.837 | 158 |
+| Idea or Concept | 0.000 | 0.000 | 0.000 | 1 |
+| Immunologic Factor | 0.674 | 0.606 | 0.638 | 434 |
+| Indicator, Reagent, or Diagnostic Aid | 0.427 | 0.451 | 0.439 | 182 |
+| Injury or Poisoning | 0.617 | 0.703 | 0.657 | 357 |
+| Inorganic Chemical | 0.611 | 0.680 | 0.643 | 256 |
+| Intellectual Product | 0.495 | 0.485 | 0.490 | 2075 |
+| Laboratory Procedure | 0.445 | 0.452 | 0.448 | 908 |
+| Laboratory or Test Result | 0.183 | 0.196 | 0.190 | 112 |
+| Mammal | 0.778 | 0.838 | 0.807 | 456 |
+| Medical Device | 0.434 | 0.437 | 0.435 | 355 |
+| Mental Process | 0.546 | 0.546 | 0.546 | 740 |
+| Mental or Behavioral Dysfunction | 0.710 | 0.774 | 0.741 | 518 |
+| Molecular Biology Research Technique | 0.500 | 0.539 | 0.519 | 206 |
+| Molecular Function | 0.504 | 0.555 | 0.528 | 719 |
+| Molecular Sequence | 0.417 | 0.556 | 0.476 | 9 |
+| Neoplastic Process | 0.761 | 0.745 | 0.753 | 918 |
+| Nucleic Acid, Nucleoside, or Nucleotide | 0.331 | 0.450 | 0.381 | 109 |
+| Nucleotide Sequence | 0.320 | 0.491 | 0.387 | 110 |
+| Organ or Tissue Function | 0.482 | 0.425 | 0.452 | 247 |
+| Organic Chemical | 0.396 | 0.464 | 0.427 | 511 |
+| Organism Function | 0.462 | 0.518 | 0.488 | 471 |
+| Organization | 0.270 | 0.442 | 0.335 | 77 |
+| Pathologic Function | 0.541 | 0.541 | 0.541 | 669 |
+| Pharmacologic Substance | 0.577 | 0.623 | 0.599 | 1258 |
+| Physiologic Function | 0.283 | 0.286 | 0.284 | 182 |
+| Plant | 0.603 | 0.618 | 0.610 | 403 |
+| Population Group | 0.710 | 0.711 | 0.711 | 1263 |
+| Professional Society | 0.000 | 0.000 | 0.000 | 7 |
+| Professional or Occupational Group | 0.599 | 0.725 | 0.656 | 360 |
+| Receptor | 0.614 | 0.686 | 0.648 | 271 |
+| Regulation or Law | 0.182 | 0.125 | 0.148 | 16 |
+| Reptile | 1.000 | 0.318 | 0.483 | 22 |
+| Research Activity | 0.559 | 0.540 | 0.549 | 1653 |
+| Self-help or Relief Organization | 0.000 | 0.000 | 0.000 | 2 |
+| Sign or Symptom | 0.629 | 0.647 | 0.638 | 340 |
+| Spatial Concept | 0.474 | 0.483 | 0.479 | 1282 |
+| Therapeutic or Preventive Procedure | 0.609 | 0.617 | 0.613 | 2036 |
+| Tissue | 0.562 | 0.525 | 0.543 | 259 |
+| Vertebrate | 0.000 | 0.000 | 0.000 | 1 |
+| Virus | 0.678 | 0.831 | 0.747 | 172 |
+| Vitamin | 0.706 | 0.511 | 0.593 | 47 |
+| macro avg | 0.507 | 0.525 | 0.512 | 40144 |
+| weighted avg | 0.571 | 0.589 | 0.578 | 40144 |
+
+## Token Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| O | 0.940 | 0.938 | 0.939 | 188640 |
+| B-Acquired Abnormality | 0.279 | 0.240 | 0.258 | 50 |
+| I-Acquired Abnormality | 0.371 | 0.283 | 0.321 | 46 |
+| B-Amino Acid Sequence | 0.368 | 0.390 | 0.379 | 82 |
+| I-Amino Acid Sequence | 0.525 | 0.503 | 0.514 | 189 |
+| B-Amino Acid, Peptide, or Protein | 0.321 | 0.325 | 0.323 | 166 |
+| I-Amino Acid, Peptide, or Protein | 0.173 | 0.211 | 0.190 | 161 |
+| B-Amphibian | 0.000 | 0.000 | 0.000 | 0 |
+| I-Amphibian | 0.000 | 0.000 | 0.000 | 0 |
+| B-Anatomical Abnormality | 0.231 | 0.158 | 0.188 | 76 |
+| I-Anatomical Abnormality | 0.296 | 0.154 | 0.203 | 104 |
+| B-Anatomical Structure | 0.154 | 0.333 | 0.211 | 12 |
+| I-Anatomical Structure | 0.000 | 0.000 | 0.000 | 11 |
+| B-Animal | 0.646 | 0.793 | 0.712 | 92 |
+| I-Animal | 0.500 | 0.512 | 0.506 | 41 |
+| B-Antibiotic | 0.846 | 0.782 | 0.813 | 147 |
+| I-Antibiotic | 0.727 | 0.566 | 0.636 | 99 |
+| B-Bacterium | 0.836 | 0.791 | 0.813 | 446 |
+| I-Bacterium | 0.904 | 0.822 | 0.861 | 1311 |
+| B-Biologic Function | 0.381 | 0.394 | 0.387 | 155 |
+| I-Biologic Function | 0.165 | 0.195 | 0.179 | 77 |
+| B-Biologically Active Substance | 0.632 | 0.675 | 0.652 | 2070 |
+| I-Biologically Active Substance | 0.581 | 0.633 | 0.606 | 2623 |
+| B-Biomedical Occupation or Discipline | 0.532 | 0.508 | 0.520 | 195 |
+| I-Biomedical Occupation or Discipline | 0.509 | 0.446 | 0.475 | 130 |
+| B-Biomedical or Dental Material | 0.431 | 0.513 | 0.468 | 195 |
+| I-Biomedical or Dental Material | 0.416 | 0.537 | 0.469 | 203 |
+| B-Bird | 0.902 | 0.892 | 0.897 | 83 |
+| I-Bird | 0.886 | 0.951 | 0.917 | 163 |
+| B-Body Location or Region | 0.455 | 0.504 | 0.479 | 232 |
+| I-Body Location or Region | 0.255 | 0.351 | 0.295 | 174 |
+| B-Body Part, Organ, or Organ Component | 0.610 | 0.643 | 0.626 | 1086 |
+| I-Body Part, Organ, or Organ Component | 0.579 | 0.630 | 0.604 | 865 |
+| B-Body Space or Junction | 0.271 | 0.299 | 0.284 | 87 |
+| I-Body Space or Junction | 0.383 | 0.500 | 0.434 | 108 |
+| B-Body Substance | 0.546 | 0.722 | 0.622 | 212 |
+| I-Body Substance | 0.529 | 0.618 | 0.570 | 102 |
+| B-Body System | 0.605 | 0.557 | 0.580 | 88 |
+| I-Body System | 0.629 | 0.619 | 0.624 | 63 |
+| B-Cell | 0.721 | 0.765 | 0.743 | 920 |
+| I-Cell | 0.747 | 0.750 | 0.748 | 1142 |
+| B-Cell Component | 0.658 | 0.688 | 0.673 | 311 |
+| I-Cell Component | 0.637 | 0.594 | 0.615 | 293 |
+| B-Cell Function | 0.545 | 0.643 | 0.590 | 498 |
+| I-Cell Function | 0.549 | 0.491 | 0.518 | 483 |
+| B-Cell or Molecular Dysfunction | 0.684 | 0.677 | 0.680 | 99 |
+| I-Cell or Molecular Dysfunction | 0.656 | 0.468 | 0.546 | 126 |
+| B-Chemical | 0.371 | 0.361 | 0.366 | 72 |
+| I-Chemical | 0.000 | 0.000 | 0.000 | 24 |
+| B-Chemical Viewed Functionally | 0.340 | 0.459 | 0.391 | 37 |
+| I-Chemical Viewed Functionally | 0.149 | 0.412 | 0.219 | 17 |
+| B-Chemical Viewed Structurally | 0.545 | 0.512 | 0.528 | 82 |
+| I-Chemical Viewed Structurally | 0.162 | 0.107 | 0.129 | 56 |
+| B-Classification | 0.556 | 0.582 | 0.569 | 306 |
+| I-Classification | 0.292 | 0.226 | 0.255 | 177 |
+| B-Clinical Attribute | 0.612 | 0.637 | 0.624 | 322 |
+| I-Clinical Attribute | 0.558 | 0.502 | 0.529 | 219 |
+| B-Congenital Abnormality | 0.500 | 0.481 | 0.490 | 79 |
+| I-Congenital Abnormality | 0.463 | 0.479 | 0.471 | 169 |
+| B-Diagnostic Procedure | 0.731 | 0.686 | 0.708 | 732 |
+| I-Diagnostic Procedure | 0.775 | 0.692 | 0.732 | 1102 |
+| B-Disease or Syndrome | 0.788 | 0.803 | 0.795 | 2185 |
+| I-Disease or Syndrome | 0.742 | 0.755 | 0.749 | 2326 |
+| B-Drug Delivery Device | 0.000 | 0.000 | 0.000 | 0 |
+| I-Drug Delivery Device | 0.000 | 0.000 | 0.000 | 0 |
+| B-Element, Ion, or Isotope | 0.737 | 0.669 | 0.702 | 381 |
+| I-Element, Ion, or Isotope | 0.714 | 0.622 | 0.665 | 225 |
+| B-Embryonic Structure | 0.595 | 0.472 | 0.526 | 53 |
+| I-Embryonic Structure | 0.545 | 0.486 | 0.514 | 37 |
+| B-Enzyme | 0.791 | 0.781 | 0.786 | 680 |
+| I-Enzyme | 0.823 | 0.779 | 0.801 | 1051 |
+| B-Eukaryote | 0.820 | 0.899 | 0.858 | 396 |
+| I-Eukaryote | 0.894 | 0.938 | 0.916 | 892 |
+| B-Experimental Model of Disease | 0.353 | 0.400 | 0.375 | 45 |
+| I-Experimental Model of Disease | 0.392 | 0.328 | 0.357 | 61 |
+| B-Finding | 0.449 | 0.424 | 0.436 | 2741 |
+| I-Finding | 0.353 | 0.323 | 0.337 | 2236 |
+| B-Fish | 0.857 | 0.947 | 0.900 | 19 |
+| I-Fish | 0.944 | 1.000 | 0.971 | 17 |
+| B-Food | 0.650 | 0.499 | 0.564 | 335 |
+| I-Food | 0.580 | 0.375 | 0.455 | 299 |
+| B-Fully Formed Anatomical Structure | 0.000 | 0.000 | 0.000 | 1 |
+| I-Fully Formed Anatomical Structure | 0.000 | 0.000 | 0.000 | 1 |
+| B-Fungus | 0.913 | 0.890 | 0.901 | 118 |
+| I-Fungus | 0.917 | 0.948 | 0.933 | 328 |
+| B-Gene or Genome | 0.653 | 0.629 | 0.641 | 911 |
+| I-Gene or Genome | 0.558 | 0.577 | 0.567 | 1329 |
+| B-Genetic Function | 0.659 | 0.683 | 0.671 | 652 |
+| I-Genetic Function | 0.523 | 0.372 | 0.435 | 489 |
+| B-Geographic Area | 0.727 | 0.763 | 0.744 | 594 |
+| I-Geographic Area | 0.672 | 0.698 | 0.685 | 557 |
+| B-Hazardous or Poisonous Substance | 0.589 | 0.570 | 0.579 | 291 |
+| I-Hazardous or Poisonous Substance | 0.427 | 0.527 | 0.472 | 300 |
+| B-Health Care Activity | 0.548 | 0.498 | 0.522 | 1055 |
+| I-Health Care Activity | 0.576 | 0.462 | 0.513 | 795 |
+| B-Health Care Related Organization | 0.582 | 0.682 | 0.628 | 296 |
+| I-Health Care Related Organization | 0.633 | 0.754 | 0.688 | 411 |
+| B-Hormone | 0.831 | 0.757 | 0.792 | 189 |
+| I-Hormone | 0.744 | 0.782 | 0.762 | 119 |
+| B-Human | 0.824 | 0.892 | 0.856 | 157 |
+| I-Human | 0.535 | 0.719 | 0.613 | 32 |
+| B-Idea or Concept | 0.000 | 0.000 | 0.000 | 1 |
+| I-Idea or Concept | 0.000 | 0.000 | 0.000 | 2 |
+| B-Immunologic Factor | 0.738 | 0.653 | 0.693 | 432 |
+| I-Immunologic Factor | 0.741 | 0.690 | 0.715 | 720 |
+| B-Indicator, Reagent, or Diagnostic Aid | 0.469 | 0.462 | 0.465 | 182 |
+| I-Indicator, Reagent, or Diagnostic Aid | 0.598 | 0.699 | 0.645 | 196 |
+| B-Injury or Poisoning | 0.672 | 0.749 | 0.708 | 347 |
+| I-Injury or Poisoning | 0.683 | 0.709 | 0.695 | 398 |
+| B-Inorganic Chemical | 0.656 | 0.707 | 0.680 | 256 |
+| I-Inorganic Chemical | 0.599 | 0.809 | 0.689 | 220 |
+| B-Intellectual Product | 0.547 | 0.517 | 0.532 | 2065 |
+| I-Intellectual Product | 0.552 | 0.541 | 0.547 | 2378 |
+| B-Laboratory Procedure | 0.538 | 0.513 | 0.525 | 907 |
+| I-Laboratory Procedure | 0.586 | 0.600 | 0.593 | 1390 |
+| B-Laboratory or Test Result | 0.283 | 0.255 | 0.268 | 110 |
+| I-Laboratory or Test Result | 0.189 | 0.174 | 0.181 | 132 |
+| B-Mammal | 0.832 | 0.859 | 0.845 | 455 |
+| I-Mammal | 0.649 | 0.633 | 0.641 | 278 |
+| B-Medical Device | 0.508 | 0.460 | 0.483 | 354 |
+| I-Medical Device | 0.451 | 0.470 | 0.460 | 566 |
+| B-Mental Process | 0.582 | 0.585 | 0.583 | 727 |
+| I-Mental Process | 0.419 | 0.545 | 0.474 | 299 |
+| B-Mental or Behavioral Dysfunction | 0.741 | 0.798 | 0.768 | 515 |
+| I-Mental or Behavioral Dysfunction | 0.670 | 0.670 | 0.670 | 412 |
+| B-Molecular Biology Research Technique | 0.570 | 0.576 | 0.573 | 205 |
+| I-Molecular Biology Research Technique | 0.621 | 0.598 | 0.609 | 353 |
+| B-Molecular Function | 0.583 | 0.610 | 0.597 | 716 |
+| I-Molecular Function | 0.526 | 0.526 | 0.526 | 741 |
+| B-Molecular Sequence | 0.455 | 0.556 | 0.500 | 9 |
+| I-Molecular Sequence | 0.600 | 0.833 | 0.698 | 18 |
+| B-Neoplastic Process | 0.807 | 0.778 | 0.792 | 914 |
+| I-Neoplastic Process | 0.792 | 0.772 | 0.782 | 1007 |
+| B-Nucleic Acid, Nucleoside, or Nucleotide | 0.394 | 0.495 | 0.439 | 109 |
+| I-Nucleic Acid, Nucleoside, or Nucleotide | 0.204 | 0.320 | 0.249 | 97 |
+| B-Nucleotide Sequence | 0.365 | 0.532 | 0.433 | 109 |
+| I-Nucleotide Sequence | 0.401 | 0.525 | 0.455 | 158 |
+| B-Organ or Tissue Function | 0.551 | 0.457 | 0.500 | 247 |
+| I-Organ or Tissue Function | 0.522 | 0.422 | 0.467 | 256 |
+| B-Organic Chemical | 0.442 | 0.491 | 0.465 | 509 |
+| I-Organic Chemical | 0.396 | 0.485 | 0.436 | 1003 |
+| B-Organism Function | 0.495 | 0.537 | 0.515 | 471 |
+| I-Organism Function | 0.314 | 0.338 | 0.326 | 207 |
+| B-Organization | 0.293 | 0.442 | 0.352 | 77 |
+| I-Organization | 0.327 | 0.467 | 0.385 | 75 |
+| B-Pathologic Function | 0.569 | 0.558 | 0.564 | 661 |
+| I-Pathologic Function | 0.517 | 0.469 | 0.492 | 512 |
+| B-Pharmacologic Substance | 0.613 | 0.659 | 0.635 | 1248 |
+| I-Pharmacologic Substance | 0.618 | 0.620 | 0.619 | 2121 |
+| B-Physiologic Function | 0.288 | 0.282 | 0.285 | 181 |
+| I-Physiologic Function | 0.393 | 0.315 | 0.350 | 111 |
+| B-Plant | 0.702 | 0.712 | 0.707 | 403 |
+| I-Plant | 0.650 | 0.763 | 0.702 | 528 |
+| B-Population Group | 0.752 | 0.738 | 0.745 | 1261 |
+| I-Population Group | 0.523 | 0.520 | 0.521 | 456 |
+| B-Professional Society | 0.000 | 0.000 | 0.000 | 7 |
+| I-Professional Society | 0.000 | 0.000 | 0.000 | 23 |
+| B-Professional or Occupational Group | 0.639 | 0.760 | 0.695 | 359 |
+| I-Professional or Occupational Group | 0.600 | 0.707 | 0.649 | 208 |
+| B-Receptor | 0.648 | 0.730 | 0.686 | 270 |
+| I-Receptor | 0.491 | 0.684 | 0.572 | 367 |
+| B-Regulation or Law | 0.273 | 0.188 | 0.222 | 16 |
+| I-Regulation or Law | 0.000 | 0.000 | 0.000 | 14 |
+| B-Reptile | 1.000 | 0.136 | 0.240 | 22 |
+| I-Reptile | 1.000 | 0.160 | 0.276 | 50 |
+| B-Research Activity | 0.607 | 0.572 | 0.589 | 1642 |
+| I-Research Activity | 0.658 | 0.619 | 0.638 | 1276 |
+| B-Self-help or Relief Organization | 0.000 | 0.000 | 0.000 | 2 |
+| I-Self-help or Relief Organization | 0.000 | 0.000 | 0.000 | 2 |
+| B-Sign or Symptom | 0.676 | 0.684 | 0.680 | 335 |
+| I-Sign or Symptom | 0.457 | 0.500 | 0.478 | 204 |
+| B-Spatial Concept | 0.504 | 0.503 | 0.503 | 1273 |
+| I-Spatial Concept | 0.480 | 0.512 | 0.495 | 512 |
+| B-Therapeutic or Preventive Procedure | 0.660 | 0.656 | 0.658 | 2017 |
+| I-Therapeutic or Preventive Procedure | 0.687 | 0.660 | 0.673 | 2012 |
+| B-Tissue | 0.633 | 0.559 | 0.593 | 256 |
+| I-Tissue | 0.620 | 0.444 | 0.517 | 239 |
+| B-Vertebrate | 1.000 | 1.000 | 1.000 | 1 |
+| I-Vertebrate | 0.000 | 0.000 | 0.000 | 1 |
+| B-Virus | 0.731 | 0.901 | 0.807 | 172 |
+| I-Virus | 0.723 | 0.923 | 0.811 | 209 |
+| B-Vitamin | 0.758 | 0.532 | 0.625 | 47 |
+| I-Vitamin | 0.582 | 0.561 | 0.571 | 57 |
+| macro avg | 0.525 | 0.527 | 0.519 | 270152 |
+| weighted avg | 0.842 | 0.841 | 0.841 | 270152 |
+
+

rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f0e2ec907ee89aeff3b80d487b543a220d4abdba877500be5e721c1d7460494e
+size 14244

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

trainer_state.json

@@ -0,0 +1,403 @@
+{
+  "best_metric": 0.5446548032758927,
+  "best_model_checkpoint": "tmp_ner_damsay_304130/run-36/checkpoint-4125",
+  "epoch": 25.0,
+  "eval_steps": 500,
+  "global_step": 4125,
+  "is_hyper_param_search": true,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 1.0,
+      "eval_accuracy": 0.7470146314639639,
+      "eval_loss": 1.2898669242858887,
+      "eval_macro_f1": 0.060475942039181926,
+      "eval_macro_precision": 0.09351501221756969,
+      "eval_macro_recall": 0.06591547068187953,
+      "eval_runtime": 3.9883,
+      "eval_samples_per_second": 220.142,
+      "eval_steps_per_second": 27.58,
+      "step": 165
+    },
+    {
+      "epoch": 2.0,
+      "eval_accuracy": 0.8084088190052983,
+      "eval_loss": 0.8025009632110596,
+      "eval_macro_f1": 0.24340420766720258,
+      "eval_macro_precision": 0.3139630221874961,
+      "eval_macro_recall": 0.24833685017157406,
+      "eval_runtime": 3.8063,
+      "eval_samples_per_second": 230.671,
+      "eval_steps_per_second": 28.9,
+      "step": 330
+    },
+    {
+      "epoch": 3.0,
+      "eval_accuracy": 0.8305009177175213,
+      "eval_loss": 0.6665005683898926,
+      "eval_macro_f1": 0.37091081081224914,
+      "eval_macro_precision": 0.4480773602522756,
+      "eval_macro_recall": 0.36393297647770273,
+      "eval_runtime": 3.7531,
+      "eval_samples_per_second": 233.938,
+      "eval_steps_per_second": 29.309,
+      "step": 495
+    },
+    {
+      "epoch": 3.0303030303030303,
+      "grad_norm": 1.6078789234161377,
+      "learning_rate": 7.735247090033069e-05,
+      "loss": 1.4056,
+      "step": 500
+    },
+    {
+      "epoch": 4.0,
+      "eval_accuracy": 0.8344504469693029,
+      "eval_loss": 0.6422820091247559,
+      "eval_macro_f1": 0.43071408069652817,
+      "eval_macro_precision": 0.48878382034336376,
+      "eval_macro_recall": 0.42148708788382616,
+      "eval_runtime": 3.8104,
+      "eval_samples_per_second": 230.421,
+      "eval_steps_per_second": 28.868,
+      "step": 660
+    },
+    {
+      "epoch": 5.0,
+      "eval_accuracy": 0.8393288401090858,
+      "eval_loss": 0.6265012621879578,
+      "eval_macro_f1": 0.47492811430942733,
+      "eval_macro_precision": 0.5136390951479204,
+      "eval_macro_recall": 0.46552778262584993,
+      "eval_runtime": 3.749,
+      "eval_samples_per_second": 234.199,
+      "eval_steps_per_second": 29.342,
+      "step": 825
+    },
+    {
+      "epoch": 6.0,
+      "eval_accuracy": 0.8411568442406724,
+      "eval_loss": 0.6596290469169617,
+      "eval_macro_f1": 0.4931684511043183,
+      "eval_macro_precision": 0.5359461122068587,
+      "eval_macro_recall": 0.48533559234357204,
+      "eval_runtime": 3.8138,
+      "eval_samples_per_second": 230.215,
+      "eval_steps_per_second": 28.842,
+      "step": 990
+    },
+    {
+      "epoch": 6.0606060606060606,
+      "grad_norm": 1.48198664188385,
+      "learning_rate": 6.926120825385258e-05,
+      "loss": 0.4174,
+      "step": 1000
+    },
+    {
+      "epoch": 7.0,
+      "eval_accuracy": 0.8377386251328275,
+      "eval_loss": 0.6801306009292603,
+      "eval_macro_f1": 0.5105261331060333,
+      "eval_macro_precision": 0.5298621091537271,
+      "eval_macro_recall": 0.5130394897405378,
+      "eval_runtime": 3.743,
+      "eval_samples_per_second": 234.574,
+      "eval_steps_per_second": 29.389,
+      "step": 1155
+    },
+    {
+      "epoch": 8.0,
+      "eval_accuracy": 0.8405512249856955,
+      "eval_loss": 0.7128350138664246,
+      "eval_macro_f1": 0.5145736070824514,
+      "eval_macro_precision": 0.5338232435701135,
+      "eval_macro_recall": 0.5148104452511267,
+      "eval_runtime": 3.8105,
+      "eval_samples_per_second": 230.418,
+      "eval_steps_per_second": 28.868,
+      "step": 1320
+    },
+    {
+      "epoch": 9.0,
+      "eval_accuracy": 0.8356208154681846,
+      "eval_loss": 0.7495563626289368,
+      "eval_macro_f1": 0.5078761754368146,
+      "eval_macro_precision": 0.5215628937215542,
+      "eval_macro_recall": 0.510509104208083,
+      "eval_runtime": 3.8142,
+      "eval_samples_per_second": 230.191,
+      "eval_steps_per_second": 28.839,
+      "step": 1485
+    },
+    {
+      "epoch": 9.090909090909092,
+      "grad_norm": 1.2219234704971313,
+      "learning_rate": 6.116994560737448e-05,
+      "loss": 0.207,
+      "step": 1500
+    },
+    {
+      "epoch": 10.0,
+      "eval_accuracy": 0.8404137531302713,
+      "eval_loss": 0.7915265560150146,
+      "eval_macro_f1": 0.5284150089162629,
+      "eval_macro_precision": 0.5652985249581868,
+      "eval_macro_recall": 0.5173389157983858,
+      "eval_runtime": 3.8115,
+      "eval_samples_per_second": 230.354,
+      "eval_steps_per_second": 28.86,
+      "step": 1650
+    },
+    {
+      "epoch": 11.0,
+      "eval_accuracy": 0.8365311020784258,
+      "eval_loss": 0.8357340097427368,
+      "eval_macro_f1": 0.5241371167696747,
+      "eval_macro_precision": 0.5418612467571973,
+      "eval_macro_recall": 0.5268409732620435,
+      "eval_runtime": 3.8104,
+      "eval_samples_per_second": 230.422,
+      "eval_steps_per_second": 28.868,
+      "step": 1815
+    },
+    {
+      "epoch": 12.0,
+      "eval_accuracy": 0.83948488924227,
+      "eval_loss": 0.860895574092865,
+      "eval_macro_f1": 0.5274542995604197,
+      "eval_macro_precision": 0.5642301005555995,
+      "eval_macro_recall": 0.5189445167674911,
+      "eval_runtime": 3.7927,
+      "eval_samples_per_second": 231.498,
+      "eval_steps_per_second": 29.003,
+      "step": 1980
+    },
+    {
+      "epoch": 12.121212121212121,
+      "grad_norm": 1.4894951581954956,
+      "learning_rate": 5.307868296089637e-05,
+      "loss": 0.1106,
+      "step": 2000
+    },
+    {
+      "epoch": 13.0,
+      "eval_accuracy": 0.837367079577627,
+      "eval_loss": 0.8975165486335754,
+      "eval_macro_f1": 0.5324199411604539,
+      "eval_macro_precision": 0.545792242605528,
+      "eval_macro_recall": 0.5377469764932828,
+      "eval_runtime": 3.7913,
+      "eval_samples_per_second": 231.585,
+      "eval_steps_per_second": 29.014,
+      "step": 2145
+    },
+    {
+      "epoch": 14.0,
+      "eval_accuracy": 0.831667570760851,
+      "eval_loss": 0.9421281218528748,
+      "eval_macro_f1": 0.5262611445373023,
+      "eval_macro_precision": 0.5347627543444692,
+      "eval_macro_recall": 0.5417363003088691,
+      "eval_runtime": 3.804,
+      "eval_samples_per_second": 230.807,
+      "eval_steps_per_second": 28.917,
+      "step": 2310
+    },
+    {
+      "epoch": 15.0,
+      "eval_accuracy": 0.8381064552324761,
+      "eval_loss": 0.9438627362251282,
+      "eval_macro_f1": 0.5348984394876934,
+      "eval_macro_precision": 0.546671824271427,
+      "eval_macro_recall": 0.5379298790499474,
+      "eval_runtime": 3.8089,
+      "eval_samples_per_second": 230.516,
+      "eval_steps_per_second": 28.88,
+      "step": 2475
+    },
+    {
+      "epoch": 15.151515151515152,
+      "grad_norm": 0.9380698800086975,
+      "learning_rate": 4.498742031441827e-05,
+      "loss": 0.0622,
+      "step": 2500
+    },
+    {
+      "epoch": 16.0,
+      "eval_accuracy": 0.835973783745625,
+      "eval_loss": 0.9927621483802795,
+      "eval_macro_f1": 0.5346150876236929,
+      "eval_macro_precision": 0.5412591223724303,
+      "eval_macro_recall": 0.5415646974890974,
+      "eval_runtime": 3.8119,
+      "eval_samples_per_second": 230.328,
+      "eval_steps_per_second": 28.857,
+      "step": 2640
+    },
+    {
+      "epoch": 17.0,
+      "eval_accuracy": 0.8385114398876446,
+      "eval_loss": 1.00917387008667,
+      "eval_macro_f1": 0.5375529028664181,
+      "eval_macro_precision": 0.5691165677941541,
+      "eval_macro_recall": 0.5349126486369854,
+      "eval_runtime": 3.8075,
+      "eval_samples_per_second": 230.599,
+      "eval_steps_per_second": 28.891,
+      "step": 2805
+    },
+    {
+      "epoch": 18.0,
+      "eval_accuracy": 0.8392285228091816,
+      "eval_loss": 1.0304529666900635,
+      "eval_macro_f1": 0.5373729743137114,
+      "eval_macro_precision": 0.5576924127069764,
+      "eval_macro_recall": 0.539334821981497,
+      "eval_runtime": 3.8101,
+      "eval_samples_per_second": 230.437,
+      "eval_steps_per_second": 28.87,
+      "step": 2970
+    },
+    {
+      "epoch": 18.181818181818183,
+      "grad_norm": 1.10272216796875,
+      "learning_rate": 3.689615766794016e-05,
+      "loss": 0.0365,
+      "step": 3000
+    },
+    {
+      "epoch": 19.0,
+      "eval_accuracy": 0.8382290652656922,
+      "eval_loss": 1.0565452575683594,
+      "eval_macro_f1": 0.5382182396542954,
+      "eval_macro_precision": 0.5617383574248755,
+      "eval_macro_recall": 0.5399115920507231,
+      "eval_runtime": 3.8119,
+      "eval_samples_per_second": 230.33,
+      "eval_steps_per_second": 28.857,
+      "step": 3135
+    },
+    {
+      "epoch": 20.0,
+      "eval_accuracy": 0.8372110304444428,
+      "eval_loss": 1.0832605361938477,
+      "eval_macro_f1": 0.5426999278930784,
+      "eval_macro_precision": 0.5539906546541987,
+      "eval_macro_recall": 0.5517989175520424,
+      "eval_runtime": 3.8083,
+      "eval_samples_per_second": 230.549,
+      "eval_steps_per_second": 28.884,
+      "step": 3300
+    },
+    {
+      "epoch": 21.0,
+      "eval_accuracy": 0.839499751064478,
+      "eval_loss": 1.0868653059005737,
+      "eval_macro_f1": 0.53917903306677,
+      "eval_macro_precision": 0.5749497192899481,
+      "eval_macro_recall": 0.5373986303762837,
+      "eval_runtime": 3.8141,
+      "eval_samples_per_second": 230.199,
+      "eval_steps_per_second": 28.84,
+      "step": 3465
+    },
+    {
+      "epoch": 21.21212121212121,
+      "grad_norm": 0.46956342458724976,
+      "learning_rate": 2.8804895021462055e-05,
+      "loss": 0.0224,
+      "step": 3500
+    },
+    {
+      "epoch": 22.0,
+      "eval_accuracy": 0.8391096282315175,
+      "eval_loss": 1.1133960485458374,
+      "eval_macro_f1": 0.5443083566401578,
+      "eval_macro_precision": 0.5641662348298077,
+      "eval_macro_recall": 0.5481264782508153,
+      "eval_runtime": 3.813,
+      "eval_samples_per_second": 230.266,
+      "eval_steps_per_second": 28.849,
+      "step": 3630
+    },
+    {
+      "epoch": 23.0,
+      "eval_accuracy": 0.838050723399196,
+      "eval_loss": 1.1350845098495483,
+      "eval_macro_f1": 0.544210275681084,
+      "eval_macro_precision": 0.5622845529634525,
+      "eval_macro_recall": 0.5482077780076627,
+      "eval_runtime": 3.8169,
+      "eval_samples_per_second": 230.032,
+      "eval_steps_per_second": 28.82,
+      "step": 3795
+    },
+    {
+      "epoch": 24.0,
+      "eval_accuracy": 0.838050723399196,
+      "eval_loss": 1.1593064069747925,
+      "eval_macro_f1": 0.5416925123679783,
+      "eval_macro_precision": 0.5680966862603153,
+      "eval_macro_recall": 0.5448186956610225,
+      "eval_runtime": 3.8224,
+      "eval_samples_per_second": 229.699,
+      "eval_steps_per_second": 28.778,
+      "step": 3960
+    },
+    {
+      "epoch": 24.242424242424242,
+      "grad_norm": 0.9432898759841919,
+      "learning_rate": 2.071363237498395e-05,
+      "loss": 0.0149,
+      "step": 4000
+    },
+    {
+      "epoch": 25.0,
+      "eval_accuracy": 0.8398712966196785,
+      "eval_loss": 1.1621172428131104,
+      "eval_macro_f1": 0.5446548032758927,
+      "eval_macro_precision": 0.572924667254898,
+      "eval_macro_recall": 0.5453082375999836,
+      "eval_runtime": 3.8202,
+      "eval_samples_per_second": 229.832,
+      "eval_steps_per_second": 28.794,
+      "step": 4125
+    }
+  ],
+  "logging_steps": 500,
+  "max_steps": 5280,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 32,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 3,
+        "early_stopping_threshold": 0.001
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 3
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.745821766419731e+16,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": {
+    "learning_rate": 7.86794379743531e-05,
+    "per_device_train_batch_size": 16,
+    "warmup_ratio": 0.07903396276412193,
+    "weight_decay": 0.06816454557507429
+  }
+}