Upload OrthoViTForImageClassification

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

config.json

@@ -0,0 +1,2038 @@
+{
+  "architectures": [
+    "OrthoViTForImageClassification"
+  ],
+  "attention_probs_dropout_prob": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_ortho_vit.OrthoViTConfig",
+    "AutoModelForImageClassification": "modeling_ortho_vit.OrthoViTForImageClassification"
+  },
+  "elementwise_affine_ln": false,
+  "encoder_stride": 16,
+  "hidden_act": "gelu_fast",
+  "hidden_dropout_prob": 0.0,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1",
+    "2": "LABEL_2",
+    "3": "LABEL_3",
+    "4": "LABEL_4",
+    "5": "LABEL_5",
+    "6": "LABEL_6",
+    "7": "LABEL_7",
+    "8": "LABEL_8",
+    "9": "LABEL_9",
+    "10": "LABEL_10",
+    "11": "LABEL_11",
+    "12": "LABEL_12",
+    "13": "LABEL_13",
+    "14": "LABEL_14",
+    "15": "LABEL_15",
+    "16": "LABEL_16",
+    "17": "LABEL_17",
+    "18": "LABEL_18",
+    "19": "LABEL_19",
+    "20": "LABEL_20",
+    "21": "LABEL_21",
+    "22": "LABEL_22",
+    "23": "LABEL_23",
+    "24": "LABEL_24",
+    "25": "LABEL_25",
+    "26": "LABEL_26",
+    "27": "LABEL_27",
+    "28": "LABEL_28",
+    "29": "LABEL_29",
+    "30": "LABEL_30",
+    "31": "LABEL_31",
+    "32": "LABEL_32",
+    "33": "LABEL_33",
+    "34": "LABEL_34",
+    "35": "LABEL_35",
+    "36": "LABEL_36",
+    "37": "LABEL_37",
+    "38": "LABEL_38",
+    "39": "LABEL_39",
+    "40": "LABEL_40",
+    "41": "LABEL_41",
+    "42": "LABEL_42",
+    "43": "LABEL_43",
+    "44": "LABEL_44",
+    "45": "LABEL_45",
+    "46": "LABEL_46",
+    "47": "LABEL_47",
+    "48": "LABEL_48",
+    "49": "LABEL_49",
+    "50": "LABEL_50",
+    "51": "LABEL_51",
+    "52": "LABEL_52",
+    "53": "LABEL_53",
+    "54": "LABEL_54",
+    "55": "LABEL_55",
+    "56": "LABEL_56",
+    "57": "LABEL_57",
+    "58": "LABEL_58",
+    "59": "LABEL_59",
+    "60": "LABEL_60",
+    "61": "LABEL_61",
+    "62": "LABEL_62",
+    "63": "LABEL_63",
+    "64": "LABEL_64",
+    "65": "LABEL_65",
+    "66": "LABEL_66",
+    "67": "LABEL_67",
+    "68": "LABEL_68",
+    "69": "LABEL_69",
+    "70": "LABEL_70",
+    "71": "LABEL_71",
+    "72": "LABEL_72",
+    "73": "LABEL_73",
+    "74": "LABEL_74",
+    "75": "LABEL_75",
+    "76": "LABEL_76",
+    "77": "LABEL_77",
+    "78": "LABEL_78",
+    "79": "LABEL_79",
+    "80": "LABEL_80",
+    "81": "LABEL_81",
+    "82": "LABEL_82",
+    "83": "LABEL_83",
+    "84": "LABEL_84",
+    "85": "LABEL_85",
+    "86": "LABEL_86",
+    "87": "LABEL_87",
+    "88": "LABEL_88",
+    "89": "LABEL_89",
+    "90": "LABEL_90",
+    "91": "LABEL_91",
+    "92": "LABEL_92",
+    "93": "LABEL_93",
+    "94": "LABEL_94",
+    "95": "LABEL_95",
+    "96": "LABEL_96",
+    "97": "LABEL_97",
+    "98": "LABEL_98",
+    "99": "LABEL_99",
+    "100": "LABEL_100",
+    "101": "LABEL_101",
+    "102": "LABEL_102",
+    "103": "LABEL_103",
+    "104": "LABEL_104",
+    "105": "LABEL_105",
+    "106": "LABEL_106",
+    "107": "LABEL_107",
+    "108": "LABEL_108",
+    "109": "LABEL_109",
+    "110": "LABEL_110",
+    "111": "LABEL_111",
+    "112": "LABEL_112",
+    "113": "LABEL_113",
+    "114": "LABEL_114",
+    "115": "LABEL_115",
+    "116": "LABEL_116",
+    "117": "LABEL_117",
+    "118": "LABEL_118",
+    "119": "LABEL_119",
+    "120": "LABEL_120",
+    "121": "LABEL_121",
+    "122": "LABEL_122",
+    "123": "LABEL_123",
+    "124": "LABEL_124",
+    "125": "LABEL_125",
+    "126": "LABEL_126",
+    "127": "LABEL_127",
+    "128": "LABEL_128",
+    "129": "LABEL_129",
+    "130": "LABEL_130",
+    "131": "LABEL_131",
+    "132": "LABEL_132",
+    "133": "LABEL_133",
+    "134": "LABEL_134",
+    "135": "LABEL_135",
+    "136": "LABEL_136",
+    "137": "LABEL_137",
+    "138": "LABEL_138",
+    "139": "LABEL_139",
+    "140": "LABEL_140",
+    "141": "LABEL_141",
+    "142": "LABEL_142",
+    "143": "LABEL_143",
+    "144": "LABEL_144",
+    "145": "LABEL_145",
+    "146": "LABEL_146",
+    "147": "LABEL_147",
+    "148": "LABEL_148",
+    "149": "LABEL_149",
+    "150": "LABEL_150",
+    "151": "LABEL_151",
+    "152": "LABEL_152",
+    "153": "LABEL_153",
+    "154": "LABEL_154",
+    "155": "LABEL_155",
+    "156": "LABEL_156",
+    "157": "LABEL_157",
+    "158": "LABEL_158",
+    "159": "LABEL_159",
+    "160": "LABEL_160",
+    "161": "LABEL_161",
+    "162": "LABEL_162",
+    "163": "LABEL_163",
+    "164": "LABEL_164",
+    "165": "LABEL_165",
+    "166": "LABEL_166",
+    "167": "LABEL_167",
+    "168": "LABEL_168",
+    "169": "LABEL_169",
+    "170": "LABEL_170",
+    "171": "LABEL_171",
+    "172": "LABEL_172",
+    "173": "LABEL_173",
+    "174": "LABEL_174",
+    "175": "LABEL_175",
+    "176": "LABEL_176",
+    "177": "LABEL_177",
+    "178": "LABEL_178",
+    "179": "LABEL_179",
+    "180": "LABEL_180",
+    "181": "LABEL_181",
+    "182": "LABEL_182",
+    "183": "LABEL_183",
+    "184": "LABEL_184",
+    "185": "LABEL_185",
+    "186": "LABEL_186",
+    "187": "LABEL_187",
+    "188": "LABEL_188",
+    "189": "LABEL_189",
+    "190": "LABEL_190",
+    "191": "LABEL_191",
+    "192": "LABEL_192",
+    "193": "LABEL_193",
+    "194": "LABEL_194",
+    "195": "LABEL_195",
+    "196": "LABEL_196",
+    "197": "LABEL_197",
+    "198": "LABEL_198",
+    "199": "LABEL_199",
+    "200": "LABEL_200",
+    "201": "LABEL_201",
+    "202": "LABEL_202",
+    "203": "LABEL_203",
+    "204": "LABEL_204",
+    "205": "LABEL_205",
+    "206": "LABEL_206",
+    "207": "LABEL_207",
+    "208": "LABEL_208",
+    "209": "LABEL_209",
+    "210": "LABEL_210",
+    "211": "LABEL_211",
+    "212": "LABEL_212",
+    "213": "LABEL_213",
+    "214": "LABEL_214",
+    "215": "LABEL_215",
+    "216": "LABEL_216",
+    "217": "LABEL_217",
+    "218": "LABEL_218",
+    "219": "LABEL_219",
+    "220": "LABEL_220",
+    "221": "LABEL_221",
+    "222": "LABEL_222",
+    "223": "LABEL_223",
+    "224": "LABEL_224",
+    "225": "LABEL_225",
+    "226": "LABEL_226",
+    "227": "LABEL_227",
+    "228": "LABEL_228",
+    "229": "LABEL_229",
+    "230": "LABEL_230",
+    "231": "LABEL_231",
+    "232": "LABEL_232",
+    "233": "LABEL_233",
+    "234": "LABEL_234",
+    "235": "LABEL_235",
+    "236": "LABEL_236",
+    "237": "LABEL_237",
+    "238": "LABEL_238",
+    "239": "LABEL_239",
+    "240": "LABEL_240",
+    "241": "LABEL_241",
+    "242": "LABEL_242",
+    "243": "LABEL_243",
+    "244": "LABEL_244",
+    "245": "LABEL_245",
+    "246": "LABEL_246",
+    "247": "LABEL_247",
+    "248": "LABEL_248",
+    "249": "LABEL_249",
+    "250": "LABEL_250",
+    "251": "LABEL_251",
+    "252": "LABEL_252",
+    "253": "LABEL_253",
+    "254": "LABEL_254",
+    "255": "LABEL_255",
+    "256": "LABEL_256",
+    "257": "LABEL_257",
+    "258": "LABEL_258",
+    "259": "LABEL_259",
+    "260": "LABEL_260",
+    "261": "LABEL_261",
+    "262": "LABEL_262",
+    "263": "LABEL_263",
+    "264": "LABEL_264",
+    "265": "LABEL_265",
+    "266": "LABEL_266",
+    "267": "LABEL_267",
+    "268": "LABEL_268",
+    "269": "LABEL_269",
+    "270": "LABEL_270",
+    "271": "LABEL_271",
+    "272": "LABEL_272",
+    "273": "LABEL_273",
+    "274": "LABEL_274",
+    "275": "LABEL_275",
+    "276": "LABEL_276",
+    "277": "LABEL_277",
+    "278": "LABEL_278",
+    "279": "LABEL_279",
+    "280": "LABEL_280",
+    "281": "LABEL_281",
+    "282": "LABEL_282",
+    "283": "LABEL_283",
+    "284": "LABEL_284",
+    "285": "LABEL_285",
+    "286": "LABEL_286",
+    "287": "LABEL_287",
+    "288": "LABEL_288",
+    "289": "LABEL_289",
+    "290": "LABEL_290",
+    "291": "LABEL_291",
+    "292": "LABEL_292",
+    "293": "LABEL_293",
+    "294": "LABEL_294",
+    "295": "LABEL_295",
+    "296": "LABEL_296",
+    "297": "LABEL_297",
+    "298": "LABEL_298",
+    "299": "LABEL_299",
+    "300": "LABEL_300",
+    "301": "LABEL_301",
+    "302": "LABEL_302",
+    "303": "LABEL_303",
+    "304": "LABEL_304",
+    "305": "LABEL_305",
+    "306": "LABEL_306",
+    "307": "LABEL_307",
+    "308": "LABEL_308",
+    "309": "LABEL_309",
+    "310": "LABEL_310",
+    "311": "LABEL_311",
+    "312": "LABEL_312",
+    "313": "LABEL_313",
+    "314": "LABEL_314",
+    "315": "LABEL_315",
+    "316": "LABEL_316",
+    "317": "LABEL_317",
+    "318": "LABEL_318",
+    "319": "LABEL_319",
+    "320": "LABEL_320",
+    "321": "LABEL_321",
+    "322": "LABEL_322",
+    "323": "LABEL_323",
+    "324": "LABEL_324",
+    "325": "LABEL_325",
+    "326": "LABEL_326",
+    "327": "LABEL_327",
+    "328": "LABEL_328",
+    "329": "LABEL_329",
+    "330": "LABEL_330",
+    "331": "LABEL_331",
+    "332": "LABEL_332",
+    "333": "LABEL_333",
+    "334": "LABEL_334",
+    "335": "LABEL_335",
+    "336": "LABEL_336",
+    "337": "LABEL_337",
+    "338": "LABEL_338",
+    "339": "LABEL_339",
+    "340": "LABEL_340",
+    "341": "LABEL_341",
+    "342": "LABEL_342",
+    "343": "LABEL_343",
+    "344": "LABEL_344",
+    "345": "LABEL_345",
+    "346": "LABEL_346",
+    "347": "LABEL_347",
+    "348": "LABEL_348",
+    "349": "LABEL_349",
+    "350": "LABEL_350",
+    "351": "LABEL_351",
+    "352": "LABEL_352",
+    "353": "LABEL_353",
+    "354": "LABEL_354",
+    "355": "LABEL_355",
+    "356": "LABEL_356",
+    "357": "LABEL_357",
+    "358": "LABEL_358",
+    "359": "LABEL_359",
+    "360": "LABEL_360",
+    "361": "LABEL_361",
+    "362": "LABEL_362",
+    "363": "LABEL_363",
+    "364": "LABEL_364",
+    "365": "LABEL_365",
+    "366": "LABEL_366",
+    "367": "LABEL_367",
+    "368": "LABEL_368",
+    "369": "LABEL_369",
+    "370": "LABEL_370",
+    "371": "LABEL_371",
+    "372": "LABEL_372",
+    "373": "LABEL_373",
+    "374": "LABEL_374",
+    "375": "LABEL_375",
+    "376": "LABEL_376",
+    "377": "LABEL_377",
+    "378": "LABEL_378",
+    "379": "LABEL_379",
+    "380": "LABEL_380",
+    "381": "LABEL_381",
+    "382": "LABEL_382",
+    "383": "LABEL_383",
+    "384": "LABEL_384",
+    "385": "LABEL_385",
+    "386": "LABEL_386",
+    "387": "LABEL_387",
+    "388": "LABEL_388",
+    "389": "LABEL_389",
+    "390": "LABEL_390",
+    "391": "LABEL_391",
+    "392": "LABEL_392",
+    "393": "LABEL_393",
+    "394": "LABEL_394",
+    "395": "LABEL_395",
+    "396": "LABEL_396",
+    "397": "LABEL_397",
+    "398": "LABEL_398",
+    "399": "LABEL_399",
+    "400": "LABEL_400",
+    "401": "LABEL_401",
+    "402": "LABEL_402",
+    "403": "LABEL_403",
+    "404": "LABEL_404",
+    "405": "LABEL_405",
+    "406": "LABEL_406",
+    "407": "LABEL_407",
+    "408": "LABEL_408",
+    "409": "LABEL_409",
+    "410": "LABEL_410",
+    "411": "LABEL_411",
+    "412": "LABEL_412",
+    "413": "LABEL_413",
+    "414": "LABEL_414",
+    "415": "LABEL_415",
+    "416": "LABEL_416",
+    "417": "LABEL_417",
+    "418": "LABEL_418",
+    "419": "LABEL_419",
+    "420": "LABEL_420",
+    "421": "LABEL_421",
+    "422": "LABEL_422",
+    "423": "LABEL_423",
+    "424": "LABEL_424",
+    "425": "LABEL_425",
+    "426": "LABEL_426",
+    "427": "LABEL_427",
+    "428": "LABEL_428",
+    "429": "LABEL_429",
+    "430": "LABEL_430",
+    "431": "LABEL_431",
+    "432": "LABEL_432",
+    "433": "LABEL_433",
+    "434": "LABEL_434",
+    "435": "LABEL_435",
+    "436": "LABEL_436",
+    "437": "LABEL_437",
+    "438": "LABEL_438",
+    "439": "LABEL_439",
+    "440": "LABEL_440",
+    "441": "LABEL_441",
+    "442": "LABEL_442",
+    "443": "LABEL_443",
+    "444": "LABEL_444",
+    "445": "LABEL_445",
+    "446": "LABEL_446",
+    "447": "LABEL_447",
+    "448": "LABEL_448",
+    "449": "LABEL_449",
+    "450": "LABEL_450",
+    "451": "LABEL_451",
+    "452": "LABEL_452",
+    "453": "LABEL_453",
+    "454": "LABEL_454",
+    "455": "LABEL_455",
+    "456": "LABEL_456",
+    "457": "LABEL_457",
+    "458": "LABEL_458",
+    "459": "LABEL_459",
+    "460": "LABEL_460",
+    "461": "LABEL_461",
+    "462": "LABEL_462",
+    "463": "LABEL_463",
+    "464": "LABEL_464",
+    "465": "LABEL_465",
+    "466": "LABEL_466",
+    "467": "LABEL_467",
+    "468": "LABEL_468",
+    "469": "LABEL_469",
+    "470": "LABEL_470",
+    "471": "LABEL_471",
+    "472": "LABEL_472",
+    "473": "LABEL_473",
+    "474": "LABEL_474",
+    "475": "LABEL_475",
+    "476": "LABEL_476",
+    "477": "LABEL_477",
+    "478": "LABEL_478",
+    "479": "LABEL_479",
+    "480": "LABEL_480",
+    "481": "LABEL_481",
+    "482": "LABEL_482",
+    "483": "LABEL_483",
+    "484": "LABEL_484",
+    "485": "LABEL_485",
+    "486": "LABEL_486",
+    "487": "LABEL_487",
+    "488": "LABEL_488",
+    "489": "LABEL_489",
+    "490": "LABEL_490",
+    "491": "LABEL_491",
+    "492": "LABEL_492",
+    "493": "LABEL_493",
+    "494": "LABEL_494",
+    "495": "LABEL_495",
+    "496": "LABEL_496",
+    "497": "LABEL_497",
+    "498": "LABEL_498",
+    "499": "LABEL_499",
+    "500": "LABEL_500",
+    "501": "LABEL_501",
+    "502": "LABEL_502",
+    "503": "LABEL_503",
+    "504": "LABEL_504",
+    "505": "LABEL_505",
+    "506": "LABEL_506",
+    "507": "LABEL_507",
+    "508": "LABEL_508",
+    "509": "LABEL_509",
+    "510": "LABEL_510",
+    "511": "LABEL_511",
+    "512": "LABEL_512",
+    "513": "LABEL_513",
+    "514": "LABEL_514",
+    "515": "LABEL_515",
+    "516": "LABEL_516",
+    "517": "LABEL_517",
+    "518": "LABEL_518",
+    "519": "LABEL_519",
+    "520": "LABEL_520",
+    "521": "LABEL_521",
+    "522": "LABEL_522",
+    "523": "LABEL_523",
+    "524": "LABEL_524",
+    "525": "LABEL_525",
+    "526": "LABEL_526",
+    "527": "LABEL_527",
+    "528": "LABEL_528",
+    "529": "LABEL_529",
+    "530": "LABEL_530",
+    "531": "LABEL_531",
+    "532": "LABEL_532",
+    "533": "LABEL_533",
+    "534": "LABEL_534",
+    "535": "LABEL_535",
+    "536": "LABEL_536",
+    "537": "LABEL_537",
+    "538": "LABEL_538",
+    "539": "LABEL_539",
+    "540": "LABEL_540",
+    "541": "LABEL_541",
+    "542": "LABEL_542",
+    "543": "LABEL_543",
+    "544": "LABEL_544",
+    "545": "LABEL_545",
+    "546": "LABEL_546",
+    "547": "LABEL_547",
+    "548": "LABEL_548",
+    "549": "LABEL_549",
+    "550": "LABEL_550",
+    "551": "LABEL_551",
+    "552": "LABEL_552",
+    "553": "LABEL_553",
+    "554": "LABEL_554",
+    "555": "LABEL_555",
+    "556": "LABEL_556",
+    "557": "LABEL_557",
+    "558": "LABEL_558",
+    "559": "LABEL_559",
+    "560": "LABEL_560",
+    "561": "LABEL_561",
+    "562": "LABEL_562",
+    "563": "LABEL_563",
+    "564": "LABEL_564",
+    "565": "LABEL_565",
+    "566": "LABEL_566",
+    "567": "LABEL_567",
+    "568": "LABEL_568",
+    "569": "LABEL_569",
+    "570": "LABEL_570",
+    "571": "LABEL_571",
+    "572": "LABEL_572",
+    "573": "LABEL_573",
+    "574": "LABEL_574",
+    "575": "LABEL_575",
+    "576": "LABEL_576",
+    "577": "LABEL_577",
+    "578": "LABEL_578",
+    "579": "LABEL_579",
+    "580": "LABEL_580",
+    "581": "LABEL_581",
+    "582": "LABEL_582",
+    "583": "LABEL_583",
+    "584": "LABEL_584",
+    "585": "LABEL_585",
+    "586": "LABEL_586",
+    "587": "LABEL_587",
+    "588": "LABEL_588",
+    "589": "LABEL_589",
+    "590": "LABEL_590",
+    "591": "LABEL_591",
+    "592": "LABEL_592",
+    "593": "LABEL_593",
+    "594": "LABEL_594",
+    "595": "LABEL_595",
+    "596": "LABEL_596",
+    "597": "LABEL_597",
+    "598": "LABEL_598",
+    "599": "LABEL_599",
+    "600": "LABEL_600",
+    "601": "LABEL_601",
+    "602": "LABEL_602",
+    "603": "LABEL_603",
+    "604": "LABEL_604",
+    "605": "LABEL_605",
+    "606": "LABEL_606",
+    "607": "LABEL_607",
+    "608": "LABEL_608",
+    "609": "LABEL_609",
+    "610": "LABEL_610",
+    "611": "LABEL_611",
+    "612": "LABEL_612",
+    "613": "LABEL_613",
+    "614": "LABEL_614",
+    "615": "LABEL_615",
+    "616": "LABEL_616",
+    "617": "LABEL_617",
+    "618": "LABEL_618",
+    "619": "LABEL_619",
+    "620": "LABEL_620",
+    "621": "LABEL_621",
+    "622": "LABEL_622",
+    "623": "LABEL_623",
+    "624": "LABEL_624",
+    "625": "LABEL_625",
+    "626": "LABEL_626",
+    "627": "LABEL_627",
+    "628": "LABEL_628",
+    "629": "LABEL_629",
+    "630": "LABEL_630",
+    "631": "LABEL_631",
+    "632": "LABEL_632",
+    "633": "LABEL_633",
+    "634": "LABEL_634",
+    "635": "LABEL_635",
+    "636": "LABEL_636",
+    "637": "LABEL_637",
+    "638": "LABEL_638",
+    "639": "LABEL_639",
+    "640": "LABEL_640",
+    "641": "LABEL_641",
+    "642": "LABEL_642",
+    "643": "LABEL_643",
+    "644": "LABEL_644",
+    "645": "LABEL_645",
+    "646": "LABEL_646",
+    "647": "LABEL_647",
+    "648": "LABEL_648",
+    "649": "LABEL_649",
+    "650": "LABEL_650",
+    "651": "LABEL_651",
+    "652": "LABEL_652",
+    "653": "LABEL_653",
+    "654": "LABEL_654",
+    "655": "LABEL_655",
+    "656": "LABEL_656",
+    "657": "LABEL_657",
+    "658": "LABEL_658",
+    "659": "LABEL_659",
+    "660": "LABEL_660",
+    "661": "LABEL_661",
+    "662": "LABEL_662",
+    "663": "LABEL_663",
+    "664": "LABEL_664",
+    "665": "LABEL_665",
+    "666": "LABEL_666",
+    "667": "LABEL_667",
+    "668": "LABEL_668",
+    "669": "LABEL_669",
+    "670": "LABEL_670",
+    "671": "LABEL_671",
+    "672": "LABEL_672",
+    "673": "LABEL_673",
+    "674": "LABEL_674",
+    "675": "LABEL_675",
+    "676": "LABEL_676",
+    "677": "LABEL_677",
+    "678": "LABEL_678",
+    "679": "LABEL_679",
+    "680": "LABEL_680",
+    "681": "LABEL_681",
+    "682": "LABEL_682",
+    "683": "LABEL_683",
+    "684": "LABEL_684",
+    "685": "LABEL_685",
+    "686": "LABEL_686",
+    "687": "LABEL_687",
+    "688": "LABEL_688",
+    "689": "LABEL_689",
+    "690": "LABEL_690",
+    "691": "LABEL_691",
+    "692": "LABEL_692",
+    "693": "LABEL_693",
+    "694": "LABEL_694",
+    "695": "LABEL_695",
+    "696": "LABEL_696",
+    "697": "LABEL_697",
+    "698": "LABEL_698",
+    "699": "LABEL_699",
+    "700": "LABEL_700",
+    "701": "LABEL_701",
+    "702": "LABEL_702",
+    "703": "LABEL_703",
+    "704": "LABEL_704",
+    "705": "LABEL_705",
+    "706": "LABEL_706",
+    "707": "LABEL_707",
+    "708": "LABEL_708",
+    "709": "LABEL_709",
+    "710": "LABEL_710",
+    "711": "LABEL_711",
+    "712": "LABEL_712",
+    "713": "LABEL_713",
+    "714": "LABEL_714",
+    "715": "LABEL_715",
+    "716": "LABEL_716",
+    "717": "LABEL_717",
+    "718": "LABEL_718",
+    "719": "LABEL_719",
+    "720": "LABEL_720",
+    "721": "LABEL_721",
+    "722": "LABEL_722",
+    "723": "LABEL_723",
+    "724": "LABEL_724",
+    "725": "LABEL_725",
+    "726": "LABEL_726",
+    "727": "LABEL_727",
+    "728": "LABEL_728",
+    "729": "LABEL_729",
+    "730": "LABEL_730",
+    "731": "LABEL_731",
+    "732": "LABEL_732",
+    "733": "LABEL_733",
+    "734": "LABEL_734",
+    "735": "LABEL_735",
+    "736": "LABEL_736",
+    "737": "LABEL_737",
+    "738": "LABEL_738",
+    "739": "LABEL_739",
+    "740": "LABEL_740",
+    "741": "LABEL_741",
+    "742": "LABEL_742",
+    "743": "LABEL_743",
+    "744": "LABEL_744",
+    "745": "LABEL_745",
+    "746": "LABEL_746",
+    "747": "LABEL_747",
+    "748": "LABEL_748",
+    "749": "LABEL_749",
+    "750": "LABEL_750",
+    "751": "LABEL_751",
+    "752": "LABEL_752",
+    "753": "LABEL_753",
+    "754": "LABEL_754",
+    "755": "LABEL_755",
+    "756": "LABEL_756",
+    "757": "LABEL_757",
+    "758": "LABEL_758",
+    "759": "LABEL_759",
+    "760": "LABEL_760",
+    "761": "LABEL_761",
+    "762": "LABEL_762",
+    "763": "LABEL_763",
+    "764": "LABEL_764",
+    "765": "LABEL_765",
+    "766": "LABEL_766",
+    "767": "LABEL_767",
+    "768": "LABEL_768",
+    "769": "LABEL_769",
+    "770": "LABEL_770",
+    "771": "LABEL_771",
+    "772": "LABEL_772",
+    "773": "LABEL_773",
+    "774": "LABEL_774",
+    "775": "LABEL_775",
+    "776": "LABEL_776",
+    "777": "LABEL_777",
+    "778": "LABEL_778",
+    "779": "LABEL_779",
+    "780": "LABEL_780",
+    "781": "LABEL_781",
+    "782": "LABEL_782",
+    "783": "LABEL_783",
+    "784": "LABEL_784",
+    "785": "LABEL_785",
+    "786": "LABEL_786",
+    "787": "LABEL_787",
+    "788": "LABEL_788",
+    "789": "LABEL_789",
+    "790": "LABEL_790",
+    "791": "LABEL_791",
+    "792": "LABEL_792",
+    "793": "LABEL_793",
+    "794": "LABEL_794",
+    "795": "LABEL_795",
+    "796": "LABEL_796",
+    "797": "LABEL_797",
+    "798": "LABEL_798",
+    "799": "LABEL_799",
+    "800": "LABEL_800",
+    "801": "LABEL_801",
+    "802": "LABEL_802",
+    "803": "LABEL_803",
+    "804": "LABEL_804",
+    "805": "LABEL_805",
+    "806": "LABEL_806",
+    "807": "LABEL_807",
+    "808": "LABEL_808",
+    "809": "LABEL_809",
+    "810": "LABEL_810",
+    "811": "LABEL_811",
+    "812": "LABEL_812",
+    "813": "LABEL_813",
+    "814": "LABEL_814",
+    "815": "LABEL_815",
+    "816": "LABEL_816",
+    "817": "LABEL_817",
+    "818": "LABEL_818",
+    "819": "LABEL_819",
+    "820": "LABEL_820",
+    "821": "LABEL_821",
+    "822": "LABEL_822",
+    "823": "LABEL_823",
+    "824": "LABEL_824",
+    "825": "LABEL_825",
+    "826": "LABEL_826",
+    "827": "LABEL_827",
+    "828": "LABEL_828",
+    "829": "LABEL_829",
+    "830": "LABEL_830",
+    "831": "LABEL_831",
+    "832": "LABEL_832",
+    "833": "LABEL_833",
+    "834": "LABEL_834",
+    "835": "LABEL_835",
+    "836": "LABEL_836",
+    "837": "LABEL_837",
+    "838": "LABEL_838",
+    "839": "LABEL_839",
+    "840": "LABEL_840",
+    "841": "LABEL_841",
+    "842": "LABEL_842",
+    "843": "LABEL_843",
+    "844": "LABEL_844",
+    "845": "LABEL_845",
+    "846": "LABEL_846",
+    "847": "LABEL_847",
+    "848": "LABEL_848",
+    "849": "LABEL_849",
+    "850": "LABEL_850",
+    "851": "LABEL_851",
+    "852": "LABEL_852",
+    "853": "LABEL_853",
+    "854": "LABEL_854",
+    "855": "LABEL_855",
+    "856": "LABEL_856",
+    "857": "LABEL_857",
+    "858": "LABEL_858",
+    "859": "LABEL_859",
+    "860": "LABEL_860",
+    "861": "LABEL_861",
+    "862": "LABEL_862",
+    "863": "LABEL_863",
+    "864": "LABEL_864",
+    "865": "LABEL_865",
+    "866": "LABEL_866",
+    "867": "LABEL_867",
+    "868": "LABEL_868",
+    "869": "LABEL_869",
+    "870": "LABEL_870",
+    "871": "LABEL_871",
+    "872": "LABEL_872",
+    "873": "LABEL_873",
+    "874": "LABEL_874",
+    "875": "LABEL_875",
+    "876": "LABEL_876",
+    "877": "LABEL_877",
+    "878": "LABEL_878",
+    "879": "LABEL_879",
+    "880": "LABEL_880",
+    "881": "LABEL_881",
+    "882": "LABEL_882",
+    "883": "LABEL_883",
+    "884": "LABEL_884",
+    "885": "LABEL_885",
+    "886": "LABEL_886",
+    "887": "LABEL_887",
+    "888": "LABEL_888",
+    "889": "LABEL_889",
+    "890": "LABEL_890",
+    "891": "LABEL_891",
+    "892": "LABEL_892",
+    "893": "LABEL_893",
+    "894": "LABEL_894",
+    "895": "LABEL_895",
+    "896": "LABEL_896",
+    "897": "LABEL_897",
+    "898": "LABEL_898",
+    "899": "LABEL_899",
+    "900": "LABEL_900",
+    "901": "LABEL_901",
+    "902": "LABEL_902",
+    "903": "LABEL_903",
+    "904": "LABEL_904",
+    "905": "LABEL_905",
+    "906": "LABEL_906",
+    "907": "LABEL_907",
+    "908": "LABEL_908",
+    "909": "LABEL_909",
+    "910": "LABEL_910",
+    "911": "LABEL_911",
+    "912": "LABEL_912",
+    "913": "LABEL_913",
+    "914": "LABEL_914",
+    "915": "LABEL_915",
+    "916": "LABEL_916",
+    "917": "LABEL_917",
+    "918": "LABEL_918",
+    "919": "LABEL_919",
+    "920": "LABEL_920",
+    "921": "LABEL_921",
+    "922": "LABEL_922",
+    "923": "LABEL_923",
+    "924": "LABEL_924",
+    "925": "LABEL_925",
+    "926": "LABEL_926",
+    "927": "LABEL_927",
+    "928": "LABEL_928",
+    "929": "LABEL_929",
+    "930": "LABEL_930",
+    "931": "LABEL_931",
+    "932": "LABEL_932",
+    "933": "LABEL_933",
+    "934": "LABEL_934",
+    "935": "LABEL_935",
+    "936": "LABEL_936",
+    "937": "LABEL_937",
+    "938": "LABEL_938",
+    "939": "LABEL_939",
+    "940": "LABEL_940",
+    "941": "LABEL_941",
+    "942": "LABEL_942",
+    "943": "LABEL_943",
+    "944": "LABEL_944",
+    "945": "LABEL_945",
+    "946": "LABEL_946",
+    "947": "LABEL_947",
+    "948": "LABEL_948",
+    "949": "LABEL_949",
+    "950": "LABEL_950",
+    "951": "LABEL_951",
+    "952": "LABEL_952",
+    "953": "LABEL_953",
+    "954": "LABEL_954",
+    "955": "LABEL_955",
+    "956": "LABEL_956",
+    "957": "LABEL_957",
+    "958": "LABEL_958",
+    "959": "LABEL_959",
+    "960": "LABEL_960",
+    "961": "LABEL_961",
+    "962": "LABEL_962",
+    "963": "LABEL_963",
+    "964": "LABEL_964",
+    "965": "LABEL_965",
+    "966": "LABEL_966",
+    "967": "LABEL_967",
+    "968": "LABEL_968",
+    "969": "LABEL_969",
+    "970": "LABEL_970",
+    "971": "LABEL_971",
+    "972": "LABEL_972",
+    "973": "LABEL_973",
+    "974": "LABEL_974",
+    "975": "LABEL_975",
+    "976": "LABEL_976",
+    "977": "LABEL_977",
+    "978": "LABEL_978",
+    "979": "LABEL_979",
+    "980": "LABEL_980",
+    "981": "LABEL_981",
+    "982": "LABEL_982",
+    "983": "LABEL_983",
+    "984": "LABEL_984",
+    "985": "LABEL_985",
+    "986": "LABEL_986",
+    "987": "LABEL_987",
+    "988": "LABEL_988",
+    "989": "LABEL_989",
+    "990": "LABEL_990",
+    "991": "LABEL_991",
+    "992": "LABEL_992",
+    "993": "LABEL_993",
+    "994": "LABEL_994",
+    "995": "LABEL_995",
+    "996": "LABEL_996",
+    "997": "LABEL_997",
+    "998": "LABEL_998",
+    "999": "LABEL_999"
+  },
+  "image_size": 224,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1,
+    "LABEL_10": 10,
+    "LABEL_100": 100,
+    "LABEL_101": 101,
+    "LABEL_102": 102,
+    "LABEL_103": 103,
+    "LABEL_104": 104,
+    "LABEL_105": 105,
+    "LABEL_106": 106,
+    "LABEL_107": 107,
+    "LABEL_108": 108,
+    "LABEL_109": 109,
+    "LABEL_11": 11,
+    "LABEL_110": 110,
+    "LABEL_111": 111,
+    "LABEL_112": 112,
+    "LABEL_113": 113,
+    "LABEL_114": 114,
+    "LABEL_115": 115,
+    "LABEL_116": 116,
+    "LABEL_117": 117,
+    "LABEL_118": 118,
+    "LABEL_119": 119,
+    "LABEL_12": 12,
+    "LABEL_120": 120,
+    "LABEL_121": 121,
+    "LABEL_122": 122,
+    "LABEL_123": 123,
+    "LABEL_124": 124,
+    "LABEL_125": 125,
+    "LABEL_126": 126,
+    "LABEL_127": 127,
+    "LABEL_128": 128,
+    "LABEL_129": 129,
+    "LABEL_13": 13,
+    "LABEL_130": 130,
+    "LABEL_131": 131,
+    "LABEL_132": 132,
+    "LABEL_133": 133,
+    "LABEL_134": 134,
+    "LABEL_135": 135,
+    "LABEL_136": 136,
+    "LABEL_137": 137,
+    "LABEL_138": 138,
+    "LABEL_139": 139,
+    "LABEL_14": 14,
+    "LABEL_140": 140,
+    "LABEL_141": 141,
+    "LABEL_142": 142,
+    "LABEL_143": 143,
+    "LABEL_144": 144,
+    "LABEL_145": 145,
+    "LABEL_146": 146,
+    "LABEL_147": 147,
+    "LABEL_148": 148,
+    "LABEL_149": 149,
+    "LABEL_15": 15,
+    "LABEL_150": 150,
+    "LABEL_151": 151,
+    "LABEL_152": 152,
+    "LABEL_153": 153,
+    "LABEL_154": 154,
+    "LABEL_155": 155,
+    "LABEL_156": 156,
+    "LABEL_157": 157,
+    "LABEL_158": 158,
+    "LABEL_159": 159,
+    "LABEL_16": 16,
+    "LABEL_160": 160,
+    "LABEL_161": 161,
+    "LABEL_162": 162,
+    "LABEL_163": 163,
+    "LABEL_164": 164,
+    "LABEL_165": 165,
+    "LABEL_166": 166,
+    "LABEL_167": 167,
+    "LABEL_168": 168,
+    "LABEL_169": 169,
+    "LABEL_17": 17,
+    "LABEL_170": 170,
+    "LABEL_171": 171,
+    "LABEL_172": 172,
+    "LABEL_173": 173,
+    "LABEL_174": 174,
+    "LABEL_175": 175,
+    "LABEL_176": 176,
+    "LABEL_177": 177,
+    "LABEL_178": 178,
+    "LABEL_179": 179,
+    "LABEL_18": 18,
+    "LABEL_180": 180,
+    "LABEL_181": 181,
+    "LABEL_182": 182,
+    "LABEL_183": 183,
+    "LABEL_184": 184,
+    "LABEL_185": 185,
+    "LABEL_186": 186,
+    "LABEL_187": 187,
+    "LABEL_188": 188,
+    "LABEL_189": 189,
+    "LABEL_19": 19,
+    "LABEL_190": 190,
+    "LABEL_191": 191,
+    "LABEL_192": 192,
+    "LABEL_193": 193,
+    "LABEL_194": 194,
+    "LABEL_195": 195,
+    "LABEL_196": 196,
+    "LABEL_197": 197,
+    "LABEL_198": 198,
+    "LABEL_199": 199,
+    "LABEL_2": 2,
+    "LABEL_20": 20,
+    "LABEL_200": 200,
+    "LABEL_201": 201,
+    "LABEL_202": 202,
+    "LABEL_203": 203,
+    "LABEL_204": 204,
+    "LABEL_205": 205,
+    "LABEL_206": 206,
+    "LABEL_207": 207,
+    "LABEL_208": 208,
+    "LABEL_209": 209,
+    "LABEL_21": 21,
+    "LABEL_210": 210,
+    "LABEL_211": 211,
+    "LABEL_212": 212,
+    "LABEL_213": 213,
+    "LABEL_214": 214,
+    "LABEL_215": 215,
+    "LABEL_216": 216,
+    "LABEL_217": 217,
+    "LABEL_218": 218,
+    "LABEL_219": 219,
+    "LABEL_22": 22,
+    "LABEL_220": 220,
+    "LABEL_221": 221,
+    "LABEL_222": 222,
+    "LABEL_223": 223,
+    "LABEL_224": 224,
+    "LABEL_225": 225,
+    "LABEL_226": 226,
+    "LABEL_227": 227,
+    "LABEL_228": 228,
+    "LABEL_229": 229,
+    "LABEL_23": 23,
+    "LABEL_230": 230,
+    "LABEL_231": 231,
+    "LABEL_232": 232,
+    "LABEL_233": 233,
+    "LABEL_234": 234,
+    "LABEL_235": 235,
+    "LABEL_236": 236,
+    "LABEL_237": 237,
+    "LABEL_238": 238,
+    "LABEL_239": 239,
+    "LABEL_24": 24,
+    "LABEL_240": 240,
+    "LABEL_241": 241,
+    "LABEL_242": 242,
+    "LABEL_243": 243,
+    "LABEL_244": 244,
+    "LABEL_245": 245,
+    "LABEL_246": 246,
+    "LABEL_247": 247,
+    "LABEL_248": 248,
+    "LABEL_249": 249,
+    "LABEL_25": 25,
+    "LABEL_250": 250,
+    "LABEL_251": 251,
+    "LABEL_252": 252,
+    "LABEL_253": 253,
+    "LABEL_254": 254,
+    "LABEL_255": 255,
+    "LABEL_256": 256,
+    "LABEL_257": 257,
+    "LABEL_258": 258,
+    "LABEL_259": 259,
+    "LABEL_26": 26,
+    "LABEL_260": 260,
+    "LABEL_261": 261,
+    "LABEL_262": 262,
+    "LABEL_263": 263,
+    "LABEL_264": 264,
+    "LABEL_265": 265,
+    "LABEL_266": 266,
+    "LABEL_267": 267,
+    "LABEL_268": 268,
+    "LABEL_269": 269,
+    "LABEL_27": 27,
+    "LABEL_270": 270,
+    "LABEL_271": 271,
+    "LABEL_272": 272,
+    "LABEL_273": 273,
+    "LABEL_274": 274,
+    "LABEL_275": 275,
+    "LABEL_276": 276,
+    "LABEL_277": 277,
+    "LABEL_278": 278,
+    "LABEL_279": 279,
+    "LABEL_28": 28,
+    "LABEL_280": 280,
+    "LABEL_281": 281,
+    "LABEL_282": 282,
+    "LABEL_283": 283,
+    "LABEL_284": 284,
+    "LABEL_285": 285,
+    "LABEL_286": 286,
+    "LABEL_287": 287,
+    "LABEL_288": 288,
+    "LABEL_289": 289,
+    "LABEL_29": 29,
+    "LABEL_290": 290,
+    "LABEL_291": 291,
+    "LABEL_292": 292,
+    "LABEL_293": 293,
+    "LABEL_294": 294,
+    "LABEL_295": 295,
+    "LABEL_296": 296,
+    "LABEL_297": 297,
+    "LABEL_298": 298,
+    "LABEL_299": 299,
+    "LABEL_3": 3,
+    "LABEL_30": 30,
+    "LABEL_300": 300,
+    "LABEL_301": 301,
+    "LABEL_302": 302,
+    "LABEL_303": 303,
+    "LABEL_304": 304,
+    "LABEL_305": 305,
+    "LABEL_306": 306,
+    "LABEL_307": 307,
+    "LABEL_308": 308,
+    "LABEL_309": 309,
+    "LABEL_31": 31,
+    "LABEL_310": 310,
+    "LABEL_311": 311,
+    "LABEL_312": 312,
+    "LABEL_313": 313,
+    "LABEL_314": 314,
+    "LABEL_315": 315,
+    "LABEL_316": 316,
+    "LABEL_317": 317,
+    "LABEL_318": 318,
+    "LABEL_319": 319,
+    "LABEL_32": 32,
+    "LABEL_320": 320,
+    "LABEL_321": 321,
+    "LABEL_322": 322,
+    "LABEL_323": 323,
+    "LABEL_324": 324,
+    "LABEL_325": 325,
+    "LABEL_326": 326,
+    "LABEL_327": 327,
+    "LABEL_328": 328,
+    "LABEL_329": 329,
+    "LABEL_33": 33,
+    "LABEL_330": 330,
+    "LABEL_331": 331,
+    "LABEL_332": 332,
+    "LABEL_333": 333,
+    "LABEL_334": 334,
+    "LABEL_335": 335,
+    "LABEL_336": 336,
+    "LABEL_337": 337,
+    "LABEL_338": 338,
+    "LABEL_339": 339,
+    "LABEL_34": 34,
+    "LABEL_340": 340,
+    "LABEL_341": 341,
+    "LABEL_342": 342,
+    "LABEL_343": 343,
+    "LABEL_344": 344,
+    "LABEL_345": 345,
+    "LABEL_346": 346,
+    "LABEL_347": 347,
+    "LABEL_348": 348,
+    "LABEL_349": 349,
+    "LABEL_35": 35,
+    "LABEL_350": 350,
+    "LABEL_351": 351,
+    "LABEL_352": 352,
+    "LABEL_353": 353,
+    "LABEL_354": 354,
+    "LABEL_355": 355,
+    "LABEL_356": 356,
+    "LABEL_357": 357,
+    "LABEL_358": 358,
+    "LABEL_359": 359,
+    "LABEL_36": 36,
+    "LABEL_360": 360,
+    "LABEL_361": 361,
+    "LABEL_362": 362,
+    "LABEL_363": 363,
+    "LABEL_364": 364,
+    "LABEL_365": 365,
+    "LABEL_366": 366,
+    "LABEL_367": 367,
+    "LABEL_368": 368,
+    "LABEL_369": 369,
+    "LABEL_37": 37,
+    "LABEL_370": 370,
+    "LABEL_371": 371,
+    "LABEL_372": 372,
+    "LABEL_373": 373,
+    "LABEL_374": 374,
+    "LABEL_375": 375,
+    "LABEL_376": 376,
+    "LABEL_377": 377,
+    "LABEL_378": 378,
+    "LABEL_379": 379,
+    "LABEL_38": 38,
+    "LABEL_380": 380,
+    "LABEL_381": 381,
+    "LABEL_382": 382,
+    "LABEL_383": 383,
+    "LABEL_384": 384,
+    "LABEL_385": 385,
+    "LABEL_386": 386,
+    "LABEL_387": 387,
+    "LABEL_388": 388,
+    "LABEL_389": 389,
+    "LABEL_39": 39,
+    "LABEL_390": 390,
+    "LABEL_391": 391,
+    "LABEL_392": 392,
+    "LABEL_393": 393,
+    "LABEL_394": 394,
+    "LABEL_395": 395,
+    "LABEL_396": 396,
+    "LABEL_397": 397,
+    "LABEL_398": 398,
+    "LABEL_399": 399,
+    "LABEL_4": 4,
+    "LABEL_40": 40,
+    "LABEL_400": 400,
+    "LABEL_401": 401,
+    "LABEL_402": 402,
+    "LABEL_403": 403,
+    "LABEL_404": 404,
+    "LABEL_405": 405,
+    "LABEL_406": 406,
+    "LABEL_407": 407,
+    "LABEL_408": 408,
+    "LABEL_409": 409,
+    "LABEL_41": 41,
+    "LABEL_410": 410,
+    "LABEL_411": 411,
+    "LABEL_412": 412,
+    "LABEL_413": 413,
+    "LABEL_414": 414,
+    "LABEL_415": 415,
+    "LABEL_416": 416,
+    "LABEL_417": 417,
+    "LABEL_418": 418,
+    "LABEL_419": 419,
+    "LABEL_42": 42,
+    "LABEL_420": 420,
+    "LABEL_421": 421,
+    "LABEL_422": 422,
+    "LABEL_423": 423,
+    "LABEL_424": 424,
+    "LABEL_425": 425,
+    "LABEL_426": 426,
+    "LABEL_427": 427,
+    "LABEL_428": 428,
+    "LABEL_429": 429,
+    "LABEL_43": 43,
+    "LABEL_430": 430,
+    "LABEL_431": 431,
+    "LABEL_432": 432,
+    "LABEL_433": 433,
+    "LABEL_434": 434,
+    "LABEL_435": 435,
+    "LABEL_436": 436,
+    "LABEL_437": 437,
+    "LABEL_438": 438,
+    "LABEL_439": 439,
+    "LABEL_44": 44,
+    "LABEL_440": 440,
+    "LABEL_441": 441,
+    "LABEL_442": 442,
+    "LABEL_443": 443,
+    "LABEL_444": 444,
+    "LABEL_445": 445,
+    "LABEL_446": 446,
+    "LABEL_447": 447,
+    "LABEL_448": 448,
+    "LABEL_449": 449,
+    "LABEL_45": 45,
+    "LABEL_450": 450,
+    "LABEL_451": 451,
+    "LABEL_452": 452,
+    "LABEL_453": 453,
+    "LABEL_454": 454,
+    "LABEL_455": 455,
+    "LABEL_456": 456,
+    "LABEL_457": 457,
+    "LABEL_458": 458,
+    "LABEL_459": 459,
+    "LABEL_46": 46,
+    "LABEL_460": 460,
+    "LABEL_461": 461,
+    "LABEL_462": 462,
+    "LABEL_463": 463,
+    "LABEL_464": 464,
+    "LABEL_465": 465,
+    "LABEL_466": 466,
+    "LABEL_467": 467,
+    "LABEL_468": 468,
+    "LABEL_469": 469,
+    "LABEL_47": 47,
+    "LABEL_470": 470,
+    "LABEL_471": 471,
+    "LABEL_472": 472,
+    "LABEL_473": 473,
+    "LABEL_474": 474,
+    "LABEL_475": 475,
+    "LABEL_476": 476,
+    "LABEL_477": 477,
+    "LABEL_478": 478,
+    "LABEL_479": 479,
+    "LABEL_48": 48,
+    "LABEL_480": 480,
+    "LABEL_481": 481,
+    "LABEL_482": 482,
+    "LABEL_483": 483,
+    "LABEL_484": 484,
+    "LABEL_485": 485,
+    "LABEL_486": 486,
+    "LABEL_487": 487,
+    "LABEL_488": 488,
+    "LABEL_489": 489,
+    "LABEL_49": 49,
+    "LABEL_490": 490,
+    "LABEL_491": 491,
+    "LABEL_492": 492,
+    "LABEL_493": 493,
+    "LABEL_494": 494,
+    "LABEL_495": 495,
+    "LABEL_496": 496,
+    "LABEL_497": 497,
+    "LABEL_498": 498,
+    "LABEL_499": 499,
+    "LABEL_5": 5,
+    "LABEL_50": 50,
+    "LABEL_500": 500,
+    "LABEL_501": 501,
+    "LABEL_502": 502,
+    "LABEL_503": 503,
+    "LABEL_504": 504,
+    "LABEL_505": 505,
+    "LABEL_506": 506,
+    "LABEL_507": 507,
+    "LABEL_508": 508,
+    "LABEL_509": 509,
+    "LABEL_51": 51,
+    "LABEL_510": 510,
+    "LABEL_511": 511,
+    "LABEL_512": 512,
+    "LABEL_513": 513,
+    "LABEL_514": 514,
+    "LABEL_515": 515,
+    "LABEL_516": 516,
+    "LABEL_517": 517,
+    "LABEL_518": 518,
+    "LABEL_519": 519,
+    "LABEL_52": 52,
+    "LABEL_520": 520,
+    "LABEL_521": 521,
+    "LABEL_522": 522,
+    "LABEL_523": 523,
+    "LABEL_524": 524,
+    "LABEL_525": 525,
+    "LABEL_526": 526,
+    "LABEL_527": 527,
+    "LABEL_528": 528,
+    "LABEL_529": 529,
+    "LABEL_53": 53,
+    "LABEL_530": 530,
+    "LABEL_531": 531,
+    "LABEL_532": 532,
+    "LABEL_533": 533,
+    "LABEL_534": 534,
+    "LABEL_535": 535,
+    "LABEL_536": 536,
+    "LABEL_537": 537,
+    "LABEL_538": 538,
+    "LABEL_539": 539,
+    "LABEL_54": 54,
+    "LABEL_540": 540,
+    "LABEL_541": 541,
+    "LABEL_542": 542,
+    "LABEL_543": 543,
+    "LABEL_544": 544,
+    "LABEL_545": 545,
+    "LABEL_546": 546,
+    "LABEL_547": 547,
+    "LABEL_548": 548,
+    "LABEL_549": 549,
+    "LABEL_55": 55,
+    "LABEL_550": 550,
+    "LABEL_551": 551,
+    "LABEL_552": 552,
+    "LABEL_553": 553,
+    "LABEL_554": 554,
+    "LABEL_555": 555,
+    "LABEL_556": 556,
+    "LABEL_557": 557,
+    "LABEL_558": 558,
+    "LABEL_559": 559,
+    "LABEL_56": 56,
+    "LABEL_560": 560,
+    "LABEL_561": 561,
+    "LABEL_562": 562,
+    "LABEL_563": 563,
+    "LABEL_564": 564,
+    "LABEL_565": 565,
+    "LABEL_566": 566,
+    "LABEL_567": 567,
+    "LABEL_568": 568,
+    "LABEL_569": 569,
+    "LABEL_57": 57,
+    "LABEL_570": 570,
+    "LABEL_571": 571,
+    "LABEL_572": 572,
+    "LABEL_573": 573,
+    "LABEL_574": 574,
+    "LABEL_575": 575,
+    "LABEL_576": 576,
+    "LABEL_577": 577,
+    "LABEL_578": 578,
+    "LABEL_579": 579,
+    "LABEL_58": 58,
+    "LABEL_580": 580,
+    "LABEL_581": 581,
+    "LABEL_582": 582,
+    "LABEL_583": 583,
+    "LABEL_584": 584,
+    "LABEL_585": 585,
+    "LABEL_586": 586,
+    "LABEL_587": 587,
+    "LABEL_588": 588,
+    "LABEL_589": 589,
+    "LABEL_59": 59,
+    "LABEL_590": 590,
+    "LABEL_591": 591,
+    "LABEL_592": 592,
+    "LABEL_593": 593,
+    "LABEL_594": 594,
+    "LABEL_595": 595,
+    "LABEL_596": 596,
+    "LABEL_597": 597,
+    "LABEL_598": 598,
+    "LABEL_599": 599,
+    "LABEL_6": 6,
+    "LABEL_60": 60,
+    "LABEL_600": 600,
+    "LABEL_601": 601,
+    "LABEL_602": 602,
+    "LABEL_603": 603,
+    "LABEL_604": 604,
+    "LABEL_605": 605,
+    "LABEL_606": 606,
+    "LABEL_607": 607,
+    "LABEL_608": 608,
+    "LABEL_609": 609,
+    "LABEL_61": 61,
+    "LABEL_610": 610,
+    "LABEL_611": 611,
+    "LABEL_612": 612,
+    "LABEL_613": 613,
+    "LABEL_614": 614,
+    "LABEL_615": 615,
+    "LABEL_616": 616,
+    "LABEL_617": 617,
+    "LABEL_618": 618,
+    "LABEL_619": 619,
+    "LABEL_62": 62,
+    "LABEL_620": 620,
+    "LABEL_621": 621,
+    "LABEL_622": 622,
+    "LABEL_623": 623,
+    "LABEL_624": 624,
+    "LABEL_625": 625,
+    "LABEL_626": 626,
+    "LABEL_627": 627,
+    "LABEL_628": 628,
+    "LABEL_629": 629,
+    "LABEL_63": 63,
+    "LABEL_630": 630,
+    "LABEL_631": 631,
+    "LABEL_632": 632,
+    "LABEL_633": 633,
+    "LABEL_634": 634,
+    "LABEL_635": 635,
+    "LABEL_636": 636,
+    "LABEL_637": 637,
+    "LABEL_638": 638,
+    "LABEL_639": 639,
+    "LABEL_64": 64,
+    "LABEL_640": 640,
+    "LABEL_641": 641,
+    "LABEL_642": 642,
+    "LABEL_643": 643,
+    "LABEL_644": 644,
+    "LABEL_645": 645,
+    "LABEL_646": 646,
+    "LABEL_647": 647,
+    "LABEL_648": 648,
+    "LABEL_649": 649,
+    "LABEL_65": 65,
+    "LABEL_650": 650,
+    "LABEL_651": 651,
+    "LABEL_652": 652,
+    "LABEL_653": 653,
+    "LABEL_654": 654,
+    "LABEL_655": 655,
+    "LABEL_656": 656,
+    "LABEL_657": 657,
+    "LABEL_658": 658,
+    "LABEL_659": 659,
+    "LABEL_66": 66,
+    "LABEL_660": 660,
+    "LABEL_661": 661,
+    "LABEL_662": 662,
+    "LABEL_663": 663,
+    "LABEL_664": 664,
+    "LABEL_665": 665,
+    "LABEL_666": 666,
+    "LABEL_667": 667,
+    "LABEL_668": 668,
+    "LABEL_669": 669,
+    "LABEL_67": 67,
+    "LABEL_670": 670,
+    "LABEL_671": 671,
+    "LABEL_672": 672,
+    "LABEL_673": 673,
+    "LABEL_674": 674,
+    "LABEL_675": 675,
+    "LABEL_676": 676,
+    "LABEL_677": 677,
+    "LABEL_678": 678,
+    "LABEL_679": 679,
+    "LABEL_68": 68,
+    "LABEL_680": 680,
+    "LABEL_681": 681,
+    "LABEL_682": 682,
+    "LABEL_683": 683,
+    "LABEL_684": 684,
+    "LABEL_685": 685,
+    "LABEL_686": 686,
+    "LABEL_687": 687,
+    "LABEL_688": 688,
+    "LABEL_689": 689,
+    "LABEL_69": 69,
+    "LABEL_690": 690,
+    "LABEL_691": 691,
+    "LABEL_692": 692,
+    "LABEL_693": 693,
+    "LABEL_694": 694,
+    "LABEL_695": 695,
+    "LABEL_696": 696,
+    "LABEL_697": 697,
+    "LABEL_698": 698,
+    "LABEL_699": 699,
+    "LABEL_7": 7,
+    "LABEL_70": 70,
+    "LABEL_700": 700,
+    "LABEL_701": 701,
+    "LABEL_702": 702,
+    "LABEL_703": 703,
+    "LABEL_704": 704,
+    "LABEL_705": 705,
+    "LABEL_706": 706,
+    "LABEL_707": 707,
+    "LABEL_708": 708,
+    "LABEL_709": 709,
+    "LABEL_71": 71,
+    "LABEL_710": 710,
+    "LABEL_711": 711,
+    "LABEL_712": 712,
+    "LABEL_713": 713,
+    "LABEL_714": 714,
+    "LABEL_715": 715,
+    "LABEL_716": 716,
+    "LABEL_717": 717,
+    "LABEL_718": 718,
+    "LABEL_719": 719,
+    "LABEL_72": 72,
+    "LABEL_720": 720,
+    "LABEL_721": 721,
+    "LABEL_722": 722,
+    "LABEL_723": 723,
+    "LABEL_724": 724,
+    "LABEL_725": 725,
+    "LABEL_726": 726,
+    "LABEL_727": 727,
+    "LABEL_728": 728,
+    "LABEL_729": 729,
+    "LABEL_73": 73,
+    "LABEL_730": 730,
+    "LABEL_731": 731,
+    "LABEL_732": 732,
+    "LABEL_733": 733,
+    "LABEL_734": 734,
+    "LABEL_735": 735,
+    "LABEL_736": 736,
+    "LABEL_737": 737,
+    "LABEL_738": 738,
+    "LABEL_739": 739,
+    "LABEL_74": 74,
+    "LABEL_740": 740,
+    "LABEL_741": 741,
+    "LABEL_742": 742,
+    "LABEL_743": 743,
+    "LABEL_744": 744,
+    "LABEL_745": 745,
+    "LABEL_746": 746,
+    "LABEL_747": 747,
+    "LABEL_748": 748,
+    "LABEL_749": 749,
+    "LABEL_75": 75,
+    "LABEL_750": 750,
+    "LABEL_751": 751,
+    "LABEL_752": 752,
+    "LABEL_753": 753,
+    "LABEL_754": 754,
+    "LABEL_755": 755,
+    "LABEL_756": 756,
+    "LABEL_757": 757,
+    "LABEL_758": 758,
+    "LABEL_759": 759,
+    "LABEL_76": 76,
+    "LABEL_760": 760,
+    "LABEL_761": 761,
+    "LABEL_762": 762,
+    "LABEL_763": 763,
+    "LABEL_764": 764,
+    "LABEL_765": 765,
+    "LABEL_766": 766,
+    "LABEL_767": 767,
+    "LABEL_768": 768,
+    "LABEL_769": 769,
+    "LABEL_77": 77,
+    "LABEL_770": 770,
+    "LABEL_771": 771,
+    "LABEL_772": 772,
+    "LABEL_773": 773,
+    "LABEL_774": 774,
+    "LABEL_775": 775,
+    "LABEL_776": 776,
+    "LABEL_777": 777,
+    "LABEL_778": 778,
+    "LABEL_779": 779,
+    "LABEL_78": 78,
+    "LABEL_780": 780,
+    "LABEL_781": 781,
+    "LABEL_782": 782,
+    "LABEL_783": 783,
+    "LABEL_784": 784,
+    "LABEL_785": 785,
+    "LABEL_786": 786,
+    "LABEL_787": 787,
+    "LABEL_788": 788,
+    "LABEL_789": 789,
+    "LABEL_79": 79,
+    "LABEL_790": 790,
+    "LABEL_791": 791,
+    "LABEL_792": 792,
+    "LABEL_793": 793,
+    "LABEL_794": 794,
+    "LABEL_795": 795,
+    "LABEL_796": 796,
+    "LABEL_797": 797,
+    "LABEL_798": 798,
+    "LABEL_799": 799,
+    "LABEL_8": 8,
+    "LABEL_80": 80,
+    "LABEL_800": 800,
+    "LABEL_801": 801,
+    "LABEL_802": 802,
+    "LABEL_803": 803,
+    "LABEL_804": 804,
+    "LABEL_805": 805,
+    "LABEL_806": 806,
+    "LABEL_807": 807,
+    "LABEL_808": 808,
+    "LABEL_809": 809,
+    "LABEL_81": 81,
+    "LABEL_810": 810,
+    "LABEL_811": 811,
+    "LABEL_812": 812,
+    "LABEL_813": 813,
+    "LABEL_814": 814,
+    "LABEL_815": 815,
+    "LABEL_816": 816,
+    "LABEL_817": 817,
+    "LABEL_818": 818,
+    "LABEL_819": 819,
+    "LABEL_82": 82,
+    "LABEL_820": 820,
+    "LABEL_821": 821,
+    "LABEL_822": 822,
+    "LABEL_823": 823,
+    "LABEL_824": 824,
+    "LABEL_825": 825,
+    "LABEL_826": 826,
+    "LABEL_827": 827,
+    "LABEL_828": 828,
+    "LABEL_829": 829,
+    "LABEL_83": 83,
+    "LABEL_830": 830,
+    "LABEL_831": 831,
+    "LABEL_832": 832,
+    "LABEL_833": 833,
+    "LABEL_834": 834,
+    "LABEL_835": 835,
+    "LABEL_836": 836,
+    "LABEL_837": 837,
+    "LABEL_838": 838,
+    "LABEL_839": 839,
+    "LABEL_84": 84,
+    "LABEL_840": 840,
+    "LABEL_841": 841,
+    "LABEL_842": 842,
+    "LABEL_843": 843,
+    "LABEL_844": 844,
+    "LABEL_845": 845,
+    "LABEL_846": 846,
+    "LABEL_847": 847,
+    "LABEL_848": 848,
+    "LABEL_849": 849,
+    "LABEL_85": 85,
+    "LABEL_850": 850,
+    "LABEL_851": 851,
+    "LABEL_852": 852,
+    "LABEL_853": 853,
+    "LABEL_854": 854,
+    "LABEL_855": 855,
+    "LABEL_856": 856,
+    "LABEL_857": 857,
+    "LABEL_858": 858,
+    "LABEL_859": 859,
+    "LABEL_86": 86,
+    "LABEL_860": 860,
+    "LABEL_861": 861,
+    "LABEL_862": 862,
+    "LABEL_863": 863,
+    "LABEL_864": 864,
+    "LABEL_865": 865,
+    "LABEL_866": 866,
+    "LABEL_867": 867,
+    "LABEL_868": 868,
+    "LABEL_869": 869,
+    "LABEL_87": 87,
+    "LABEL_870": 870,
+    "LABEL_871": 871,
+    "LABEL_872": 872,
+    "LABEL_873": 873,
+    "LABEL_874": 874,
+    "LABEL_875": 875,
+    "LABEL_876": 876,
+    "LABEL_877": 877,
+    "LABEL_878": 878,
+    "LABEL_879": 879,
+    "LABEL_88": 88,
+    "LABEL_880": 880,
+    "LABEL_881": 881,
+    "LABEL_882": 882,
+    "LABEL_883": 883,
+    "LABEL_884": 884,
+    "LABEL_885": 885,
+    "LABEL_886": 886,
+    "LABEL_887": 887,
+    "LABEL_888": 888,
+    "LABEL_889": 889,
+    "LABEL_89": 89,
+    "LABEL_890": 890,
+    "LABEL_891": 891,
+    "LABEL_892": 892,
+    "LABEL_893": 893,
+    "LABEL_894": 894,
+    "LABEL_895": 895,
+    "LABEL_896": 896,
+    "LABEL_897": 897,
+    "LABEL_898": 898,
+    "LABEL_899": 899,
+    "LABEL_9": 9,
+    "LABEL_90": 90,
+    "LABEL_900": 900,
+    "LABEL_901": 901,
+    "LABEL_902": 902,
+    "LABEL_903": 903,
+    "LABEL_904": 904,
+    "LABEL_905": 905,
+    "LABEL_906": 906,
+    "LABEL_907": 907,
+    "LABEL_908": 908,
+    "LABEL_909": 909,
+    "LABEL_91": 91,
+    "LABEL_910": 910,
+    "LABEL_911": 911,
+    "LABEL_912": 912,
+    "LABEL_913": 913,
+    "LABEL_914": 914,
+    "LABEL_915": 915,
+    "LABEL_916": 916,
+    "LABEL_917": 917,
+    "LABEL_918": 918,
+    "LABEL_919": 919,
+    "LABEL_92": 92,
+    "LABEL_920": 920,
+    "LABEL_921": 921,
+    "LABEL_922": 922,
+    "LABEL_923": 923,
+    "LABEL_924": 924,
+    "LABEL_925": 925,
+    "LABEL_926": 926,
+    "LABEL_927": 927,
+    "LABEL_928": 928,
+    "LABEL_929": 929,
+    "LABEL_93": 93,
+    "LABEL_930": 930,
+    "LABEL_931": 931,
+    "LABEL_932": 932,
+    "LABEL_933": 933,
+    "LABEL_934": 934,
+    "LABEL_935": 935,
+    "LABEL_936": 936,
+    "LABEL_937": 937,
+    "LABEL_938": 938,
+    "LABEL_939": 939,
+    "LABEL_94": 94,
+    "LABEL_940": 940,
+    "LABEL_941": 941,
+    "LABEL_942": 942,
+    "LABEL_943": 943,
+    "LABEL_944": 944,
+    "LABEL_945": 945,
+    "LABEL_946": 946,
+    "LABEL_947": 947,
+    "LABEL_948": 948,
+    "LABEL_949": 949,
+    "LABEL_95": 95,
+    "LABEL_950": 950,
+    "LABEL_951": 951,
+    "LABEL_952": 952,
+    "LABEL_953": 953,
+    "LABEL_954": 954,
+    "LABEL_955": 955,
+    "LABEL_956": 956,
+    "LABEL_957": 957,
+    "LABEL_958": 958,
+    "LABEL_959": 959,
+    "LABEL_96": 96,
+    "LABEL_960": 960,
+    "LABEL_961": 961,
+    "LABEL_962": 962,
+    "LABEL_963": 963,
+    "LABEL_964": 964,
+    "LABEL_965": 965,
+    "LABEL_966": 966,
+    "LABEL_967": 967,
+    "LABEL_968": 968,
+    "LABEL_969": 969,
+    "LABEL_97": 97,
+    "LABEL_970": 970,
+    "LABEL_971": 971,
+    "LABEL_972": 972,
+    "LABEL_973": 973,
+    "LABEL_974": 974,
+    "LABEL_975": 975,
+    "LABEL_976": 976,
+    "LABEL_977": 977,
+    "LABEL_978": 978,
+    "LABEL_979": 979,
+    "LABEL_98": 98,
+    "LABEL_980": 980,
+    "LABEL_981": 981,
+    "LABEL_982": 982,
+    "LABEL_983": 983,
+    "LABEL_984": 984,
+    "LABEL_985": 985,
+    "LABEL_986": 986,
+    "LABEL_987": 987,
+    "LABEL_988": 988,
+    "LABEL_989": 989,
+    "LABEL_99": 99,
+    "LABEL_990": 990,
+    "LABEL_991": 991,
+    "LABEL_992": 992,
+    "LABEL_993": 993,
+    "LABEL_994": 994,
+    "LABEL_995": 995,
+    "LABEL_996": 996,
+    "LABEL_997": 997,
+    "LABEL_998": 998,
+    "LABEL_999": 999
+  },
+  "layer_norm_eps": 1e-06,
+  "model_type": "ortho_vit",
+  "num_attention_heads": 12,
+  "num_channels": 3,
+  "num_hidden_layers": 12,
+  "orthogonal_method": "channel",
+  "patch_size": 16,
+  "pooler_act": "tanh",
+  "pooler_output_size": 768,
+  "qkv_bias": true,
+  "residual_connection": "orthogonal",
+  "residual_connection_dim": -1,
+  "residual_eps": 1e-06,
+  "residual_perturbation": null,
+  "torch_dtype": "float32",
+  "transformers_version": "4.52.3"
+}

configuration_ortho_vit.py

@@ -0,0 +1,92 @@
+# coding=utf-8
+# Copyright 2021 Google AI, Ross Wightman, The HuggingFace Inc. team. All rights reserved.
+# Copyright 2025 The Authors of Revisiting Residual Connections. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" OrthoViT model configuration"""
+
+from transformers.models.vit.configuration_vit import ViTConfig
+from transformers.utils import logging
+
+
+logger = logging.get_logger(__name__)
+
+class OrthoViTConfig(ViTConfig):
+    r"""
+    This is the configuration class to store the configuration of an [`OrthoViTModel`].
+    It is used to instantiate a ViT model with orthogonal residual connections
+    according to the specified arguments, defining the model architecture. Instantiating a
+    configuration with the defaults will yield a similar configuration to that of
+    the ViT-base model.
+
+    Configuration objects inherit from [`ViTConfig`] and can be used to control the model outputs.
+    Read the documentation from [`ViTConfig`] for more information.
+
+    Args:
+        residual_connection (`str`, *optional*, defaults to `"linear"`):
+            The type of residual connection to use. Can be "linear" or "orthogonal".
+        orthogonal_method (`str`, *optional*, defaults to `"channel"`):
+            The method for orthogonalization if `residual_connection` is "orthogonal".
+            Can be "channel" or "global". For ViT, "channel" is typically used for token embeddings.
+        residual_connection_dim (`int`, *optional*, defaults to -1):
+            The dimension along which to compute orthogonality. Defaults to -1 (last dimension).
+        residual_eps (`float`, *optional*, defaults to 1e-6):
+            Epsilon value for numerical stability in orthogonalization.
+        residual_perturbation (`float`, *optional*, defaults to `None`):
+            Magnitude of random perturbation to add to the module output before connection.
+
+    Example:
+
+    ```python
+    >>> from modeling_ortho_vit import OrthoViTModel
+    >>> from configuration_ortho_vit import OrthoViTConfig
+
+    >>> # Initializing a ViT-base style configuration with orthogonal connections
+    >>> configuration = OrthoViTConfig(residual_connection="orthogonal")
+
+    >>> # Initializing a model (with random weights) from the ViT-base style configuration
+    >>> model = OrthoViTModel(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+    model_type = "ortho_vit"
+
+    def __init__(
+        self,
+        residual_connection="linear",
+        orthogonal_method="channel", # For ViT, 'channel' typically means along the hidden_dim
+        residual_connection_dim=-1,
+        residual_eps=1e-6,
+        residual_perturbation=None,
+        elementwise_affine_ln=False, # To match user's OrthoBlock norm1/norm2
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.residual_connection = residual_connection
+        self.orthogonal_method = orthogonal_method
+        self.residual_connection_dim = residual_connection_dim
+        self.residual_eps = residual_eps
+        self.residual_perturbation = residual_perturbation
+        self.elementwise_affine_ln = elementwise_affine_ln # For ViTLayer norms
+
+    @property
+    def residual_kwargs(self) -> dict:
+        # This property can be used by the model to easily access all residual connection parameters
+        return dict(
+            method=self.residual_connection,
+            orthogonal_method=self.orthogonal_method,
+            dim=self.residual_connection_dim,
+            perturbation=self.residual_perturbation,
+            # eps is handled via a registered buffer in the layer
+        )

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a0d99e8aebb9719333f101b00f1110d881d3e084c80afbf8ca96c45891449b0a
+size 346141064

modeling_ortho_vit.py

@@ -0,0 +1,455 @@
+# coding=utf-8
+# Copyright 2021 Google AI, Ross Wightman, The HuggingFace Inc. team. All rights reserved.
+# Copyright 2025 The Authors of Revisiting Residual Connections. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""PyTorch OrthoViT model. 
+From Revisiting Residual Connections: Orthogonal Residual Connections for Vision Transformers.
+preprint arXiv:2505.11881
+
+Authors: Giyeong Oh, Woohyun Cho, Siyeol Kim, Suhwan Choi, Younjae Yu
+"""
+
+import collections.abc
+import math
+from typing import Dict, List, Optional, Set, Tuple, Union
+
+import torch
+import torch.utils.checkpoint
+from torch import nn
+from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss
+
+from transformers.activations import ACT2FN
+from transformers.modeling_outputs import (
+    BaseModelOutput,
+    BaseModelOutputWithPooling,
+    ImageClassifierOutput,
+)
+from transformers.modeling_utils import PreTrainedModel
+from transformers.utils import logging, auto_docstring
+from transformers.models.vit.modeling_vit import ViTEmbeddings, ViTPatchEmbeddings, ViTSelfAttention, ViTSelfOutput, ViTAttention, ViTIntermediate, ViTPooler # Use HF implementations for these
+
+# Assuming connect.py is in the same directory
+# from .connect import connect # For use as a package
+# For standalone script, you might need to adjust import path or ensure connect.py is findable
+logger = logging.get_logger(__name__)
+try:
+    from .residual import connect
+except ImportError:
+    logger.warning("residual.py not found, using fallback linear connect function.")
+    def connect(x, f_x, *args, **kwargs): # Linear connection fallback
+        """
+        Fallback connection function if connect.py is not available.
+        This will simply add the input x to the function output f_x.
+        This is a simple linear connection, not orthogonal.
+        """
+        return x + f_x 
+
+from .configuration_ortho_vit import OrthoViTConfig # Changed from ViTConfig
+
+
+logger = logging.get_logger(__name__)
+
+_CONFIG_FOR_DOC = "OrthoViTConfig"
+
+# Copied from transformers.models.vit.modeling_vit.ViTOutput
+# with the residual connection modified to use the `connect` function.
+class OrthoViTOutput(nn.Module):
+    def __init__(self, config: OrthoViTConfig) -> None:
+        super().__init__()
+        self.dense = nn.Linear(config.intermediate_size, config.hidden_size)
+        self.dropout = nn.Dropout(config.hidden_dropout_prob)
+        # Orthogonal connection parameters are stored in config and passed to connect
+        self.config = config
+        # Register buffer for eps, ensuring it's on the correct device and non-persistent
+        self.register_buffer("residual_eps", torch.tensor([config.residual_eps], dtype=torch.float32), persistent=False)
+
+
+    def forward(self, hidden_states: torch.Tensor, input_tensor: torch.Tensor, residual_input: torch.Tensor) -> torch.Tensor:
+        hidden_states = self.dense(hidden_states)
+        hidden_states = self.dropout(hidden_states)
+
+        # Use the connect function for the residual connection
+        # The 'input_tensor' to ViTOutput is the output of the intermediate layer.
+        # The 'residual_input' is the output of the attention block (hidden_states before layernorm_after).
+        hidden_states = connect(
+            x=residual_input,
+            f_x=hidden_states,
+            eps=self.residual_eps,
+            **self.config.residual_kwargs
+        )
+        return hidden_states
+
+class OrthoViTLayer(nn.Module):
+    """This corresponds to the Block class in the timm implementation."""
+
+    def __init__(self, config: OrthoViTConfig) -> None:
+        super().__init__()
+        self.chunk_size_feed_forward = config.chunk_size_feed_forward
+        self.seq_len_dim = 1
+        self.attention = ViTAttention(config) # Standard HF ViTAttention
+        self.intermediate = ViTIntermediate(config) # Standard HF ViTIntermediate
+        # self.output = ViTOutput(config) # Standard HF ViTOutput
+        self.output = OrthoViTOutput(config) # Use OrthoViTOutput for the second residual connection
+
+        self.layernorm_before = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps, elementwise_affine=config.elementwise_affine_ln)
+        self.layernorm_after = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps, elementwise_affine=config.elementwise_affine_ln)
+        self.config = config
+        # Register buffer for eps, ensuring it's on the correct device and non-persistent
+        self.register_buffer("residual_eps", torch.tensor([config.residual_eps], dtype=torch.float32), persistent=False)
+        self.residual_kwargs = self.config.residual_kwargs # Use the config's residual_kwargs
+
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        head_mask: Optional[torch.Tensor] = None,
+        output_attentions: bool = False,
+    ) -> Union[Tuple[torch.Tensor, torch.Tensor], Tuple[torch.Tensor]]:
+        
+        # Input to Attention block
+        normed_hidden_states = self.layernorm_before(hidden_states)
+        self_attention_outputs = self.attention(
+            normed_hidden_states,
+            head_mask,
+            output_attentions=output_attentions,
+        )
+        attention_output = self_attention_outputs[0] # Output of ViTAttention (includes SelfAttention + SelfOutput)
+        outputs = self_attention_outputs[1:]  # add self attentions if we output attention weights
+
+        # First residual connection (after Attention)
+        # The `connect` function expects the module output (f_x) and the stream input (x)
+        # Here, hidden_states is x, and attention_output is f_x from the attention block
+        hidden_states = connect(
+            x=hidden_states, 
+            f_x=attention_output,
+            eps=self.residual_eps, # Use the buffer
+            **self.residual_kwargs
+        )
+        
+        # MLP block
+        # In ViT, layernorm is also applied after self-attention (and first residual)
+        # This `hidden_states` is now `x_{n+1}` from the attention block
+        mlp_input = self.layernorm_after(hidden_states)
+        intermediate_output = self.intermediate(mlp_input)
+
+        # Second residual connection (after MLP) is handled by OrthoViTOutput
+        # `hidden_states` is the residual stream input to the MLP's residual connection
+        # `intermediate_output` is the direct output of the MLP's main path (before adding to residual)
+        layer_output = self.output(intermediate_output, mlp_input, hidden_states) # mlp_input here is just for consistency with ViTOutput's original signature, OrthoViTOutput will use residual_input
+
+        outputs = (layer_output,) + outputs
+
+        return outputs
+
+
+class OrthoViTEncoder(nn.Module):
+    def __init__(self, config: OrthoViTConfig) -> None:
+        super().__init__()
+        self.config = config
+        self.layer = nn.ModuleList([OrthoViTLayer(config) for _ in range(config.num_hidden_layers)]) # Use OrthoViTLayer
+        self.gradient_checkpointing = False
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        head_mask: Optional[torch.Tensor] = None,
+        output_attentions: bool = False,
+        output_hidden_states: bool = False,
+        return_dict: bool = True,
+    ) -> Union[tuple, BaseModelOutput]:
+        all_hidden_states = () if output_hidden_states else None
+        all_self_attentions = () if output_attentions else None
+
+        for i, layer_module in enumerate(self.layer):
+            if output_hidden_states:
+                all_hidden_states = all_hidden_states + (hidden_states,)
+
+            layer_head_mask = head_mask[i] if head_mask is not None else None
+
+            if self.gradient_checkpointing and self.training:
+                layer_outputs = self._gradient_checkpointing_func(
+                    layer_module.__call__,
+                    hidden_states,
+                    layer_head_mask,
+                    output_attentions,
+                )
+            else:
+                layer_outputs = layer_module(hidden_states, layer_head_mask, output_attentions)
+
+            hidden_states = layer_outputs[0]
+
+            if output_attentions:
+                all_self_attentions = all_self_attentions + (layer_outputs[1],)
+
+        if output_hidden_states:
+            all_hidden_states = all_hidden_states + (hidden_states,)
+
+        if not return_dict:
+            return tuple(v for v in [hidden_states, all_hidden_states, all_self_attentions] if v is not None)
+        return BaseModelOutput(
+            last_hidden_state=hidden_states,
+            hidden_states=all_hidden_states,
+            attentions=all_self_attentions,
+        )
+
+# @add_start_docstrings(
+#     "The bare ViT MAE transformer outputting raw hidden-states without any specific head on top.",
+#     # VIT_START_DOCSTRING, # OrthoViT is custom
+# )
+# @auto_docstring
+class OrthoViTPreTrainedModel(PreTrainedModel): # Copied from ViTPreTrainedModel
+    config_class = OrthoViTConfig # Use OrthoViTConfig
+    base_model_prefix = "vit"
+    main_input_name = "pixel_values"
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["ViTEmbeddings", "OrthoViTLayer"] # Use OrthoViTLayer
+    _supports_sdpa = True # Inherit from ViT
+    _supports_flash_attn_2 = True # Inherit from ViT
+
+
+    def _init_weights(self, module: Union[nn.Linear, nn.Conv2d, nn.LayerNorm]) -> None:
+        """Initialize the weights"""
+        if isinstance(module, (nn.Linear, nn.Conv2d)):
+            # Upcast the input in `fp32` and cast it back to desired `dtype` to avoid
+            # `trunc_normal_cpu` not implemented in `half` issues
+            module.weight.data = nn.init.trunc_normal_(
+                module.weight.data.to(torch.float32), mean=0.0, std=self.config.initializer_range
+            ).to(module.weight.dtype)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.LayerNorm):
+            if module.elementwise_affine: # Only init if affine
+                module.bias.data.zero_()
+                module.weight.data.fill_(1.0)
+        elif isinstance(module, ViTEmbeddings): # Standard ViTEmbeddings init
+            module.position_embeddings.data = nn.init.trunc_normal_(
+                module.position_embeddings.data.to(torch.float32),
+                mean=0.0,
+                std=self.config.initializer_range,
+            ).to(module.position_embeddings.dtype)
+
+            module.cls_token.data = nn.init.trunc_normal_(
+                module.cls_token.data.to(torch.float32),
+                mean=0.0,
+                std=self.config.initializer_range,
+            ).to(module.cls_token.dtype)
+
+VIT_START_DOCSTRING = r"""
+    This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use
+    it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
+    behavior.
+
+    Parameters:
+        config ([`OrthoViTConfig`]): Model configuration class with all the parameters of the model.
+            Initializing with a config file does not load the weights associated with the model, only the
+            configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights.
+"""
+
+VIT_INPUTS_DOCSTRING = r"""
+    Args:
+        pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`):
+            Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See
+            [`ViTImageProcessor.__call__`] for details.
+
+        head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*):
+            Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
+
+            - 1 indicates the head is **not masked**,
+            - 0 indicates the head is **masked**.
+
+        output_attentions (`bool`, *optional*):
+            Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
+            tensors for more detail.
+        output_hidden_states (`bool`, *optional*):
+            Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
+            more detail.
+        interpolate_pos_encoding (`bool`, *optional*):
+            Whether to interpolate the positional encoding if the input image resolution is different from the one
+            used during pre-training.
+        return_dict (`bool`, *optional*):
+            Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
+"""
+
+# @add_start_docstrings(
+#     "The bare OrthoViT transformer outputting raw hidden-states without any specific head on top.",
+#     VIT_START_DOCSTRING,
+# )
+# @auto_docstring
+class OrthoViTModel(OrthoViTPreTrainedModel):
+    def __init__(self, config: OrthoViTConfig, add_pooling_layer: bool = True, use_mask_token: bool = False):
+        super().__init__(config)
+        self.config = config
+
+        self.embeddings = ViTEmbeddings(config, use_mask_token=use_mask_token)
+        self.encoder = OrthoViTEncoder(config) # Use OrthoViTEncoder
+
+        self.layernorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps) # No affine
+        self.pooler = ViTPooler(config) if add_pooling_layer else None
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self) -> ViTPatchEmbeddings:
+        return self.embeddings.patch_embeddings
+
+    def _prune_heads(self, heads_to_prune: Dict[int, List[int]]) -> None:
+        """
+        Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base
+        class PreTrainedModel
+        """
+        for layer, heads in heads_to_prune.items():
+            self.encoder.layer[layer].attention.prune_heads(heads)
+
+    # @add_start_docstrings_to_model_forward(VIT_INPUTS_DOCSTRING)
+    # @replace_return_docstrings(output_type=BaseModelOutputWithPooling, config_class=_CONFIG_FOR_DOC)
+    # @auto_docstring
+    def forward(
+        self,
+        pixel_values: Optional[torch.Tensor] = None,
+        bool_masked_pos: Optional[torch.BoolTensor] = None, # For MAE
+        head_mask: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        interpolate_pos_encoding: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> Union[Tuple, BaseModelOutputWithPooling]:
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        if pixel_values is None:
+            raise ValueError("You have to specify pixel_values")
+
+        head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)
+
+        expected_dtype = self.embeddings.patch_embeddings.projection.weight.dtype
+        if pixel_values.dtype != expected_dtype:
+            pixel_values = pixel_values.to(expected_dtype)
+
+        embedding_output = self.embeddings(
+            pixel_values, bool_masked_pos=bool_masked_pos, interpolate_pos_encoding=interpolate_pos_encoding
+        )
+
+        encoder_outputs = self.encoder(
+            embedding_output,
+            head_mask=head_mask,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+        sequence_output = encoder_outputs[0]
+        sequence_output = self.layernorm(sequence_output) # Final LayerNorm
+        pooled_output = self.pooler(sequence_output) if self.pooler is not None else None
+
+        if not return_dict:
+            head_outputs = (sequence_output, pooled_output) if pooled_output is not None else (sequence_output,)
+            return head_outputs + encoder_outputs[1:]
+
+        return BaseModelOutputWithPooling(
+            last_hidden_state=sequence_output,
+            pooler_output=pooled_output,
+            hidden_states=encoder_outputs.hidden_states,
+            attentions=encoder_outputs.attentions,
+        )
+
+# @auto_docstring(
+    # """
+    # OrthoViT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of
+    # the [CLS] token) e.g. for ImageNet.
+    # """
+# )
+class OrthoViTForImageClassification(OrthoViTPreTrainedModel):
+    def __init__(self, config: OrthoViTConfig) -> None:
+        super().__init__(config)
+
+        self.num_labels = config.num_labels
+        self.vit = OrthoViTModel(config, add_pooling_layer=False) # Use OrthoViTModel
+
+        # Classifier head (standard HF: just a Linear layer)
+        # The user's original classifier was nn.Sequential(nn.LayerNorm(dim), nn.Linear(dim, num_classes))
+        # The OrthoViTModel's output (CLS token) is already layernormed by self.vit.layernorm.
+        # So, if that self.vit.layernorm corresponds to the user's classifier.0 (LayerNorm),
+        # then this classifier head should just be Linear.
+        # If an *additional* LN is needed on the CLS token, this needs to be nn.Sequential.
+        # For now, assume vit.layernorm covers the user's classifier.0.
+        self.classifier = nn.Linear(config.hidden_size, config.num_labels) if config.num_labels > 0 else nn.Identity()
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    # @auto_docstring
+    def forward(
+        self,
+        pixel_values: Optional[torch.Tensor] = None,
+        head_mask: Optional[torch.Tensor] = None,
+        labels: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        interpolate_pos_encoding: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> Union[Tuple, ImageClassifierOutput]:
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        outputs = self.vit(
+            pixel_values,
+            head_mask=head_mask,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            interpolate_pos_encoding=interpolate_pos_encoding,
+            return_dict=return_dict,
+        )
+
+        sequence_output = outputs[0]
+        # Takes the CLS token representation for classification
+        logits = self.classifier(sequence_output[:, 0, :])
+
+
+        loss = None
+        if labels is not None:
+            labels = labels.to(logits.device)
+            if self.config.problem_type is None:
+                if self.num_labels == 1:
+                    self.config.problem_type = "regression"
+                elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
+                    self.config.problem_type = "single_label_classification"
+                else:
+                    self.config.problem_type = "multi_label_classification"
+
+            if self.config.problem_type == "regression":
+                loss_fct = MSELoss()
+                if self.num_labels == 1:
+                    loss = loss_fct(logits.squeeze(), labels.squeeze())
+                else:
+                    loss = loss_fct(logits, labels)
+            elif self.config.problem_type == "single_label_classification":
+                loss_fct = CrossEntropyLoss()
+                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
+            elif self.config.problem_type == "multi_label_classification":
+                loss_fct = BCEWithLogitsLoss()
+                loss = loss_fct(logits, labels)
+
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return ((loss,) + output) if loss is not None else output
+
+        return ImageClassifierOutput(
+            loss=loss,
+            logits=logits,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+__all__ = ["OrthoViTConfig", "OrthoViTModel", "OrthoViTForImageClassification", "OrthoViTPreTrainedModel"]

residual.py

@@ -0,0 +1,60 @@
+import torch
+
+def _identity(x: torch.Tensor, f_x: torch.Tensor) -> torch.Tensor:
+    """
+    linear residual connection
+    x   : residual stream
+    f_x : attention/MLP/conv(if channel-wise) output
+    """
+    return f_x
+
+def _orthogonal_channel(x: torch.Tensor, f_x: torch.Tensor, dim: int, eps: float) -> torch.Tensor:
+    """
+    orthogonal residual connection
+    x   : residual stream
+    f_x : attention/MLP/conv(if channel-wise) output
+    """
+    dot      = (x * f_x).sum(dim, keepdim=True)
+    norm_x2  = (x * x  ).sum(dim, keepdim=True) + eps
+    scale = (dot / norm_x2).to(x.dtype) # for amp
+    proj_out = scale * x
+    return f_x - proj_out
+
+def _orthogonal_global(x: torch.Tensor, f_x: torch.Tensor, dim: int, eps: float) -> torch.Tensor:
+    """
+    orthogonal residual connection
+    x   : residual stream
+    f_x : conv output
+    """    
+    original_shape = x.shape
+    positive_dim = dim if dim >= 0 else len(original_shape) + dim
+
+    x_view   = x.flatten(dim)          # [B, CHW...]
+    f_view   = f_x.flatten(dim)        # same
+    dot      = (x_view * f_view).sum(dim=dim, keepdim=True)  # [B,1]
+    norm_sq  = (x_view * x_view).sum(dim=dim, keepdim=True) + eps
+
+    scale = (dot / norm_sq).to(x.dtype) # for amp
+    unsqueeze_times = len(original_shape) - positive_dim - 1
+    for _ in range(unsqueeze_times):
+        scale = scale.unsqueeze(-1)
+    proj_out = scale * x  # broadcast
+    return f_x - proj_out
+
+def connect(x, f_x, *, 
+            method="linear", orthogonal_method="global",
+            dim=-1, eps=1e-6, perturbation=None):
+    if perturbation is not None:
+        raise NotImplementedError("perturbation is not implemented yet")
+    if method == "linear":
+        return x + _identity(x, f_x)
+    elif method == "orthogonal":
+        if orthogonal_method == "global":
+            return x + _orthogonal_global(x, f_x, dim, eps)
+        elif orthogonal_method == "channel":
+            return x + _orthogonal_channel(x, f_x, dim, eps)
+        else:
+            raise ValueError(f"unknown orthogonal  method: {method}")
+    else:
+        raise ValueError(f"unknown connect method: {method}")
+