areflesh commited on
Commit
8836a24
Β·
verified Β·
1 Parent(s): e0054e4

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +140 -1
README.md CHANGED
@@ -7,4 +7,143 @@ base_model:
7
  pipeline_tag: object-detection
8
  tags:
9
  - cultural
10
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7
  pipeline_tag: object-detection
8
  tags:
9
  - cultural
10
+ ---
11
+ # πŸ–ΌοΈ Saint George on a Bike – Mask R-CNN for Iconographic Object Detection
12
+
13
+ ## Model Summary
14
+
15
+ This model uses the [Matterport Mask R-CNN](https://github.com/matterport/Mask_RCNN) implementation fine-tuned for detecting iconographic and symbolic elements in religious artworks. It is developed as part of the **Saint George on a Bike** project to enable semantic enrichment and understanding of historical imagery.
16
+
17
+ ---
18
+
19
+ ## 🧠 Model Details
20
+
21
+ - **Architecture**: Mask R-CNN with ResNet backbone
22
+ - **Framework**: TensorFlow 1.14.0 + Keras 2.2.5
23
+ - **Source**: https://github.com/matterport/Mask_RCNN
24
+ - **Configuration**:
25
+ - `NUM_CLASSES`: 69+1 (background)
26
+ - `DETECTION_MIN_CONFIDENCE`: 0.76
27
+
28
+ ---
29
+
30
+ ## 🎯 Use Cases
31
+
32
+ - Iconography detection in religious paintings
33
+ - Digital humanities and art historical research
34
+ - Training multimodal models for cultural heritage
35
+ - Enriching metadata in museum and archive collections
36
+
37
+ ---
38
+
39
+ ## 🏷️ Labels (Selected)
40
+
41
+ The model detects over 40 iconographic concepts including:
42
+
43
+ - `crucifixion`
44
+ - `angel`
45
+ - `crown of thorns`
46
+ - `monk`
47
+ - `sword`
48
+ - `chalice`
49
+ - `dove`
50
+ - `lion`, `shepherd`, `scroll`, `key of heaven`, `mitre`, and more
51
+
52
+ > Full class list is available in the source notebook.
53
+
54
+ ---
55
+
56
+ ## πŸ“Š Training Data
57
+
58
+ - The model was trained on a DEArt dataset curated for the **Saint George on a Bike** project.
59
+ - Dataset contains annotated religious artworks with rich symbolic content.
60
+ - Format and exact size unspecified; annotations PascalXML structure.
61
+
62
+ ---
63
+
64
+ ## πŸ§ͺ Example Usage
65
+
66
+ ```python
67
+ from mrcnn.config import Config
68
+ from mrcnn.model import MaskRCNN
69
+ from mrcnn.model import mold_image
70
+ from keras.preprocessing.image import load_img, img_to_array
71
+ from numpy import expand_dims
72
+ import matplotlib.pyplot as plt
73
+ from matplotlib.patches import Rectangle
74
+
75
+ # Define class labels (shortened list)
76
+ classids=["BG","crucifixion","angel","person","crown of thorns", "horse", "dragon","bird","dog","boat","cat","book",
77
+ "sheep","shepherd","elephant","zebra","crown","tiara","camauro","zucchetto","mitre","saturno","skull",
78
+ "orange","apple","banana","nude","monk","lance","key of heaven", "banner","chalice","palm","sword","rooster",
79
+ "knight","scroll","lily","horn","prayer","tree","arrow","crozier","deer","devil","dove","eagle","hands",
80
+ "head","lion","serpent","stole","trumpet","judith","halo","helmet","shield","jug","holy shroud","god the father",
81
+ "swan", "butterfly", "bear", "centaur","pegasus","donkey","mouse","monkey","cow","unicorn"]
82
+
83
+ # Define the inference config
84
+ class PredictionConfig(Config):
85
+ NAME = "PREDICTION_cfg"
86
+ NUM_CLASSES = len(classids)
87
+ GPU_COUNT = 1
88
+ IMAGES_PER_GPU = 1
89
+ DETECTION_MIN_CONFIDENCE = 0.76
90
+
91
+ # Initialize model
92
+ cfg = PredictionConfig()
93
+ model = MaskRCNN(mode='inference', model_dir='./', config=cfg)
94
+ model.load_weights('<weights of model>', by_name=True)
95
+
96
+ # Load and process image
97
+ img = load_img("example.jpg")
98
+ image = img_to_array(img)
99
+ scaled_image = mold_image(image, cfg)
100
+ sample = expand_dims(scaled_image, 0)
101
+
102
+ # Run detection
103
+ yhat = model.detect(sample, verbose=0)[0]
104
+
105
+ # Visualize detections
106
+ fig = plt.figure(figsize=(12, 12))
107
+ ax = fig.add_subplot(111)
108
+ ax.imshow(img)
109
+ for i in range(len(yhat['rois'])):
110
+ y1, x1, y2, x2 = yhat['rois'][i]
111
+ width, height = x2 - x1, y2 - y1
112
+ rect = Rectangle((x1, y1), width, height, fill=False, color='red')
113
+ ax.add_patch(rect)
114
+ ax.text(x1 + 5, y1 + 10, classids[yhat['class_ids'][i]], fontsize=12, color='white')
115
+ plt.show()
116
+ ```
117
+
118
+ ---
119
+
120
+ ## πŸ“Œ Limitations
121
+
122
+ - Accuracy on modern images or non-religious art is not guaranteed
123
+ - Requires legacy versions of TensorFlow and Keras
124
+
125
+ ---
126
+
127
+ ## πŸ“š Citation
128
+
129
+ If you use this model, please cite:
130
+
131
+ - The Matterport Mask R-CNN repository: https://github.com/matterport/Mask_RCNN
132
+ - DEArt Dataset
133
+ ```
134
+ @misc{reshetnikov2022deartdataseteuropeanart,
135
+ title={DEArt: Dataset of European Art},
136
+ author={Artem Reshetnikov and Maria-Cristina Marinescu and Joaquim More Lopez},
137
+ year={2022},
138
+ eprint={2211.01226},
139
+ archivePrefix={arXiv},
140
+ primaryClass={cs.CV},
141
+ url={https://arxiv.org/abs/2211.01226},
142
+ }
143
+ ```
144
+
145
+ ---
146
+
147
+ ## πŸ™ Acknowledgements
148
+
149
+ This research has been supported by the Saint George on a Bike project 2018-EU-IA-0104, co-financed by the Connecting Europe Facility of the European Union.