neonforestmist commited on
Commit
1d3d0b8
Β·
verified Β·
1 Parent(s): 40f149f

Add Clover inpainting support

Browse files
Files changed (1) hide show
  1. Clover-iOS/README.md +188 -0
Clover-iOS/README.md ADDED
@@ -0,0 +1,188 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Clover Image Tiny for iOS
2
+
3
+ Native, private image generation on iPhone with SwiftUI, Core ML, and
4
+ [Clover Image Tiny](https://huggingface.co/neonforestmist/Clover-Image-Tiny).
5
+
6
+ ## Screenshots
7
+
8
+ | Create | Models & Styles | Library |
9
+ |:---:|:---:|:---:|
10
+ | ![Create screen with a generated image](Screenshots/create-output.png) | ![Model picker: Clover downloadable, styles locked until it is installed](Screenshots/models.png) | ![Library grid of generated images](Screenshots/library.png) |
11
+ | On-device generation | Base first, styles opt-in | Saved artwork |
12
+
13
+ ## Features
14
+
15
+ - Runs image generation on the device after model installation
16
+ - Apple-style SwiftUI interface for iPhone and iPad
17
+ - Prompt and negative-prompt controls
18
+ - 4–100 inference steps with both a slider and precise stepper
19
+ - Saved generation timelines with previews every 1–10 steps
20
+ - Guidance from 1.0–20.0
21
+ - Reproducible seeds with seed randomization
22
+ - One to four images per generation
23
+ - Native 512 Γ— 512 output without post-generation cropping
24
+ - PNDM and DPM-Solver++ schedulers
25
+ - NumPy and PyTorch-compatible random generators
26
+ - Neural Engine, automatic, and GPU compute preferences
27
+ - Local artwork library and Photos export
28
+ - Base-first downloads: install Clover, then add only the styles you want
29
+ - Import a Clover-compatible `.safetensors` style from **On My iPhone β†’ Clover β†’ Imported Styles**
30
+ - Visible downloads in **On My iPhone β†’ Clover β†’ Models**
31
+ - Immutable model revisions with byte-count and SHA-256 verification
32
+
33
+ Output resolution is fixed at 512 Γ— 512 by the converted Core ML models.
34
+
35
+ ## Requirements
36
+
37
+ - Xcode 16 or newer
38
+ - iOS 18 or newer
39
+ - A physical iPhone or iPad for Core ML image generation
40
+ - Enough free storage for the selected Core ML model
41
+
42
+ The repository and app bundle do not contain the model weights.
43
+
44
+ ## Run the app
45
+
46
+ 1. Clone this repository.
47
+ 2. Open `Clover.xcodeproj`.
48
+ 3. Select the `Clover` scheme and your Apple development team.
49
+ 4. Choose a connected iPhone or iPad. Simulator is suitable for UI work.
50
+ 5. Build and run.
51
+ 6. Open **Models & Styles** in the app and download a model.
52
+
53
+ The app reads its versioned catalog from
54
+ [`neonforestmist/Clover-Image-Tiny-CoreML`](https://huggingface.co/neonforestmist/Clover-Image-Tiny-CoreML).
55
+ Installed files are checked against the catalog before use.
56
+ Downloaded files are visible in the Files app under
57
+ **On My iPhone β†’ Clover β†’ Models**. Existing downloads from earlier builds are
58
+ migrated into that folder when possible.
59
+
60
+ Clover installs first: its shared components (text encoder, VAE, safety
61
+ checker, tokenizer) plus one stateful base U-Net, about 1.5 GB total. Monet,
62
+ Pointillism, and Watercolor Anime are then **optional 6,927,128-byte LoRA
63
+ style downloads**: `Monet.safetensors`, `Pointillism.safetensors`, and
64
+ `Watercolor-Anime.safetensors`. The Swift pipeline converts the selected style
65
+ weights to FP16 and loads them into the U-Net's mutable Core ML state; it never
66
+ downloads another ~648 MB U-Net. Each style comes from its compact
67
+ `-lora-coreml` Hugging Face repo, which contains only the named style weights,
68
+ state mapping, model card, and license. Styles stay locked until Clover is
69
+ installed. You can also side-load a Clover-compatible LoRA by placing its
70
+ `.safetensors` file directly in **On My iPhone β†’ Clover β†’ Imported Styles**.
71
+ The app detects the filename and loads its tensors into the installed Clover
72
+ U-Net. Older full Core ML model folders remain supported.
73
+
74
+ Output generation remains 512 Γ— 512. The app can save centered 1:1, 4:5, 5:4,
75
+ 9:16, 16:9, and custom width:height crops without downloading additional U-Net
76
+ variants.
77
+
78
+ ## Inpainting resources
79
+
80
+ Inpainting is a separate, standalone Core ML pipeline because its U-Net has a
81
+ 9-channel input and needs a VAE encoder. Download the companion resource
82
+ bundle from
83
+ [`neonforestmist/Clover-Image-Tiny-Inpaint-CoreML`](https://huggingface.co/neonforestmist/Clover-Image-Tiny-Inpaint-CoreML),
84
+ then place its `Resources` directory in an app-managed model folder. The
85
+ runtime entry point is `CoreMLInpaintingService`:
86
+
87
+ ```swift
88
+ let service = CoreMLInpaintingService()
89
+ var settings = GenerationSettings()
90
+ settings.prompt = "replace the masked area with a tiny glass greenhouse"
91
+ let result = try await service.generate(
92
+ resourcesURL: modelFolder,
93
+ request: InpaintingRequest(image: sourceImage, mask: whiteMeansRegenerateMask),
94
+ settings: settings,
95
+ cancellation: GenerationCancellationToken(),
96
+ progress: { _ in }
97
+ )
98
+ ```
99
+
100
+ The service encodes the masked image locally, sends
101
+ `[noisy latent, mask, masked-image latent]` to the batch-one U-Net, and
102
+ composites untouched pixels from the original image. The bundle is converted
103
+ for iOS 18 and is intended for iPhone 15-class hardware; validate final
104
+ latency and memory on a physical device rather than Simulator.
105
+
106
+ ## On-device behavior
107
+
108
+ After installation, prompts and generated images stay on the device. Network
109
+ access is used to refresh the catalog and download model files from Hugging
110
+ Face.
111
+
112
+ Live Step Previews render a lightweight RGB approximation directly from the
113
+ current denoising latent every 1–10 steps. This keeps the VAE decoder and safety
114
+ checker out of the denoising loop, avoiding repeated Core ML memory spikes on
115
+ iPhone; the finished image still uses the full VAE decoder. Preview frames are
116
+ stored as compressed JPEGs inside the finished artwork's folder. The Library
117
+ still shows one tile per final image; opening that imageβ€”or using the controls
118
+ below the newest Create outputβ€”reveals a slider for scrubbing through the saved
119
+ steps. Share and Save operate on the currently selected frame. The Library
120
+ detail screen can also download the complete timeline as a ZIP containing each
121
+ saved preview and the full-resolution final step. More frequent previews still
122
+ increase generation time, storage use, and battery use.
123
+
124
+ The stateful U-Net uses a batch-one input and runs classifier-free guidance in
125
+ two serial passes, cutting the largest activation peak roughly in half. It runs
126
+ with CPU and GPU compute because its mutable style buffers are not
127
+ execution-planned reliably on the Neural Engine. Other pipeline components
128
+ still honor the selected compute preference. Validate real generation on an
129
+ iPhone or iPad; the Simulator is suitable for UI testing. The first generation
130
+ may take longer while Core ML compiles and caches its graphs.
131
+
132
+ The current Core ML U-Net exposes one rank-16 mutable LoRA state, so Clover
133
+ applies one style per generation. Combining arbitrary rank-16 styles exactly
134
+ can require rank 32; supporting that safely needs a new rank-expanded Core ML
135
+ base export and catalog format rather than simply adding two adapter states.
136
+
137
+ ## Project structure
138
+
139
+ ```text
140
+ Clover/
141
+ β”œβ”€β”€ App/ App entry point and navigation
142
+ β”œβ”€β”€ Features/ Create, model picker, settings, and library screens
143
+ β”œβ”€β”€ Models/ Generation settings, artwork, and catalog types
144
+ β”œβ”€β”€ Services/ Core ML inference, downloads, storage, and Photos export
145
+ └── Support/ Assets, colors, and app configuration
146
+ ```
147
+
148
+ The app vendors the small Swift runtime from Apple's
149
+ [`ml-stable-diffusion`](https://github.com/apple/ml-stable-diffusion) project.
150
+ Its model loader is adapted to create an iOS 18 `MLState`, populate 144 mutable
151
+ LoRA tensors from a standard Diffusers safetensors file, and keep that state for
152
+ every denoising step. Apple's license is retained in the vendored package.
153
+
154
+ The model-picker artwork comes from the MIT-licensed Phosphor collection via
155
+ Iconify. See [THIRD_PARTY_ASSETS.md](THIRD_PARTY_ASSETS.md) for attribution.
156
+
157
+ ### Local safety-checker override
158
+
159
+ Safety checking is enabled by default. To disable it only for a local Xcode
160
+ run, duplicate the `Clover` scheme, make the duplicate unshared, then add this
161
+ Run argument:
162
+
163
+ ```text
164
+ -DisableSafetyChecker YES
165
+ ```
166
+
167
+ The unshared scheme is stored under `xcuserdata` and is not staged to GitHub.
168
+
169
+ ## Regenerate the Xcode project
170
+
171
+ The checked-in project is generated from `project.yml` with
172
+ [XcodeGen](https://github.com/yonaskolb/XcodeGen):
173
+
174
+ ```bash
175
+ brew install xcodegen
176
+ xcodegen generate
177
+ ```
178
+
179
+ ## License
180
+
181
+ The iOS source code in this repository is licensed under the
182
+ [Apache License 2.0](LICENSE).
183
+
184
+ Downloadable Clover Image Tiny model weights and derivative style models are
185
+ licensed separately under
186
+ [CreativeML Open RAIL-M](https://huggingface.co/neonforestmist/Clover-Image-Tiny/blob/main/LICENSE).
187
+ The Apache-2.0 license for this application does not relicense those model
188
+ weights. Third-party dependencies retain their respective licenses.