Text-to-Image
Diffusers
Safetensors
StableDiffusionPipeline
local-ai
image-generation
lora
apple-silicon
clover-image
diffusion
stable-diffusion
knowledge-distillation
compact
small-model
local-inference
edge-inference
mobile-inference
core-ml
iphone
sd-1.4-class
Instructions to use neonforestmist/Clover-Image-Tiny with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use neonforestmist/Clover-Image-Tiny with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("nota-ai/bk-sdm-tiny-2m", dtype=torch.bfloat16, device_map="cuda") pipe.load_lora_weights("neonforestmist/Clover-Image-Tiny") prompt = "a glass of red wine" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- Draw Things
- DiffusionBee
Add Clover inpainting support
Browse files- Clover-iOS/README.md +188 -0
Clover-iOS/README.md
ADDED
|
@@ -0,0 +1,188 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Clover Image Tiny for iOS
|
| 2 |
+
|
| 3 |
+
Native, private image generation on iPhone with SwiftUI, Core ML, and
|
| 4 |
+
[Clover Image Tiny](https://huggingface.co/neonforestmist/Clover-Image-Tiny).
|
| 5 |
+
|
| 6 |
+
## Screenshots
|
| 7 |
+
|
| 8 |
+
| Create | Models & Styles | Library |
|
| 9 |
+
|:---:|:---:|:---:|
|
| 10 |
+
|  |  |  |
|
| 11 |
+
| On-device generation | Base first, styles opt-in | Saved artwork |
|
| 12 |
+
|
| 13 |
+
## Features
|
| 14 |
+
|
| 15 |
+
- Runs image generation on the device after model installation
|
| 16 |
+
- Apple-style SwiftUI interface for iPhone and iPad
|
| 17 |
+
- Prompt and negative-prompt controls
|
| 18 |
+
- 4β100 inference steps with both a slider and precise stepper
|
| 19 |
+
- Saved generation timelines with previews every 1β10 steps
|
| 20 |
+
- Guidance from 1.0β20.0
|
| 21 |
+
- Reproducible seeds with seed randomization
|
| 22 |
+
- One to four images per generation
|
| 23 |
+
- Native 512 Γ 512 output without post-generation cropping
|
| 24 |
+
- PNDM and DPM-Solver++ schedulers
|
| 25 |
+
- NumPy and PyTorch-compatible random generators
|
| 26 |
+
- Neural Engine, automatic, and GPU compute preferences
|
| 27 |
+
- Local artwork library and Photos export
|
| 28 |
+
- Base-first downloads: install Clover, then add only the styles you want
|
| 29 |
+
- Import a Clover-compatible `.safetensors` style from **On My iPhone β Clover β Imported Styles**
|
| 30 |
+
- Visible downloads in **On My iPhone β Clover β Models**
|
| 31 |
+
- Immutable model revisions with byte-count and SHA-256 verification
|
| 32 |
+
|
| 33 |
+
Output resolution is fixed at 512 Γ 512 by the converted Core ML models.
|
| 34 |
+
|
| 35 |
+
## Requirements
|
| 36 |
+
|
| 37 |
+
- Xcode 16 or newer
|
| 38 |
+
- iOS 18 or newer
|
| 39 |
+
- A physical iPhone or iPad for Core ML image generation
|
| 40 |
+
- Enough free storage for the selected Core ML model
|
| 41 |
+
|
| 42 |
+
The repository and app bundle do not contain the model weights.
|
| 43 |
+
|
| 44 |
+
## Run the app
|
| 45 |
+
|
| 46 |
+
1. Clone this repository.
|
| 47 |
+
2. Open `Clover.xcodeproj`.
|
| 48 |
+
3. Select the `Clover` scheme and your Apple development team.
|
| 49 |
+
4. Choose a connected iPhone or iPad. Simulator is suitable for UI work.
|
| 50 |
+
5. Build and run.
|
| 51 |
+
6. Open **Models & Styles** in the app and download a model.
|
| 52 |
+
|
| 53 |
+
The app reads its versioned catalog from
|
| 54 |
+
[`neonforestmist/Clover-Image-Tiny-CoreML`](https://huggingface.co/neonforestmist/Clover-Image-Tiny-CoreML).
|
| 55 |
+
Installed files are checked against the catalog before use.
|
| 56 |
+
Downloaded files are visible in the Files app under
|
| 57 |
+
**On My iPhone β Clover β Models**. Existing downloads from earlier builds are
|
| 58 |
+
migrated into that folder when possible.
|
| 59 |
+
|
| 60 |
+
Clover installs first: its shared components (text encoder, VAE, safety
|
| 61 |
+
checker, tokenizer) plus one stateful base U-Net, about 1.5 GB total. Monet,
|
| 62 |
+
Pointillism, and Watercolor Anime are then **optional 6,927,128-byte LoRA
|
| 63 |
+
style downloads**: `Monet.safetensors`, `Pointillism.safetensors`, and
|
| 64 |
+
`Watercolor-Anime.safetensors`. The Swift pipeline converts the selected style
|
| 65 |
+
weights to FP16 and loads them into the U-Net's mutable Core ML state; it never
|
| 66 |
+
downloads another ~648 MB U-Net. Each style comes from its compact
|
| 67 |
+
`-lora-coreml` Hugging Face repo, which contains only the named style weights,
|
| 68 |
+
state mapping, model card, and license. Styles stay locked until Clover is
|
| 69 |
+
installed. You can also side-load a Clover-compatible LoRA by placing its
|
| 70 |
+
`.safetensors` file directly in **On My iPhone β Clover β Imported Styles**.
|
| 71 |
+
The app detects the filename and loads its tensors into the installed Clover
|
| 72 |
+
U-Net. Older full Core ML model folders remain supported.
|
| 73 |
+
|
| 74 |
+
Output generation remains 512 Γ 512. The app can save centered 1:1, 4:5, 5:4,
|
| 75 |
+
9:16, 16:9, and custom width:height crops without downloading additional U-Net
|
| 76 |
+
variants.
|
| 77 |
+
|
| 78 |
+
## Inpainting resources
|
| 79 |
+
|
| 80 |
+
Inpainting is a separate, standalone Core ML pipeline because its U-Net has a
|
| 81 |
+
9-channel input and needs a VAE encoder. Download the companion resource
|
| 82 |
+
bundle from
|
| 83 |
+
[`neonforestmist/Clover-Image-Tiny-Inpaint-CoreML`](https://huggingface.co/neonforestmist/Clover-Image-Tiny-Inpaint-CoreML),
|
| 84 |
+
then place its `Resources` directory in an app-managed model folder. The
|
| 85 |
+
runtime entry point is `CoreMLInpaintingService`:
|
| 86 |
+
|
| 87 |
+
```swift
|
| 88 |
+
let service = CoreMLInpaintingService()
|
| 89 |
+
var settings = GenerationSettings()
|
| 90 |
+
settings.prompt = "replace the masked area with a tiny glass greenhouse"
|
| 91 |
+
let result = try await service.generate(
|
| 92 |
+
resourcesURL: modelFolder,
|
| 93 |
+
request: InpaintingRequest(image: sourceImage, mask: whiteMeansRegenerateMask),
|
| 94 |
+
settings: settings,
|
| 95 |
+
cancellation: GenerationCancellationToken(),
|
| 96 |
+
progress: { _ in }
|
| 97 |
+
)
|
| 98 |
+
```
|
| 99 |
+
|
| 100 |
+
The service encodes the masked image locally, sends
|
| 101 |
+
`[noisy latent, mask, masked-image latent]` to the batch-one U-Net, and
|
| 102 |
+
composites untouched pixels from the original image. The bundle is converted
|
| 103 |
+
for iOS 18 and is intended for iPhone 15-class hardware; validate final
|
| 104 |
+
latency and memory on a physical device rather than Simulator.
|
| 105 |
+
|
| 106 |
+
## On-device behavior
|
| 107 |
+
|
| 108 |
+
After installation, prompts and generated images stay on the device. Network
|
| 109 |
+
access is used to refresh the catalog and download model files from Hugging
|
| 110 |
+
Face.
|
| 111 |
+
|
| 112 |
+
Live Step Previews render a lightweight RGB approximation directly from the
|
| 113 |
+
current denoising latent every 1β10 steps. This keeps the VAE decoder and safety
|
| 114 |
+
checker out of the denoising loop, avoiding repeated Core ML memory spikes on
|
| 115 |
+
iPhone; the finished image still uses the full VAE decoder. Preview frames are
|
| 116 |
+
stored as compressed JPEGs inside the finished artwork's folder. The Library
|
| 117 |
+
still shows one tile per final image; opening that imageβor using the controls
|
| 118 |
+
below the newest Create outputβreveals a slider for scrubbing through the saved
|
| 119 |
+
steps. Share and Save operate on the currently selected frame. The Library
|
| 120 |
+
detail screen can also download the complete timeline as a ZIP containing each
|
| 121 |
+
saved preview and the full-resolution final step. More frequent previews still
|
| 122 |
+
increase generation time, storage use, and battery use.
|
| 123 |
+
|
| 124 |
+
The stateful U-Net uses a batch-one input and runs classifier-free guidance in
|
| 125 |
+
two serial passes, cutting the largest activation peak roughly in half. It runs
|
| 126 |
+
with CPU and GPU compute because its mutable style buffers are not
|
| 127 |
+
execution-planned reliably on the Neural Engine. Other pipeline components
|
| 128 |
+
still honor the selected compute preference. Validate real generation on an
|
| 129 |
+
iPhone or iPad; the Simulator is suitable for UI testing. The first generation
|
| 130 |
+
may take longer while Core ML compiles and caches its graphs.
|
| 131 |
+
|
| 132 |
+
The current Core ML U-Net exposes one rank-16 mutable LoRA state, so Clover
|
| 133 |
+
applies one style per generation. Combining arbitrary rank-16 styles exactly
|
| 134 |
+
can require rank 32; supporting that safely needs a new rank-expanded Core ML
|
| 135 |
+
base export and catalog format rather than simply adding two adapter states.
|
| 136 |
+
|
| 137 |
+
## Project structure
|
| 138 |
+
|
| 139 |
+
```text
|
| 140 |
+
Clover/
|
| 141 |
+
βββ App/ App entry point and navigation
|
| 142 |
+
βββ Features/ Create, model picker, settings, and library screens
|
| 143 |
+
βββ Models/ Generation settings, artwork, and catalog types
|
| 144 |
+
βββ Services/ Core ML inference, downloads, storage, and Photos export
|
| 145 |
+
βββ Support/ Assets, colors, and app configuration
|
| 146 |
+
```
|
| 147 |
+
|
| 148 |
+
The app vendors the small Swift runtime from Apple's
|
| 149 |
+
[`ml-stable-diffusion`](https://github.com/apple/ml-stable-diffusion) project.
|
| 150 |
+
Its model loader is adapted to create an iOS 18 `MLState`, populate 144 mutable
|
| 151 |
+
LoRA tensors from a standard Diffusers safetensors file, and keep that state for
|
| 152 |
+
every denoising step. Apple's license is retained in the vendored package.
|
| 153 |
+
|
| 154 |
+
The model-picker artwork comes from the MIT-licensed Phosphor collection via
|
| 155 |
+
Iconify. See [THIRD_PARTY_ASSETS.md](THIRD_PARTY_ASSETS.md) for attribution.
|
| 156 |
+
|
| 157 |
+
### Local safety-checker override
|
| 158 |
+
|
| 159 |
+
Safety checking is enabled by default. To disable it only for a local Xcode
|
| 160 |
+
run, duplicate the `Clover` scheme, make the duplicate unshared, then add this
|
| 161 |
+
Run argument:
|
| 162 |
+
|
| 163 |
+
```text
|
| 164 |
+
-DisableSafetyChecker YES
|
| 165 |
+
```
|
| 166 |
+
|
| 167 |
+
The unshared scheme is stored under `xcuserdata` and is not staged to GitHub.
|
| 168 |
+
|
| 169 |
+
## Regenerate the Xcode project
|
| 170 |
+
|
| 171 |
+
The checked-in project is generated from `project.yml` with
|
| 172 |
+
[XcodeGen](https://github.com/yonaskolb/XcodeGen):
|
| 173 |
+
|
| 174 |
+
```bash
|
| 175 |
+
brew install xcodegen
|
| 176 |
+
xcodegen generate
|
| 177 |
+
```
|
| 178 |
+
|
| 179 |
+
## License
|
| 180 |
+
|
| 181 |
+
The iOS source code in this repository is licensed under the
|
| 182 |
+
[Apache License 2.0](LICENSE).
|
| 183 |
+
|
| 184 |
+
Downloadable Clover Image Tiny model weights and derivative style models are
|
| 185 |
+
licensed separately under
|
| 186 |
+
[CreativeML Open RAIL-M](https://huggingface.co/neonforestmist/Clover-Image-Tiny/blob/main/LICENSE).
|
| 187 |
+
The Apache-2.0 license for this application does not relicense those model
|
| 188 |
+
weights. Third-party dependencies retain their respective licenses.
|