FluidInference/canary-1b-v2-coreml

Canary-1B-v2 — CoreML (ANE)

CoreML conversion of nvidia/canary-1b-v2 for Apple Silicon / Neural Engine, packaged for FluidAudio.

Canary is a FastConformer encoder + Transformer attention encoder-decoder (AED) ASR model (25 European languages, 16384-token SentencePiece BPE). It is decoded autoregressively: the transformer decoder cross-attends to the encoder output and emits tokens greedily until EOS (id 3), with a 1024→16384 projection head.

Files

File	Role	Precision
Preprocessor.mlmodelc	waveform [1,240000] → mel [1,128,1501]	fp32 (CPU)
EncoderInt4.mlmodelc	mel → encoder [1,1024,188]	int4 (ANE, iOS18)
DecoderInt4.mlmodelc	autoregressive transformer → hidden [1,256,1024]	int4 (ANE, iOS18)
Projection.mlmodelc	hidden [1,1024] → logits [1,16384]	fp16 (ANE)
vocab.json	16384 SentencePiece pieces (id → piece)	—
projection_weights.npz	raw projection weights (for Python reference pipelines)	fp32
metadata.json	shapes, sample rate, special token ids	—

Contract: 15 s window (240000 samples @ 16 kHz), 256 decoder steps, eos=3, pad=2, bos=4. int4 weight payloads require iOS 18 / macOS 15.

Variants

• int4 (this default): ANE-runnable, ~573 MB, fastest. Per-block-32 symmetric.
• fp16: exact parity with PyTorch, iOS 17, ~1.8 GB (not included here by default).
• int8 per-channel decodes correctly only on CPU (crashes the GPU/ANE MPSGraph backend), so it is not recommended; use int4 for an ANE-resident small build.

Accuracy / speed (LibriSpeech test-clean, ≤15 s, int4, M-series ANE)

Metric	Value
WER	~2.1%
RTFx	~7x

fp16 CoreML output is byte-identical to the NeMo PyTorch greedy decode.

Usage (FluidAudio)

let manager = try await CanaryManager.load(precision: .int4)
let text = try await manager.transcribe(audioURL: url)

Speech translation (en ↔ 24 languages)

The AED decoder picks its task from the two language slots in the prompt: source == target transcribes, source != target translates. The same model files translate speech with no change — only one prompt token differs. canary-1b-v2 is trained on en ↔ X pairs only.

let translator = try await CanaryManager.load(source: .english, target: .german)
let german = try await translator.transcribe(audioURL: url)

// Audio longer than 15 s: split at pauses (Silero VAD) instead of overlap
// chunking — translated text is not token-stable across an overlap, so the
// seam merge used for transcripts degrades translations.
let vad = try await VadManager()
let longGerman = try await translator.transcribe(audioURL: longURL, vad: vad)

swift run fluidaudiocli canary-transcribe audio.wav --translate-to de [--source-lang en] [--vad]

Prompt token map, FLEURS benchmark hypotheses and scores: FluidInference/canary-speech-translation-coreml.

Conversion

See the mobius conversion pipeline (models/stt/canary-1b-v2/coreml/): convert-coreml.py (NeMo→CoreML), quantize_int4.py, build_projection.py, validate.py, stage_hf.py.

License

Inherits cc-by-4.0 from the base model nvidia/canary-1b-v2.