CARDIAC-FM

Multimodal foundation model for cardiovascular risk prediction from 12-lead ECG and cardiac MRI.

• Paper: CARDIAC-FM: A Multimodal Foundation Model for Cardiovascular Risk Prediction Using ECG and Cardiac MRI
• Code: https://github.com/lst627/CARDIAC-FM

The headline model (m75) is a cross-view CineMA masked autoencoder pretrained on UK Biobank cardiac MRI, then contrastively aligned (InfoNCE) with an ECG-FM encoder. It is developed on UK Biobank and validated externally on CHS and MESA.

Files

file	what
stage1_cinema_m75.pth	the headline model — ECG↔MRI contrastively aligned encoder
cinema_mae_m75.pth	self-supervised cardiac MRI encoder (CineMA MAE), used to build stage 1
downstream_m75/af5_ecg.pth	UKB-fine-tuned 5-year AF model, ECG only
downstream_m75/af5_ecg_mri.pth	UKB-fine-tuned 5-year AF model, ECG + MRI
downstream_m75/hf5_ecg.pth	UKB-fine-tuned 5-year HF model, ECG only
downstream_m75/hf5_ecg_mri.pth	UKB-fine-tuned 5-year HF model, ECG + MRI

Architecture (must match at load time): ViT-base — embed_dim 768, depth 12, heads 12, img_size 224, n_frames 8, view_encoder conv, n_sa_slices 3, mask ratio 0.75.

stage1_cinema_m75.pth contains four sub-modules: mri_encoder.* (the CineMA MAE), ecg_encoder.* (ECG-FM), and the two projection heads mri_projection.* / ecg_projection.*, plus the learned InfoNCE temperature.

Not included

The ECG-FM backbone (mimic_iv_ecg_physionet_pretrained.pt) is a third-party checkpoint and is not redistributed here. Obtain it from the upstream ECG-FM release; it is required to build the ECG encoder architecture before the aligned weights load on top.

Usage

git clone https://github.com/lst627/CARDIAC-FM
cd CARDIAC-FM

conda create -n cardiacfm python=3.12 -y && conda activate cardiacfm
pip install torch torchvision          # match your CUDA
pip install -r requirements.txt        # includes fairseq-signals (pinned git install)

Download the aligned and fine-tuned checkpoints:

from huggingface_hub import hf_hub_download

stage1 = hf_hub_download("lst627/CARDIAC-FM", "stage1_cinema_m75.pth")
fine_tuned = hf_hub_download("lst627/CARDIAC-FM", "downstream_m75/af5_ecg_mri.pth")

Continue fine-tuning based on our fine-tuned model

Based on our fine-tuned model: pass the matching downstream_m75/ checkpoint with --finetuned_ckpt. The outcome and input mode must match the selected checkpoint.

python UKBB/downstream/downstream_ecgmri_cinema.py \
  --outcome my_outcome --mode ecg_mri --training_type senc_proj \
  --cl_ckpt stage1_cinema_m75.pth \
  --ecg_ckpt /path/to/ecgfm_mimic_iv_physionet.pt \
  --finetuned_ckpt downstream_m75/af5_ecg_mri.pth \
  --mri_dir prepared_mri/ --ecg_dir prepared/ECG_manifest \
  --csv_train splits/train.csv --csv_val splits/valid.csv --csv_test splits/test.csv \
  --out_dir runs/my_outcome \
  --view_encoder conv --n_sa_slices 3 \
  --embed_dim 768 --encoder_depth 12 --encoder_heads 12 --pool per_view

For ECG-only continued fine-tuning, use --mode ecg and the corresponding _ecg.pth checkpoint. --view_encoder, --pool, and encoder dimensions must match the checkpoint architecture.

For inference without additional training, use infer.py with the matching fine-tuned checkpoint; see the code repository README for ECG and ECG+MRI examples.

Supported input configurations: ECG only; ECG + MRI; either one combined with clinical risk scores (CHARGE-AF / PREVENT-HF).

Data

Model development used UK Biobank; external validation used CHS and MESA. Access to each requires a separate application to that study. No participant data is distributed here.

Intended use and limitations

Research use only. This model is not a medical device and has not been prospectively validated or approved for clinical decision-making.

• Developed on UK Biobank, whose participants are predominantly of European ancestry and healthier than the general population; external validation on CHS and MESA covers older and more diverse cohorts, but performance in other populations is unverified.
• Requires ECG preprocessed to the pipeline's expected format (12-lead, 500 Hz, baseline-corrected) and, for the multimodal arms, cardiac MRI preprocessed as described in the repository README.

License

MIT, matching the code repository. Third-party components (ECG-FM, CineMA, fairseq-signals) carry their own licenses.

Citation

@article{li2026cardiacfm,
  title={CARDIAC-FM: A Multimodal Foundation Model for Cardiovascular Risk Prediction Using ECG and Cardiac MRI},
  author={Li, Fumin and Li, Siting and Qian, Yuhan and Chen, Bojun and Brody, Jennifer A and Yogeswaran, Vidhushei and Wiggins, Kerri L and Sitlani, Colleen M and Bis, Joshua C and Shojaie, Ali and others},
  journal={medRxiv},
  year={2026},
  doi={10.64898/2026.03.16.26348526}
}