mlx-community/sam-3d-body-dinov3-bf16

This model was converted to MLX format from facebook/sam-3d-body-dinov3 (revision 11aaa346) for the sam3d_body model in mlx-vlm. Refer to the original model card for more details on the model.

This is an unofficial derivative of Meta's SAM 3D Body and its Momentum Human Rig (MHR) body model. It holds weights and metadata only; the inference code lives in mlx-vlm. SAM 3D Body estimates a full-body human mesh (18,439 vertices, 127 joints, 70 keypoints) from a single image and a person box.

Contents

model.safetensors holds 1,203 tensors (2.80 GB): 615 bfloat16, 574 float32, 11 int32, 3 uint8. Dtypes match Meta's release exactly. The DINOv3-H+ backbone, the bulk of the weights, is bfloat16 as Meta ships it, including its RoPE periods. The decoders, heads, prompt encoder, ray conditioning and MHR body model stay float32. Index tensors are int32, and MHR's parameter masks are uint8. Nothing is cast to float16 or quantized.

Component	Prefixes	Tensors	Dtypes
DINOv3-H+ backbone	backbone	615	bfloat16
Hand-box detection	bbox_embed, hand_box_embedding, hand_cls_embed, hand_pe_layer	10	float32
Body decoder	decoder	194	float32
Hand decoder, heads, tokens and ray conditioning	decoder_hand, head_camera_hand, head_pose_hand, init_camera_hand, init_pose_hand, init_to_token_mhr_hand, keypoint3d_embedding_hand, keypoint3d_posemb_linear_hand, keypoint_embedding_hand, keypoint_feat_linear_hand, keypoint_posemb_linear_hand, prev_to_token_mhr_hand, ray_cond_emb_hand	223	float32
Camera head	head_camera	4	float32
MHR body model	head_pose.body_model	25	float32, int32, uint8
Pose head	head_pose	17	float32, int32
Tokens and keypoint embeddings	init_camera, init_pose, init_to_token_mhr, keypoint3d_embedding, keypoint3d_posemb_linear, keypoint_embedding, keypoint_feat_linear, keypoint_posemb_linear, prev_to_token_mhr	18	float32
Prompt encoder	prompt_encoder, prompt_to_token	94	float32
Ray conditioning	ray_cond_emb	3	float32

The checkpoint is complete: it includes the hand decoder and heads (*_hand), hand-box detection (hand_cls_embed, bbox_embed, hand_pe_layer), the prompt encoder's point and mask embeddings with mask_downscaling, the MHR expression basis (head_pose.body_model.face_shape_vectors), mesh UVs and parameter limits. Keys are mlx-vlm module paths in MLX layout: Conv2d weights are (O, H, W, I), and each backbone qkv projection is split into q_proj, k_proj and v_proj. Only exact duplicates, empty tensors and unused placeholders were left out (47 source tensors). conversion.json lists every tensor's source and transform, every skipped tensor, and MHR's joint and parameter names.

config.json follows mlx-vlm's SAM3DConfig, with Meta's values, including image_size: [512, 512]. model_config.yaml is Meta's original configuration.

Use with mlx

pip install -U mlx-vlm

import numpy as np
from mlx_vlm.models.sam3d_body.generate import SAM3DPredictor
from mlx_vlm.utils import get_model_path, load_model
from PIL import Image

model = load_model(get_model_path("mlx-community/sam-3d-body-dinov3-bf16"), strict=False)
predictor = SAM3DPredictor(model, model.config)
image = np.asarray(Image.open("person.jpg").convert("RGB"))
result = predictor.predict(image, bbox=[x1, y1, x2, y2])

Pass strict=False because these tensors have no mlx-vlm module yet:

• head_pose.body_model.parameter_limits
• head_pose.body_model.pmi
• head_pose.body_model.texcoord_faces
• head_pose.body_model.texcoords
• init_camera_hand
• init_pose_hand
• keypoint3d_embedding_hand
• keypoint_embedding_hand

mlx-vlm's Model.sanitize drops the hand branch and mask_downscaling. SAM3DPredictor.from_pretrained can't load this checkpoint because it reads tensors through NumPy, which has no bfloat16 type.

mlx-vlm's current sam3d_body inference code departs from Meta's reference in several places. Its default is a 512×384 anisotropic crop, where Meta uses a 512×512 isotropic one; this repository's config.json sets 512×512. Its decoder FFN uses ReLU instead of GELU. Its CLIFF condition, camera translation and ray map use different box sizes. Expect its outputs to differ from Meta's. The weights here are faithful to Meta's release either way.

Conversion and validation

The converter read model.ckpt and assets/mhr_model.pt at revision 11aaa346c7204874a1cbafe3d39a979080b2c55a. It renamed keys to mlx-vlm module paths, split the fused qkv weights, and merged each effective bias (bias * bias_mask) into q_proj and v_proj. k_proj has no bias; its masked bias is exactly zero. It also moved convolution channels last and narrowed int64 indices to int32. Every stored tensor was read back and matches its source bit for bit after the layout transform is undone. With mlx-vlm 0.7.2 and MLX 0.32.2, the checkpoint loads through mlx_vlm.utils.load_model: every model parameter matches its tensor in the file, none is missing, and no shape differs. A forward pass on a synthetic image produced finite outputs. This doesn't claim parity with Meta's PyTorch model; see the warning above.

License and credits

SAM 3D Body is by Meta Superintelligence Labs (code, paper). Its weights are under the SAM License, a copy of which is in LICENSE. The MHR body model is Meta's Momentum Human Rig (paper), under the Apache License 2.0 in LICENSE-APACHE. The MHR tensors here are renamed and re-laid out as conversion.json describes.

@article{yang2026sam3dbody,
  title={SAM 3D Body: Robust Full-Body Human Mesh Recovery},
  author={Yang, Xitong and Kukreja, Devansh and Pinkus, Don and Sagar, Anushka and Fan, Taosha
          and Park, Jinhyung and Shin, Soyong and Cao, Jinkun and Liu, Jiawei and Ugrinovic,
          Nicolas and Feiszli, Matt and Malik, Jitendra and Dollar, Piotr and Kitani, Kris},
  journal={arXiv preprint arXiv:2602.15989},
  year={2026}
}

@misc{MHR:2025,
  title={MHR: Momentum Human Rig},
  author={Ferguson, Aaron and others},
  year={2025},
  eprint={2511.15586},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2511.15586}
}