Spaces:
Running
Running
Commit History
24 days ago
-
Only remap moves that are listed
RemiFabreAssisted-by: Claude:claude-opus-5-5f78ce97
-
Rich feature extraction + weight-based mapping system
RemiFabreExtract 7 new face features (per-eye openness, gaze direction, mouth pucker, cheek squint, brow furrow) using iris and facial landmarks. Replace the 15-field MappingPreset dataclass with a weight-based architecture: LinearMapping (bias + weighted feature sum) and NonLinearFn (per-frame callable for triggers/velocity effects). 12 creative presets: full, expressive, antenna_party, zen, mirror, wink_reactor, gaze_follower, puppet, mood_ring, bobblehead, sleepy, surprised. Old JSON data loads unchanged (new fields have defaults, "full" preset produces identical output).0c4640f -
Elapsed-time video sync and creative mapping presets
RemiFabreVideo sync: server reports playback_elapsed (seconds since motion loop started). Frontend syncs video.currentTime to it on each poll, with drift correction if >0.4s. Eliminates all timing hacks and constant delays. Consistent on first and subsequent plays regardless of video size or cache state. Presets: removed 9 debug presets (axis swaps, negations), replaced with 8 creative presets: full — all features (default) head_and_body — head + body sway head_only — head rotation only expressive — 2.5× amplification, big reactions subtle — reduced for calm content puppet — exaggerated everything mirror — inverted yaw for face-to-face antenna_party — all expressions drive antennas 2× Added antenna_intensity multiplier to MappingPreset for per-preset antenna reactivity scaling.10bdb65 -
Fix video sync race condition, boost amplitudes, add pitch offset
RemiFabreVideo sync: buffer video fully before API call (consistent first/Nth play timing), add startingPlayback flag to prevent poller from killing video during startup sequence. Amplitudes: pitch_scale 1.0→1.8, roll_scale 0.8→1.8, body_yaw_ratio 0.4→0.6, mouth_boost 0.15→0.25, eye_dip 0.10→0.15. Yaw unchanged. Pitch offset: subtract PITCH_OFFSET_RAD (0.10 rad ≈ 5.7°) from pitch signal so a neutral face maps to pitch ≈ 0 instead of looking down.3d57293 -
Fix video-robot sync: start buffering before API call, play on response
RemiFabrePreviously, video playback only started when polling detected the "playing" state (0-500ms after robot started). Now playMove() starts buffering the video immediately, calls the API (robot starts), then plays video as soon as the response arrives. The API round-trip gives the robot a natural ~50ms head start. VIDEO_DELAY_MS remains tunable for further adjustment.026a381 -
Add 300ms video delay to sync playback with robot start
RemiFabreVideo was playing slightly ahead of robot motion. Delay video start by VIDEO_DELAY_MS (300ms) after canplay fires, giving the robot time to begin moving first. Easy to hand-tune the constant.ca9f6f7 -
Negate pitch/roll signs and fix video sizing during playback
RemiFabreInvert pitch and roll extracted from solvePnP to match robot joint conventions. Widen video panel to 1200px and use fetch-based cache warming instead of setting video.src on hidden elements (which caused tiny intrinsic sizes).98db6e2 -
Fix axis labeling from camera frame and improve video-robot sync
RemiFabreEuler axis fix: as_euler("xyz") returns [rot_X, rot_Y, rot_Z] in OpenCV camera frame (X=right, Y=down, Z=forward). In that frame: rot_X = pitch (nod), rot_Y = yaw (turn), rot_Z = roll (tilt) Previously labeled positionally as (roll, pitch, yaw) causing pitch→roll and yaw→pitch cross-mapping on the robot. Video sync: preload video when Play is clicked (before backend responds), use preload="auto" on video elements, and wait for canplay event before starting playback. This eliminates the 0.5-1s delay between robot start and video start.286ac47 -
Fix Euler angle convention mismatch between solvePnP and create_head_pose
RemiFabreThe custom ZYX Euler decomposition produced roll/pitch/yaw labels that didn't match create_head_pose's intrinsic XYZ convention (scipy). This caused yaw→pitch and pitch→roll cross-mapping on the robot. Fix: use scipy.spatial.transform.Rotation.as_euler("xyz") — the exact same convention as create_head_pose uses with R.from_euler("xyz") — so the round-trip is guaranteed correct.094b6dd -
Tighten robot safety limits to match SafetyMixer from ultradancemix_9000
RemiFabreUse the proven motion-safe limits instead of hardware limits: - Yaw: ±175° → ±40° (0.7 rad, SafetyMixer) - Pitch: ±38° → ±23° (0.4 rad, SafetyMixer) - Roll: ±38° → ±29° (0.5 rad, SafetyMixer) - Body: ±155° → ±46° (0.8 rad, SafetyMixer) Prevents aggressive movements from noisy face detection, camera cuts, or characters doing full head rotations in the video.087e6a2 -
Fix dropdown and button hover reset caused by polling re-render
RemiFabrerenderMoves() was rebuilding the entire DOM every 500ms-1.5s, which destroyed open <select> dropdowns and reset CSS hover/active states. Fix: fingerprint the moves list state and skip re-render when unchanged. Also skip re-render when user is interacting with a preset dropdown.b8075c3 -
Add selectable mapping presets for axis debugging and expression tuning
RemiFabre12 mapping presets from simple (single-axis) to full expressive: - yaw_only, pitch_only, roll_only: isolate one rotation axis - swap_yaw_pitch, swap_yaw_roll, swap_pitch_roll: test axis ordering - neg_pitch, neg_yaw, neg_roll: test sign conventions - head_only, head_and_body: standard mapping without expressions - full: all features (eyebrows, mouth, eyes, smile, body) Remap without re-processing: face data saved in JSON, remap endpoint re-applies a different preset instantly. UI shows preset dropdown per move card with Remap button. replay.py accepts --preset flag.565ee8d -
Raise YouTube limits to 10min/1080p, add video title naming and NEW badge
RemiFabre- YouTube: 10 min max (was 2), 1080p (was 720p), 500MB, 300s timeout - Extract YouTube video title via yt-dlp for readable move names - Show blue NEW badge on unplayed moves in the library - Played state persisted in move JSON, badge disappears after first play - Updated README: mapping table, FAQ on simultaneous faces, YouTube limitsd284602 -
Add rich face-to-robot mapping: mouth, eyes, individual eyebrows, face-switch detection
RemiFabre- Extract mouth open/smile, individual eyebrow raise, eye openness from FaceMesh - Map left/right eyebrows independently to left/right antennas - Mouth open boosts antennas (excitement), eye blinks dip antennas - Smile drives subtle body yaw oscillation for liveliness - Detect face switch (nose jump >15%) and reset solvePnP to fix freeze on character change - Add replay.py script for quick process+play iteration on simulator - All 40 tests passadf6f5c -
Add FAQ section to README covering face detection, multi-face, video
RemiFabreformats, YouTube, file storage, annotated video, and known limitationsda890c5 -
Save processed videos to dataset/ inside app directory
RemiFabreChanged default output from ~/reachy_mini_video_mimicry/ to video_mimicry/../dataset/, matching create_move.py's default. Keeps all data (JSON, WAV, video, annotated video) in one place.9d5054f -
Fix solvePnP 180-degree yaw ambiguity and annotated video codec
RemiFabreprocessor.py: Negate z-coordinates in the 3D face model so the face points into the camera frame. The old model had the face pointing away (+z), causing solvePnP to return yaw ~170 degrees for frontal faces (especially on portrait/vertical YouTube videos). annotator.py: OpenCV's avc1 fourcc doesn't work on Linux, so the annotated video was written as mp4v (MPEG-4 Part 2) which browsers can't play. Now writes mp4v first, then re-encodes to H.264 via ffmpeg with libx264. Falls back to mp4v if ffmpeg is unavailable.af1f22d -
Force H.264 codec in yt-dlp to avoid AV1 decode failures
RemiFabreYouTube increasingly serves AV1-encoded video, which OpenCV cannot decode (produces "Missing Sequence Header" errors and zero frames). Add vcodec filter to yt-dlp format string to select only H.264/AVC.cb474bf -
Fix README colorFrom metadata for HuggingFace Spaces
RemiFabre"orange" is not a valid HuggingFace Space color — changed to "yellow".98042ef -
Add annotated video overlay, auto-process on upload, and YouTube URL support
RemiFabre- New annotator.py: draws FACEMESH_CONTOURS overlay on video frames - Upload now auto-queues processing (no manual "Extract" button click) - POST /api/youtube downloads via yt-dlp (720p max, 60s timeout) - Side-by-side video playback (Original + Face Tracking) in UI - Tabbed upload panel (Upload File / YouTube URL) - Process panel removed; progress bar moved into upload panel - Delete cleans up _annotated.mp4; metadata includes has_annotated - 9 new tests (annotator, YouTube API, auto-process, annotated endpoint)4e9284e -
Fix imports to use absolute paths (works from any directory)
RemiFabre1dcf7c1 -
Fix import error and add video replay during playback
RemiFabre- Fix relative import crash when running as `python video_mimicry/main.py` (fallback to absolute imports when no parent package) - Keep original video after processing (moved to output dir instead of deleted) - Add GET /api/videos/{move_id} endpoint to serve original video - Frontend plays the original video in sync when Play is clicked - Video panel appears during playback, auto-hides when done - Move cards show "video" badge when original video is available - Delete also cleans up video files41c9522 -
Add standalone create_move and play_move scripts
RemiFabre- create_move.py: offline video → JSON + WAV pipeline (no robot needed) - play_move.py: plays moves using standard SDK play_move() (robot or sim) - play_move.py --verify: checks motor positions moved during playback - README.md: rewritten to feature the two-script workflow - Tested end-to-end: video with audio → JSON + WAV → simulator playback with verified motor motion (0.70 rad head delta)fb21312 -
Add testing documentation and automated test suite
RemiFabre- TESTING.md: comprehensive guide covering manual testing, automated tests, API testing with curl/Python, output inspection, architecture overview - tests/test_pipeline.py: 31 tests covering FaceProcessor, mapper, audio extraction, FastAPI API endpoints, and full pipeline integration - tests/conftest.py: shared fixtures (synthetic video, real test videos) - pyproject.toml: add pytest config - .gitignore: exclude test_videos/ and .pytest_cache/15f5b04
-
Initial implementation of video_mimicry app
RemiFabreExtract face motion from video using MediaPipe FaceMesh + OpenCV solvePnP, map to Reachy Mini's 9 DOF (head rotation, body yaw, antennas from eyebrows), and replay with synchronized audio. Features: - Upload video via web UI with drag-and-drop - Offline face processing at 25fps, upsampled to 100Hz for playback - Direct DOF mapping: head rotation 1:1, eyebrow→antenna, body yaw dampened - Save processed moves as RecordedMove-compatible JSON for replay - Works without robot (processing only) or with robot/simulator - 3-step UI: Upload → Process (progress bar) → Play from library00a86a2