MotionGraphicsBench: Evaluating Motion Graphics Generation with Programmatic Verification
Abstract
Large language models can generate motion graphics from static images and natural language requests, but it remains difficult to evaluate how well models can follow requests that combine spatial, appearance, and temporal requirements. We introduce MotionGraphicsBench, an expert-curated benchmark of 100 instances for animating an input static SVG with a motion edit prompt. By supplying all required visual elements, our editing task reflects a designer-like workflow and isolates motion generation from graphic generation. Derived from real-world Lottie motion designs, our benchmark contains diverse combinations of spatio-temporal animation behaviors. Each benchmark instance pairs its model input with a deterministic, human-readable verification program that accepts different valid implementations and reports which prompt requirements pass or fail, rather than comparing against a reference animation or using a VLM judge. We construct these programs with a multi-agent pipeline using human-labeled animations and synthesized test cases. On a balanced test set of 180 animations, our programs achieve 93.3% raw agreement with human judgments, compared with 83.3% for a GPT-6 Astra VLM baseline, and outperform the baseline in both approval and rejection recall. Across 47 configurations from 32 LLMs and 12 providers, pass@1 ranges from 6.0% to 67.7%. Across 15 models evaluated at their highest available reasoning effort with 32 attempts per instance, additional attempts improve every model, but pass@32 still ranges from 38.0% to 95.0%. Our analysis of verification programs reveals that generated animations often fail to match reference poses, follow requested motion trajectories, or coordinate related events. We release our benchmark at https://huggingface.co/datasets/figma-ai/motion-graphics-bench.
Community
We introduce MotionGraphicsBench, the first motion graphics benchmark that frames animation generation as adding motions to input static images, rather than generating the visual elements from scratch.
Our 100 expert-curated tasks pair static SVGs and motion-edit prompts with deterministic, human-readable verification programs that check whether generated animations satisfy the requested motions.
Blog: https://huggingface.co/blog/figma-ai/motion-graphics-bench
I really like that the verification program accepts any valid implementation instead of matching one reference; it feels like the fair way to grade animation. Which requirement trips models up most: spatial, appearance, or timing?
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- ManimCode: Learning Visual Self-Correction for Educational Animation (2026)
- 3DCodeEdit: Benchmarking Instruction-Guided 3D Program Editing (2026)
- Text2Sim: Agentic Physics-Based Simulation Generation with Distilled Expertise (2026)
- BlenderWorld: Benchmarking Visual Agents for 3D Creation from Real-World References (2026)
- CineRecon: Benchmarking Frontier Models for Agentic Cinematic Reconstruction (2026)
- MaLiang-Harness: A Programmable Path to Image and Video Generation (2026)
- ChartRevise: A Dataset and Evaluation Protocol for Exact Chart Editing via Code (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Hello. One part of your paper caught my attention: the analysis of failed animations in Appendix D.2. Almost all failed animations pass some of the checks, many fail exactly one, and in 125 pools of 32 candidates every check passes in at least one candidate, yet no single animation passes all of them. You call such cases near misses and describe the failures by type: reference poses, motion trajectories and the coordination of related events.
It seems to me that this picture says something not only about the difficulty of individual requirements but also about a limit on how many requirements a model can satisfy at once. The model can apparently satisfy each requirement of your prompt: different checks pass in different candidates. What fails is satisfying all of them in one animation, and a prompt combines 7 to 14 kinds of requirements. Load of this kind leaves a characteristic signature. While the requirements stay below the model's limit, it satisfies all of them; once they exceed it, one or two requirements drop out in each attempt, and different ones each time. This, I think, is what your numbers show: 93.6% of failed animations pass at least one check, 31.6% fail exactly one, and in 8.3% of pools every check passes separately but never all together.
Such a limit can be described as the capacity of a model. A study of the behavior and degradation of language models under controlled load (arXiv:2610.36095) shows that, in the measured model, quality falls as the number of simultaneous instructions grows, following its own law: by roughly the same amount with each doubling of their number. Instruction capacity is derived from these laws: it is the number of simultaneous instructions that a model can follow at a given level of quality. In your benchmark this load seems to correspond to the number of kinds of requirements in the prompt and the number of checks in the program: each requirement is an instruction the model must satisfy in the same animation.
If so, the difficulty of a task in your benchmark depends also on how many requirements it places on the model at once, and it would make sense to report this number next to each task. The spread of difficulty is large: sixteen tasks have pass@1 below 5% and twenty-five have 50% or more, and part of this spread may come from the number of requirements. This also bears on your idea of repairing animations from the check reports: if the model is at its limit, fixing one check may break others. For now this is only a conjecture: in the study mentioned above the laws were measured on simple word-substitution rules, and their transfer to animation has not been shown. It could be tested by adding requirements to the same scene in steps, two, four, eight and so on up to all of them: the scene and the kinds of requirements stay the same, and only their number grows. If the failures come from load, the share of failed checks should grow regularly with each step, and the check reports would show which requirements the model loses first. If difficulty is set only by the kind of requirement, the steps will not increase the share of failed checks.
Get this paper in your agent:
hf papers read 2611.10592 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper