JST-2 Motion: full-body 3D motion from text or video
Full-body 3D motion from text or video, applied directly to JST-2 characters without a separate retargeting step.
Viggle Research
This is the full motion benchmark behind the JST-2 launch: how JST-2 generates full-body 3D motion from text or video, how we tested it, and where it still falls short.
Key Takeaways#
- Full-body 3D motion from text or video. For video inputs, JST-2 reconstructs visible movement and uses learned dynamics to infer how unseen parts of the body move.
- Text-to-motion. JST-2 leads both baselines across the evaluated criteria, and performs particularly well on hand gestures and dance.
- Video-to-motion. JST-2 leads both baselines overall. The reconstructed 3D motion retains the source performance's rhythm and changes of speed, along with fine hand and finger movements.
- Ready for characters. JST-2 motions can be applied directly to JST-2 characters, without a separate retargeting step, and are natively compatible with standard 3D character rigs.
Full-body 3D motion from text or video#
JST-2 generates full-body 3D motion from text or video. For video inputs, it reconstructs visible movement and uses learned dynamics to infer how unseen parts of the body move.
From text, one sentence returns a motion. Artists and developers stop digging through a motion library and stop hand-tuning keyframes, and an AI agent can write the motion instruction itself.
From video, every motion that has already happened in the real world becomes a source. For games and animation that cuts the cost of motion capture; for embodied-AI training it supplies human-motion data. The model does not need the subject facing the camera: seen from the side, at an angle, or with limbs occluded, it infers the hidden part of the movement from context.
JST-2 motions can be applied directly to JST-2 characters, without a separate retargeting step. They are also natively compatible with standard 3D character rigs. Together with generated scenes, these characters and motions form the worlds of the JST-2 launch.
How we evaluated JST-2 Motion#
Human evaluators assessed both tasks in blind comparisons. All clips were played on the same untextured MetaHuman rig, keeping the comparison focused on motion rather than appearance.
For text-to-motion, JST-2 was compared with HY-Motion-1.0 and Cartwheel Swing on prompts across eight buckets: daily locomotion and posture, daily object and self-care, upper-body and hand gesture, fast combat and stunts, slow controlled and emotional motion, sports, performance and dance, and social and health interaction.
For video-to-motion, JST-2 was compared with DeepMotion Animate 3D and Cartwheel Comic 4 on real footage across locomotion, sports and athletics, fitness and outdoor activity, daily activities, social interaction and leisure, and game character actions: different people, viewpoints, ranges of movement and directions of travel. Each video motion in the viewer below plays beside its source footage, with a link to the creator.
Average Score is the equal-weight mean of all criteria, including Overall Preference. Draws count as half a win. A score of 50% indicates equal preference between JST-2 and the comparison model.
Text-to-motion#
For text-to-motion, JST-2 leads both baselines across the evaluated criteria. The model performs particularly well on hand gestures and dance, following the intent of the prompt while producing natural, continuous movement.
vs Cartwheel Swing
vs HY-Motion-1.0
Video-to-motion#
JST-2 leads both video-to-motion baselines overall. The reconstructed 3D motion retains the source performance's rhythm and changes of speed, along with fine hand and finger movements. The cadence of the performance comes with the motion, rather than a replay of its key poses, and detail at the end of the limb stays in the asset instead of being averaged away.
vs Cartwheel Comic 4
vs DeepMotion Animate 3D
What comes next#
Upcoming releases will reconstruct a person's path through the scene from video, alongside their body motion. We will also improve motion accuracy. Today a few limits remain.
Fast, full-body, strongly periodic motion is where the most headroom is. Prompt control can go further too: direction of travel, the transition between two motions, and the rhythm of a motion.
Fast sports and athletics footage is where Cartwheel Comic 4 still leads in video-to-motion.
Foot contact remains the thing to keep working on. Feet can slide slightly on plant, on lift-off and while they should be held on the ground.
In JST-2, character motion is driven by learned dynamics. Scene collisions and object interactions still rely on rule-based physics. We are developing JST-3 to replace those rules with learned dynamics that predict how objects interact and how those interactions change the world.
Read the JST-2 launch