Viggle-Animate: Character Replacement in Video from a Single Repainted Frame#

Demo · Weights · viggle.ai/h3

Key Takeaways#

• Viggle-Animate is an open video model that replaces a character in a video using a single repainted frame. Given a driving video and a repainted frame sampled from it, the model propagates the character edit while preserving the source motion and camera trajectory.
• Once the repainted frame is available, video inference needs no auxiliary models — no pose estimator, segmentation model, face tracker, or user-provided text prompt. Two inputs and three forward passes produce a 124-frame clip in 26 s on one GPU.
• Its largest qualitative advantages appear in difficult cases, including fast motion and extreme poses. The same interface also supports animals, stylized characters, and selected non-humanoid objects.

Abstract#

Viggle-Animate replaces a character throughout a video using one repainted frame. First, a frame from the driving video is edited with an image editor such as gpt-image. The video and edited frame are then provided to Viggle-Animate, which propagates the replacement while preserving the source motion and camera trajectory.

Unlike methods that require pose skeletons, segmentation masks, face crops, or background plates, Viggle-Animate uses no intermediate representations or user-provided text prompt during video inference. Since the reference comes from the driving video, its pose, camera, lighting, and background are aligned with the footage. Viggle-Animate is a full-parameter fine-tune of MiniMax-H3's 33.1B-parameter ref2va transformer, distilled with Distribution Matching Distillation (DMD) to three forward passes. In a matched comparison on the same machine and B200 GPU, using the same source videos, output resolution, and frame count, it renders 124 frames in 26 s, 6.1× faster than Wan2.2-Animate-14B.

Method: Repainted-Frame Conditioning#

Character replacement must specify both the appearance of the new character and how that character moves through the shot. Methods conditioned on a standalone character image must align two different sources: appearance from the reference image, and pose, motion, camera, and lighting from the driving video. Existing systems often use pose skeletons, masks, face crops, or background representations to perform this alignment.

Viggle-Animate instead uses an edited frame sampled directly from the driving video. An image editor first replaces the character in that frame while preserving its pose, lighting, and background. This produces a reference that is already spatially aligned with the source video. Viggle-Animate then performs the second stage: propagating the replacement across time.

The video stage reduces inference cost in two ways. First, once the repainted frame has been prepared, it does not run a pose estimator, segmenter, face tracker, or text encoder. Second, the teacher's thirty-pass sampler is distilled to three forward passes. The complete video inference stage therefore runs with one model on one GPU.

Dual-Teacher Distillation#

We distill the student from two teachers, using a different teacher at each end of the noise schedule. At high noise levels, we use our task-specific fine-tuned model to emphasize replacement fidelity. At low noise levels, we use the original MiniMax-H3 model to preserve fine detail and texture. This design aims to combine the replacement capability of the fine-tuned model with the visual quality of the original model while reducing the transfer of visual regressions to the student.

Topology-Agnostic Conditioning#

The inference pipeline does not rely on human-specific pose representations, so its conditioning does not explicitly restrict the replacement to a human topology. Qualitative examples show that the model can animate animals, stylized characters, and some non-humanoid objects. In practice, the range of possible replacements depends on what can be represented clearly in the repainted frame.

Inference Efficiency#

• 26 s per render — 124 frames at 24 fps, 480×832, a single B200.
• 3 forward passes — --steps 4 sets four sigma boundaries, so three passes between them.
• 2 inputs — a clip, and one of its own frames repainted.
• 0 auxiliary models during video inference — no pose estimator, segmenter, face tracker, or text encoder after the repainted frame has been prepared.

Both models are evaluated on the same machine and B200 GPU, using the same source videos at 124 frames, 24 fps, and 480×832 resolution. Each model uses the sampling configuration shown below.

6.1× faster per render and 10.3× faster in sampling alone. Parameter counts use the same counting convention for both models. The reported 160 s for Wan does not include its additional preprocessing pass.

Fast Motion and Pose Fidelity#

Each comparison contains four panels: the repainted reference frame, the driving video, Viggle-Animate, and Wan2.2-Animate-14B. We render Wan2.2-Animate-14B from the official unmodified weights using its documented character-replacement settings.

In our qualitative comparisons, the largest difference appears during fast motion. Wan2.2-Animate-14B often loses detail or introduces smearing around rapidly moving heads and limbs, whereas Viggle-Animate retains sharper character details. Viggle-Animate also follows the source pose more closely in these examples, including stance width, arm extension, limb displacement, and the timing of extreme poses, despite not receiving a pose skeleton.

Non-Human Character Results#

The model receives no explicit character category, so its range of supported characters is determined empirically. Each example contains three panels: the repainted reference frame, the driving video, and the Viggle-Animate output.

Limitations#

The output is constrained by both the repainted frame and the driving video. Visual attributes that are not visible in the reference frame may not be reproduced consistently. When the edited geometry conflicts with the source motion, the model tends to preserve the source body structure.

Lip-sync is weak. In close-ups, mouth shapes do not closely track speech, although identity and expression remain stable. We hypothesize that this primarily reflects limitations in the training data rather than the model design.

Complex scenes are harder than single-subject scenes. Quality decreases with multiple characters, close interactions, and videos that contain shot changes. We do not consider these cases solved.

Ongoing work targets these three limitations. This release represents the current model rather than the expected limit of the approach.

Open Weights and Usage#

Try the demo · Weights on Hugging Face · viggle.ai/h3

This is our first open model release, and the first of several. Follow us on Hugging Face for what comes next.

The weights are released under the MiniMax H3 Community License; the inference code is Apache 2.0. The repainted reference frame determines the character identity. Use reference material only from people who have consented, and label generated outputs as AI-generated.

Citation#

@misc{viggle2026animate,
  title  = {Viggle-Animate: Character Replacement in Video from a Single Repainted Frame},
  author = {Viggle Research},
  year   = {2026},
  url    = {https://huggingface.co/Viggle/Viggle-Animate}
}

Powered by MiniMax H3. Music in the teaser: "Electrodoodle" by Kevin MacLeod (incompetech.com), licensed under Creative Commons: By Attribution 4.0.